CN113297132A

CN113297132A - Generation method and device of migration file list, electronic equipment and computer readable medium

Info

Publication number: CN113297132A
Application number: CN202010109037.3A
Authority: CN
Inventors: 白玉宾
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2020-02-21
Filing date: 2020-02-21
Publication date: 2021-08-24
Anticipated expiration: 2040-02-21
Also published as: CN113297132B

Abstract

The embodiment of the application provides a method and a device for generating a migration file list, electronic equipment and a computer readable medium, and relates to the field of data storage. Wherein the method comprises the following steps: acquiring information of a plurality of directory partitions in a directory to be scanned, which is included in a data source in a migration task; starting a plurality of scanning tasks to perform concurrent scanning on a plurality of target partitions indicated by the information of the plurality of directory partitions; and filtering the scanning result of concurrent scanning, and generating a migration file list of the data source according to the filtering result. According to the embodiment of the application, the multiple scanning tasks are started to concurrently scan the multiple target partitions indicated by the information of the multiple directory partitions, so that the scanning speed of the data source is increased, the scanning performance of the data source is greatly improved, the migration file list is rapidly generated, and the generation efficiency of the migration file list is effectively improved.

Description

Generation method and device of migration file list, electronic equipment and computer readable medium

Technical Field

The embodiment of the application relates to the field of data storage, in particular to a method and a device for generating a migration file list, electronic equipment and a computer readable medium.

Background

With the penetration of cloud computing into various industries, the tendency of cloud acquisition is high, and particularly, storage systems are migrated to cloud storage, for example, an object storage service applied to the cloud storage. The single user migrates to the cloud, the data volume reaches the hundred PB level, and the number of files reaches the billion level. Data migration first requires the generation of a list of migrated files, and then migration from file to file. The migration file list is generated by scanning a data source, and the generation efficiency of the migration file list directly influences the data migration efficiency. Particularly, during incremental migration, time is consumed in scanning a data source, finding out an incremental file, and generating an incremental migration file list. For billions of files, a migration file list is generated by scanning a data source through a single thread according to a conventional method, the generation efficiency of the migration file list is low, the efficient cloud requirement of a user cannot be met at all, and incremental migration is silent. Therefore, how to effectively improve the generation efficiency of the migration file list becomes a technical problem to be solved urgently at present.

Disclosure of Invention

The application aims to provide a method and a device for generating a migration file list, an electronic device and a computer readable medium, which are used for solving the technical problem of how to effectively improve the generation efficiency of the migration file list in the prior art.

According to a first aspect of embodiments of the present application, a method for generating a migration file list is provided. The method comprises the following steps: acquiring information of a plurality of directory partitions in a directory to be scanned, which is included in a data source in a migration task; starting a plurality of scanning tasks to perform concurrent scanning on a plurality of target partitions indicated by the information of the plurality of directory partitions; and filtering the scanning result of concurrent scanning, and generating a migration file list of the data source according to the filtering result.

According to a second aspect of the embodiments of the present application, there is provided a device for generating a migration file list. The device comprises: the acquisition module is used for acquiring information of a plurality of directory partitions in the directory to be scanned, which is included in the data source in the migration task; the starting module is used for starting a plurality of scanning tasks to perform concurrent scanning on a plurality of target partitions indicated by the information of the plurality of directory partitions; and the first generation module is used for filtering the scanning result of the concurrent scanning and generating a migration file list of the data source according to the filtering result.

According to a third aspect of embodiments of the present application, there is provided an electronic apparatus, including: one or more processors; a computer readable medium configured to store one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the method for generating a migration file list as described in the first aspect of the embodiments.

According to a fourth aspect of embodiments of the present application, there is provided a computer-readable medium on which a computer program is stored, which when executed by a processor, implements the method for generating a migration file list as described in the first aspect of the embodiments above.

According to the generation scheme of the migration file list provided by the embodiment of the application, the information of a plurality of directory partitions in the directory to be scanned, which is included in the data source in the migration task, is obtained, the plurality of scanning tasks are started to concurrently scan a plurality of target partitions indicated by the information of the plurality of directory partitions, then the scanning results of concurrent scanning are filtered, and the migration file list of the data source is generated according to the filtering results.

Drawings

Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:

fig. 1A is a flowchart illustrating steps of a method for generating a migration file list according to an embodiment of the present application;

fig. 1B is a schematic diagram of a process of generating a migration file list according to an embodiment of the present application;

fig. 1C is a schematic diagram of a scanning cost tree according to an embodiment of the present application;

FIG. 1D is a schematic diagram of a connectivity graph provided in accordance with an embodiment of the present application;

FIG. 1E is a schematic diagram of an optimal scan path according to an embodiment of the present application;

fig. 2 is a schematic structural diagram of a device for generating a migration file list according to a second embodiment of the present application;

fig. 3 is a schematic structural diagram of a device for generating a migration file list according to a third embodiment of the present application;

fig. 4 is a schematic structural diagram of a device for generating a migration file list according to a fourth embodiment of the present application;

fig. 5 is a schematic structural diagram of an electronic device in a fifth embodiment of the present application;

fig. 6 is a hardware structure of an electronic device according to a sixth embodiment of the present application.

Detailed Description

The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.

It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.

Referring to fig. 1A, a flowchart of steps of a migration file list generation method according to a first embodiment of the present application is shown.

Specifically, the method for generating the migration file list provided by this embodiment includes the following steps:

in step S101, information of a plurality of directory partitions in a directory to be scanned included in a data source in a migration task is acquired.

In the embodiment of the present application, the migration task may be understood as a data migration task between storage spaces in the object storage service system, and may also be understood as a data migration task between the object storage service systems. The object storage service can be understood as a service for providing object storage, and is widely applied to the field of cloud computing or cloud storage at present. The so-called Object Storage (Object Storage) is a new type of network Storage architecture compared to the traditional block Storage and file Storage, and is characterized in that the Object Storage is an infinite-capacity cheap Storage system supporting remote access of a large number of users. The basic unit of object storage is an object, and the objects in the object storage are collectively referred to as files. The data source may be understood as a data source having data migration requirements, for example, an object storage service system, a storage space of the object storage service system. The directory to be scanned may be understood as a data storage directory to be scanned, e.g., a prefix at/at the end of an object storage, collectively referred to as a directory, in an object storage service system. The information of the plurality of directory partitions may be understood as information that divides the directory to be scanned into regions. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In some optional embodiments, when information of a plurality of directory partitions in a directory to be scanned, which is included in a data source in a migration task, is acquired, the directory to be scanned is divided into a plurality of sections according to names of files or subdirectories in the directory to be scanned; determining a division step size of each of the plurality of sections according to the weighted value of each of the plurality of sections; and dividing each section according to the dividing step length of each section so as to obtain the information of a plurality of directory partitions in the directory to be scanned. Therefore, the information of the plurality of directory partitions in the directory to be scanned can be accurately obtained by determining the division step size of each zone according to the weighted value of each zone in the plurality of zones and dividing each zone according to the division step size of each zone. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, the directory to be scanned is divided into a plurality of sections according to the section where the ASCII code of the first character of the name of the file or the subdirectory in the directory to be scanned is located. And then, determining the division step size of each section according to the weighted value of each section in the plurality of sections. The weighted value of each section in the plurality of sections is the weighted value of the section where the corresponding ASCII code is located, and the weighted value of the section where the corresponding ASCII code is located is configured in advance. In general, the larger the weight value of each of the plurality of sections is, the smaller the division step size of each of the plurality of sections is, and the smaller the weight value of each of the plurality of sections is, the larger the division step size of each of the plurality of sections is. And finally, dividing each section according to the dividing step length of each section to obtain the information of a plurality of directory partitions in the directory to be scanned. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In step S102, a plurality of scanning tasks are started to concurrently scan a plurality of target partitions indicated by the information of the plurality of directory partitions.

In the embodiment of the present application, the scanning task may be understood as a task of scanning a file or a directory of a target partition indicated by the information of the directory partition. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In some optional embodiments, when the starting of the multiple scanning tasks concurrently scans the multiple target partitions indicated by the information of the multiple directory partitions, the multiple scanning tasks are started to concurrently scan the multiple target partitions according to the scanning context information of the multiple target partitions. Specifically, after the problem such as the program exception is recovered or the program is abnormally started, the last scanning can be continued through the scanning context information of the target partition, and the effect of breakpoint continuation is achieved. Therefore, the effect of continuous scanning of the breakpoint can be realized by starting a plurality of scanning tasks to concurrently scan a plurality of target partitions according to the scanning context information of the plurality of target partitions. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, the scan context information includes at least one of: scanning position information of a target partition, scanning filter information of the target partition, and scanning task information of the target partition. The scanning location information of the target partition may be location information formed by combining a name of a directory to which the target partition belongs, an identifier of the target partition, and a scanning start location of the target partition. The scanning task information of the target partition may be current limiting information, attribute information, and the like of the scanning task of the target partition. The scan filter information of the target partition may be a scan filter condition of the target partition. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In some optional embodiments, before the starting of the multiple scanning tasks to concurrently scan the multiple target partitions according to the scanning context information of the multiple target partitions, the method further includes: and generating scanning filter information of the target partition according to the migration limiting condition in the migration task. Thus, the scan filter information of the target partition can be accurately generated by the migration restriction condition in the migration task. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In one specific example, the migration restrictions may be migration restrictions entered by the user when creating the migration task, such as including or excluding certain directories, not filtering certain directories, file modification time restrictions, file name regular matching conditions, file creation time restrictions, and the like. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, the scan filter information includes scan file filter information and scan directory filter information. The scan file filter information may be understood as a scan file filter condition of the target partition, and the scan directory filter information may be understood as a scan directory filter condition of the target partition. When generating the scanning filter information of the target partition according to the migration limiting condition in the migration task, generating the scanning directory filter information of the target partition according to the migration directory limiting condition in the migration limiting condition; and generating the scanning file filter information of the target partition according to the limited conditions of the migration files in the limited conditions of the migration. The migration directory restriction condition may include or exclude some directories, not filter some directories, and the like, and the migration file restriction condition may be a file modification time restriction, a file name regular matching condition, a file creation time restriction, and the like. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In some optional embodiments, when a plurality of scanning tasks are started to concurrently scan a plurality of target partitions indicated by information of the plurality of directory partitions, the plurality of scanning tasks are started to concurrently scan the plurality of target partitions with breadth first, and concurrently scan directories or files in the plurality of target partitions with depth first. Therefore, through breadth-first scanning between the target partitions and depth-first scanning inside the target partitions, directories in the target partitions and the target partitions are scanned concurrently, the scanning performance of the data source is greatly improved, and the migration file list can be generated quickly. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, the breadth-first scanning between the target partitions may be understood as a strategy of lateral scanning priority between the target partitions. Depth-first scanning inside the target partition may be understood as a policy of longitudinal scanning priority inside the target partition. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In step S103, filtering the scanning result of the concurrent scanning, and generating a migration file list of the data source according to the filtering result.

In this embodiment, the migration file list of the data source may be a total migration file list of the data source. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In some optional embodiments, when filtering the scanning result of the concurrent scanning and generating the migration file list of the data source according to the filtering result, if the scanning result is a file, filtering the file by using the scanned file filter information in the scanning context information to obtain a file meeting the migration file definition condition, and writing the metadata of the file meeting the migration file definition condition into the migration file list corresponding to the scanning task. Wherein the metadata of the file meeting the migration file definition condition may include an address, a file name, a file size (Byte), and a last modification time of the file meeting the migration file definition condition. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In some optional embodiments, after writing the metadata of the file meeting the migration file definition condition into the migration file list corresponding to the scanning task, the method further includes: adjusting the target partition according to the file quantity and the data volume of the target partition to obtain an adjusted target partition; and updating the scanning position information in the scanning context information according to the adjusted scanning position information of the target partition. Therefore, the target partition is adjusted according to the file number and the data volume of the target partition, the scanning position information in the scanning context information is updated according to the adjusted scanning position information of the target partition, the scanning strategy can be dynamically adjusted, the concurrent scanning task is reasonably arranged, the scanning time of the concurrent scanning task has no long tail effect, the overall scanning time of the data source is shortest, and the effect of quickly generating the migration file list is achieved. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, when the target partition is adjusted according to the file number and the data amount of the target partition, the scanning duration of the target partition is determined according to the file number and the data amount of the target partition, and the target partition is adjusted according to the scanning time of the target partition, so that the scanning time of the adjusted target partition is balanced. And then, updating the scanning position information in the scanning context information according to the adjusted scanning position information of the target partition. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In some optional embodiments, when filtering the scanning result of the concurrent scanning and generating the migration file list of the data source according to the filtering result, if the scanning result is a directory, the scanning directory filter information in the scanning context information is used to filter the directory to obtain a directory meeting the migration directory restriction condition, and generate the migration file list corresponding to the directory meeting the migration directory restriction condition. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, when a migration file list corresponding to the directory meeting the migration directory definition condition is generated, the directory meeting the migration directory definition condition is partitioned to obtain a plurality of directory partitions corresponding to the directory meeting the migration directory definition condition, and according to the scanning context information of the directory partitions, scanning tasks of the directory partitions are started to scan files or directories of the directory partitions until the data source is scanned, so as to generate a migration file list corresponding to the directory meeting the migration directory definition condition. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, when the directory meeting the migration directory definition condition is partitioned, the directory meeting the migration directory definition condition is divided into a plurality of sections according to the names of files or subdirectories in the directory meeting the migration directory definition condition; determining a division step size of each of the plurality of sections according to the weighted value of each of the plurality of sections; and dividing each section according to the division step size of each section to obtain a plurality of directory partitions in the directory which meet the migration directory definition conditions. Therefore, the division step size of each section is determined according to the weighted value of each section in the plurality of sections, and each section is divided according to the division step size of each section, so that the plurality of directory partitions in the directory which meet the migration directory limitation condition can be accurately obtained. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, the scan context information of the plurality of directory partitions includes at least one of: scanning position information of the directory partition, scanning filter information of the directory partition, and scanning task information of the directory partition. The scanning location information of the directory partition may be location information formed by combining a name of a directory to which the directory partition belongs, an identifier of the directory partition, and a scanning start location of the directory partition. The scanning task information of the directory partition may be current limiting information and attribute information of the scanning task of the directory partition. The scan filter information for the directory partition may be a scan filter condition for the directory partition. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, before starting the scanning tasks of the plurality of directory partitions according to the scanning context information of the plurality of directory partitions, the method further includes: and generating scanning filter information of the plurality of directory partitions according to the migration limiting conditions in the migration task. Thus, the scan filter information of the directory partition can be accurately generated by the migration restriction condition in the migration task. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, the specific implementation of generating the scanning filter information of the plurality of directory partitions according to the migration limiting condition in the migration task is similar to the specific implementation of generating the scanning filter information of the target partition according to the migration limiting condition in the migration task, and is not described herein again. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In some optional embodiments, after the generating the scan filter information for the plurality of directory partitions, the method further comprises: and updating the migration directory tree of the data source according to the directory which meets the migration directory limit condition. Specifically, after the problem such as the program exception is recovered or the program is abnormally started, the last scanning can be continued through the migration directory tree of the data source, and the effect of breakpoint continuation is achieved. Therefore, by updating the migration directory tree of the data source, the effect of continuous scanning of the breakpoints can be realized. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In some optional embodiments, when filtering the scanning result of the concurrent scanning and generating the migration file list of the data source according to the filtering result, the metadata of the files meeting the migration file definition condition in the migration file lists respectively corresponding to the scanning task and the directories meeting the migration directory definition condition are merged to generate a total migration file list of the data source. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, after the generating the migration file list of the data source, the method further includes: and summarizing the scanning context information of the scanning tasks of the target partitions and the directory partitions to generate a scanning directory tree of the data source. Therefore, the scanning directory tree of the data source can be generated by summarizing the scanning context information of the scanning tasks of the target partitions and the directory partitions, and the storage of the scanning information of the data source can be realized. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, after the generating the migration file list of the data source, the method further includes: and generating an optimal scanning path for scanning the data source next time according to the file number and the file size of the target partitions and the directory partitions and during scanning. Therefore, the optimal scanning path of the next scanning data source is generated according to the file number and the file size of the target partitions and the directory partitions and during scanning, the scanning strategy of the next scanning data source can be dynamically adjusted according to the scanning result, the concurrent scanning task is reasonably arranged, the scanning time of the concurrent scanning task has no long tail effect, the overall scanning time of the next scanning data source is shortest, and the effect of quickly generating the migration file list is achieved. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, as shown in fig. 1B, the process of generating the migration file list of the data source is as follows: 1. an initial scan location, directory partition, is generated. According to historical experience, different step lengths are determined according to weighted values, and the directory to be scanned is divided into a plurality of areas. The scanning position comprises the name of the directory to which the directory partition belongs, the scanning starting position of the directory partition, the identification of the directory partition and the like. 2. A scan condition filter is generated. And respectively generating a file filter and a directory filter according to the migration limiting conditions, such as inclusion or investigation of some directories, file modification time limitation, file name regular matching and the like, and searching out the directories and files meeting the migration conditions. 3. A scan task is initiated according to a scan context. The scan context includes scan location, scan filters, scan job information, and the like. The scanning tasks are executed concurrently. 4. And (5) processing a scanning result. The scan task scans out files and directories from the data source according to the scan context. 5-7, and processing the scanning result as a file. If the scanning result is a file, searching the file meeting the migration condition by using the specified file filter, and writing the file into a migration file list. Then, the directory partition is adjusted according to the number of partition files and the data size, and the scanning position is updated. 8-11, and processing the scanning result as a directory. If the result of the scanning task is a directory, the result directory is retrieved by using the specified directory filter, and a corresponding scanning position, directory partition and scanning filter are generated. Then, updating the migration directory tree, starting a scanning task according to the scanning context, and scanning the files or the directories under the directory. 12. And generating a migration file list. And the directories and the directory partitions are scanned concurrently, and the results of all scanning tasks are merged to generate a migration file list. 13. And summarizing the context information of the scanning task to generate a scanning directory tree. And merging directory partitions according to the number of scanned files, the size of the files and the scanning time to generate an optimal scanning strategy for the next scanning. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In some optional embodiments, when an optimal scanning path for scanning the data source next time is generated according to the file number and the file size of the target partitions and the directory partitions and during scanning, the target partitions and the directory partitions are merged according to the file number and the file size of the target partitions and the directory partitions and during scanning, so as to generate a scanning cost tree of the data source, wherein leaf nodes of the scanning cost tree are merged directory partitions, and non-leaf nodes of the scanning cost tree are directories to which the merged directory partitions belong; generating a connection graph between the merged directory partitions according to the scanning cost tree and the scanning time of the merged directory partitions; searching the connection graph between the merged directory partitions to generate a minimum spanning tree of the connection graph, and taking the minimum spanning tree as the optimal scanning path. Therefore, the minimum spanning tree of the connected graph is generated by generating the connected graph among the merged directory partitions and searching the connected graph among the merged directory partitions, so that the optimal scanning path of the data source scanned next time can be accurately obtained. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, when the connection graph between the merged directory partitions is generated according to the scanning cost tree and the merged directory partitions for scanning, the merged directory partitions of the scanning cost tree are connected later, and the scanning time of the merged directory partitions is used as the weight of the connection edge between the merged directory partitions to generate the connection graph between the merged directory partitions. The data structure composed of vertices and edges is collectively referred to as a graph data structure. The graph data structure is composed of a finite set of nodes, V, and a set of edges, E. To distinguish from tree structures, nodes are often referred to in graph data structures as vertices, edges are ordered pairs of vertices, and an edge between two vertices indicates that the two vertices have an adjacent relationship. The length of an edge between two vertices is called a weight. Any two points in the undirected graph data structure have paths to communicate, and the graph data structure of the type is called a connected graph. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In some optional embodiments, after the generating the optimal scan path for scanning the data source next time, the method further includes: and generating a concurrent scanning task for scanning the data source next time according to the optimal scanning path so as to generate a migration file list for scanning the data source next time. Therefore, the concurrent scanning task of scanning the data source next time is generated according to the optimal scanning path, the overall scanning time of scanning the data source next time can be shortened, and the effect of quickly generating the migration file list when scanning the data source next time is achieved. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, when the data source is scanned for the second time or more to generate the migration file list, that is, the incremental migration scenario, the next optimal scan path may be generated according to the previous scan result. The specific process is as follows: from the last scan result, the directory partition with smaller merging time is used to generate the scan cost tree shown in fig. 1C, where nodes A, B, C and the like are directories, leaf nodes B1, B2, B3, C1 and the like are directory partitions, and the number under a directory partition is the scan time for scanning the directory partition in the last scan result. On the basis of the directory partition of the scanning cost tree shown in fig. 1C, a scanning Start point (Start) and a scanning End point (End) are added, and a connected graph as shown in fig. 1D is generated. Specifically, the directory partitions of the scanning cost tree shown in fig. 1C are connected later, and the scanning time of the directory partitions is used as the weight of the connection edge between the directory partitions, so as to generate a connection graph corresponding to the scanning cost tree, as shown in fig. 1D. On the connected graph shown in fig. 1D, a minimum spanning tree as shown in fig. 1E is generated using the Prim algorithm. The minimum spanning tree is the optimal scan path. And generating a concurrent scanning task according to the optimal scanning path shown in fig. 1E, and generating a new scanning file list. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In practical application, the generation scheme of the migration file list provided by the embodiment of the application accelerates the scanning speed by multi-directory partition concurrent scanning and dynamically adjusting the scanning strategy, the speed of the primary scanning performance is improved by more than 3 times compared with that of the conventional scheme, and the scanning performance of the secondary scanning performance is improved by more than 5 times compared with that of the conventional scheme. The rapid migration and incremental file migration of massive files become reality. The scheme is suitable for all object storage systems in the current market and has universality.

According to the method for generating the migration file list provided by the embodiment of the application, the information of the directory partitions in the directory to be scanned, which is included in the data source in the migration task, is obtained, the scanning tasks are started to concurrently scan the target partitions indicated by the information of the directory partitions, the scanning results of the concurrent scanning are filtered, and the migration file list of the data source is generated according to the filtering results.

The method for generating the migration file list of the present embodiment may be executed by any suitable device with data processing capability, including but not limited to: cameras, terminals, mobile terminals, PCs, servers, in-vehicle devices, entertainment devices, advertising devices, Personal Digital Assistants (PDAs), tablet computers, notebook computers, handheld game consoles, glasses, watches, wearable devices, virtual display devices or display enhancement devices (such as Google Glass, Oculus rise, HoloLens, Gear VR), and the like.

Referring to fig. 2, a schematic structural diagram of a generation apparatus for migrating a file list in the second embodiment of the present application is shown.

The device for generating the migration file list provided by the embodiment comprises: an obtaining module 201, configured to obtain information of a plurality of directory partitions in a directory to be scanned, where the directory partitions are included in a data source in a migration task; a starting module 202, configured to start multiple scanning tasks to perform concurrent scanning on multiple target partitions indicated by the information of the multiple directory partitions; the first generating module 203 is configured to perform filtering processing on the scanning result of the concurrent scanning, and generate a migration file list of the data source according to the filtering processing result.

The device for generating a migration file list of this embodiment is used to implement the method for generating a corresponding migration file list in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described herein again.

Referring to fig. 3, a schematic structural diagram of a generation apparatus for migrating a file list in the third embodiment of the present application is shown.

The device for generating the migration file list provided by the embodiment comprises: an obtaining module 301, configured to obtain information of multiple directory partitions in a directory to be scanned, where the directory partitions are included in a data source in a migration task; a starting module 302, configured to start multiple scanning tasks to perform concurrent scanning on multiple target partitions indicated by the information of the multiple directory partitions; the first generating module 303 is configured to perform filtering processing on the scanning result of the concurrent scanning, and generate a migration file list of the data source according to the filtering processing result.

Optionally, the obtaining module 301 is specifically configured to: dividing the directory to be scanned into a plurality of sections according to the names of files or subdirectories in the directory to be scanned; determining a division step size of each of the plurality of sections according to the weighted value of each of the plurality of sections; and dividing each section according to the dividing step length of each section so as to obtain the information of a plurality of directory partitions in the directory to be scanned.

Optionally, the starting module 302 includes: a starting module 3022, configured to start the multiple scanning tasks to perform concurrent scanning on the multiple target partitions according to the scanning context information of the multiple target partitions.

Optionally, the scan context information comprises at least one of: scanning position information of a target partition, scanning filter information of the target partition, and scanning task information of the target partition.

Optionally, before the starting sub-module 3022, the starting module 302 further includes: a first generating sub-module 3021, configured to generate scan filter information of the target partition according to a migration limiting condition in the migration task.

Optionally, the scan filter information includes scan file filter information and scan directory filter information, and the first generating sub-module 3021 is specifically configured to: generating scanning directory filter information of the target partition according to the migration directory limiting conditions in the migration limiting conditions; and generating the scanning file filter information of the target partition according to the limited conditions of the migration files in the limited conditions of the migration.

Optionally, the first generating module 303 includes: the second generating sub-module 3031 is configured to, if the scanning result is a file, perform filtering processing on the file by using the scanned file filter information in the scanning context information to obtain a file meeting a migration file definition condition, and write metadata of the file meeting the migration file definition condition into a migration file list corresponding to the scanning task.

Optionally, after the second generating sub-module 3031, the first generating module 303 further includes: an adjusting submodule 3032, configured to adjust the target partition according to the file number and the data size of the target partition, so as to obtain an adjusted target partition; and an updating submodule 3033, configured to update the scanning position information in the scanning context information according to the adjusted scanning position information of the target partition.

Referring to fig. 4, a schematic structural diagram of a generation apparatus for migrating a file list in the fourth embodiment of the present application is shown.

The device for generating the migration file list provided by the embodiment comprises: an obtaining module 401, configured to obtain information of multiple directory partitions in a directory to be scanned, where the directory partitions are included in a data source in a migration task; a starting module 402, configured to start multiple scanning tasks to perform concurrent scanning on multiple target partitions indicated by information of the multiple directory partitions; a first generating module 403, configured to perform filtering processing on the scanning result of the concurrent scanning, and generate a migration file list of the data source according to the filtering processing result.

Optionally, the first generating module 403 includes: a third generation sub-module 4031, configured to, if the scan result is a directory, filter the directory using the scan directory filter information in the scan context information to obtain a directory meeting a migration directory restriction condition, and generate a migration file list corresponding to the directory meeting the migration directory restriction condition.

Optionally, the third generation sub-module 4031 includes: a partitioning unit 4032, configured to partition the directory meeting the migration directory restriction condition to obtain a plurality of directory partitions corresponding to the directory meeting the migration directory restriction condition; a starting unit 4035, configured to start a scanning task of the multiple directory partitions according to the scanning context information of the multiple directory partitions, so as to scan files or directories of the multiple directory partitions until the data source is completely scanned, so as to generate a migration file list corresponding to the directory meeting the migration directory definition condition.

Optionally, the scan context information of the plurality of directory partitions includes at least one of: scanning position information of the directory partition, scanning filter information of the directory partition, and scanning task information of the directory partition.

Optionally, before the starting unit 4035, the third generating sub-module 4031 further includes: a generating unit 4033, configured to generate scan filter information of the multiple directory partitions according to the migration restriction condition in the migration task.

Optionally, after the generating unit 4033, the third generating sub-module 4031 further includes: an updating unit 4034, configured to update the migration directory tree of the data source according to the directory meeting the migration directory definition condition.

Optionally, the first generating module 403 is specifically configured to: and merging metadata of files meeting the migration file limiting conditions in migration file lists respectively corresponding to the scanning task and the directories meeting the migration directory limiting conditions so as to generate a migration file list of the data source.

Optionally, after the first generating module 403, the apparatus further includes: a summarizing module 404, configured to summarize the scanning context information of the scanning tasks of the plurality of target partitions and the plurality of directory partitions, and generate a scanning directory tree of the data source.

Optionally, after the first generating module 403, the apparatus further includes: a second generating module 405, configured to generate an optimal scanning path for scanning the data source next time according to the number of files, the file sizes, and the scanning time of the multiple target partitions and the multiple directory partitions.

Optionally, the second generating module 405 includes: a fourth generating sub-module 4051, configured to combine the multiple target partitions and the multiple directory partitions according to the file numbers and file sizes of the multiple target partitions and the multiple directory partitions, and during scanning, to generate a scanning cost tree of the data source, where leaf nodes of the scanning cost tree are merged directory partitions, and non-leaf nodes of the scanning cost tree are directories to which the merged directory partitions belong; a fifth generating sub-module 4052, configured to generate a connection graph between the merged directory partitions according to the scanning cost tree and the scanning time of the merged directory partitions; a sixth generating sub-module 4053, configured to search the connected graph between the merged directory partitions to generate a minimum spanning tree of the connected graph, and use the minimum spanning tree as the optimal scanning path.

Optionally, the fifth generation sub-module 4052 is specifically configured to: and then communicating the merged directory partitions of the scanning cost tree, and taking the scanning time of the merged directory partitions as the weight of a connecting edge between the merged directory partitions to generate a communication graph between the merged directory partitions.

Optionally, after the second generating module 405, the apparatus further includes: a third generating module 406, configured to generate a concurrent scanning task for scanning the data source next time according to the optimal scanning path, so as to generate a migration file list for scanning the data source next time.

Optionally, the starting module 402 is specifically configured to: and starting the multiple scanning tasks to perform breadth-first concurrent scanning on the multiple target partitions, and performing depth-first concurrent scanning on directories or files in the multiple target partitions.

Fig. 5 is a schematic structural diagram of an electronic device in a fifth embodiment of the present application; the electronic device may include:

one or more processors 501;

a computer-readable medium 502, which may be configured to store one or more programs,

when the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating the migration file list according to the embodiment.

Fig. 6 is a hardware structure of an electronic device according to a sixth embodiment of the present application; as shown in fig. 6, the hardware structure of the electronic device may include: a processor 601, a communication interface 602, a computer-readable medium 603, and a communication bus 604;

wherein the processor 601, the communication interface 602, and the computer readable medium 603 communicate with each other via a communication bus 604;

alternatively, the communication interface 602 may be an interface of a communication module, such as an interface of a GSM module;

the processor 601 may be specifically configured to: acquiring information of a plurality of directory partitions in a directory to be scanned, which is included in a data source in a migration task; starting a plurality of scanning tasks to perform concurrent scanning on a plurality of target partitions indicated by the information of the plurality of directory partitions; and filtering the scanning result of concurrent scanning, and generating a migration file list of the data source according to the filtering result.

The Processor 601 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The computer-readable medium 603 may be, but is not limited to, a Random Access Memory (RAM), a Read-Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and the like.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code configured to perform the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication section, and/or installed from a removable medium. The computer program, when executed by a Central Processing Unit (CPU), performs the above-described functions defined in the method of the present application. It should be noted that the computer readable medium described herein can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access storage media (RAM), a read-only storage media (ROM), an erasable programmable read-only storage media (EPROM or flash memory), an optical fiber, a portable compact disc read-only storage media (CD-ROM), an optical storage media piece, a magnetic storage media piece, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

Computer program code configured to carry out operations for the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may operate over any of a variety of networks: including a Local Area Network (LAN) or a Wide Area Network (WAN) -to the user's computer, or alternatively, to an external computer (e.g., through the internet using an internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions configured to implement the specified logical function(s). In the above embodiments, specific precedence relationships are provided, but these precedence relationships are only exemplary, and in particular implementations, the steps may be fewer, more, or the execution order may be modified. That is, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present application may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes an acquisition module, a start module, and a first generation module. The names of these modules do not constitute a limitation to the module itself in some cases, for example, the obtaining module may also be described as a module that obtains information of a plurality of directory partitions in a directory to be scanned included in a data source in a migration task.

As another aspect, the present application further provides a computer-readable medium, on which a computer program is stored, which when executed by a processor, implements the method for generating a migration file list as described in the first embodiment.

As another aspect, the present application also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be present separately and not assembled into the device. The computer readable medium carries one or more programs which, when executed by the apparatus, cause the apparatus to: acquiring information of a plurality of directory partitions in a directory to be scanned, which is included in a data source in a migration task; starting a plurality of scanning tasks to perform concurrent scanning on a plurality of target partitions indicated by the information of the plurality of directory partitions; and filtering the scanning result of concurrent scanning, and generating a migration file list of the data source according to the filtering result.

The expressions "first", "second", "said first" or "said second" used in various embodiments of the present disclosure may modify various components regardless of order and/or importance, but these expressions do not limit the respective components. The above description is only configured for the purpose of distinguishing elements from other elements. For example, the first user equipment and the second user equipment represent different user equipment, although both are user equipment. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of the present disclosure.

When an element (e.g., a first element) is referred to as being "operably or communicatively coupled" or "connected" (operably or communicatively) to "another element (e.g., a second element) or" connected "to another element (e.g., a second element), it is understood that the element is directly connected to the other element or the element is indirectly connected to the other element via yet another element (e.g., a third element). In contrast, it is understood that when an element (e.g., a first element) is referred to as being "directly connected" or "directly coupled" to another element (a second element), no element (e.g., a third element) is interposed therebetween.

The above description is only a preferred embodiment of the application and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention herein disclosed is not limited to the particular combination of features described above, but also encompasses other arrangements formed by any combination of the above features or their equivalents without departing from the spirit of the invention. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.

Claims

1. A method of generating a list of migrated files, the method comprising:

acquiring information of a plurality of directory partitions in a directory to be scanned, which is included in a data source in a migration task;

starting a plurality of scanning tasks to perform concurrent scanning on a plurality of target partitions indicated by the information of the plurality of directory partitions;

and filtering the scanning result of concurrent scanning, and generating a migration file list of the data source according to the filtering result.

2. The method according to claim 1, wherein the obtaining information of a plurality of directory partitions in a directory to be scanned, which is included in a data source in the migration task, includes:

dividing the directory to be scanned into a plurality of sections according to the names of files or subdirectories in the directory to be scanned;

determining a division step size of each of the plurality of sections according to the weighted value of each of the plurality of sections;

and dividing each section according to the dividing step length of each section so as to obtain the information of a plurality of directory partitions in the directory to be scanned.

3. The method of claim 1, wherein the initiating the plurality of scan tasks to concurrently scan a plurality of target partitions indicated by the information of the plurality of directory partitions comprises:

and starting the multiple scanning tasks to perform concurrent scanning on the multiple target partitions according to the scanning context information of the multiple target partitions.

4. The method of claim 3, wherein the scan context information comprises at least one of:

scanning position information of a target partition, scanning filter information of the target partition, and scanning task information of the target partition.

5. The method of claim 4, wherein prior to initiating the concurrent scanning of the plurality of target partitions by the plurality of scanning tasks according to the scanning context information of the plurality of target partitions, the method further comprises:

and generating scanning filter information of the target partition according to the migration limiting condition in the migration task.

6. The method of claim 5, wherein the scanning filter information includes scanning file filter information and scanning directory filter information,

the generating of the scanning filter information of the target partition according to the migration limiting condition in the migration task comprises:

generating scanning directory filter information of the target partition according to the migration directory limiting conditions in the migration limiting conditions;

and generating the scanning file filter information of the target partition according to the limited conditions of the migration files in the limited conditions of the migration.

7. The method according to claim 3, wherein the filtering the scanning result of the concurrent scanning and generating the migration file list of the data source according to the filtering result comprises:

and if the scanning result is a file, filtering the file by using the scanned file filter information in the scanning context information to obtain a file meeting the migration file limiting condition, and writing the metadata of the file meeting the migration file limiting condition into a migration file list corresponding to the scanning task.

8. The method according to claim 7, wherein after writing the metadata of the file meeting the migration file definition condition into the migration file list corresponding to the scanning task, the method further comprises:

adjusting the target partition according to the file quantity and the data volume of the target partition to obtain an adjusted target partition;

and updating the scanning position information in the scanning context information according to the adjusted scanning position information of the target partition.

9. The method according to claim 8, wherein the filtering the scanning result of the concurrent scanning and generating the migration file list of the data source according to the filtering result comprises:

and if the scanning result is a directory, filtering the directory by using the scanning directory filter information in the scanning context information to obtain a directory meeting the migration directory definition condition, and generating a migration file list corresponding to the directory meeting the migration directory definition condition.

10. The method according to claim 9, wherein the generating a migration file list corresponding to the directory meeting the migration directory qualification includes:

partitioning the directories meeting the migration directory restriction condition to obtain a plurality of directory partitions corresponding to the directories meeting the migration directory restriction condition, and starting scanning tasks of the directory partitions according to the scanning context information of the directory partitions to scan files or directories of the directory partitions until the data source is completely scanned to generate a migration file list corresponding to the directories meeting the migration directory restriction condition.

11. The method of claim 10, wherein the scan context information for the plurality of directory partitions comprises at least one of:

scanning position information of the directory partition, scanning filter information of the directory partition, and scanning task information of the directory partition.

12. The method of claim 11, wherein prior to initiating the scan tasks for the plurality of directory partitions based on the scan context information for the plurality of directory partitions, the method further comprises:

and generating scanning filter information of the plurality of directory partitions according to the migration limiting conditions in the migration task.

13. The method of claim 12, wherein after the generating scan filter information for the plurality of directory partitions, the method further comprises:

and updating the migration directory tree of the data source according to the directory which meets the migration directory limit condition.

14. The method according to claim 13, wherein the filtering the scanning result of the concurrent scanning, and generating the migration file list of the data source according to the filtering result includes:

and merging metadata of files meeting the migration file limiting conditions in migration file lists respectively corresponding to the scanning task and the directories meeting the migration directory limiting conditions so as to generate a migration file list of the data source.

15. The method of claim 14, wherein after the generating the list of migrated files for the data source, the method further comprises:

and summarizing the scanning context information of the scanning tasks of the target partitions and the directory partitions to generate a scanning directory tree of the data source.

16. The method of claim 14, wherein after the generating the list of migrated files for the data source, the method further comprises:

and generating an optimal scanning path for scanning the data source next time according to the file number and the file size of the target partitions and the directory partitions and during scanning.

17. The method of claim 16, wherein the generating an optimal scan path for scanning the data source next time according to the number of files, the file sizes, and the scan times of the target partitions and the directory partitions comprises:

merging the target partitions and the directory partitions according to the file number and the file size of the target partitions and the directory partitions and during scanning to generate a scanning cost tree of the data source, wherein leaf nodes of the scanning cost tree are merged directory partitions, and non-leaf nodes of the scanning cost tree are directories to which the merged directory partitions belong;

generating a connection graph between the merged directory partitions according to the scanning cost tree and the scanning time of the merged directory partitions;

searching the connection graph between the merged directory partitions to generate a minimum spanning tree of the connection graph, and taking the minimum spanning tree as the optimal scanning path.

18. The method of claim 17, wherein the generating a connectivity graph between the merged directory partitions from the scan cost tree and the scan age of the merged directory partitions comprises:

and then communicating the merged directory partitions of the scanning cost tree, and taking the scanning time of the merged directory partitions as the weight of a connecting edge between the merged directory partitions to generate a communication graph between the merged directory partitions.

19. The method of claim 16, wherein after the generating the optimal scan path for the next time the data source is scanned, the method further comprises:

and generating a concurrent scanning task for scanning the data source next time according to the optimal scanning path so as to generate a migration file list for scanning the data source next time.

20. The method of claim 1, wherein the initiating the plurality of scan tasks to concurrently scan a plurality of target partitions indicated by the information of the plurality of directory partitions comprises:

and starting the multiple scanning tasks to perform breadth-first concurrent scanning on the multiple target partitions, and performing depth-first concurrent scanning on directories or files in the multiple target partitions.

21. An apparatus for generating a migration file list, the apparatus comprising:

the acquisition module is used for acquiring information of a plurality of directory partitions in the directory to be scanned, which is included in the data source in the migration task;

the starting module is used for starting a plurality of scanning tasks to perform concurrent scanning on a plurality of target partitions indicated by the information of the plurality of directory partitions;

and the first generation module is used for filtering the scanning result of the concurrent scanning and generating a migration file list of the data source according to the filtering result.

22. An electronic device, the device comprising:

one or more processors;

a computer readable medium configured to store one or more programs,

when executed by the one or more processors, cause the one or more processors to implement a method of generating a list of migrated files as claimed in any of claims 1-20.

23. A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out the method of generating a list of migration files according to any one of claims 1 to 20.