CN117370295A - File copying method, device, server and storage medium - Google Patents

Abstract

The application provides a file copying method, a device, a server and a storage medium, relates to the technical field of networks, and solves the technical problems of low file copying speed and low efficiency in related technologies. The method comprises the following steps: acquiring a file copy request sent by a terminal, wherein the file copy request comprises a file name of a file to be copied, a position of the file to be copied and creation time of the file to be copied; determining a first position and a second position based on the file name of the file to be copied, the position of the file to be copied and the creation time of the file to be copied, wherein the first position is a storage position of file data of the file to be copied, and the second position is a storage position of metadata of the file to be copied; acquiring file data of the file to be copied based on the first position and acquiring metadata of the file to be copied based on the second position; and generating the target copy file based on the file data of the file to be copied and the metadata of the file to be copied.

Description

File copying method, device, server and storage medium

Technical Field

The present disclosure relates to the field of network technologies, and in particular, to a file copying method, a device, a server, and a storage medium.

Background

At present, in a distributed storage system, when copying a file, metadata of an original file is copied to a new file, file data of the original file is acquired based on the copied metadata, the file data of the original file is copied to the new file, and copying of the original file is completed, so that the obtained new file is the copied file.

However, in the above method, since some files have more metadata, the metadata of the original file is copied first, and then the file data of the original file is copied, which has problems of slow file copying speed and low efficiency.

Disclosure of Invention

The application provides a file copying method, a file copying device, a server and a storage medium, which solve the technical problems of relatively low file copying speed and relatively low efficiency in the related art that metadata of some files are more, and file data of original files are copied after metadata of the original files are copied.

In a first aspect, the present application provides a file copying method, including: acquiring a file copy request sent by a terminal, wherein the file copy request comprises a file name of a file to be copied, a position of the file to be copied and creation time of the file to be copied; determining a first position and a second position based on the file name of the file to be copied, the position of the file to be copied and the creation time of the file to be copied, wherein the first position is a storage position of file data of the file to be copied, the second position is a storage position of metadata of the file to be copied, the metadata of the file to be copied is used for representing attribute information of the file to be copied, and the attribute information of the file to be copied is used for indicating access and management of the file data of the file to be copied; acquiring file data of the file to be copied based on the first position, and acquiring metadata of the file to be copied based on the second position; and generating a target copy file based on the file data of the file to be copied and the metadata of the file to be copied.

Optionally, generating the target copy file based on the file data of the file to be copied and the metadata of the file to be copied includes: writing the file data of the file to be copied into the file data of the target copy file, and writing the target metadata of the file to be copied into the metadata of the target copy file, wherein the target metadata comprises attribute information except for the file name of the file to be copied, the position of the file to be copied and the creation time of the file to be copied in the attribute information of the metadata of the file to be copied.

In the application, the server writes the file data of the file to be copied into the file data of the target copy file, so that the target copy file and the file data between the target copy file are identical, the server writes the target metadata of the file to be copied into the metadata of the target copy file, and the target metadata comprises attribute information except the file name of the file to be copied, the position of the file to be copied and the creation time of the file to be copied in the attribute information of the metadata of the file to be copied, so that the metadata of the target copy file except the file name, the position and the creation time are identical to the file to be copied, and the copying of the file to be copied can be completed.

Optionally, the method further comprises: adding a first operation to a target mapping queue, wherein the first operation is that target metadata of the file to be copied is written into metadata of the target copy file, and the target mapping queue is used for indicating files corresponding to one or more operations respectively; receiving a service request sent by the terminal, wherein the service request is used for requesting to process metadata of the target copy file; adding the service request to a request waiting queue under the condition that the first operation is not completed, wherein the request waiting queue is used for storing the next service request which comes when the current operation of a file is not completed; deleting the service request from the request wait queue and deleting the first operation from the target map queue and adding a second operation to the target map queue, the second operation being to process the target copy file based on the service request, if the first operation is complete; the second operation is performed based on the target mapping queue.

In the application, a server adds a first operation to a target mapping queue, a terminal sends a service request to the server, the server receives the service request sent by the terminal, in the case that the first operation is not completed, if the service request is executed at the same time, because the first operation and the service request process metadata of the target copy file at the same time, there may be a conflict, at this time, the server adds the service request to a request waiting queue, that is, temporarily does not execute the service request, in the case that the first operation is completed, and the service request is not executed at the same time, that is, there is no operation conflict, at this time, the server processes the target copy file based on the service request, and at the same time, because the request waiting queue is used for storing the next incoming service request when the last operation is not executed, and when the service request is executed, the server deletes the service request from the request waiting queue to avoid the server to repeatedly acquire the service request in the request waiting queue, because the second operation is located in the target mapping queue, and the target mapping queue is used for indicating one or more operations corresponding to each other, at the same time, the target copy file can be processed based on the service request, and the target copy file can not be generated when the second operation is executed on the target copy file, and the target copy file can be processed at the same time, the availability of access and management to the target copy file is reliably ensured while the quick copy file is ensured.

Optionally, the file copy request further includes a file system to which the file to be copied belongs, where the file system to which the file to be copied belongs is an asynchronous file system or a synchronous file system, and the method further includes: determining the second position based on the file name of the file to be copied, the position of the file to be copied and the creation time of the file to be copied under the condition that the file system to which the file to be copied belongs is a synchronous file system; acquiring metadata of the file to be copied based on the second position; and acquiring file data of the file to be copied based on the metadata of the file to be copied.

In the application, under the condition that the file system to which the file to be copied belongs is a synchronous file system, it is stated that a user chooses not to perform asynchronous copying operation on the file to be copied, at this time, the server determines the second position, that is, the storage position of the metadata to be copied, based on the file name of the file to be copied, the position of the file to be copied, and the creation time of the file to be copied, and the server obtains the metadata of the file to be copied based on the second position.

In a second aspect, the present application provides a file copying apparatus, including: the device comprises an acquisition module, a determination module and a processing module; the acquisition module is used for acquiring a file copy request sent by the terminal, wherein the file copy request comprises a file name of a file to be copied, a position of the file to be copied and creation time of the file to be copied; the determining module is configured to determine a first location and a second location based on a file name of the file to be copied, a location of the file to be copied, and a creation time of the file to be copied, where the first location is a storage location of file data of the file to be copied, the second location is a storage location of metadata of the file to be copied, the metadata of the file to be copied is used for characterizing attribute information of the file to be copied, and the attribute information of the file to be copied is used for indicating access and management of the file data of the file to be copied; the acquisition module is further used for acquiring file data of the file to be copied based on the first position and acquiring metadata of the file to be copied based on the second position; the processing module is used for generating a target copy file based on the file data of the file to be copied and the metadata of the file to be copied.

Optionally, the processing module is specifically configured to write file data of the file to be copied into file data of the target copy file, and write target metadata of the file to be copied into metadata of the target copy file, where the target metadata includes attribute information, except for a file name of the file to be copied, a location of the file to be copied, and a creation time of the file to be copied, in attribute information of the metadata of the file to be copied.

Optionally, the apparatus further comprises a receiving module; the processing module is further configured to add the first operation to a target mapping queue, where the first operation is writing target metadata of the file to be copied into metadata of the target copy file, the first operation corresponds to the target copy file, and the target mapping queue is used to indicate files corresponding to one or more operations respectively; the receiving module is used for receiving a service request sent by the terminal, wherein the service request is used for requesting to process metadata of the target copy file; the processing module is further configured to add the service request to a request waiting queue when the first operation is not completed, where the request waiting queue is configured to store a next service request that has come when a current operation of a file is not completed; the processing module is further configured to delete the service request from the request wait queue and delete the first operation from the target map queue and add a second operation to the target map queue, where the second operation is to process the target copy file based on the service request, if the first operation is complete; the processing module is further configured to perform the second operation based on the target mapping queue.

Optionally, the file copy request further includes a file system to which the file to be copied belongs, where the file system to which the file to be copied belongs is an asynchronous file system or a synchronous file system; the determining module is further configured to determine, when the file system to which the file to be copied belongs is a synchronous file system, the second location based on a file name of the file to be copied, a location of the file to be copied, and a creation time of the file to be copied; the acquisition module is further used for acquiring metadata of the file to be copied based on the second position; the acquisition module is further used for acquiring file data of the file to be copied based on metadata of the file to be copied.

In a third aspect, the present application provides a server comprising: a processor and a memory configured to store processor-executable instructions; wherein the processor is configured to execute the instructions to implement any of the above-described optional file copy methods of the first aspect.

In a fourth aspect, the present application provides a computer readable storage medium having instructions stored thereon that, when executed by a server, enable the server to perform any one of the above-described optional file copy methods of the first aspect.

According to the file copying method, device, server and storage medium, a terminal sends a file copying request to a server, the server obtains the file copying request sent by the terminal, the server determines a first position and a second position based on the file name of a file to be copied, the position of the file to be copied and the creation time of the file to be copied, and the server obtains file data of the file to be copied based on the first position; the server obtains metadata of the file to be copied based on the second position, and generates the target copy file based on the file data of the file to be copied and the metadata of the file to be copied, so that the file data and the metadata of the file to be copied can be obtained simultaneously without obtaining the metadata of the file to be copied first and then obtaining the file data of the file to be copied, further, the target copy file corresponding to the file to be copied is generated rapidly and accurately, and the speed and the efficiency of file copying are guaranteed.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.

FIG. 1 is a schematic diagram of a network architecture of a file copy system according to an embodiment of the present disclosure;

FIG. 2 is a schematic flow chart of a file copying method according to an embodiment of the present application;

FIG. 3 is a flowchart illustrating another method for copying files according to an embodiment of the present disclosure;

FIG. 4 is a flowchart illustrating another method for copying files according to an embodiment of the present disclosure;

FIG. 5 is a flowchart illustrating another method for copying files according to an embodiment of the present disclosure;

fig. 6 is a schematic diagram of an internal structure of a server according to an embodiment of the present application;

FIG. 7 is a schematic diagram of the structure of the target mapping queue and the request waiting queue according to the embodiment of the present application;

FIG. 8 is a schematic diagram of a document copying apparatus according to an embodiment of the present disclosure;

fig. 9 is a schematic structural diagram of another document copying apparatus according to an embodiment of the present application.

Detailed Description

The file copying method, device, server and storage medium provided in the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

The terms "first" and "second" and the like in the description and in the drawings are used for distinguishing between different objects and not for describing a particular sequence of objects, e.g., first and second operations and the like are used for distinguishing between different operations and not for describing a particular sequence of operations.

Furthermore, references to the terms "comprising" and "having" and any variations thereof in the description of the present application are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed but may optionally include other steps or elements not listed or inherent to such process, method, article, or apparatus.

It should be noted that, in the embodiments of the present application, words such as "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "for example" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.

In the description of the present application, unless otherwise indicated, the meaning of "a plurality" means two or more.

Based on the description in the background art, since the related art generally performs regular backup on data of an application program, copying is performed on a data set directory. The copy command flow creates a new file and then copies the file data by reading and writing. For copying a directory set containing a large amount of data, more time is often required, especially in services like AI and machine learning, an application data set is often a directory containing a large amount of small files, and for small files, the file data copy takes up less time, and the creation time of a large amount of small files takes up more time. Based on the above, the embodiment of the application provides a file copying method, device, server and storage medium, which can simultaneously obtain the file data and metadata of the file to be copied without obtaining the metadata of the file to be copied first and then obtaining the file data of the file to be copied, so as to quickly and accurately generate the target copy file corresponding to the file to be copied, and ensure the speed and efficiency of file copying.

The method, the device, the server and the storage medium for copying files provided in the embodiments of the present application may be applied to a file copying system, as shown in fig. 1, where the file copying system includes a terminal 101, a server 102 and a storage device 103. In general, in practical application, the connection between the above devices may be a wireless connection, and for convenience and intuitionistic representation of the connection relationship between the devices, a solid line is used in fig. 1 to illustrate.

Wherein the terminal 101 is configured to send a file processing request to the server 102 to access or manage a file stored in the storage device 103, the file processing request being a file copy request, a file delete request, a file modify request, or the like.

A server 102 for receiving a file processing request transmitted from the terminal 101 and processing a file stored in the storage device 103 based on the file processing request.

And a storage device 103 for storing one or more files, wherein a file includes metadata of the file and file data of the file, the metadata of the file is used for representing attribute information of the file, and the attribute information of the file is used for indicating access and management of the file data of the file.

The terminal for executing the file copying method provided in the embodiment of the present application may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (personal digital assistant, PDA), an augmented reality (augmented reality, AR) \virtual reality (VR) device, and the specific form of the terminal is not limited in the embodiment of the present application. The system can perform man-machine interaction with a user through one or more modes of a keyboard, a touch pad, a touch screen, a remote controller, voice interaction or handwriting equipment and the like.

The server for executing the file copying method provided by the embodiment of the application may be an independent physical server, may be a server cluster or a distributed system formed by a plurality of physical servers, or may be a cloud server for providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, network acceleration services (content delivery network, CDN), basic cloud computing services such as big data and artificial intelligence platforms, and the like.

The file copying method, device, server and storage medium provided by the embodiment of the application are applied to a scene of fast file copying in a distributed storage system. Specifically, the server can copy the metadata of the file to be copied and the file data of the file to be copied at the same time, so as to quickly and accurately obtain the copy file corresponding to the file to be copied, and complete the copying of the file to be copied.

As shown in fig. 2, the file copying method provided in the embodiment of the present application may include S101-S105.

S101, the terminal sends a file copy request to the server.

The file copy request includes a file name of a file to be copied, a location of the file to be copied, and a creation time of the file to be copied.

Optionally, the terminal sends the file copy request to the server through cephFS (distributed file system, ceph file storage, a subsystem of ceph (distributed storage system) for storing, accessing and managing files).

S102, the server acquires a file copy request sent by the terminal.

In one implementation manner of the embodiment of the present application, after receiving a file copy request sent by a terminal, a server creates a new file first, reads file data of a file to be copied and metadata of the file to be copied from a storage device (specifically, the file data of the file to be copied and the metadata of the file to be copied are both stored in rados, where rados is a distributed object storage system, and is composed of multiple OSDs (object storage cluster, object storage clusters, and data for managing one disk space storage)), and writes the file data of the file to be copied and the metadata of the file to be copied into the new file, so as to complete copying of the file to be copied.

Alternatively, the data is stored in rados in terms of a 4 Mbyte block of memory.

Optionally, the server includes an MDS (metadata server cluster, metadata server for caching and synchronizing distributed metadata), and reading and writing of metadata of the file to be copied is performed by the MDS.

In one implementation of the embodiments of the present application, creating the new file is accomplished by an asynchronous operation module in the server.

S103, the server determines a first position and a second position based on the file name of the file to be copied, the position of the file to be copied and the creation time of the file to be copied.

The first location is a storage location of file data of the file to be copied, the second location is a storage location of metadata of the file to be copied, the metadata of the file to be copied is used for representing attribute information of the file to be copied, and the attribute information of the file to be copied is used for indicating access and management of the file data of the file to be copied.

In one implementation manner of the embodiment of the present application, the server generates a file handle of the file to be copied based on a file name of the file to be copied, a location of the file to be copied, and a creation time of the file to be copied, where the file handle is used to indicate a storage location of file data of the file to be copied, and the server obtains a storage location of metadata of the file to be copied based on the file name of the file to be copied.

Optionally, the MDS includes a metadata processing module and a caching module, the server caches the file handle in a memory of the server, and sends a file creation request to the MDS, where the metadata processing module generates metadata of a new file based on the file creation request and metadata of the file to be copied.

In one implementation manner of the embodiment of the present application, the above-mentioned caching module further stores the metadata of the new file cached in the MDS memory in rados, that is, completes the persistence of the metadata of the new file.

In an implementation manner of the embodiment of the present application, when the above-mentioned caching module caches metadata of a new file in the MDS memory, the server generates a journ log corresponding to the new file, so as to ensure transactional performance of metadata operation.

S104, the server acquires file data of the file to be copied based on the first position, and acquires metadata of the file to be copied based on the second position.

In one implementation manner of the embodiment of the present application, the MDS in the server obtains metadata of the file to be copied from the rados based on the storage location of the metadata to be copied, and the server obtains file data of the file to be copied from the rados based on the storage location of the file data to be copied.

S105, the server generates a target copy file based on file data of the file to be copied and metadata of the file to be copied.

It should be understood that the target copy file includes file data of the target copy file and metadata of the target copy file.

In combination with fig. 2, as shown in fig. 3, in one implementation, the server generates a target copy file based on file data of the file to be copied and metadata of the file to be copied, including S1051.

S1051, the server writes the file data of the file to be copied into the file data of the target copy file, and writes the target metadata of the file to be copied into the metadata of the target copy file.

The target metadata includes attribute information except for a file name of the file to be copied, a position of the file to be copied and creation time of the file to be copied in the attribute information of the metadata of the file to be copied.

It should be understood that, the server writes the file data of the file to be copied into the file data of the target copy file, so that the file data between the target copy file and the file to be copied are identical, the server writes the target metadata of the file to be copied into the metadata of the target copy file, and the target metadata includes attribute information except for the file name of the file to be copied, the position of the file to be copied and the creation time of the file to be copied in the attribute information of the metadata of the file to be copied, so that other attribute information except for the file name, the position and the creation time in the metadata of the target copy file are identical to the file to be copied, thereby completing the copying of the file to be copied.

It will be appreciated that, because the file name of the target copy file, the location of the target copy file, and the file creation time of the target copy file are all different from those of the file to be copied, the server writes the target metadata into the metadata of the target copy file, instead of writing all the metadata of the file to be copied into the metadata of the target copy file.

The technical scheme provided by the embodiment at least has the following beneficial effects: from S101 to S105, it is known that: the method comprises the steps that a terminal sends a file copying request to a server, the server obtains the file copying request sent by the terminal, the server determines a first position and a second position based on a file name of a file to be copied, a position of the file to be copied and creation time of the file to be copied, and the server obtains file data of the file to be copied based on the first position; the server obtains metadata of the file to be copied based on the second position, and generates the target copy file based on the file data of the file to be copied and the metadata of the file to be copied, so that the file data and the metadata of the file to be copied can be obtained simultaneously without obtaining the metadata of the file to be copied first and then obtaining the file data of the file to be copied, further, the target copy file corresponding to the file to be copied is generated rapidly and accurately, and the speed and the efficiency of file copying are guaranteed.

In connection with fig. 3, as shown in fig. 4, in one implementation, the file copying method further includes S106-S111.

S106, the server adds the first operation to the target mapping queue.

The first operation is writing the target metadata of the file to be copied into the metadata of the target copy file, and the target mapping queue is used for indicating the file corresponding to each of the one or more operations.

Illustratively, the object mapping queue stores operations corresponding to the object copy file based on the form of key-value pairs.

Optionally, the target mapping queue includes at least one service request and a hash value corresponding to each service request, where the hash value corresponding to one service request is used to represent a path name of a file, and the path name of a file is used to represent a storage location of the file.

It may be appreciated that the server adds the first operation to an operation mapping queue, where the operation mapping queue is used to indicate files corresponding to one or more operations respectively, so that the server may determine files corresponding to the operations respectively based on the operation mapping queue.

And S107, the terminal sends a service request to the server.

Wherein the service request is for requesting processing of metadata of the target copy file.

Alternatively, the service request may be used to request that metadata of the target copy file be processed in the manner of access, deletion, modification, and the like.

S108, the server receives a service request sent by the terminal.

S109, in the case where the first operation is not completed, the server adds the service request to the request waiting queue.

The request waiting queue is used for storing the operation corresponding to the next service request when the current operation of one file is not completed.

It should be appreciated that in the case where the first operation is not completed, if the service request is executed at the same time, there may be a conflict due to the first operation and the service request processing the metadata of the target copy file at the same time, and at this time, the server adds the service request to the request waiting queue, i.e. does not execute the service request temporarily.

For example, if the first operation is used to delete the metadata of the target copy file, the first operation is incomplete, that is, the metadata of the file to be copied is not yet written into the metadata of the target copy file, and at this time, the first operation cannot be executed, that is, the first operation and the service request are executed simultaneously, which may result in a conflict.

S110, deleting the service request from the request waiting queue and deleting the first operation from the target mapping queue and adding the second operation to the target mapping queue when the first operation is completed.

Wherein the second operation is to process the target copy file based on the service request.

In connection with the above S109, it should be understood that, in the case where the first operation is completed, there is no occurrence of a case where the first operation and the service request are simultaneously executed, that is, there is no operation conflict, at this time, the server processes the target copy file based on the service request, and at the same time, since the request waiting queue is used to store an operation corresponding to a next service request that has come when the current operation of one file is not completed, when the service request is executed, the server deletes the service request from the request waiting queue to avoid the server from repeatedly acquiring the service request in the request waiting queue.

S111, the server executes a second operation based on the target mapping queue.

It will be appreciated that, since the second operation is located in the target mapping queue, and the target mapping queue is used to indicate files corresponding to one or more operations, the server may determine, based on the target mapping queue, that the service request requests processing of the target copy file, and then the second operation is performed by the server, that is, the server performs the second operation on the target copy file.

In one implementation manner of the embodiment of the present application, the first operation is completed by the MDS in the server, and when the first operation is completed, the MDS sends a first operation completion response to the server, and the server wakes up the service request in the request waiting queue to perform the subsequent steps.

In this embodiment, a server adds a first operation to a target mapping queue, a terminal sends a service request to the server, the server receives the service request sent by the terminal, in the case that the first operation is incomplete, if the service request is executed simultaneously, because there may be a conflict between the first operation and metadata of the target copy file being processed by the service request, at this time, the server adds the service request to a request waiting queue, that is, temporarily does not execute the service request, in the case that the first operation is completed, there is no operation conflict, at this time, the server processes the target copy file based on the service request, and, in the case that the first operation is completed, the server is used to store the next service request that has arrived when the last operation is not completed, and when the service request is executed, the server deletes the service request from the request waiting queue to avoid the server from repeatedly obtaining the service request in the request waiting queue, in the target mapping queue, because the second operation is located in the target mapping queue, and the target mapping queue is used to indicate one or more corresponding operations, in the case that there is no operation conflict between the first operation and the target copy file being processed by the server, the target copy file can be processed when the second operation is not completed, the target copy file can be processed by the server based on the service request, and the target copy file can be further processed when the copy file is not processed by the target copy queue, the availability of access and management to the target copy file is reliably ensured while the quick copy file is ensured.

In an implementation manner of this embodiment of the present application, the file copy request further includes a file system to which the file to be copied belongs, where the file system to which the file to be copied belongs is an asynchronous file system or a synchronous file system. In connection with fig. 1, as shown in fig. 5, in one implementation, the file copying method further includes S112-S114.

And S112, determining a second position based on the file name of the file to be copied, the position of the file to be copied and the creation time of the file to be copied under the condition that the file system to which the file to be copied belongs is a synchronous file system.

It should be understood that, in the case that the file system to which the file to be copied belongs is an asynchronous file system, it is described that the user selects to perform asynchronous copying operation on the file to be copied, that is, the user selects to use a mode of simultaneously acquiring file data of the file to be copied and metadata of the file to be copied and generating a target copy file, at this time, the server determines the first position and the second position based on the file name of the file to be copied, the position of the file to be copied and the creation time of the file to be copied; and under the condition that the file system to which the file to be copied belongs is a synchronous file system, the user is informed of selecting not to carry out asynchronous copying operation on the file to be copied, and at the moment, the server determines the second position, namely the storage position of the metadata to be copied, based on the file name of the file to be copied, the position of the file to be copied and the creation time of the file to be copied.

In one implementation of the embodiments of the present application, the server sets file system creation operation async based on ceph fs set < filesystem > async command; specifically, the server modifies the fsmap (a map topology containing a plurality of file system attributes) and synchronizes the fsmap to each cluster by a monitor (a topology for the storage cluster) via the paxos protocol; the server judges a file system to which a file requested in a user request belongs based on the fsmap, if the file system belongs to an asynchronous file system, the file request is created to carry out asynchronous processing, and if the file system to which the file to be copied belongs is not synchronous (namely, if the file system to which the file to be copied belongs is synchronous file system, the server determines a storage position of the metadata to be copied based on a file name of the file to be copied, a position of the file to be copied and creation time of the file to be copied, the server acquires the metadata to be copied based on the storage position of the metadata to be copied, acquires the file data to be copied based on the metadata to be copied, and further generates the target copy file based on the metadata to be copied and the file data to be copied).

S113, the server acquires metadata of the file to be copied based on the second position.

S114, the server acquires file data of the file to be copied based on metadata of the file to be copied.

In connection with S112 above, it should be understood that, in the case where the file system to which the file to be copied belongs is a synchronous file system, the server determines the storage location of the metadata to be copied based on the file name of the file to be copied, the location of the file to be copied, and the creation time of the file to be copied, and the server obtains the metadata to be copied based on the storage location of the metadata to be copied.

It may be understood that the attribute information of the file to be copied is used to indicate access and management to the file data of the file to be copied, and indicates that the attribute information of the file to be copied (i.e., the metadata of the file to be copied) includes a storage location of the file data of the file to be copied, so that the server may access the file data of the file to be copied based on the storage location of the file data of the file to be copied.

Specifically, the server obtains the storage position of the file data of the file to be copied from the metadata of the file to be copied, and obtains the file data of the file to be copied based on the storage position of the file data of the file to be copied.

In this embodiment, when the file system to which the file to be copied belongs is a synchronous file system, it is described that the user chooses not to perform asynchronous copying operation on the file to be copied, at this time, the server determines the second location, that is, the storage location of the metadata to be copied, based on the file name of the file to be copied, the location of the file to be copied, and the creation time of the file to be copied, and the server obtains the metadata of the file to be copied based on the second location.

Fig. 6 is a schematic diagram of an internal structure of a server according to an embodiment of the present application, including a main server module, an MDS module, and a rados module.

The server is used for generating metadata of the target copy file and file data of the target copy file.

The MDS module comprises a metadata processing module and an MDS cache module, wherein the metadata processing module is used for constructing metadata of the target copy file, the MDS cache module is used for storing the metadata of the target copy file in a memory of the MDS, the MDS cache module is used for sending the metadata of the target copy file to the main server cache module, and the MDS cache module is also used for storing the metadata of the target copy file in the rados module so as to realize persistence of the target copy file.

And the rados module is used for storing the file data of the target copy file, the metadata of the target copy file, the file data to be copied and the metadata to be copied, and providing reading, writing or deleting services of the file data or the metadata.

Fig. 7 is a schematic structural diagram of the target mapping queue and the request waiting queue according to the embodiment of the present application.

The target mapping queue comprises a plurality of files and operations corresponding to the files respectively.

The request waiting queue comprises a plurality of service requests, wherein one service request comprises the operation of the service request, a processing file of the service request and the request time of the service request, and the plurality of service requests in the request waiting queue are arranged according to the time sequence.

It should be understood that when an operation of a file in the target mapping queue is completed, the server obtains a service request corresponding to the file from the request waiting queue, and when a plurality of service requests corresponding to the file are obtained, the server obtains an earliest service request from the plurality of service requests corresponding to the file, and based on the service request, the server determines an operation corresponding to the service request and adds the operation to the target mapping queue, the operation being located at a position corresponding to the file in the target mapping queue.

The embodiment of the present application may divide the functional modules of the server or the like according to the above method example, for example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The integrated modules may be implemented in hardware or in software functional modules. It should be noted that, in the embodiment of the present application, the division of the modules is schematic, which is merely a logic function division, and other division manners may be implemented in actual implementation.

In the case of dividing the respective function modules with the respective functions, fig. 8 shows a schematic diagram of one possible configuration of the file copying apparatus involved in the above-described embodiment, and as shown in fig. 8, the file copying apparatus 20 may include: an acquisition module 201, a determination module 202 and a processing module 203.

The obtaining module 201 is configured to obtain a file copy request sent by a terminal, where the file copy request includes a file name of a file to be copied, a location of the file to be copied, and a creation time of the file to be copied.

The determining module 202 is configured to determine a first location and a second location based on a file name of the file to be copied, a location of the file to be copied, and a creation time of the file to be copied, where the first location is a storage location of file data of the file to be copied, the second location is a storage location of metadata of the file to be copied, the metadata of the file to be copied is used for characterizing attribute information of the file to be copied, and the attribute information of the file to be copied is used for indicating access and management of the file data of the file to be copied.

The obtaining module 201 is further configured to obtain file data of the file to be copied based on the storage location of the file data to be copied, and obtain metadata of the file to be copied based on the storage location of the metadata to be copied.

And the processing module 203 is configured to generate a target copy file based on the file data of the file to be copied and the metadata of the file to be copied.

Optionally, the processing module 203 is specifically configured to write file data of the file to be copied into file data of the target copy file, and write target metadata of the file to be copied into metadata of the target copy file, where the target metadata includes attribute information, except for a file name of the file to be copied, a location of the file to be copied, and a creation time of the file to be copied, in attribute information of the metadata of the file to be copied.

Optionally, the apparatus further comprises a receiving module 204.

The processing module 203 is further configured to add the first operation to a target mapping queue, where the first operation is writing target metadata of the file to be copied into metadata of the target copy file, the first operation corresponds to the target copy file, and the target mapping queue is used to indicate files corresponding to one or more operations respectively.

And the receiving module 204 is configured to receive a service request sent by the terminal, where the service request is used to request processing of metadata of the target copy file.

The processing module 203 is further configured to add the service request to a request waiting queue, where the first operation is not completed, and the request waiting queue is configured to store a next service request that has come when a current operation of a file is not completed.

The processing module 203 is further configured to delete the service request from the request waiting queue and delete the first operation from the target mapping queue and add a second operation to the target mapping queue, where the second operation is to process the target copy file based on the service request, and the second operation corresponds to the target copy file.

The processing module 203 is further configured to perform the second operation based on the target mapping queue.

Optionally, the file copy request further includes a file system to which the file to be copied belongs, where the file system to which the file to be copied belongs is an asynchronous file system or a synchronous file system.

The determining module 202 is further configured to determine, when the file system to which the file to be copied belongs is a synchronous file system, the second location based on the file name of the file to be copied, the location of the file to be copied, and the creation time of the file to be copied.

The obtaining module 201 is further configured to obtain metadata of the file to be copied based on the second location.

The obtaining module 201 is further configured to obtain file data of the file to be copied based on metadata of the file to be copied.

In case of an integrated unit, fig. 9 shows a schematic diagram of a possible structure of the file copying apparatus involved in the above-described embodiment. As shown in fig. 9, the file copying apparatus 30 may include: a processing module 301 and a communication module 302. The processing module 301 may be used to control and manage the actions of the file copying apparatus 30. The communication module 302 may be used to support communication of the file copy device 30 with other entities. Optionally, as shown in fig. 9, the file copying apparatus 30 may further include a storage module 303 for storing program codes and data of the file copying apparatus 30.

Wherein the processing module 301 may be a processor or a controller. The communication module 302 may be a transceiver, a transceiver circuit, a communication interface, or the like. The storage module 303 may be a memory.

When the processing module 301 is a processor, the communication module 302 is a transceiver, and the storage module 303 is a memory, the processor, the transceiver, and the memory may be connected through a bus. The bus may be a peripheral component interconnect standard (peripheral component interconnect, PCI) bus or an extended industry standard architecture (extended industry standard architecture, EISA) bus, or the like. The buses may be divided into address buses, data buses, control buses, etc.

It should be understood that, in various embodiments of the present application, the sequence numbers of the foregoing processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present application.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in accordance with embodiments of the present application are produced in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber terminal line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device including one or more servers, data centers, etc. that can be integrated with the medium. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a Solid State Disk (SSD)), or the like.

The foregoing is merely specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (10)

1. A method of copying a file, the method comprising:

acquiring a file copy request sent by a terminal, wherein the file copy request comprises a file name of a file to be copied, a position of the file to be copied and creation time of the file to be copied;

determining a first position and a second position based on the file name of the file to be copied, the position of the file to be copied and the creation time of the file to be copied, wherein the first position is a storage position of file data of the file to be copied, the second position is a storage position of metadata of the file to be copied, the metadata of the file to be copied is used for representing attribute information of the file to be copied, and the attribute information of the file to be copied is used for indicating access and management of the file data of the file to be copied;

Acquiring file data of the file to be copied based on the first position, and acquiring metadata of the file to be copied based on the second position;

and generating a target copy file based on the file data of the file to be copied and the metadata of the file to be copied.

2. The file copying method according to claim 1, wherein generating the target copy file based on file data of the file to be copied and metadata of the file to be copied, comprises:

writing the file data of the file to be copied into the file data of the target copy file, and writing the target metadata of the file to be copied into the metadata of the target copy file, wherein the target metadata comprises attribute information except for the file name of the file to be copied, the position of the file to be copied and the creation time of the file to be copied in the attribute information of the metadata of the file to be copied.

3. The method of copying a file according to claim 2, wherein said method further comprises:

adding a first operation into a target mapping queue, wherein the first operation is the writing of target metadata of the file to be copied into metadata of the target copy file, and the target mapping queue is used for indicating files corresponding to one or more operations respectively;

Receiving a service request sent by the terminal, wherein the service request is used for requesting to process metadata of the target copy file;

when the first operation is not completed, adding the service request into a request waiting queue, wherein the request waiting queue is used for storing the next service request which comes when the current operation of one file is not completed;

deleting the service request from the request wait queue and deleting the first operation from the target map queue and adding a second operation to the target map queue, the second operation being to process the target copy file based on the service request, if the first operation is complete;

and executing the second operation based on the target mapping queue.

4. A file copying method according to any of claims 1-3, wherein said file copying request further includes a file system to which said file to be copied belongs, said file system to which said file to be copied belongs being an asynchronous file system or a synchronous file system, said method further comprising:

determining the second position based on the file name of the file to be copied, the position of the file to be copied and the creation time of the file to be copied under the condition that the file system to which the file to be copied belongs is a synchronous file system;

Acquiring metadata of the file to be copied based on the second position;

and acquiring file data of the file to be copied based on the metadata of the file to be copied.

5. A document copying apparatus, comprising: the device comprises an acquisition module, a determination module and a processing module;

the acquisition module is used for acquiring a file copy request sent by the terminal, wherein the file copy request comprises a file name of a file to be copied, a position of the file to be copied and creation time of the file to be copied;

the determining module is configured to determine a first location and a second location based on a file name of the file to be copied, a location of the file to be copied, and creation time of the file to be copied, where the first location is a storage location of file data of the file to be copied, the second location is a storage location of metadata of the file to be copied, the metadata of the file to be copied is used to characterize attribute information of the file to be copied, and the attribute information of the file to be copied is used to indicate access and management of the file data of the file to be copied;

the acquisition module is further configured to acquire file data of the file to be copied based on the first location, and acquire metadata of the file to be copied based on the second location;

And the processing module is used for generating a target copy file based on the file data of the file to be copied and the metadata of the file to be copied.

6. The document copying apparatus according to claim 5, wherein,

the processing module is specifically configured to write file data of the file to be copied into file data of the target copy file, and write target metadata of the file to be copied into metadata of the target copy file, where the target metadata includes attribute information, except for a file name of the file to be copied, a location of the file to be copied, and creation time of the file to be copied, in attribute information of the metadata of the file to be copied.

7. The file copying apparatus according to claim 6, wherein said apparatus further comprises a receiving module;

the processing module is further configured to add a first operation to a target mapping queue, where the first operation is the writing of target metadata of the file to be copied into metadata of the target copy file, the first operation corresponds to the target copy file, and the target mapping queue is used to indicate files corresponding to one or more operations respectively;

The receiving module is used for receiving a service request sent by the terminal, wherein the service request is used for requesting to process metadata of the target copy file;

the processing module is further configured to add the service request to a request waiting queue when the first operation is not completed, where the request waiting queue is configured to store a next service request that has come when a current operation of a file is not completed;

the processing module is further configured to delete the service request from the request waiting queue and delete the first operation from the target mapping queue and add a second operation to the target mapping queue if the first operation is completed, where the second operation is to process the target copy file based on the service request;

the processing module is further configured to perform the second operation based on the target mapping queue.

8. The file copying apparatus according to any one of claims 5 to 7, wherein the file copying request further includes a file system to which the file to be copied belongs, and the file system to which the file to be copied belongs is an asynchronous file system or a synchronous file system;

The determining module is further configured to determine, when the file system to which the file to be copied belongs is a synchronous file system, the second position based on a file name of the file to be copied, a position of the file to be copied, and a creation time of the file to be copied;

the acquisition module is further used for acquiring metadata of the file to be copied based on the second position;

the acquisition module is further used for acquiring file data of the file to be copied based on metadata of the file to be copied.

9. A server, the server comprising:

a processor;

a memory configured to store the processor-executable instructions;

wherein the processor is configured to execute the instructions to implement the file copy method of any of claims 1-4.

10. A computer readable storage medium having instructions stored thereon, which when executed by a server, cause the server to perform the file copy method of any of claims 1-4.