WO2018059222A1 - 一种文件切片上传方法、装置及云存储系统 - Google Patents

一种文件切片上传方法、装置及云存储系统 Download PDF

Info

Publication number
WO2018059222A1
WO2018059222A1 PCT/CN2017/101197 CN2017101197W WO2018059222A1 WO 2018059222 A1 WO2018059222 A1 WO 2018059222A1 CN 2017101197 W CN2017101197 W CN 2017101197W WO 2018059222 A1 WO2018059222 A1 WO 2018059222A1
Authority
WO
WIPO (PCT)
Prior art keywords
file
slice
server
available
object storage
Prior art date
Application number
PCT/CN2017/101197
Other languages
English (en)
French (fr)
Inventor
卢洲
Original Assignee
杭州海康威视数字技术股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 杭州海康威视数字技术股份有限公司 filed Critical 杭州海康威视数字技术股份有限公司
Publication of WO2018059222A1 publication Critical patent/WO2018059222A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Definitions

  • the present application relates to the field of computer and Internet data processing technologies, and in particular, to a file slice uploading method, device, and cloud storage system.
  • cloud storage systems have become the trend of Internet development. From the perspective of application architecture, the difference between cloud storage systems and traditional distributed storage systems is not obvious. The more essential difference is reflected in the internal software architecture.
  • the cloud storage system builds a large-scale storage cluster based on a large number of servers and storage devices. To provide storage capacity and to expand capacity online, the overall cost of building a large-capacity storage system is much lower than that of a traditional storage architecture. As the number of mobile users increases, the amount of data in cloud storage systems increases. Therefore, the file uploading technology of cloud storage systems has become a new challenge in the development of cloud storage systems.
  • the client uploads the file to the server or the corresponding storage device for storage.
  • the file when the file is uploaded, it will occupy a large amount of resources, such as the CPU (Central Processing Unit), port, traffic, memory, and disk I/O (input/output, input/output port) of the server. If the load on the server is too large, it will easily cause excessive delay or packet loss when the file is uploaded, which will result in file upload failure. At the same time, the network environment is bad, which is also an important factor causing file upload failure.
  • the present invention provides a file slice uploading method, device, and cloud storage system, and when a server load is too large or a bad network environment occurs, files can be uploaded normally.
  • the specific technical solutions are as follows:
  • the embodiment of the present application provides a file slice uploading method, which is applied to a metadata server in a cloud storage system, where the cloud storage system further includes a plurality of slice servers and a plurality of object storage devices, and the method includes :
  • the allocation request carries: a file identifier of the target file, and an identification signal of the available slice server and the available object storage device;
  • the client Sending a device identifier of the available slicing server to the client, so that the client uploads a slice file obtained by slicing the target file with the file identifier to the available based on the device identifier Slice server, such that the available tile server stores a plurality of first slice files of the target file to the available object storage device, wherein the first slice file is for the target file
  • the slice file performs the slice file obtained by the raw data calculation method.
  • an embodiment of the present application provides a metadata server, including: a processor, a memory, a communication interface, and a bus;
  • the processor, the memory, and the communication interface are connected by the bus and complete communication with each other;
  • the memory is for storing executable program code
  • the processor is configured to execute a program corresponding to the executable program code by reading the executable program code stored in the memory, to execute the file slice uploading method provided by the first aspect of the embodiment of the present application .
  • an embodiment of the present application provides a cloud storage system, where the system includes a metadata server, multiple slice servers, and multiple object storage devices;
  • the metadata server is configured to receive an allocation request of the target file to be uploaded sent by the client, where the allocation request carries: a file identifier of the target file, and a slice server and an available object storage device that are available for application Identification signal; determining, according to current load information of the plurality of slice servers, an available slice server corresponding to the target file having the file identifier; Determining, according to the available load information of the available slice server and the plurality of object storage devices, an available object storage device corresponding to the target file having the file identifier; sending the device identifier of the available slice server to the Describe the client, so that the client uploads a slice file obtained by slicing the target file with the file identifier to the available slicing server based on the device identifier;
  • the slice server is configured to collect and send current load information of the slice server and current load information of the object storage device corresponding to itself to the metadata server; and receive the target file sent by the client a slice file storing a plurality of first slice files of the target file to the available object storage device, wherein the first slice file is a raw data calculation method performed on a slice file of the target file The resulting slice file.
  • the embodiment of the present application provides a storage medium for storing executable code, where the executable code is used to execute a file slice uploading method provided by the first aspect of the embodiment of the present application at runtime.
  • an embodiment of the present application provides an application program for performing a file slice uploading method provided by the first aspect of the embodiment of the present application at runtime.
  • the file slice uploading method, device, and cloud storage system can reasonably allocate the slice server and the object storage device according to the load information of the slice server and the object storage device by using the metadata server, and upload the file server and the object storage device.
  • the target file when the server load is too large, effectively avoids the file upload failure by selecting a reasonable slicing server and the object storage device; and by slicing the target file to be uploaded, when the network environment is bad, If the slice file is lost, the upload of the remaining slice files is not affected; and the original data calculation method is executed during the file uploading process to ensure that the file stored by the object storage device is a recoverable target file to be uploaded.
  • FIG. 1 is a flowchart of a file slice uploading method according to an embodiment of the present application
  • FIG. 2 is a flowchart of a file slice uploading method according to still another embodiment of the present application.
  • FIG. 3 is a flowchart of a file slice uploading method according to still another embodiment of the present application.
  • FIG. 4 is a flowchart of a file slice uploading method according to still another embodiment of the present application.
  • FIG. 5 is a flowchart of a file slice uploading method according to still another embodiment of the present application.
  • FIG. 6 is a schematic diagram of a logical structure of a file slice uploading apparatus according to an embodiment of the present application.
  • FIG. 7 is a schematic diagram of a logical structure of a file slice uploading apparatus according to still another embodiment of the present application.
  • FIG. 8 is a schematic diagram of a logical structure of a file slice uploading apparatus according to still another embodiment of the present application.
  • FIG. 9 is a schematic diagram showing the logical structure of a file slice uploading apparatus according to still another embodiment of the present application.
  • FIG. 10 is a schematic diagram showing the logical structure of a file slice uploading apparatus according to still another embodiment of the present application.
  • FIG. 11 is a schematic structural diagram of a cloud storage system according to an embodiment of the present application.
  • FIG. 12 is a schematic structural diagram of a metadata server according to an embodiment of the present application.
  • the embodiment of the present application provides a file slice uploading method, device, and cloud storage system.
  • An execution body of a file slice uploading method provided by an embodiment of the present application is a metadata server in a cloud storage system.
  • the function software for implementing the file slice uploading method provided by the embodiment is software provided in the metadata server.
  • a file slice uploading method provided by an embodiment of the present application may include the following steps:
  • S101 Receive an allocation request of a target file to be uploaded sent by a client.
  • the client Before uploading the target file, the client needs to send an allocation request to the metadata server for applying for an available slicing server and an available object storage device; and because at the same time, the client may simultaneously send multiple allocations for multiple target files.
  • the request in order to ensure that the slice server and the object storage device allocated by the metadata server correspond to the target file, the file identifier of the target file needs to be carried in the allocation request; the allocation request may be an HTTP (Hyper Transfer Protocol) message.
  • the metadata server provides services to the client in the form of the HTTP REST API (Hyper Transfer Protocol Representational State Transfer Application Programming Interface).
  • the client can be a dedicated client software on a desktop computer or a mobile phone, or a browser software on a desktop computer or a mobile phone, etc., which is reasonable.
  • the file identifier is identification information capable of uniquely determining the target file in the server, for example, a name of the target file, or a number assigned by the metadata server to the target file, and the like.
  • the metadata server is used for unified management of business applications and storage system metadata
  • the slicing server is used for data calculation
  • the object storage device is used for storing files.
  • the target file to be uploaded may be a video file, a picture file, or a voice file, and the like.
  • S102 Determine, according to current load information of the plurality of slice servers, an available slice server corresponding to the target file having the file identifier.
  • the load information of the slicing server needs to be obtained; the slicing server collects and sends the current load information of the slicing server and the current load information of the object storage device corresponding to itself to the metadata server; The metadata server receives the load information.
  • the load information of the slicing server may be the load size of the slicing server or the proportion of the load of the slicing server, and is of course not limited thereto.
  • the cloud storage system includes multiple slicing servers, based on the load information of each slicing server.
  • the metadata server selects multiple slice servers from different slice servers according to the current load information of each slice server.
  • the selection method may be a rotation method, a random method, or the like, and selects available.
  • Slice server may be a rotation method, a random method, or the like, and selects available.
  • the current load information of the slicing server there are various implementations of determining the available slicing servers corresponding to the target file having the file identifier.
  • the specific implementation manner of determining the available slicing server corresponding to the target file with the file identifier is described later.
  • the available slicing server and the object storage device may be used.
  • the current load information is used to determine an available object storage device corresponding to the object file having the file identifier.
  • the load information of the object storage device is the load size of the object storage device or the load proportion of the object storage device, which is of course not limited thereto.
  • the cloud storage system includes a plurality of object storage devices, and the object storage device can have a correspondence with the slice server.
  • the so-called correspondence specifically means that the object storage device can be set in the slice server, or the object storage device is associated with the slice server. ,and many more.
  • the object storage device is set in the slicing server, the object storage device and the slicing server have the same IP (Internet Protocol) address, and the load information is the same.
  • IP Internet Protocol
  • the available object storage device corresponding to the target file having the file identifier is determined according to the current load information of the available slice server and the object storage device.
  • S104 Send the device identifier of the available slicing server to the client, so that the client uploads the slice file obtained by slicing the target file with the file identifier to the available slicing server based on the device identifier, thereby making the available slicing server
  • a plurality of first slice files of the target file are stored to an available object storage device.
  • the metadata server obtains the device identifier of the available slicing server after determining the available slicing server, and the metadata server sends the device identifier to the client, and the client is connected. After receiving the device identifier, it can determine which available slice servers to send the target file; in the process of sending the target file, the client slices the target file according to the preset slice policy, and then uploads the sliced slice file. After the slice server receives the slice file, the slice data is calculated by the original data calculation method to obtain a plurality of first slice files to ensure that the original target file can be restored through the plurality of first slice files. And then sending the plurality of first slice files to the available object storage device; after receiving the plurality of first slice files, the available object storage device, according to the preset disk selection method, the plurality of first slice files Store to a different disk.
  • the slice file is a file obtained by the client performing a slice operation on the target file according to a preset slice policy in the process of sending the target file, and the first slice file is obtained by calculating the original data calculation method of the slice file by the slice server. file.
  • the device identifier is identification information that uniquely identifies the available slicing servers in the server and client, such as the IP address of the available slicing server, or the number assigned by the metadata server to the available slicing servers, and so on.
  • the method for slicing the target file and the method for selecting the disk belong to the prior art, and are not described here.
  • the metadata server allocates the slice server and the object storage device according to the load information of the slice server and the object storage device, and uploads the target file.
  • the server load is too large, the reasonable slice server and the object storage device are selected. Effectively avoiding file upload failures; and by slicing the target file to be uploaded, if the network file environment is bad, if the slice file is lost, the upload of the remaining slice files will not be affected; and during the file upload process
  • the original data calculation method is executed to ensure that the file stored by the object storage device is a recoverable target file to be uploaded.
  • the file slice uploading method may further include the following steps before receiving the allocation request of the target file to be uploaded sent by the client:
  • the client Before sending the allocation request and uploading the target file, the client may send an initialization request to the metadata server for requesting the metadata server to perform an initialization operation before the target file is uploaded; the initialization operation includes: storing the basic information of the target file, and creating the target file. Version information and creation The initial identifier of the target file; the basic information of the target file may be attribute information of the target file or description information of the target file; the initialization request may be an HTTP message.
  • S202 Generate an initial identifier, and send the initial identifier to the client, so that the client uses the received initial identifier as a file identifier of the target file.
  • the initial identifier is sent to the client; after receiving the initial identifier, the client uses the initial identifier as the file identifier of the target file. It is used to carry the file identifier in the subsequent application allocation and uploading the slice file, so as to avoid the occurrence of the situation that the metadata server allocates the corresponding slice server or the object storage device.
  • the client Upon receiving the initial identifier, the client indicates that the initialization operation of the metadata server is completed, that is, the metadata server has stored basic information of the target file, version information of the created target file, and an initial identifier of the target file.
  • the metadata server stores the basic information of the target file for quickly indexing the metadata of the slice file corresponding to the target file in a subsequent step; the version information of the target file is sent to the slice server after being created, and is used for data calculation by the slice server. When used.
  • S203 to S206 are the same as S101 to S104 of the embodiment shown in FIG. 1, and details are not described herein again.
  • the metadata server allocates the slice server and the object storage device according to the load information of the slice server and the object storage device, and uploads the target file.
  • the server load is too large, the reasonable slice server and the object storage device are selected. Effectively avoiding file upload failures; by slicing the target file to be uploaded, if the network file is bad, if the slice file is lost, the upload of the remaining slice files will not be affected; the original is executed during the file upload process.
  • the data calculation method ensures that the file stored by the object storage device is a recoverable target file to be uploaded; and before the client sends the allocation request, first applies for initialization, and the metadata server corresponds to the target file production initial identifier, and the initial
  • the file identifier sent by the identifier to the client as the target file can avoid the occurrence of a meta-server that does not correspond to the slicing server or the object storage device, and the file identifier is more convenient by the unified numbering of the metadata server. Small memory.
  • a file slice uploading method in the embodiment of the present application determines an available object corresponding to the target file having the file identifier. After storing the device, it can also include:
  • the device identifier of the available slicing server is sent to the client, so that the client uploads the slice file obtained by slicing the target file with the file identifier to the available slicing server based on the device identifier, thereby making the available slice
  • the server stores the plurality of first slice files of the target file to the available object storage device, which may include:
  • the received object identifier stores a plurality of first slice files of the target file to an available object storage device.
  • the object identifier of the available object storage device is obtained, and the metadata server sends the object identifier to the available slice server, and the available slice server receives the object.
  • the identifier it can be determined which available object storage devices can store the first slice file; the available object storage device, after receiving the plurality of first slice files, according to a preset disk selection method, the first slice The files are stored on different disks.
  • the object identifier is identification information that can uniquely determine the available object storage device in the server and the storage device, for example: the IP address of the available object storage device, or the number assigned by the metadata server to the available object storage device, etc. .
  • the metadata server sends the object identifier of the available object storage device to the available slice server when determining the available object storage device, so that the available slice server sends the target file corresponding to the available object storage device.
  • the first slice file ensures the accuracy of the file upload.
  • the available slice server may receive the After the file is sliced, the object identifier of the available object storage device is requested from the metadata server.
  • the device identifier of the available slicing server is sent to the client, so that the client uploads the slice file obtained by slicing the target file with the file identifier to the available slicing server based on the device identifier. Therefore, the available slicing server stores the plurality of first slice files of the target file to the available object storage device, which may include:
  • the data server requests an object identifier of the available object storage device and stores a plurality of first slice files of the target file to the available object storage device based on the requested object identifier.
  • the object identifier is identification information that can uniquely determine the available object storage device in the server and the storage device, for example: the IP address of the available object storage device, or the number assigned by the metadata server to the available object storage device, etc. .
  • the metadata server determines the available object storage device
  • the object identifier of the available object storage device is stored or generated
  • the available slice server obtains the first slice file after performing the calculation of the slice file.
  • the metadata server applies to store the first slice file, and the metadata server sends the object identifier to the available slice server, so that the available slice server sends the first slice file corresponding to the target file to the available object storage device. Guarantee the accuracy of file uploading.
  • the allocation of the slicing server is performed reasonably, and according to the current load information of the plurality of slicing servers, the available slicing server corresponding to the file having the file identifier may be included, which may include :
  • load values of the plurality of slice servers wherein the load value is at least one of a usage rate of the central processing unit in the slice no service, a usage rate of the memory, and a usage rate of the input/output device;
  • the available tile server corresponding to the object file having the file identifier is selected.
  • the preset first load threshold is set according to the actual load condition of the slice server. If the load value of the slice server is less than the preset first load threshold, the success rate of sending the slice file of the target file to the slice server is large. The slice file may be sent to the slice server. If the load value of the slice server is greater than the preset first load threshold, the load of the slice server is too large, and the slice file is not sent to the slice server.
  • the random method, or the round robin method, or other selection methods may be used for selection.
  • the allocation of the object storage device is performed, and the target file with the file identifier is determined according to the available load information of the slice server and the object storage device.
  • Corresponding available object storage devices which may include:
  • the first object storage device is used as an available object storage device
  • the available object storage device corresponding to the object file having the file identifier is selected.
  • the load information of the object storage device is the same as the load information of the available slice server, and the object storage can be directly determined.
  • the device is an available object storage device, which simplifies the process of determining the load information of the object storage device. If there is no object storage device with the same IP address as the available slice server, the object storage device needs to be The load information of the storage device is judged, and then the available object storage device is determined.
  • the object storage device when there is no object storage device with the same IP address as the available slice server IP address in the plurality of object storage devices, the object storage device is allocated reasonably in order to determine the available object storage device.
  • the determining, based on the current load information of the plurality of object storage devices, the available object storage devices corresponding to the target file having the file identifier from the plurality of object storage devices may include:
  • the available object storage device corresponding to the object file having the file identifier is selected.
  • the preset second load threshold is set according to the actual load condition of the object storage device. If the load value of the target storage device is less than the preset second load threshold, the first slice file of the target file is sent to the target storage device. If the success rate of the object storage device is greater than the preset second load threshold, the allocation may be performed according to the number of times the target storage device is allocated, and the preset allocation is performed. The number of times thresholds can be set according to historical data. The greater the number of allocated times, the higher the reliability of the first slice file of the object storage device storing the target file, the object storage device can be selected as the available object storage device storage. The first slice file of the target file.
  • a random method or a rotation method may be adopted. Or other options to choose.
  • the original data calculation method in order to restore the original target file when the slice file of the target file is lost, is an EC (Erasure Code) algorithm.
  • the specific principle of the EC algorithm is to add n copies of the original slice file to the m-check file, and restore the original slice file by any n files in the n+m share.
  • the EC algorithm includes two processes of encoding and decoding.
  • the original n pieces of slice files are changed into n+m files, and the n+m files can be distributed and stored in different positions of the available slice files, if any less than m
  • the file is invalid and can still be restored by the remaining data.
  • the process of restoration is decoding.
  • the specific process of the EC algorithm belongs to the prior art and will not be described here.
  • a file slice uploading method provided by an embodiment of the present application sends a device identifier of an available slicing server to a client, so that the client is based on the device identifier.
  • the method may further include:
  • the first metadata is the metadata of the first slice file; the metadata is the data of the data, that is, the index information, the attribute information, or the description information of the file can be mapped.
  • the metadata includes the disk serial number.
  • the disk serial number is the disk serial number storing the first slice file, and the disk is determined by the available object storage device according to the memory size of the first slice file.
  • the metadata table in the metadata server is used to record information of the first slice file stored in the available object storage device, including the slice file identifier of the first slice file and the disk serial number, as needed When the target file is called, it only needs to apply to the metadata server, and can index the first slice file stored in the available object storage device according to the slice file identifier and the disk serial number, and call it through the index.
  • S101 to S104 are method steps of the embodiment shown in FIG. 1, and details are not described herein again.
  • the metadata server allocates the slice server and the object storage device according to the load information of the slice server and the object storage device, and uploads the target file.
  • the server load is too large, the reasonable slice server and the object storage device are selected. Effectively avoiding file upload failures; and by slicing the target file to be uploaded, if the network file environment is bad, if the slice file is lost, the upload of the remaining slice files will not be affected; and during the file upload process
  • the original data calculation method is executed to ensure that the file stored by the object storage device is a recoverable target file to be uploaded.
  • the metadata server stores the first metadata to the metadata table, so as to facilitate aggregation and invocation of subsequent target files.
  • a file slice uploading method provided by an embodiment of the present application may further include the following, after storing the first metadata sent by the available slicing server to the metadata table. step:
  • S106 Send a storage success instruction corresponding to the plurality of first metadata to an available slice server, so that the available slice server sends an upload end instruction of the target file to the client after receiving the storage success instruction.
  • the storage success instruction may store the success status information for the first metadata or any instruction signal indicating that the storage is successful. After the first metadata is successfully stored, it indicates that the target file has been uploaded, and the upload end instruction should be sent to the client, so that the client performs the subsequent steps; the upload completion command may be the status information that the target file has been uploaded, or Any command signal that has been uploaded.
  • the client After receiving the upload completion command, the client needs to send an aggregation request to the metadata server for applying the first metadata of the target file corresponding to the aggregate file identifier; and because the client may be targeted at the same time Target files send multiple aggregate requests at the same time, in order to Ensure that the first metadata of the metadata server is the same as the target file, and the aggregation request needs to carry the file identifier of the target file; the aggregation request can be an HTTP message.
  • a plurality of first metadata corresponding to the target file having the file identifier in the metadata table is aggregated based on the file identifier of the target file.
  • the first metadata corresponding to the object file having the file identifier in the aggregation metadata table may be the first metadata stored in the adjacent table in the data table, or the first metadata may be It is reasonable to bind the entries stored in the data table.
  • S101 to S105 are the method steps of the embodiment shown in FIG. 3, and details are not described herein again.
  • the metadata server allocates the slice server and the object storage device according to the load information of the slice server and the object storage device, and uploads the target file.
  • the server load is too large, the reasonable slice server and the object storage device are selected. Effectively avoiding file upload failures; and by slicing the target file to be uploaded, if the network file environment is bad, if the slice file is lost, the upload of the remaining slice files will not be affected; and during the file upload process
  • the original data calculation method is executed to ensure that the file stored by the object storage device is a recoverable target file to be uploaded.
  • the metadata server stores the first metadata to the metadata table, so as to facilitate aggregation and calling of the subsequent target files; and the metadata information of the first slice file is aggregated by the metadata server, and the file call is raised when the file is called. speed.
  • a file slice uploading method provided by the embodiment of the present application may further include:
  • the client After receiving the upload completion instruction, the client sends a termination request to the metadata server for applying the metadata to release the remaining space; and because the client may simultaneously send multiple termination requests for multiple target files at the same time, In order to ensure that the information deleted by the metadata server corresponds to the target file, the termination request needs to carry the file identifier of the target file; the termination request may be an HTTP message.
  • S101 to S108 are the method steps of the embodiment shown in FIG. 4, and details are not described herein again.
  • the metadata server allocates the slice server and the object storage device according to the load information of the slice server and the object storage device, and uploads the target file.
  • the server load is too large, the reasonable slice server and the object storage device are selected. Effectively avoiding file upload failures; and by slicing the target file to be uploaded, if the network file environment is bad, if the slice file is lost, the upload of the remaining slice files will not be affected; and during the file upload process
  • the original data calculation method is executed to ensure that the file stored by the object storage device is a recoverable target file to be uploaded.
  • the metadata server stores the first metadata to the metadata table, so as to facilitate aggregation and calling of the subsequent target files; and the metadata information of the first slice file is aggregated by the metadata server, and the file call is raised when the file is called.
  • the speed of the metadata server is effectively released after the upload of the target file is completed, thereby avoiding wasted space of the metadata server.
  • the file slice uploading method provided by the embodiment of the present application is introduced in the following with reference to a specific application example.
  • the file slice upload process is as follows:
  • the user selects the video file A to upload through the browser on the desktop computer.
  • the metadata server detects the upload instruction, which is equivalent to obtaining the initialization request, and the metadata server allocates the initial identifier 0001 as the video file A.
  • the file identifier, and then the file identifier 0001 is sent to the browser.
  • the browser When the browser receives the file identifier 0001, it is equivalent to obtaining the initialization completion flag, and then sends an allocation request to the metadata server.
  • the metadata server receives the allocation request, according to the load information of the slice server and the object storage device, Assign slice server 1, slice server 2
  • the slice server 3 and the object storage device 1, the object storage device 2, and the object storage device 3 are used for uploading and storing the video file A, wherein the IP address of the slice server 1 is 168.1.1.1, and the IP address of the slice server 2 is 202.1.1.1, the IP address of the slice server 3 is 155.1.1, the IP address of the object storage device 1 is 168.1.1.1, the IP address of the object storage device 2 is 202.1.1.26, and the IP address of the object storage device 3 is 155.1.1.125.
  • the metadata server transmits the IP addresses of the slicing server 1, the slicing server 2, and the slicing server 3 to the browser, and transmits the IP addresses of the object storage device 1, the object storage device 2, and the object storage device 3 to the slicing server.
  • the browser When receiving the IP addresses of the slicing server 1, the slicing server 2, and the slicing server 3, the browser starts uploading the video file A.
  • the video file A is fragmented according to the fragmentation method to obtain the fragment file A1.
  • the server 3 stores the received fragment file in the stripe buffer, and performs EC calculation to obtain the slice file B1, the slice file B2, and the slice file B3 respectively; then, the slice server 1, the slice server 2, and the slice server 3 respectively
  • the slice file B1, the slice file B2, and the slice file B3 are sent to the object storage device 1, the object storage device 2, the object storage device 3, the object storage device 1, the object storage device 2, and the object storage device 3 to select an appropriate disk pair slice file B1.
  • slice file B2, slice file B3 for storage, and return the disk serial number to the slice server; the slice server will slice the file A1, The file A2, the fragment file A3 and the corresponding disk serial number are sent to the metadata server, and then stored to the metadata table; after the storage is completed, the metadata server sends the storage completion instruction to the slicing server, and the slicing server further goes to the desktop computer browser. Send the upload result.
  • the metadata server allocates the slice server and the object storage device reasonably according to the load information of the slice server and the object storage device, uploads the target file, and selects a reasonable slice when the server load is too large.
  • the server and the object storage device can effectively prevent the file uploading failure from occurring; and the object file to be uploaded is sliced, so that if the network file environment is bad, if the slice file is lost, the uploading of the remaining slice files is not affected;
  • the original data calculation method is executed during the file uploading process to ensure that the file stored by the object storage device is a recoverable target file to be uploaded.
  • the embodiment of the present application provides a file slice uploading device.
  • the apparatus can include:
  • the first receiving module 610 is configured to receive an allocation request of the target file to be uploaded sent by the client, where the allocation request carries: a file identifier of the target file, and a slice server and an available object storage device that are available for application Identification signal;
  • the first determining module 620 is configured to determine, according to current load information of the multiple slice servers, an available slice server corresponding to the target file having the file identifier;
  • a second determining module 630 configured to determine, according to the available workload information of the available slice server and the plurality of object storage devices, an available object storage device corresponding to the target file having the file identifier;
  • a first sending module 640 configured to send a device identifier of the available slicing server to the client, so that the client uploads a target file slice with the file identifier based on the device identifier Obtaining a slice file to the available slice server such that the available slice server stores a plurality of first slice files of the target file to the available object storage device, wherein the first slice
  • the slice file is a slice file obtained by performing a raw data calculation on a slice file of the target file.
  • the metadata server allocates the slice server and the object storage device according to the load information of the slice server and the object storage device, and uploads the target file.
  • the server load is too large, the reasonable slice server and the object storage device are selected. Effectively avoiding file upload failures; and by slicing the target file to be uploaded, if the network file environment is bad, if the slice file is lost, the upload of the remaining slice files will not be affected; and during the file upload process
  • the original data calculation method is executed to ensure that the file stored by the object storage device is a recoverable target file to be uploaded.
  • a file slice uploading apparatus provided by the embodiment of the present application may further include:
  • the second receiving module 710 is configured to receive an initialization request of the target file sent by the client.
  • a second sending module 720 configured to generate an initial identifier, and send the initial identifier to the client, so that the client uses the received initial identifier as the target file File identifier.
  • the second determining module 630 and the first sending module 640 are the same, and are not described here.
  • the metadata server allocates the slice server and the object storage device according to the load information of the slice server and the object storage device, and uploads the target file.
  • the server load is too large, the reasonable slice server and the object storage device are selected. Effectively avoiding file upload failures; by slicing the target file to be uploaded, if the network file is bad, if the slice file is lost, the upload of the remaining slice files will not be affected; the original is executed during the file upload process.
  • the data calculation method ensures that the file stored by the object storage device is a recoverable target file to be uploaded; and before the client sends the allocation request, first applies for initialization, and the metadata server corresponds to the target file production initial identifier, and the initial
  • the file identifier sent by the identifier to the client as the target file can avoid the occurrence of a slice server or an object storage device in which the metadata allocation does not correspond, and the file identifier is more easily made by the metadata server to make the file identifier easier and occupy the memory. .
  • the file slice uploading apparatus of the embodiment of the present application may further include:
  • a third sending module after the second determining module 750 determines an available object storage device corresponding to the target file that has the file identifier, sending an object identifier of the available object storage device to The available slicing server;
  • the first sending module 760 is specifically configured to send a device identifier of the available slicing server to the client, so that the client uploads the file identifier based on the device identifier.
  • a slice file obtained by slicing the target file to the available slicing server, thereby causing the available slicing server to store a plurality of first slice files of the target file to the based object identifier based on the received object identifier
  • the available object storage device is specifically configured to send a device identifier of the available slicing server to the client, so that the client uploads the file identifier based on the device identifier.
  • the first sending module 760 is specifically configured to send a device identifier of the available slicing server to the client, so that the client uploads the file based on the device identifier.
  • a slice file obtained by slicing the target file of the identifier to the available slice service So that the available slicing server requests the object identifier of the available object storage device from the metadata server, and based on the requested object identifier, the first slice of the target file The file is stored to the available object storage device.
  • the first determining module 740 may include:
  • a first obtaining submodule configured to acquire a load value of the plurality of slicing servers, wherein the load value is at least one of a usage rate of a central processing unit, a usage rate of the memory, and a usage rate of the input/output device in the slicing server Species
  • a first determining sub-module configured to determine whether a plurality of first slicing servers exist in the plurality of slicing servers, wherein a load value of the first slicing server is less than a preset first load threshold;
  • a first determining submodule configured to determine, if the output of the first determining submodule is YES, selecting, from the plurality of first slicing servers, an available slicing server corresponding to the target file having the file identifier;
  • a second determining submodule configured to determine, if the output of the first determining submodule is no, select, from the plurality of slicing servers, an available slicing server corresponding to the target file having the file identifier.
  • the second determining module 750 may include:
  • a second obtaining submodule configured to obtain a network protocol IP address of the available slicing server and an IP address of the plurality of object storage devices
  • a second determining sub-module configured to determine whether a first object storage device exists in the plurality of object storage devices, wherein an IP address of the first object storage device is the same as an IP address of the available slicing server;
  • a third determining submodule configured to determine, if the output of the second determining submodule is YES, using the first object storage device as an available object storage device;
  • a fourth determining submodule configured to determine, if the output of the second determining submodule is no, based on current load information of the plurality of object storage devices, select a target file having the file identifier from the plurality of object storage devices Corresponding available object storage devices.
  • the fourth determining submodule may include:
  • An obtaining unit configured to acquire a load value of multiple object storage devices
  • a first determining unit configured to determine whether a plurality of second object storage devices exist in the plurality of object storage devices, wherein a load value of the second object storage device is less than a preset second load threshold
  • a fifth determining unit configured to determine, if the output of the first determining unit is YES, selecting, from the plurality of second object storage devices, an available object storage device corresponding to the target file having the file identifier;
  • a sixth determining unit configured to determine, if the output of the first determining unit is no, obtain the number of times of allocation of the plurality of object storage devices
  • a second determining unit configured to determine whether a plurality of third object storage devices exist in the plurality of object storage devices, wherein the number of times of allocating the third object storage device is greater than a preset number of allocation times threshold;
  • a seventh determining unit configured to determine, if the output of the second determining unit is YES, select an available object storage device corresponding to the target file having the file identifier from the plurality of third object storage devices.
  • the original data calculation method is an erasure code EC algorithm.
  • a file slice uploading apparatus provided by the embodiment of the present application may further include:
  • a storage module 650 configured to store first metadata sent by the available slicing server to a metadata table, where the first metadata is metadata of the first slice file, including storing the The disk serial number of a slice file.
  • the first receiving module 610, the first determining module 620, the second determining module 630, and the first sending module 640 in this embodiment are modules of the embodiment shown in FIG. 6, and details are not described herein again.
  • a file slice uploading apparatus provided by the embodiment of the present application may further include:
  • a fourth sending module 660 configured to send, after the first metadata storage is successful, a storage success instruction corresponding to the plurality of first metadata to the available slicing server, so that the available slicing server is receiving After the storing success instruction, sending an upload end instruction of the target file to The client;
  • the third receiving module 670 is configured to receive an aggregation request of the target file sent by the client, where the aggregation request carries: a file identifier of the target file and an identification signal requesting aggregation of the plurality of first metadata
  • the aggregation request is sent by the client after receiving an upload end instruction of the target file;
  • the aggregation module 680 is configured to aggregate, according to the file identifier of the target file, a plurality of first metadata corresponding to the target file having the file identifier in the metadata table.
  • the first receiving module 610, the first determining module 620, the second determining module 630, the first sending module 640, and the storage module 650 in this embodiment are modules of the embodiment shown in FIG. 8, and details are not described herein again.
  • a file slice uploading apparatus provided by the embodiment of the present application may further include:
  • the fourth receiving module 690 is configured to receive the termination request sent by the client, where the termination request carries: a file identifier of the target file and an identifier signal for requesting termination of the slice file upload, where the termination request is The client sends after receiving the upload end instruction of the target file;
  • the deleting module 6110 is configured to delete the file identifier after determining that the plurality of first metadata corresponding to the target file having the file identifier in the metadata table is aggregated.
  • the first receiving module 610, the first determining module 620, the second determining module 630, the first sending module 640, the storage module 650, the fourth sending module 660, the third receiving module 670, and the aggregation module 680 in this embodiment are as shown in the figure.
  • the module of the embodiment shown in FIG. 9 will not be described again here.
  • the file slice uploading apparatus may include: a first receiving module 610, a first determining module 620, a second determining module 630, a first sending module 640, a second receiving module 710, and a first The two sending module 720, the storage module 650, the fourth sending module 660, the third receiving module 670, the aggregation module 680, the fourth receiving module 690, and the deleting module 6110.
  • the embodiment of the present application further provides a cloud storage system, where the system includes a metadata server 1110 and multiple slice services.
  • the system includes a metadata server 1110 and multiple slice services.
  • the metadata server 1110 is configured to receive an allocation request of an object file to be uploaded sent by a client, where the allocation request carries: a file identifier of the target file, and a slice server and an available object that are available for application.
  • An identification signal of the storage device determining, according to current load information of the plurality of slice servers, an available slice server corresponding to the target file having the file identifier; according to the available load of the slice server and the plurality of object storage devices Information determining an available object storage device corresponding to the object file having the file identifier; transmitting a device identifier of the available tile server to the client to cause the client to be based on the device identifier Uploading a slice file obtained by slicing the target file with the file identifier to the available slicing server;
  • the slice server 1120 is configured to collect and send current load information of the slice server and current load information of the object storage device corresponding to itself to the metadata server; when receiving the target sent by the client a slice file of the file, storing a plurality of first slice files of the target file to the available object storage device, wherein the first slice file is a raw data calculation method for a slice file of the target file The resulting slice file.
  • the metadata server appropriately allocates the slice server and the object storage device according to the load information of the slice server and the object storage device, uploads the target file, and selects a reasonable slice server and object storage when the server load is too large.
  • the device can effectively prevent file upload failures from occurring; and the target file to be uploaded is sliced, so that when the network environment is bad, if the slice file is lost, the upload of the remaining slice files is not affected; and the file upload is performed.
  • the original data calculation method is executed in the process to ensure that the file stored by the object storage device is a recoverable target file to be uploaded.
  • the metadata server 1110 is further configured to:
  • the metadata server 1110 is further configured to:
  • the slicing server is specifically configured to store the first slice file of the target file to the available object storage device based on the received object identifier.
  • the slicing server 1110 is specifically configured to: request an object identifier of the available object storage device from the metadata server, and set the target file according to the object identifier obtained by the request A slice file is stored to the available object storage device.
  • the metadata server 1110 is specifically configured to:
  • the load value is at least one of a usage rate of a central processing unit in the slice server, a usage rate of the memory, and a usage rate of the input/output device;
  • the available tile server corresponding to the object file having the file identifier is selected.
  • the metadata server 1110 is specifically configured to:
  • the first object storage device is used as an available object storage device
  • an available object storage device corresponding to the object file having the file identifier is selected.
  • the metadata server 1110 is specifically configured to:
  • the available object storage device corresponding to the object file having the file identifier is selected.
  • the original data calculation method is an erasure code EC algorithm.
  • the metadata server 1110 is further configured to: after transmitting the device identifier of the available slicing server to the client, storing the first metadata sent by the available slicing server to a metadata table, wherein the first metadata is metadata of the first slice file, and includes a disk serial number storing the first slice file.
  • the metadata server 1110 is further configured to: after the first metadata is successfully stored, send a storage success instruction corresponding to the multiple first metadata to the available slicing server; An aggregation request of the target file sent by the client, where the aggregation request carries: a file identifier of the target file and an identification signal requesting aggregation of multiple first metadata, where the aggregation request is And sending, by the client, the first metadata corresponding to the target file in the metadata table, based on the file identifier of the target file;
  • the slicing server 1110 is further configured to: after receiving the storage success instruction of the first metadata of the plurality of first slice files, send an upload end instruction of the target file to the client.
  • the metadata server 1110 is further configured to:
  • the termination request carries: the destination a file identifier of the target file and an identification signal requesting termination of the upload of the slice file, the termination request being sent by the client after receiving the upload end instruction of the target file;
  • the file identifier is deleted.
  • the embodiment of the present application provides a storage medium for storing executable code, where the executable code is used to execute at the runtime as in the embodiment of the present application.
  • the file slice uploading method is provided; specifically, the file slice uploading method includes:
  • the allocation request carries: a file identifier of the target file, and an identification signal of the available slice server and the available object storage device;
  • the client Sending a device identifier of the available slicing server to the client, so that the client uploads a slice file obtained by slicing the target file with the file identifier to the available based on the device identifier Slice server, such that the available tile server stores a plurality of first slice files of the target file to the available object storage device, wherein the first slice file is for the target file
  • the slice file performs the slice file obtained by the raw data calculation method.
  • the storage medium stores an application that executes the file slice uploading method provided by the embodiment of the present application at runtime, so that the metadata server can reasonably allocate the slice according to the load information of the slice server and the object storage device.
  • the server and the object storage device upload the target file, and when the server load is too large, the file server and the object storage device are selected to effectively avoid the file upload failure; and the target file to be uploaded is sliced.
  • the network environment is bad, if the slice file is lost, it will not affect the rest.
  • the uploading of the slice file is performed; and the original data calculation method is executed during the file uploading process to ensure that the file stored by the object storage device is a recoverable target file to be uploaded.
  • the embodiment of the present application provides an application program for performing the file slice uploading method provided by the embodiment of the present application at runtime;
  • the file slice uploading method provided by the embodiment includes:
  • the allocation request carries: a file identifier of the target file, and an identification signal of the available slice server and the available object storage device;
  • the client Sending a device identifier of the available slicing server to the client, so that the client uploads a slice file obtained by slicing the target file with the file identifier to the available based on the device identifier Slice server, such that the available tile server stores a plurality of first slice files of the target file to the available object storage device, wherein the first slice file is for the target file
  • the slice file performs the slice file obtained by the raw data calculation method.
  • the application performs the file slice uploading method provided by the embodiment of the present application at runtime, so that the metadata server can reasonably allocate the slice server and the object storage device according to the load information of the slice server and the object storage device.
  • Uploading the target file when the server load is too large, by selecting a reasonable slicing server and the object storage device, the file uploading failure is effectively avoided; and the target file to be uploaded is sliced, so that the network environment is bad. If the loss of the slice file occurs, the upload of the remaining slice files is not affected; and the original data calculation method is executed during the file uploading process to ensure that the file stored by the object storage device is a recoverable target file to be uploaded.
  • the embodiment of the present application provides a metadata server, as shown in FIG. 12, including: a processor 1210, a memory 1220, a communication interface 1230, and a bus 1240;
  • the processor 1210, the memory 1220, and the communication interface 1230 are connected by the bus 1240 and complete communication with each other;
  • the memory 1220 is configured to store executable program code
  • the processor 1210 is configured to execute a program corresponding to the executable program code by reading the executable program code stored in the memory 1220 to execute the file slice uploading method provided by the embodiment of the present application.
  • the file slice uploading method includes:
  • the allocation request carries: a file identifier of the target file, and an identification signal of the available slice server and the available object storage device;
  • the client Sending a device identifier of the available slicing server to the client, so that the client uploads a slice file obtained by slicing the target file with the file identifier to the available based on the device identifier Slice server, such that the available tile server stores a plurality of first slice files of the target file to the available object storage device, wherein the first slice file is for the target file
  • the slice file performs the slice file obtained by the raw data calculation method.
  • the processor of the metadata server runs a program corresponding to the executable program code by reading executable program code stored in the memory, and the program executes the file provided by the embodiment of the present application at runtime.
  • the slice uploading method can realize that the metadata server can reasonably allocate the slice server and the object storage according to the load information of the slice server and the object storage device.
  • the storage device uploads the target file, and when the server load is too large, the file server and the object storage device are selected to effectively avoid the file upload failure; and the target file to be uploaded is sliced to make the network
  • the original data calculation method is executed during the file upload process to ensure that the file stored by the object storage device is a recoverable target file to be uploaded.
  • the embodiment of the present invention provides a storage medium for storing a computer program, and when the computer program is executed by the processor, the step of implementing the file slice uploading method is implemented. .
  • the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Transfer Between Computers (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种文件切片上传方法、装置及云存储系统。其中，文件切片上传方法包括：元数据服务器接收客户端发送的待上传的目标文件的分配请求；根据多个切片服务器当前的负载信息，确定具有文件标识符的目标文件所对应的可用的切片服务器；根据可用的切片服务器及多个对象存储设备当前的负载信息，确定具有文件标识符的目标文件所对应的可用的对象存储设备；发送可用的切片服务器的设备标识符至客户端，以使该客户端基于该设备标识符，上传目标文件的切片文件至可用的切片服务器，从而使得该可用的切片服务器将目标文件的多个第一切片文件存储至可用的对象存储设备。通过本方案可以实现在服务器负载过大或出现恶劣的网络环境时，文件仍可正常上传。

Description

一种文件切片上传方法、装置及云存储系统
本申请要求于2016年09月28日提交中国专利局、申请号为201610858544.0发明名称为“一种文件切片上传方法、装置及云存储系统”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机和互联网数据处理技术领域，特别是涉及一种文件切片上传方法、装置及云存储系统。
背景技术
随着互联网网速的不断提升及移动互联网、智能终端的快速普及，使得云存储系统已成为当前互联网发展的趋势。从应用架构上看，云存储系统与传统分布式存储系统差异并不明显，更本质的区别体现在内部软件架构上，云存储系统以大量的服务器和存储设备为基础，构建一个大规模存储集群，提供存储容量，并能够在线进行容量的扩充，由此搭建大容量存储系统整体成本远低于传统存储架构。随着移动用户数量的增加，云存储系统的数据量也越来越大，因此，云存储系统的文件上传技术俨然成为云存储系统发展的新的挑战。
客户端将文件上传至服务器或相应的存储设备进行存储。相关文件上传技术中，在文件上传时，会占用大量的资源，如服务器的CPU(Central Processing Unit，中央处理单元)、端口、流量、内存以及磁盘I/O(input/output，输入/输出端口)等，一旦服务器的负载过大，则很容易造成文件在上传时产生过大的时延或者丢包，从而导致文件上传失败；同时，网络环境恶劣，也是导致文件上传失败的重要因素。
发明内容
本申请提供了一种文件切片上传方法、装置及云存储系统，实现服务器负载过大或出现恶劣的网络环境时，文件可以正常上传。具体技术方案如下：
第一方面，本申请实施例提供了一种文件切片上传方法，应用于云存储系统中的元数据服务器，所述云存储系统还包括多个切片服务器及多个对象存储设备，所述方法包括：
接收客户端发送的待上传的目标文件的分配请求，其中，所述分配请求中携带：目标文件的文件标识符、及申请可用的切片服务器和可用的对象存储设备的标识信号；
根据多个切片服务器当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的切片服务器；
根据所述可用的切片服务器及多个对象存储设备当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的对象存储设备；
发送所述可用的切片服务器的设备标识符至所述客户端，以使所述客户端基于所述设备标识符，上传具有所述文件标识符的目标文件切片后得到的切片文件至所述可用的切片服务器，从而使得所述可用的切片服务器将所述目标文件的多个第一切片文件存储至所述可用的对象存储设备，其中，所述第一切片文件为对所述目标文件的切片文件执行原始数据计算法所得到的切片文件。
第二方面，本申请实施例提供了一种元数据服务器，包括：处理器、存储器、通信接口和总线；
所述处理器、所述存储器和所述通信接口通过所述总线连接并完成相互间的通信；
所述存储器用于存储可执行程序代码；
所述处理器，用于通过读取所述存储器中存储的可执行程序代码来运行与所述可执行程序代码对应的程序，以执行如本申请实施例第一方面所提供的文件切片上传方法。
第三方面，本申请实施例提供了一种云存储系统，所述系统包括元数据服务器、多个切片服务器及多个对象存储设备；
所述元数据服务器，用于接收客户端发送的待上传的目标文件的分配请求，其中，所述分配请求中携带：目标文件的文件标识符、及申请可用的切片服务器和可用的对象存储设备的标识信号；根据多个切片服务器当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的切片服务器； 根据所述可用的切片服务器及多个对象存储设备当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的对象存储设备；发送所述可用的切片服务器的设备标识符至所述客户端，以使所述客户端基于所述设备标识符，上传具有所述文件标识符的目标文件切片后得到的切片文件至所述可用的切片服务器；
所述切片服务器，用于采集并发送所述切片服务器当前的负载信息以及与自身对应的对象存储设备当前的负载信息至所述元数据服务器；当接收到所述客户端发送的所述目标文件的切片文件，存储所述目标文件的多个第一切片文件至所述可用的对象存储设备，其中，所述第一切片文件为对所述目标文件的切片文件执行原始数据计算法所得到的切片文件。
第四方面，本申请实施例提供了一种存储介质，用于存储可执行代码，所述可执行代码用于在运行时执行如本申请实施例第一方面所提供的文件切片上传方法。
第五方面，本申请实施例提供了一种应用程序，用于在运行时执行如本申请实施例第一方面所提供的文件切片上传方法。
由上述的技术方案可见，本申请实施例提供的文件切片上传方法、装置及云存储系统，可以通过元数据服务器根据切片服务器和对象存储设备的负载信息，合理分配切片服务器和对象存储设备，上传目标文件，在服务器负载过大时，通过选择合理的切片服务器和对象存储设备，有效的避免文件上传失败的情况发生；并且通过将待上传的目标文件进行切片处理，使得在网络环境恶劣时，如果发生切片文件丢失，不影响其余切片文件的上传；且在文件上传的过程中执行原始数据计算法，保证对象存储设备存储的文件为可恢复的待上传的目标文件。
附图说明
为了更清楚地说明本申请实施例和相关技术的技术方案，下面对实施例和相关技术中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本申请一实施例的文件切片上传方法的流程图；
图2为本申请再一实施例的文件切片上传方法的流程图；
图3为本申请又一实施例的文件切片上传方法的流程图；
图4为本申请又一实施例的文件切片上传方法的流程图；
图5为本申请又一实施例的文件切片上传方法的流程图；
图6为本申请一实施例的文件切片上传装置的逻辑结构示意图；
图7为本申请再一实施例的文件切片上传装置的逻辑结构示意图；
图8为本申请又一实施例的文件切片上传装置的逻辑结构示意图；
图9为本申请又一实施例的文件切片上传装置的逻辑结构示意图；
图10为本申请又一实施例的文件切片上传装置的逻辑结构示意图；
图11为本申请实施例的云存储系统的结构示意图；
图12为本申请实施例的元数据服务器的结构示意图。
具体实施方式
为使本申请的目的、技术方案、及优点更加清楚明白，以下参照附图并举实施例，对本申请进一步详细说明。显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
为了保证在服务器负载过大或出现恶劣的网络环境时，文件可以正常上传，本申请实施例提供了一种文件切片上传方法、装置及云存储系统。
下面首先对本申请实施例所提供的文件切片上传方法进行介绍。
本申请实施例所提供的一种文件切片上传方法的执行主体为云存储系统中的元数据服务器。其中，实现本实施例所提供的一种文件切片上传方法的功能软件为设置于元数据服务器的软件。
如图1所示，本申请实施例所提供的一种文件切片上传方法，可以包括如下步骤：
S101，接收客户端发送的待上传的目标文件的分配请求。
客户端在上传目标文件之前，需要向元数据服务器发送分配请求，用于申请可用的切片服务器和可用的对象存储设备；并且由于在同一时刻，客户端可能针对多个目标文件同时发送多个分配请求，为了保证元数据服务器分配的切片服务器和对象存储设备与目标文件相对应，分配请求中需携带目标文件的文件标识符；分配请求可以为HTTP(Hyper Transfer Protocol，超文本传输协议)报文，元数据服务器以HTTP REST API(Hyper Transfer Protocol Representational State Transfer Application Programming Interface，超文本传输协议表述性状态转移应用程序编程接口)的形式对客户端提供服务。其中，客户端可以为台式电脑或者手机上专门的客户端软件，也可以为台式电脑或者手机上的浏览器软件等等，这都是合理的。
并且，该文件标识符为在服务器中能够唯一确定出该目标文件的标识信息，例如：目标文件的名称，或者元数据服务器为目标文件分配的编号，等等。
云存储系统中，元数据服务器用于对业务应用及存储系统元数据统一管理，切片服务器用于数据计算，对象存储设备用于存储文件。待上传的目标文件可以是视频文件、图片文件或者语音文件，等等。
S102，根据多个切片服务器当前的负载信息，确定具有文件标识符的目标文件所对应的可用的切片服务器。
其中，为了消除服务器负载过大对文件上传的影响，需要获得切片服务器的负载信息；切片服务器采集并发送切片服务器当前的负载信息以及与自身对应的对象存储设备当前的负载信息至元数据服务器；元数据服务器接收该负载信息。切片服务器的负载信息可以为切片服务器的负载大小，或者为切片服务器的负载占有比例，当然并不局限于此。
云存储系统中包括有多个切片服务器，根据每个切片服务器的负载信息。元数据服务器根据各个切片服务器当前的负载信息，从所有的切片服务器中，采用不同的选择方法对多个切片服务器进行选择；选择方法可以为轮转法，或者随机法，等等，选择出可用的切片服务器。
根据切片服务器当前的负载信息，确定具有文件标识符的目标文件所对应的可用的切片服务器的具体实现方式存在多种。为了方案布局清楚，根据切片服务器当前的负载信息，确定具有文件标识符的目标文件所对应的可用的切片服务器的具体实现方式后续进行介绍。
S103，根据可用的切片服务器及多个对象存储设备当前的负载信息，确定具有文件标识符的目标文件所对应的可用的对象存储设备。
其中，为了保证文件正常上传，在确定出可用的切片服务器后，由于存储文件的设备为对象存储设备，且对象存储设备与切片服务器具有对应性，因此，可以根据可用的切片服务器及对象存储设备当前的负载信息，来确定具有文件标识符的目标文件所对应的可用的对象存储设备。其中，对象存储设备的负载信息为对象存储设备的负载大小，或者为对象存储设备的负载占有比例，当然并不局限于此。
云存储系统中包括多个对象存储设备，并且，对象存储设备可以与切片服务器具有对应性，所谓的对应性具体指：对象存储设备可以设置于切片服务器内，或者，对象存储设备关联到切片服务器，等等。其中，如果对象存储设备设置于切片服务器内，此时对象存储设备与切片服务器具有相同的IP(Internet Protocol，网络协议)地址，且负载信息相同。
所述的根据可用的切片服务器及对象存储设备当前的负载信息，确定具有文件标识符的目标文件所对应的可用的对象存储设备的具体实现方式存在多种。为了方案布局清楚，所述的根据可用的切片服务器及对象存储设备当前的负载信息，确定具有文件标识符的目标文件所对应的可用的对象存储设备在后续进行举例介绍。
S104，发送可用的切片服务器的设备标识符至客户端，以使客户端基于设备标识符，上传具有文件标识符的目标文件切片后得到的切片文件至可用的切片服务器，从而使得可用的切片服务器将目标文件的多个第一切片文件存储至可用的对象存储设备。
其中，元数据服务器在确定可用的切片服务器后，得到可用的切片服务器的设备标识符，元数据服务器将该设备标识符发送给客户端，客户端在接 收到该设备标识符之后，能够确定向哪些可用的切片服务器发送目标文件；客户端在发送目标文件的过程中，根据预设的切片策略对目标文件进行切片，然后将切片后的切片文件上传至切片服务器；切片服务器在接收到切片文件后，对切片文件进行原始数据计算法的计算，得到多个第一切片文件，以保证通过多个第一切片文件可以恢复原始的目标文件，然后将多个第一切片文件发送至可用的对象存储设备；可用的对象存储设备在接收到多个第一切片文件后，根据预设的磁盘选择方法，将多个第一切片文件存储至不同的磁盘中。
切片文件为客户端在发送目标文件的过程中，根据预设的切片策略对目标文件进行切片操作得到的文件，第一切片文件为切片服务器对切片文件进行原始数据计算法的计算后得到的文件。设备标识符为在服务器和客户端中能够唯一确定出可用的切片服务器的标识信息，例如：可用的切片服务器的IP地址，或者元数据服务器为可用的切片服务器分配的编号，等等。目标文件的切片方法及磁盘的选择方法属于现有技术，这里不再一一赘述。
应用本实施例，元数据服务器根据切片服务器和对象存储设备的负载信息，合理分配切片服务器和对象存储设备，上传目标文件，在服务器负载过大时，通过选择合理的切片服务器和对象存储设备，有效的避免文件上传失败的情况发生；并且通过将待上传的目标文件进行切片处理，使得在网络环境恶劣时，如果发生切片文件丢失，不影响其余切片文件的上传；且在文件上传的过程中执行原始数据计算法，保证对象存储设备存储的文件为可恢复的待上传的目标文件。
如图2所示，本申请实施例所提供的一种文件切片上传方法，在接收由客户端发送的待上传的目标文件的分配请求之前，还可以包括如下步骤：
S201，接收客户端发送的目标文件的初始化请求。
客户端在发送分配请求、上传目标文件之前，可以向元数据服务器发送初始化请求，用于申请元数据服务器进行目标文件上传前的初始化操作；初始化操作包括：存储目标文件的基本信息、创建目标文件的版本信息及创建 目标文件的初始标识符；目标文件的基本信息可以是目标文件的属性信息，或者是目标文件的描述信息；初始化请求可以为HTTP报文。
S202，生成初始标识符，并发送初始标识符至客户端，以使该客户端将所接收到的初始标识符作为目标文件的文件标识符。
其中，元数据服务器在创建完目标文件的初始标识符后，将该初始标识符发送给客户端；客户端在接收到该初始标识符后，将该初始标识符作为目标文件的文件标识符，用于在后续的申请分配、上传切片文件的步骤中，携带着该文件标识符，以避免出现元数据服务器分配不对应的切片服务器或对象存储设备的情况出现。
客户端在接收到初始标识符时，表示元数据服务器的初始化操作完成，即元数据服务器已存储目标文件的基本信息、已创建目标文件的版本信息和目标文件的初始标识符。元数据服务器存储目标文件的基本信息用于在后续的步骤中快速的索引目标文件对应的切片文件的元数据；目标文件的版本信息在创建后发送给切片服务器，用于切片服务器在进行数据计算时使用。
本实施例中，S203至S206与图1所示实施例的S101至S104相同，这里不再赘述。
应用本实施例，元数据服务器根据切片服务器和对象存储设备的负载信息，合理分配切片服务器和对象存储设备，上传目标文件，在服务器负载过大时，通过选择合理的切片服务器和对象存储设备，有效的避免文件上传失败的情况发生；通过将待上传的目标文件进行切片处理，使得在网络环境恶劣时，如果发生切片文件丢失，不影响其余切片文件的上传；在文件上传的过程中执行原始数据计算法，保证对象存储设备存储的文件为可恢复的待上传的目标文件；并且在客户端发送分配请求之前，先申请初始化，元数据服务器对应于目标文件生产初始标识符，并将该初始标识符发送给客户端作为目标文件的文件标识符，能够避免出现元数据服务器分配不对应的切片服务器或对象存储设备的情况出现，且由元数据服务器统一编号使得文件标识符更为简便、占内存小。
一种具体实现方式中，为了使切片服务器发送第一切片文件至可用的对象存储设备，本申请实施例的一种文件切片上传方法在确定具有文件标识符的目标文件所对应的可用的对象存储设备之后，还可以包括：
将可用的对象存储设备的对象标识符发送至可用的切片服务器。
相应地，发送可用的切片服务器的设备标识符至客户端，以使客户端基于设备标识符，上传具有文件标识符的目标文件切片后得到的切片文件至可用的切片服务器，从而使得可用的切片服务器将目标文件的多个第一切片文件存储至可用的对象存储设备，可以包括：
发送可用的切片服务器的设备标识符至客户端，以使客户端基于设备标识符，上传具有文件标识符的目标文件切片后得到的切片文件至可用的切片服务器，从而使得可用的切片服务器基于所接收到的对象标识符，将目标文件的多个第一切片文件存储至可用的对象存储设备。
其中，元数据服务器在分配确定可用的对象存储设备后，得到可用的对象存储设备的对象标识符，元数据服务器将该对象标识符发送给可用的切片服务器，可用的切片服务器在接收到该对象标识符之后，能够确定哪些可用的对象存储设备可以存储第一切片文件；可用的对象存储设备在接收到多个第一切片文件后，根据预设的磁盘选择方法，将第一切片文件存储至不同的磁盘中。
对象标识符为在服务器和存储设备中能够唯一确定出可用的对象存储设备的标识信息，例如：可用的对象存储设备的IP地址，或者元数据服务器为可用的对象存储设备分配的编号，等等。
应用本实施例，元数据服务器在确定可用的对象存储设备时，将可用的对象存储设备的对象标识符发送给可用的切片服务器，以使可用的切片服务器向可用的对象存储设备发送目标文件对应的第一切片文件，保证文件上传的准确性。
另一种具体实现方式中，除了元数据服务器向可用的切片服务器发送可用的对象存储设备的对象标识符之外，还可以是可用的切片服务器在接收到 切片文件后向元数据服务器请求可用的对象存储设备的对象标识符。对于该种实现方式而言，发送可用的切片服务器的设备标识符至客户端，以使客户端基于设备标识符，上传具有文件标识符的目标文件切片后得到的切片文件至可用的切片服务器，从而使得可用的切片服务器将目标文件的多个第一切片文件存储至所述可用的对象存储设备，可以包括：
发送可用的切片服务器的设备标识符至客户端，以使客户端基于设备标识符，上传具有文件标识符的目标文件切片后得到的切片文件至可用的切片服务器，从而使得可用的切片服务器向元数据服务器请求可用的对象存储设备的对象标识符，并基于请求得到的对象标识符，将目标文件的多个第一切片文件存储至可用的对象存储设备。
对象标识符为在服务器和存储设备中能够唯一确定出可用的对象存储设备的标识信息，例如：可用的对象存储设备的IP地址，或者元数据服务器为可用的对象存储设备分配的编号，等等。
应用本实施例，元数据服务器在确定可用的对象存储设备时，将可用的对象存储设备的对象标识进行存储或生成，可用的切片服务器在进行完切片文件的计算得到第一切片文件之后向元数据服务器申请存储第一切片文件，元数据服务器再将该对象标识符发送给可用的切片服务器，以使可用的切片服务器向可用的对象存储设备发送目标文件对应的第一切片文件，保证文件上传的准确性。
再一种具体实现方式中，为了确定可用的切片服务器，合理地进行切片服务器的分配，根据多个切片服务器当前的负载信息，确定具有文件标识符的文件所对应的可用的切片服务器，可以包括：
获取多个切片服务器的负载值，其中，负载值为切片无服务内中央处理单元的使用率、内存的使用率和输入/输出设备的使用率中的至少一种；
判断多个切片服务器中是否存在多个第一切片服务器，其中，第一切片服务器的负载值小于预设第一负载阈值；
如果是，从多个第一切片服务器中，选择具有文件标识符的目标文件所 对应的可用的切片服务器；
如果否，从多个切片服务器中，选择具有文件标识符的目标文件所对应的可用的切片服务器。
其中，预设第一负载阈值根据切片服务器的实际负载情况进行设定，如果切片服务器的负载值小于预设第一负载阈值，说明向该切片服务器发送目标文件的切片文件的成功率较大，则可以向该切片服务器发送切片文件；如果切片服务器的负载值大于预设第一负载阈值，说明该切片服务器的负载已经过大，不宜向该切片服务器发送切片文件。
在确定具有文件标识符的目标文件所对应的可用的切片服务器时，在多个切片服务器或多个第一切片服务器中，均可以采用随机法，或者轮转法，或者其他选择方法进行选择。
又一种具体实现方式中，为了确定可用的对象存储设备，合理地进行对象存储设备的分配，所述根据可用的切片服务器及对象存储设备当前的负载信息，确定具有文件标识符的目标文件所对应的可用的对象存储设备，可以包括：
获取可用的切片服务器的网络协议IP地址及多个对象存储设备的IP地址；
判断多个对象存储设备中是否存在第一对象存储设备，其中，第一对象存储设备的IP地址与可用的切片服务器的IP地址相同；
如果是，将第一对象存储设备作为可用的对象存储设备；
如果否，基于多个对象存储设备当前的负载信息，从多个对象存储设备中，选择具有文件标识符的目标文件所对应的可用的对象存储设备。
其中，如果多个对象存储设备中存在IP地址与可用的切片服务器的IP地址相同的对象存储设备，说明该对象存储设备的负载信息与可用的切片服务器的负载信息相同，可以直接确定该对象存储设备为可用的对象存储设备，简化了对象存储设备再进行负载信息判断的过程；如果多个对象存储设备中不存在IP地址与可用的切片服务器的IP地址相同的对象存储设备，则需要进行对 象存储设备的负载信息判断，然后确定可用的对象存储设备。
又一种具体实现方式中，在多个对象存储设备中不存在IP地址与可用的切片服务器的IP地址相同的对象存储设备时，为了确定可用的对象存储设备，合理地进行对象存储设备的分配，所述基于多个对象存储设备当前的负载信息，从多个对象存储设备中，确定具有文件标识符的目标文件所对应的可用的对象存储设备，可以包括：
获取多个对象存储设备的负载值；
判断多个对象存储设备中是否存在多个第二对象存储设备，其中，第二对象存储设备的负载值小于预设第二负载阈值；
如果是，从多个第二对象存储设备中，选择具有文件标识符的目标文件所对应的可用的对象存储设备；
如果否，获取多个对象存储设备的分配次数；
判断多个对象存储设备中是否存在多个第三对象存储设备，其中，第三对象存储设备的分配次数大于预设分配次数阈值；
如果是，从多个第三对象存储设备中，选择具有文件标识符的目标文件所对应的可用的对象存储设备。
其中，预设第二负载阈值根据对象存储设备的实际负载情况进行设定，如果对象存储设备的负载值小于预设第二负载阈值，说明向该对象存储设备发送目标文件的第一切片文件的成功率较大，则可以向该对象存储设备发送第一切片文件；如果对象存储设备的负载值大于预设第二负载阈值，则可以根据对象存储设备的分配次数进行分配，预设分配次数阈值可根据历史数据进行设定，已分配次数越大，说明该对象存储设备存储目标文件的第一切片文件的可靠性越高，则可以选择该对象存储设备作为可用的对象存储设备存储目标文件的第一切片文件。
在确定具有文件标识符的目标文件所对应的可用的对象存储设备时，在第二对象存储设备或者第三对象存储设备中，可以采用随机法，或者轮转法， 或者其他选择方法进行选择。
又一种具体实现方式中，为了在目标文件的切片文件丢失时，能够恢复原始目标文件，从而保证文件存储的可靠性，所述原始数据计算法为EC(Erasure Code，纠删码)算法。
其中，EC算法的具体原理为，将n份原始切片文件，增加m份校验文件，并能通过n+m份中的任意n份文件，还原为原始切片文件。EC算法包含了编码和解码两个过程，将原始的n份切片文件变为n+m份文件是编码，n+m份文件可分散存放在可用的切片文件的不同位置，如果有任意小于m份的文件失效，仍然能通过剩下的数据还原出来，还原的过程为解码。EC算法的具体过程属于现有技术，这里不再赘述。
通过EC算法，只保留原始切片文件和校验文件，当切片文件丢失时可以通过EC算法恢复原始切片文件，在保证切片文件存储的可靠性的同时较少存储资源及存储成本。
基于图1所示实施例，如图3所示，本申请实施例所提供的一种文件切片上传方法，在发送可用的切片服务器的设备标识符至客户端，以使客户端基于设备标识符，发送目标文件的切片文件至可用的切片服务器，从而使得可用的切片服务器将目标文件的第一切片文件存储至可用的对象存储设备之后，还可以包括：
S105，存储由可用的切片服务器发送的多个第一元数据至元数据表。
其中，第一元数据为第一切片文件的元数据；元数据为数据的数据，即能够映射文件的索引信息、属性信息或者描述信息等，本实施例中，元数据包括磁盘序列号，磁盘序列号为存储第一切片文件的磁盘序列号，磁盘由可用的对象存储设备根据第一切片文件的内存大小确定。
元数据服务器中的元数据表用于记录第一切片文件存储于可用的对象存储设备的信息，包括第一切片文件的切片文件标识符及磁盘序列号，在需要 调用目标文件时，只需要向元数据服务器申请，便可根据切片文件标识符及磁盘序列号索引到可用的对象存储设备中存储的第一切片文件，通过索引进行调用。
本实施例中，S101至S104为图1所示实施例的方法步骤，这里不再赘述。
应用本实施例，元数据服务器根据切片服务器和对象存储设备的负载信息，合理分配切片服务器和对象存储设备，上传目标文件，在服务器负载过大时，通过选择合理的切片服务器和对象存储设备，有效的避免文件上传失败的情况发生；并且通过将待上传的目标文件进行切片处理，使得在网络环境恶劣时，如果发生切片文件丢失，不影响其余切片文件的上传；且在文件上传的过程中执行原始数据计算法，保证对象存储设备存储的文件为可恢复的待上传的目标文件。并且，元数据服务器将第一元数据存储至元数据表，便于后续目标文件的聚合、调用。
基于图3所示实施例，如图4所示，本申请实施例所提供的一种文件切片上传方法，存储由可用的切片服务器发送的第一元数据至元数据表之后，还可以包括如下步骤：
S106，发送多个第一元数据所对应的存储成功指令至可用的切片服务器，以使可用的切片服务器在接收到存储成功指令后，发送目标文件的上传结束指令至客户端。
其中，存储成功指令可以为第一元数据存储成功状态信息，或者为表示存储成功的任何指令信号。第一元数据存储成功后，说明目标文件已上传结束，应将上传结束指令发送至客户端，以使客户端执行后续步骤；上传完成指令可以为目标文件已完成上传的状态信息，或者为表示已完成上传的任何指令信号。
S107，接收客户端发送的目标文件的聚合请求。
其中，客户端在接收到上传完成指令之后，需要向元数据服务器发送聚合请求，用于申请聚合文件标识符所对应的目标文件的第一元数据；并且由于在同一时刻，客户端可能针对多个目标文件同时发送多个聚合请求，为了 保证元数据服务器聚合的第一元数据与目标文件相对应，聚合请求中需携带目标文件的文件标识符；聚合请求可以为HTTP报文。
S108，基于目标文件的文件标识符，聚合元数据表中的具有文件标识符的目标文件对应的多个第一元数据。
其中，聚合元数据表中的具有文件标识符的目标文件对应的多个第一元数据，可以为将第一元数据存储至数据表中相邻的表项，也可以为将第一元数据存储在数据表中的表项进行绑定，这都是合理的。
本实施例中，S101至S105为图3所示实施例的方法步骤，这里不再赘述。
应用本实施例，元数据服务器根据切片服务器和对象存储设备的负载信息，合理分配切片服务器和对象存储设备，上传目标文件，在服务器负载过大时，通过选择合理的切片服务器和对象存储设备，有效的避免文件上传失败的情况发生；并且通过将待上传的目标文件进行切片处理，使得在网络环境恶劣时，如果发生切片文件丢失，不影响其余切片文件的上传；且在文件上传的过程中执行原始数据计算法，保证对象存储设备存储的文件为可恢复的待上传的目标文件。并且，元数据服务器将第一元数据存储至元数据表，便于后续目标文件的聚合、调用；通过元数据服务器对第一切片文件的元数据信息进行聚合，在调用文件时，提高文件调用的速度。
基于图4所示实施例，如图5所示，本申请实施例所提供的一种文件切片上传方法，还可以包括：
S109，接收客户端发送的终止请求。
其中，客户端在接收到上传完成指令之后，向元数据服务器发送终止请求，用于申请元数据释放剩余空间；并且由于在同一时刻，客户端可能针对多个目标文件同时发送多个终止请求，为了保证元数据服务器删除的信息与目标文件相对应，终止请求中需携带目标文件的文件标识符；终止请求可以为HTTP报文。
S110，判断具有文件标识符的目标文件对应的多个第一元数据是否聚合 完毕。如果是，则执行S111；否则等待直至聚合完毕。
S111，删除目标文件的文件标识符。
释放元数据服务器的剩余空间，除了删除目标文件的文件标识符之外，还应删除目标文件的基本信息。
本实施例中，S101至S108为图4所示实施例的方法步骤，这里不再赘述。
应用本实施例，元数据服务器根据切片服务器和对象存储设备的负载信息，合理分配切片服务器和对象存储设备，上传目标文件，在服务器负载过大时，通过选择合理的切片服务器和对象存储设备，有效的避免文件上传失败的情况发生；并且通过将待上传的目标文件进行切片处理，使得在网络环境恶劣时，如果发生切片文件丢失，不影响其余切片文件的上传；且在文件上传的过程中执行原始数据计算法，保证对象存储设备存储的文件为可恢复的待上传的目标文件。并且，元数据服务器将第一元数据存储至元数据表，便于后续目标文件的聚合、调用；通过元数据服务器对第一切片文件的元数据信息进行聚合，在调用文件时，提高文件调用的速度；在完成目标文件的上传之后，有效释放元数据服务器的剩余空间，避免元数据服务器的空间浪费。
下面结合具体的应用实例，对本申请实施例所提供的文件切片上传方法进行介绍。
假设客户端为台式电脑上的浏览器软件，文件切片上传过程如下：
用户通过台式电脑上的浏览器选择视频文件A进行上传，在点击上传按键时，元数据服务器检测到上传指令，即相当于获得了初始化请求，进而元数据服务器分配初始标识符0001作为视频文件A的文件标识符，然后将文件标识符0001发送给浏览器。
浏览器在接收到文件标识符0001时，即相当于获得了初始化完成标志，则发送分配请求给元数据服务器，元数据服务器在接收到分配请求时，根据切片服务器及对象存储设备的负载信息，分配切片服务器1、切片服务器2、 切片服务器3及对象存储设备1、对象存储设备2、对象存储设备3，用于视频文件A的上传与存储，其中，假设切片服务器1的IP地址为168.1.1.1、切片服务器2的IP地址为202.1.1.1、切片服务器3的IP地址为155.1.1，对象存储设备1的IP地址为168.1.1.1、对象存储设备2的IP地址为202.1.1.26、对象存储设备3的IP地址为155.1.1.125；元数据服务器将切片服务器1、切片服务器2及切片服务器3的IP地址发送给浏览器，将对象存储设备1、对象存储设备2及对象存储设备3的IP地址发送给切片服务器。
浏览器在接收到切片服务器1、切片服务器2及切片服务器3的IP地址时，开始上传视频文件A，在上传的过程中，按照分片方法对视频文件A进行分片，得到分片文件A1、分片文件A2及分片文件A3，并分别上传分片文件A1、分片文件A2、分片文件A3至切片服务器1、切片服务器2、切片服务器3，切片服务器1、切片服务器2及切片服务器3分别将接收到的分片文件存储至条带缓存，并进行EC计算，分别得到切片文件B1、切片文件B2及切片文件B3；然后，切片服务器1、切片服务器2、切片服务器3分别将切片文件B1、切片文件B2、切片文件B3发送至对象存储设备1、对象存储设备2、对象存储设备3，对象存储设备1、对象存储设备2、对象存储设备3选择适当的磁盘对切片文件B1、切片文件B2、切片文件B3进行存储，并将磁盘序列号返回至切片服务器；切片服务器将分片文件A1、分片文件A2、分片文件A3及相应的磁盘序列号发送至元数据服务器，进而存储至元数据表；存储完成后，元数据服务器发送存储完成指令至切片服务器，切片服务器进而向台式电脑的浏览器发送上传结果。
与现有技术相比，本方案中，元数据服务器根据切片服务器和对象存储设备的负载信息，合理分配切片服务器和对象存储设备，上传目标文件，在服务器负载过大时，通过选择合理的切片服务器和对象存储设备，有效的避免文件上传失败的情况发生；并且通过将待上传的目标文件进行切片处理，使得在网络环境恶劣时，如果发生切片文件丢失，不影响其余切片文件的上传；且在文件上传的过程中执行原始数据计算法，保证对象存储设备存储的文件为可恢复的待上传的目标文件。
相应于上述方法实施例，本申请实施例提供了一种文件切片上传装置， 如图6所述，所述装置可以包括：
第一接收模块610，用于接收客户端发送的待上传的目标文件的分配请求，其中，所述分配请求中携带：目标文件的文件标识符、及申请可用的切片服务器和可用的对象存储设备的标识信号；
第一确定模块620，用于根据多个切片服务器当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的切片服务器；
第二确定模块630，用于根据所述可用的切片服务器及多个对象存储设备当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的对象存储设备；
第一发送模块640，用于发送所述可用的切片服务器的设备标识符至所述客户端，以使所述客户端基于所述设备标识符，上传具有所述文件标识符的目标文件切片后得到的切片文件至所述可用的切片服务器，从而使得所述可用的切片服务器将所述目标文件的多个第一切片文件存储至所述可用的对象存储设备，其中，所述第一切片文件为对所述目标文件的切片文件执行原始数据计算法所得到的切片文件。
应用本实施例，元数据服务器根据切片服务器和对象存储设备的负载信息，合理分配切片服务器和对象存储设备，上传目标文件，在服务器负载过大时，通过选择合理的切片服务器和对象存储设备，有效的避免文件上传失败的情况发生；并且通过将待上传的目标文件进行切片处理，使得在网络环境恶劣时，如果发生切片文件丢失，不影响其余切片文件的上传；且在文件上传的过程中执行原始数据计算法，保证对象存储设备存储的文件为可恢复的待上传的目标文件。
更进一步的，基于图6所示实施例，如图7所示，本申请实施例所提供的一种文件切片上传装置还可以包括：
第二接收模块710，用于接收所述客户端发送的所述目标文件的初始化请求；
第二发送模块720，用于生成初始标识符，并发送所述初始标识符至所述客户端，以使所述客户端将所接收到的所述初始标识符作为所述目标文件的 文件标识符。
本实施例中的第一接收模块730、第一确定模块740、第二确定模块750、第一发送模块760，与图6所示实施例的第一接收模块610、第一确定模块620、第二确定模块630、第一发送模块640相同，这里不再赘述。
应用本实施例，元数据服务器根据切片服务器和对象存储设备的负载信息，合理分配切片服务器和对象存储设备，上传目标文件，在服务器负载过大时，通过选择合理的切片服务器和对象存储设备，有效的避免文件上传失败的情况发生；通过将待上传的目标文件进行切片处理，使得在网络环境恶劣时，如果发生切片文件丢失，不影响其余切片文件的上传；在文件上传的过程中执行原始数据计算法，保证对象存储设备存储的文件为可恢复的待上传的目标文件；并且在客户端发送分配请求之前，先申请初始化，元数据服务器对应于目标文件生产初始标识符，并将该初始标识符发送给客户端作为目标文件的文件标识符，能够避免出现元数据分配不对应的切片服务器或对象存储设备的情况出现，且由元数据服务器统一编号使得文件标识符更为简便、占内存小。
可选的，为了使切片服务器发送第一切片文件至可用的对象存储设备，本申请实施例的一种文件切片上传装置还可以包括：
第三发送模块，用于所述第二确定模块750确定具有所述文件标识符的所述目标文件所对应的可用的对象存储设备之后，将所述可用的对象存储设备的对象标识符发送至所述可用的切片服务器；
相应地，所述第一发送模块760，具体用于发送所述可用的切片服务器的设备标识符至所述客户端，以使所述客户端基于所述设备标识符，上传具有所述文件标识符的目标文件切片后得到的切片文件至所述可用的切片服务器，从而使得所述可用的切片服务器基于所接收到的对象标识符，将所述目标文件的多个第一切片文件存储至所述可用的对象存储设备。
可选的，所述第一发送模块760，具体用于发送所述可用的切片服务器的设备标识符至所述客户端，以使所述客户端基于所述设备标识符，上传具有所述文件标识符的目标文件切片后得到的切片文件至所述可用的切片服务 器，从而使得所述可用的切片服务器向所述元数据服务器请求所述可用的对象存储设备的对象标识符，并基于请求得到的对象标识符，将所述目标文件的多个第一切片文件存储至所述可用的对象存储设备。
可选的，所述第一确定模块740，可以包括：
第一获取子模块，用于获取多个切片服务器的负载值，其中，所述负载值为切片服务器内中央处理单元的使用率、内存的使用率和输入/输出设备的使用率中的至少一种；
第一判断子模块，用于判断所述多个切片服务器中是否存在多个第一切片服务器，其中，所述第一切片服务器的负载值小于预设第一负载阈值；
第一确定子模块，用于判断如果第一判断子模块的输出为是，从多个第一切片服务器中，选择具有所述文件标识符的目标文件所对应的可用的切片服务器；
第二确定子模块，用于判断如果第一判断子模块的输出为否，从多个切片服务器中，选择具有所述文件标识符的目标文件所对应的可用的切片服务器。
可选的，所述第二确定模块750，可以包括：
第二获取子模块，用于获取所述可用的切片服务器的网络协议IP地址及多个对象存储设备的IP地址；
第二判断子模块，用于判断多个对象存储设备中是否存在第一对象存储设备，其中，所述第一对象存储设备的IP地址与所述可用的切片服务器的IP地址相同；
第三确定子模块，用于判断如果第二判断子模块的输出为是，将所述第一对象存储设备作为可用的对象存储设备；
第四确定子模块，用于判断如果第二判断子模块的输出为否，基于多个对象存储设备当前的负载信息，从多个对象存储设备中，选择具有所述文件标识符的目标文件所对应的可用的对象存储设备。
可选的，所述第四确定子模块，可以包括：
获取单元，用于获取多个对象存储设备的负载值；
第一判断单元，用于判断多个对象存储设备中是否存在多个第二对象存储设备，其中，所述第二对象存储设备的负载值小于预设第二负载阈值；
第五确定单元，用于判断如果第一判断单元的输出为是，从多个第二对象存储设备中，选择具有所述文件标识符的目标文件所对应的可用的对象存储设备；
第六确定单元，用于判断如果第一判断单元的输出为否，获取多个对象存储设备的分配次数；
第二判断单元，用于判断所述多个对象存储设备中是否存在多个第三对象存储设备，其中，所述第三对象存储设备的分配次数大于预设分配次数阈值；
第七确定单元，用于判断如果第二判断单元的输出为是，从多个第三对象存储设备中，选择具有所述文件标识符的目标文件所对应的可用的对象存储设备。
可选的，所述第一发送模块760中，所述原始数据计算法为纠删码EC算法。
更进一步的，基于图6所示实施例，如图8所示，本申请实施例所提供的一种文件切片上传装置还可以包括：
存储模块650，用于存储由所述可用的切片服务器发送的第一元数据至元数据表，其中，所述第一元数据为所述第一切片文件的元数据，包括存储所述第一切片文件的磁盘序列号。
本实施例中的第一接收模块610、第一确定模块620、第二确定模块630、第一发送模块640为图6所示实施例的模块，这里不再赘述。
更进一步的，基于图8所示实施例，如图9所示，本申请实施例所提供的一种文件切片上传装置还可以包括：
第四发送模块660，用于在所述第一元数据存储成功后，发送多个第一元数据所对应的存储成功指令至所述可用的切片服务器，以使所述可用的切片服务器在接收到所述存储成功指令后，发送所述目标文件的上传结束指令至 所述客户端；
第三接收模块670，用于接收所述客户端发送的所述目标文件的聚合请求，其中，所述聚合请求中携带：目标文件的文件标识符及请求聚合多个第一元数据的标识信号，所述聚合请求为所述客户端在接收到所述目标文件的上传结束指令后所发送的；
聚合模块680，用于基于所述目标文件的文件标识符，聚合所述元数据表中的具有所述文件标识符的目标文件对应的多个第一元数据。
本实施例中的第一接收模块610、第一确定模块620、第二确定模块630、第一发送模块640、存储模块650为图8所示实施例的模块，这里不再赘述。
更进一步的，基于图9所示实施例，如图10所示，本申请实施例所提供的一种文件切片上传装置还可以包括：
第四接收模块690，用于接收所述客户端发送的终止请求，其中，所述终止请求中携带：所述目标文件的文件标识符及请求终止切片文件上传的标识信号，所述终止请求为所述客户端在接收到所述目标文件的上传结束指令后所发送的；
删除模块6110，用于在判断出聚合完毕所述元数据表中具有所述文件标识符的目标文件对应的多个第一元数据后，删除所述文件标识符。
本实施例中的第一接收模块610、第一确定模块620、第二确定模块630、第一发送模块640、存储模块650、第四发送模块660、第三接收模块670、聚合模块680为图9所示实施例的模块，这里不再赘述。
本申请实施例的另一实施例中，文件切片上传装置可以同时包括：第一接收模块610、第一确定模块620、第二确定模块630、第一发送模块640、第二接收模块710、第二发送模块720、存储模块650、第四发送模块660、第三接收模块670、聚合模块680、第四接收模块690和删除模块6110。
相应于上述文件切片上传方法及装置实施例，如图11所示，本申请实施例还提供一种云存储系统，所述系统包括元数据服务器1110、多个切片服务 器1120及多个对象存储设备1130；
所述元数据服务器1110，用于接收由客户端发送的待上传的目标文件的分配请求，其中，所述分配请求中携带：目标文件的文件标识符、及申请可用的切片服务器和可用的对象存储设备的标识信号；根据多个切片服务器当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的切片服务器；根据所述可用的切片服务器及多个对象存储设备当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的对象存储设备；发送所述可用的切片服务器的设备标识符至所述客户端，以使所述客户端基于所述设备标识符，上传具有所述文件标识符的目标文件切片后得到的切片文件至所述可用的切片服务器；
所述切片服务器1120，用于采集并发送所述切片服务器当前的负载信息以及与自身对应的对象存储设备当前的负载信息至所述元数据服务器；当接收到所述客户端发送的所述目标文件的切片文件，存储所述目标文件的多个第一切片文件至所述可用的对象存储设备，其中，所述第一切片文件为对所述目标文件的切片文件执行原始数据计算法所得到的切片文件。
在本申请实施例中，元数据服务器根据切片服务器和对象存储设备的负载信息，合理分配切片服务器和对象存储设备，上传目标文件，在服务器负载过大时，通过选择合理的切片服务器和对象存储设备，有效的避免文件上传失败的情况发生；并且通过将待上传的目标文件进行切片处理，使得在网络环境恶劣时，如果发生切片文件丢失，不影响其余切片文件的上传；且在文件上传的过程中执行原始数据计算法，保证对象存储设备存储的文件为可恢复的待上传的目标文件。
可选的，所述元数据服务器1110，还可以用于：
在所述接收由客户端发送的待上传的目标文件的分配请求之前，接收所述客户端发送的所述目标文件的初始化请求；
生成初始标识符，并发送所述初始标识符至所述客户端，以使所述客户端将所接收到的所述初始标识符作为所述目标文件的文件标识符。
可选的，所述元数据服务器1110，还可以用于：
在所述确定具有所述文件标识符的所述目标文件所对应的可用的对象存储设备之后，将所述可用的对象存储设备的对象标识符发送至所述可用的切片服务器；
相应地，所述切片服务器具体用于基于所接收到的对象标识符，将所述目标文件的第一切片文件存储至所述可用的对象存储设备。
可选的，所述切片服务器1110，具体可以用于：向所述元数据服务器请求所述可用的对象存储设备的对象标识符，并基于请求得到的对象标识符，将所述目标文件的第一切片文件存储至所述可用的对象存储设备。
可选的，所述元数据服务器1110，具体可以用于：
获取多个切片服务器的负载值，其中，所述负载值为切片服务器内中央处理单元的使用率、内存的使用率和输入/输出设备的使用率中的至少一种；
判断多个切片服务器中是否存在多个第一切片服务器，其中，所述第一切片服务器的负载值小于预设第一负载阈值；
如果是，从多个第一切片服务器中，选择具有所述文件标识符的目标文件所对应的可用的切片服务器；
如果否，从多个切片服务器中，选择具有所述文件标识符的目标文件所对应的可用的切片服务器。
可选的，所述元数据服务器1110，具体可以用于：
获取所述可用的切片服务器的网络协议IP地址及多个对象存储设备的IP地址；
判断多个对象存储设备中是否存在第一对象存储设备，其中，所述第一对象存储设备的IP地址与所述可用的切片服务器的IP地址相同；
如果是，将所述第一对象存储设备作为可用的对象存储设备；
如果否，基于多个对象存储设备当前的负载信息，从多个对象存储设备中，选择具有所述文件标识符的目标文件所对应的可用的对象存储设备。
可选的，所述元数据服务器1110，具体可以用于：
获取多个对象存储设备的负载值；
判断多个对象存储设备中是否存在多个第二对象存储设备，其中，所述第二对象存储设备的负载值小于预设第二负载阈值；
如果是，从多个第二对象存储设备中，选择具有所述文件标识符的目标文件所对应的可用的对象存储设备；
如果否，获取多个对象存储设备的分配次数；
判断所述多个对象存储设备中是否存在多个第三对象存储设备，其中，所述第三对象存储设备的分配次数大于预设分配次数阈值；
如果是，从多个第三对象存储设备中，选择具有所述文件标识符的目标文件所对应的可用的对象存储设备。
可选的，所述原始数据计算法为纠删码EC算法。
可选的，所述元数据服务器1110，还可以用于：在发送所述可用的切片服务器的设备标识符至所述客户端之后，存储由所述可用的切片服务器发送的第一元数据至元数据表，其中，所述第一元数据为所述第一切片文件的元数据，包括存储所述第一切片文件的磁盘序列号。
可选的，所述元数据服务器1110，还可以用于：在所述第一元数据存储成功后，发送多个第一元数据所对应的存储成功指令至所述可用的切片服务器；接收所述客户端发送的所述目标文件的聚合请求，其中，所述聚合请求中携带：所述目标文件的文件标识符及请求聚合多个第一元数据的标识信号，所述聚合请求为所述客户端在接收到所述目标文件的上传结束指令后所发送的；基于所述目标文件的文件标识符，聚合所述元数据表中的所述目标文件对应的第一元数据；
相应地，所述切片服务器1110，还可以用于：在接收到多个第一切片文件的第一元数据的存储成功指令后，发送所述目标文件的上传结束指令至所述客户端。
可选的，所述元数据服务器1110，还可以用于：
接收所述客户端发送的终止请求，其中，所述终止请求中携带：所述目 标文件的文件标识符及请求终止切片文件上传的标识信号，所述终止请求为所述客户端在接收到所述目标文件的上传结束指令后所发送的；
在判断出聚合完毕所述元数据表中具有所述文件标识符的目标文件对应的多个第一元数据后，删除所述文件标识符。
另外，相应于上述实施例所提供的文件切片上传方法，本申请实施例提供了一种存储介质，用于存储可执行代码，所述可执行代码用于在运行时执行如本申请实施例所提供的文件切片上传方法；具体的，所述文件切片上传方法，包括：
接收客户端发送的待上传的目标文件的分配请求，其中，所述分配请求中携带：目标文件的文件标识符、及申请可用的切片服务器和可用的对象存储设备的标识信号；
根据多个切片服务器当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的切片服务器；
根据所述可用的切片服务器及多个对象存储设备当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的对象存储设备；
发送所述可用的切片服务器的设备标识符至所述客户端，以使所述客户端基于所述设备标识符，上传具有所述文件标识符的目标文件切片后得到的切片文件至所述可用的切片服务器，从而使得所述可用的切片服务器将所述目标文件的多个第一切片文件存储至所述可用的对象存储设备，其中，所述第一切片文件为对所述目标文件的切片文件执行原始数据计算法所得到的切片文件。
本实施例中，存储介质存储有在运行时执行本申请实施例所提供的文件切片上传方法的应用程序，因此能够实现：通过元数据服务器根据切片服务器和对象存储设备的负载信息，合理分配切片服务器和对象存储设备，上传目标文件，在服务器负载过大时，通过选择合理的切片服务器和对象存储设备，有效的避免文件上传失败的情况发生；并且通过将待上传的目标文件进行切片处理，使得在网络环境恶劣时，如果发生切片文件丢失，不影响其余 切片文件的上传；且在文件上传的过程中执行原始数据计算法，保证对象存储设备存储的文件为可恢复的待上传的目标文件。
另外，相应于上述实施例所提供的文件切片上传方法，本申请实施例提供了一种应用程序，用于在运行时执行如本申请实施例所提供的文件切片上传方法；具体的，本申请实施例所提供的文件切片上传方法，包括：
接收客户端发送的待上传的目标文件的分配请求，其中，所述分配请求中携带：目标文件的文件标识符、及申请可用的切片服务器和可用的对象存储设备的标识信号；
根据多个切片服务器当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的切片服务器；
根据所述可用的切片服务器及多个对象存储设备当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的对象存储设备；
发送所述可用的切片服务器的设备标识符至所述客户端，以使所述客户端基于所述设备标识符，上传具有所述文件标识符的目标文件切片后得到的切片文件至所述可用的切片服务器，从而使得所述可用的切片服务器将所述目标文件的多个第一切片文件存储至所述可用的对象存储设备，其中，所述第一切片文件为对所述目标文件的切片文件执行原始数据计算法所得到的切片文件。
本实施例中，应用程序在运行时执行本申请实施例所提供的文件切片上传方法，因此能够实现：通过元数据服务器根据切片服务器和对象存储设备的负载信息，合理分配切片服务器和对象存储设备，上传目标文件，在服务器负载过大时，通过选择合理的切片服务器和对象存储设备，有效的避免文件上传失败的情况发生；并且通过将待上传的目标文件进行切片处理，使得在网络环境恶劣时，如果发生切片文件丢失，不影响其余切片文件的上传；且在文件上传的过程中执行原始数据计算法，保证对象存储设备存储的文件为可恢复的待上传的目标文件。
另外，相应于上述实施例提供的文件切片上传方法，本申请实施例提供了一种元数据服务器，如图12所示，包括：处理器1210、存储器1220、通信接口1230和总线1240；
所述处理器1210、所述存储器1220和所述通信接口1230通过所述总线1240连接并完成相互间的通信；
所述存储器1220用于存储可执行程序代码；
所述处理器1210，用于通过读取所述存储器1220中存储的可执行程序代码来运行与所述可执行程序代码对应的程序，以执行本申请实施例所提供的文件切片上传方法。
具体的，本申请实施例所提供的文件切片上传方法，包括：
接收客户端发送的待上传的目标文件的分配请求，其中，所述分配请求中携带：目标文件的文件标识符、及申请可用的切片服务器和可用的对象存储设备的标识信号；
根据多个切片服务器当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的切片服务器；
根据所述可用的切片服务器及多个对象存储设备当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的对象存储设备；
发送所述可用的切片服务器的设备标识符至所述客户端，以使所述客户端基于所述设备标识符，上传具有所述文件标识符的目标文件切片后得到的切片文件至所述可用的切片服务器，从而使得所述可用的切片服务器将所述目标文件的多个第一切片文件存储至所述可用的对象存储设备，其中，所述第一切片文件为对所述目标文件的切片文件执行原始数据计算法所得到的切片文件。
本实施例中，该元数据服务器的处理器通过读取存储器中存储的可执行程序代码来运行与所述可执行程序代码对应的程序，该程序在运行时执行本申请实施例所提供的文件切片上传方法，因此能够实现：通过元数据服务器根据切片服务器和对象存储设备的负载信息，合理分配切片服务器和对象存 储设备，上传目标文件，在服务器负载过大时，通过选择合理的切片服务器和对象存储设备，有效的避免文件上传失败的情况发生；并且通过将待上传的目标文件进行切片处理，使得在网络环境恶劣时，如果发生切片文件丢失，不影响其余切片文件的上传；且在文件上传的过程中执行原始数据计算法，保证对象存储设备存储的文件为可恢复的待上传的目标文件。
另外，相应于上述实施例所提供的文件切片上传方法，本发明实施例提供了一种存储介质，用于存储计算机程序，所述计算机程序被处理器执行时，实现上述文件切片上传方法的步骤。
对于元数据服务器、应用程序以及存储介质实施例而言，由于其所涉及的方法内容基本相似于前述的方法实施例，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
需要说明的是，在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个......”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
本说明书中的各个实施例均采用相关的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于系统实施例而言，由于其基本相似于方法实施例，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
以上所述仅为本申请的较佳实施例而已，并不用以限制本申请，凡在本申请的精神和原则之内，所做的任何修改、等同替换、改进等，均应包含在本申请保护的范围之内。

Claims (18)

  1. 一种文件切片上传方法，其特征在于，应用于云存储系统中的元数据服务器，所述云存储系统还包括多个切片服务器及多个对象存储设备，所述方法包括：
    接收客户端发送的待上传的目标文件的分配请求，其中，所述分配请求中携带：目标文件的文件标识符、及申请可用的切片服务器和可用的对象存储设备的标识信号；
    根据多个切片服务器当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的切片服务器；
    根据所述可用的切片服务器及多个对象存储设备当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的对象存储设备；
    发送所述可用的切片服务器的设备标识符至所述客户端，以使所述客户端基于所述设备标识符，上传具有所述文件标识符的目标文件切片后得到的切片文件至所述可用的切片服务器，从而使得所述可用的切片服务器将所述目标文件的多个第一切片文件存储至所述可用的对象存储设备，其中，所述第一切片文件为对所述目标文件的切片文件执行原始数据计算法所得到的切片文件。
  2. 根据权利要求1所述的方法，其特征在于，在接收客户端发送的待上传的目标文件的分配请求之前，所述方法还包括：
    接收所述客户端发送的所述目标文件的初始化请求；
    生成初始标识符，并发送所述初始标识符至所述客户端，以使所述客户端将所接收到的所述初始标识符作为所述目标文件的文件标识符。
  3. 根据权利要求1所述的方法，其特征在于，在确定具有所述文件标识符的目标文件所对应的可用的对象存储设备之后，所述方法还包括：
    将所述可用的对象存储设备的对象标识符发送至所述可用的切片服务器；
    相应地，发送所述可用的切片服务器的设备标识符至所述客户端，以使 所述客户端基于所述设备标识符，上传具有所述文件标识符的目标文件切片后得到的切片文件至所述可用的切片服务器，从而使得所述可用的切片服务器将所述目标文件的多个第一切片文件存储至所述可用的对象存储设备，包括：
    发送所述可用的切片服务器的设备标识符至所述客户端，以使所述客户端基于所述设备标识符，上传具有所述文件标识符的目标文件切片后得到的切片文件至所述可用的切片服务器，从而使得所述可用的切片服务器基于所接收到的对象标识符，将所述目标文件的多个第一切片文件存储至所述可用的对象存储设备。
  4. 根据权利要求1所述的方法，其特征在于，发送所述可用的切片服务器的设备标识符至所述客户端，以使所述客户端基于所述设备标识符，上传具有所述文件标识符的目标文件切片后得到的切片文件至所述可用的切片服务器，从而使得所述可用的切片服务器将所述目标文件的多个第一切片文件存储至所述可用的对象存储设备，包括：
    发送所述可用的切片服务器的设备标识符至所述客户端，以使所述客户端基于所述设备标识符，上传具有所述文件标识符的目标文件切片后得到的切片文件至所述可用的切片服务器，从而使得所述可用的切片服务器向所述元数据服务器请求所述可用的对象存储设备的对象标识符，并基于请求得到的对象标识符，将所述目标文件的多个第一切片文件存储至所述可用的对象存储设备。
  5. 根据权利要求1所述的方法，其特征在于，根据多个切片服务器当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的切片服务器，包括：
    获取多个切片服务器的负载值，其中，所述负载值为切片服务器内中央处理单元的使用率、内存的使用率和输入/输出设备的使用率中的至少一种；
    判断多个切片服务器中是否存在多个第一切片服务器，其中，所述第一切片服务器的负载值小于预设第一负载阈值；
    如果是，从多个第一切片服务器中，选择具有所述文件标识符的目标文 件所对应的可用的切片服务器；
    如果否，从多个切片服务器中，选择具有所述文件标识符的目标文件所对应的可用的切片服务器。
  6. 根据权利要求1所述的方法，其特征在于，根据所述可用的切片服务器及多个对象存储设备当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的对象存储设备，包括：
    获取所述可用的切片服务器的网络协议IP地址及多个对象存储设备的IP地址；
    判断多个对象存储设备中是否存在第一对象存储设备，其中，所述第一对象存储设备的IP地址与所述可用的切片服务器的IP地址相同；
    如果是，将所述第一对象存储设备作为可用的对象存储设备；
    如果否，基于多个对象存储设备当前的负载信息，从多个对象存储设备中，选择具有所述文件标识符的目标文件所对应的可用的对象存储设备。
  7. 根据权利要求6所述的方法，其特征在于，基于多个对象存储设备当前的负载信息，从多个对象存储设备中，选择具有所述文件标识符的目标文件所对应的可用的对象存储设备，包括：
    获取多个对象存储设备的负载值；
    判断多个对象存储设备中是否存在多个第二对象存储设备，其中，所述第二对象存储设备的负载值小于预设第二负载阈值；
    如果是，从多个第二对象存储设备中，选择具有所述文件标识符的目标文件所对应的可用的对象存储设备；
    如果否，获取多个对象存储设备的分配次数；
    判断多个对象存储设备中是否存在多个第三对象存储设备，其中，所述第三对象存储设备的分配次数大于预设分配次数阈值；
    如果是，从多个第三对象存储设备中，选择具有所述文件标识符的目标文件所对应的可用的对象存储设备。
  8. 根据权利要求1所述的方法，其特征在于，所述原始数据计算法为纠删码EC算法。
  9. 根据权利要求1所述的方法，其特征在于，在发送所述可用的切片服务器的设备标识符至所述客户端之后，所述方法还包括：
    存储由所述可用的切片服务器发送的多个第一元数据至元数据表，其中，所述第一元数据为第一切片文件的元数据，包括：存储第一切片文件的磁盘序列号。
  10. 根据权利要求9所述的方法，其特征在于，在存储由所述可用的切片服务器发送的多个第一元数据至元数据表之后，所述方法还包括：
    发送多个第一元数据所对应的存储成功指令至所述可用的切片服务器，以使所述可用的切片服务器在接收到所述存储成功指令后，发送所述目标文件的上传结束指令至所述客户端；
    接收所述客户端发送的目标文件的聚合请求，其中，所述聚合请求中携带：目标文件的文件标识符及请求聚合多个第一元数据的标识信号，所述聚合请求为所述客户端在接收到所述目标文件的上传结束指令后所发送的；
    基于所述目标文件的文件标识符，聚合所述元数据表中的具有所述文件标识符的目标文件对应的多个第一元数据。
  11. 根据权利要求10所述的方法，其特征在于，在发送多个第一元数据所对应的存储成功指令至所述可用的切片服务器之后，所述方法还包括：
    接收所述客户端发送的终止请求，其中，所述终止请求中携带：目标文件的文件标识符及请求终止切片文件上传的标识信号，所述终止请求为所述客户端在接收到所述目标文件的上传结束指令后所发送的；
    在判断出聚合完毕所述元数据表中具有所述文件标识符的目标文件对应的多个第一元数据后，删除所述文件标识符。
  12. 一种元数据服务器，其特征在于，包括：处理器、存储器、通信接口和总线；
    所述处理器、所述存储器和所述通信接口通过所述总线连接并完成相互 间的通信；
    所述存储器用于存储可执行程序代码；
    所述处理器，用于通过读取所述存储器中存储的可执行程序代码来运行与所述可执行程序代码对应的程序，以执行如权利要求1-11任一项所述的文件切片上传方法。
  13. 一种云存储系统，其特征在于，所述系统包括元数据服务器、多个切片服务器及多个对象存储设备；
    所述元数据服务器，用于接收客户端发送的待上传的目标文件的分配请求，其中，所述分配请求中携带：目标文件的文件标识符、及申请可用的切片服务器和可用的对象存储设备的标识信号；根据多个切片服务器当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的切片服务器；根据所述可用的切片服务器及多个对象存储设备当前的负载信息，确定具有所述文件标识符的目标文件所对应的可用的对象存储设备；发送所述可用的切片服务器的设备标识符至所述客户端，以使所述客户端基于所述设备标识符，上传具有所述文件标识符的目标文件切片后得到的切片文件至所述可用的切片服务器；
    所述切片服务器，用于采集并发送所述切片服务器当前的负载信息以及与自身对应的对象存储设备当前的负载信息至所述元数据服务器；当接收到所述客户端发送的所述目标文件的切片文件，存储所述目标文件的多个第一切片文件至所述可用的对象存储设备，其中，所述第一切片文件为对所述目标文件的切片文件执行原始数据计算法所得到的切片文件。
  14. 根据权利要求13所述的系统，其特征在于，所述元数据服务器，具体用于：
    获取多个切片服务器的负载值，其中，所述负载值为切片服务器内中央处理单元的使用率、内存的使用率和输入/输出设备的使用率中的至少一种；
    判断多个切片服务器中是否存在多个第一切片服务器，其中，所述第一切片服务器的负载值小于预设第一负载阈值；
    如果是，从多个第一切片服务器中，选择具有所述文件标识符的目标文件所对应的可用的切片服务器；
    如果否，从多个切片服务器中，选择具有所述文件标识符的目标文件所对应的可用的切片服务器。
  15. 根据权利要求13所述的系统，其特征在于，所述元数据服务器，具体用于：
    获取所述可用的切片服务器的网络协议IP地址及多个对象存储设备的IP地址；
    判断多个对象存储设备中是否存在第一对象存储设备，其中，所述第一对象存储设备的IP地址与所述可用的切片服务器的IP地址相同；
    如果是，将所述第一对象存储设备作为可用的对象存储设备；
    如果否，基于多个对象存储设备当前的负载信息，从多个对象存储设备中，选择具有所述文件标识符的目标文件所对应的可用的对象存储设备。
  16. 根据权利要求15所述的系统，其特征在于，所述元数据服务器，具体还用于：
    获取多个对象存储设备的负载值；
    判断多个对象存储设备中是否存在多个第二对象存储设备，其中，所述第二对象存储设备的负载值小于预设第二负载阈值；
    如果是，从多个第二对象存储设备中，选择具有所述文件标识符的目标文件所对应的可用的对象存储设备；
    如果否，获取多个对象存储设备的分配次数；
    判断多个对象存储设备中是否存在多个第三对象存储设备，其中，所述第三对象存储设备的分配次数大于预设分配次数阈值；
    如果是，从多个第三对象存储设备中，选择具有所述文件标识符的目标文件所对应的可用的对象存储设备。
  17. 一种存储介质，其特征在于，用于存储可执行代码，所述可执行代 码用于在运行时执行如权利要求1-11任一项所述的文件切片上传方法。
  18. 一种应用程序，其特征在于，用于在运行时执行如权利要求1-11任一项所述的文件切片上传方法。
PCT/CN2017/101197 2016-09-28 2017-09-11 一种文件切片上传方法、装置及云存储系统 WO2018059222A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610858544.0A CN107872489B (zh) 2016-09-28 2016-09-28 一种文件切片上传方法、装置及云存储系统
CN201610858544.0 2016-09-28

Publications (1)

Publication Number Publication Date
WO2018059222A1 true WO2018059222A1 (zh) 2018-04-05

Family

ID=61761909

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/101197 WO2018059222A1 (zh) 2016-09-28 2017-09-11 一种文件切片上传方法、装置及云存储系统

Country Status (2)

Country Link
CN (1) CN107872489B (zh)
WO (1) WO2018059222A1 (zh)

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110688073A (zh) * 2019-10-10 2020-01-14 深圳市网心科技有限公司 数据存储方法、电子设备、系统及介质
WO2020043203A1 (zh) * 2018-08-31 2020-03-05 杭州海康威视系统技术有限公司 一种基于纠删码的数据存储方法、装置及电子设备
CN111404990A (zh) * 2020-02-14 2020-07-10 Oppo（重庆）智能科技有限公司 文件传输方法、装置、客户端及存储介质
CN111897791A (zh) * 2020-07-15 2020-11-06 银联商务股份有限公司 一种业务分流方法、装置、设备和存储介质
CN113704027A (zh) * 2021-10-29 2021-11-26 苏州浪潮智能科技有限公司 文件聚合兼容方法、装置、计算机设备和存储介质
CN113726842A (zh) * 2021-07-09 2021-11-30 北京房江湖科技有限公司 一种文件上传方法和装置、电子设备及存储介质
CN113852642A (zh) * 2021-10-09 2021-12-28 珠海迈科智能科技股份有限公司 一种基于dvb标准的ts流分片上传方法及其装置
CN114598697A (zh) * 2022-03-29 2022-06-07 浪潮云信息技术股份公司 一种传输文件的方法、装置及计算机可读存储介质
CN115150388A (zh) * 2022-06-24 2022-10-04 中国建设银行股份有限公司 文件上传和文件上传状态查询方法及其设备、存储介质
CN115412543A (zh) * 2022-04-24 2022-11-29 联通沃悦读科技文化有限公司 一种基于MapReduce的超大文件批量上传方法
CN117156172A (zh) * 2023-10-30 2023-12-01 江西云眼视界科技股份有限公司 视频切片上报方法、系统、存储介质及计算机

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112039932B (zh) * 2019-06-03 2023-10-20 北京国电通网络技术有限公司 文件传输方法和装置
CN111131450B (zh) * 2019-12-23 2022-06-21 北京奇艺世纪科技有限公司 文件上传方法、重组方法及装置
CN111147332B (zh) * 2019-12-29 2022-04-29 北京浪潮数据技术有限公司 存储系统云备份的上传进度确定方法、系统及相关装置
CN111158606B (zh) * 2019-12-31 2020-11-03 广州鼎甲计算机科技有限公司 存储方法、装置、计算机设备和存储介质
CN112583904B (zh) * 2020-12-04 2023-05-19 北京百度网讯科技有限公司 文件上传方法、装置、设备以及存储介质
CN115378930B (zh) * 2021-05-17 2024-08-13 成都鼎桥通信技术有限公司 文件分发方法及装置
CN113612735B (zh) * 2021-07-15 2022-09-02 中国联合网络通信集团有限公司 安全存储系统
CN113901356B (zh) * 2021-09-28 2022-08-09 北京五八信息技术有限公司 一种信息处理方法及装置
CN114301931B (zh) * 2022-03-11 2022-07-08 上海凯翔信息科技有限公司 一种基于云端nas的数据同步系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102420854A (zh) * 2011-11-14 2012-04-18 西安电子科技大学 面向云存储的分布式文件系统
CN104731516A (zh) * 2013-12-18 2015-06-24 腾讯科技（深圳）有限公司 一种存取文件的方法、装置及分布式存储系统
CN105100146A (zh) * 2014-05-07 2015-11-25 腾讯科技（深圳）有限公司 数据存储方法、装置及系统
US20160134684A1 (en) * 2013-06-14 2016-05-12 T-Data Systems (S) Pte Ltd System and method for uploading, showcasing and selling news footage
CN105791353A (zh) * 2014-12-23 2016-07-20 深圳市腾讯计算机系统有限公司 基于纠删码的分布式数据存储方法及系统

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101741730B (zh) * 2009-12-02 2012-05-30 成都市华为赛门铁克科技有限公司 文件下载方法及设备、提供文件下载服务的方法及系统
CN102316127B (zh) * 2010-06-29 2014-04-23 阿尔卡特朗讯 无线通信系统中基于分布式存储的文件传输方法
CN102833294B (zh) * 2011-06-17 2015-05-20 阿里巴巴集团控股有限公司 基于云存储的文件处理方法、系统及服务器集群系统
CN103731451B (zh) * 2012-10-12 2018-10-19 腾讯科技（深圳）有限公司 一种文件上传的方法及系统
CN103780658B (zh) * 2012-10-25 2017-09-12 中国电信股份有限公司 文件下载方法、装置和系统
CN103136338B (zh) * 2013-02-04 2016-02-10 中国科学院信息工程研究所 一种基于目录划分的元数据分布方法及装置
CN105025053A (zh) * 2014-04-24 2015-11-04 苏宁云商集团股份有限公司 基于云存储技术的分布式文件的上传方法及其系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102420854A (zh) * 2011-11-14 2012-04-18 西安电子科技大学 面向云存储的分布式文件系统
US20160134684A1 (en) * 2013-06-14 2016-05-12 T-Data Systems (S) Pte Ltd System and method for uploading, showcasing and selling news footage
CN104731516A (zh) * 2013-12-18 2015-06-24 腾讯科技（深圳）有限公司 一种存取文件的方法、装置及分布式存储系统
CN105100146A (zh) * 2014-05-07 2015-11-25 腾讯科技（深圳）有限公司 数据存储方法、装置及系统
CN105791353A (zh) * 2014-12-23 2016-07-20 深圳市腾讯计算机系统有限公司 基于纠删码的分布式数据存储方法及系统

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020043203A1 (zh) * 2018-08-31 2020-03-05 杭州海康威视系统技术有限公司 一种基于纠删码的数据存储方法、装置及电子设备
CN110688073A (zh) * 2019-10-10 2020-01-14 深圳市网心科技有限公司 数据存储方法、电子设备、系统及介质
CN110688073B (zh) * 2019-10-10 2023-07-14 深圳市网心科技有限公司 数据存储方法、电子设备、系统及介质
CN111404990A (zh) * 2020-02-14 2020-07-10 Oppo（重庆）智能科技有限公司 文件传输方法、装置、客户端及存储介质
CN111404990B (zh) * 2020-02-14 2023-06-02 Oppo（重庆）智能科技有限公司 文件传输方法、装置、客户端及存储介质
CN111897791A (zh) * 2020-07-15 2020-11-06 银联商务股份有限公司 一种业务分流方法、装置、设备和存储介质
CN111897791B (zh) * 2020-07-15 2024-01-12 银联商务股份有限公司 一种业务分流方法、装置、设备和存储介质
CN113726842A (zh) * 2021-07-09 2021-11-30 北京房江湖科技有限公司 一种文件上传方法和装置、电子设备及存储介质
CN113852642B (zh) * 2021-10-09 2023-05-09 珠海迈科智能科技股份有限公司 一种基于dvb标准的ts流分片上传方法及其装置
CN113852642A (zh) * 2021-10-09 2021-12-28 珠海迈科智能科技股份有限公司 一种基于dvb标准的ts流分片上传方法及其装置
CN113704027A (zh) * 2021-10-29 2021-11-26 苏州浪潮智能科技有限公司 文件聚合兼容方法、装置、计算机设备和存储介质
CN114598697A (zh) * 2022-03-29 2022-06-07 浪潮云信息技术股份公司 一种传输文件的方法、装置及计算机可读存储介质
CN114598697B (zh) * 2022-03-29 2024-06-07 浪潮云信息技术股份公司 一种传输文件的方法、装置及计算机可读存储介质
CN115412543A (zh) * 2022-04-24 2022-11-29 联通沃悦读科技文化有限公司 一种基于MapReduce的超大文件批量上传方法
CN115150388A (zh) * 2022-06-24 2022-10-04 中国建设银行股份有限公司 文件上传和文件上传状态查询方法及其设备、存储介质
CN115150388B (zh) * 2022-06-24 2024-04-26 中国建设银行股份有限公司 文件上传和文件上传状态查询方法及其设备、存储介质
CN117156172A (zh) * 2023-10-30 2023-12-01 江西云眼视界科技股份有限公司 视频切片上报方法、系统、存储介质及计算机
CN117156172B (zh) * 2023-10-30 2024-01-16 江西云眼视界科技股份有限公司 视频切片上报方法、系统、存储介质及计算机

Also Published As

Publication number Publication date
CN107872489A (zh) 2018-04-03
CN107872489B (zh) 2020-06-05

Similar Documents

Publication Publication Date Title
WO2018059222A1 (zh) 一种文件切片上传方法、装置及云存储系统
WO2021036228A1 (zh) 文件上传方法、装置、终端、服务器、系统及存储介质
US9998531B2 (en) Computer-based, balanced provisioning and optimization of data transfer resources for products and services
US9804981B2 (en) Method, controller, and system for service flow control in object-based storage system
AU2014235793B2 (en) Automatic tuning of virtual data center resource utilization policies
US8769269B2 (en) Cloud data management
TWI528191B (zh) File Handling Method Based on Cloud Storage, System and Server Cluster System
US9772801B2 (en) Performing volume expansion in storage management system
US10764132B2 (en) Scale-out association method and apparatus, and system
WO2015067117A1 (zh) 一种云上传方法及系统、调度设备、客户端
US11262916B2 (en) Distributed storage system, data processing method, and storage node
US8949430B2 (en) Clustered computer environment partition resolution
WO2019041738A1 (zh) 客户资源获取方法、装置、终端设备及存储介质
CN111585887B (zh) 基于多个网络的通信方法、装置、电子设备及存储介质
WO2014194869A1 (zh) 一种请求处理方法、装置及系统
US10776173B1 (en) Local placement of resource instances in a distributed system
US10505862B1 (en) Optimizing for infrastructure diversity constraints in resource placement
WO2021139264A1 (zh) 对象存储控制方法、装置、计算机设备及存储介质
US11079968B1 (en) Queue management in multi-site storage systems
WO2014183417A1 (zh) 管理内存的方法、装置和系统
US20210081352A1 (en) Internet small computer interface systems extension for remote direct memory access (rdma) for distributed hyper-converged storage systems
CN107301090B (zh) 在存储服务器中为应用设置数据处理路径的方法及装置
WO2013104181A1 (zh) 管理资源的方法和装置
US10812408B1 (en) Preventing concentrated selection of resource hosts for placing resources
US20150120913A1 (en) Dynamic cloning of application infrastructures

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17854672

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17854672

Country of ref document: EP

Kind code of ref document: A1