CN112866409A

CN112866409A - Object transmission and collection method based on object storage fragment uploading and terminal equipment

Info

Publication number: CN112866409A
Application number: CN202110184218.7A
Authority: CN
Inventors: 陈坚; 王东; 邱尚高
Original assignee: Shenzhen Sandstone Data Technology Co ltd
Current assignee: Shenzhen Sandstone Data Technology Co ltd
Priority date: 2021-02-08
Filing date: 2021-02-08
Publication date: 2021-05-28
Anticipated expiration: 2041-02-08
Also published as: CN112866409B

Abstract

The invention discloses an object transmission and collection method based on object storage fragment uploading and a terminal device, wherein the object transmission method comprises the following steps: sending an object initialization request and receiving an object identifier; dividing the object into fragments, writing data, and sending fragment uploading requests; receiving fragment data; sending a fragment uploading completion request; sending a request for modifying object data, receiving a segment interval to be modified, uploading a segment again by using the segment interval to be modified, covering old data by using new segment data, and sending a segment uploading completion request again when the uploading of the new data is completed. The method and the device realize that the object is visible in the process of uploading the large object in the fragments, directly cover and modify the object data in the mode of uploading the new fragments, do not need to upload the whole object again to update and modify, save modification time and resources, and further solve the problems of invisible object and inconvenient modification in the uploading process.

Description

Object transmission and collection method based on object storage fragment uploading and terminal equipment

Technical Field

The invention relates to the technical field of storage, in particular to an object transmission and collection method and terminal equipment based on uploading of object storage fragments.

Background

With the rapid development of information technology, the data volume of the internet is more and more huge, and the distributed object storage technology comes along with the development. The technology is very suitable for storing unstructured data such as audio and video files and the like, and has the characteristics of directly and efficiently accessing a disk and meeting file sharing. Thus, the object storage technology is taken as a key technology for realizing storage products by manufacturers of public clouds and private clouds.

In the object storage system, the object which is the most basic unit is divided into two parts of data and metadata, and the two parts are respectively stored in different positions of a disk. Wherein the user accesses the object through the metadata. The metadata also contains two parts of content. A portion of the metadata describes attribute information for the data, and generally includes: object name, object size, creation time, creator, and user-defined attributes. Another part of the metadata is called manifest, which describes the distribution of the location of the object data on the disk. The user can read the data of the object on the disk through the metadata information of the object.

Generally, an object storage system supports two ways of uploading objects to the storage system, namely, integral uploading and fragmented uploading. The whole uploading is used for uploading scenes with smaller objects and shorter uploading time; the fragment uploading is used for uploading a large object in a long-time scene, such as video monitoring, streaming log analysis and the like, a single file is generally large and can be continuously written for a period of time, for example, a single video monitoring file needs to support video stream writing for about 30 minutes, and the video monitoring data cannot be read according to the object. But in video surveillance scenarios, the user sometimes needs to play back <30 minutes of data, and a method is needed to be able to read the file being written. However, whether the whole uploading or the fragmented uploading is performed, the object storage system generates the metadata after the data uploading is completed. This results in large objects that are not visible to the user during upload through the shards because there is no metadata. Since the large object data portion is large, the object may not be visible to the user for a long time.

For an object stored in the object storage system, if the data of the object needs to be modified, the object can only be processed by uploading the same-name object. The method is acceptable for the objects uploaded integrally, because the general data of the objects uploaded integrally is smaller, and the uploading requires less resources. However, for a large object uploaded by fragmentation, it is obviously unreasonable to upload the object again because the data is large. Therefore, it is desirable to be able to directly modify portions of data of an object to achieve the goal of object modification.

The patent with publication number CN111835867A provides a file uploading method and system, the method of the present invention includes the following steps: s1, acquiring a file and initializing the acquired file; s2, dividing the file into fragments, filtering the uploaded fragments, and uploading the rest fragments; and S3, merging the uploaded fragments into an original file. The system comprises a file acquisition module, a file initialization module, a file fragmentation module, a file uploading module and an object storage module. According to the invention, through a fragmentation uploading and breakpoint continuous uploading mechanism, the file is uploaded by using a multithreading technology, so that the uploading speed of the file is greatly increased, the uploading time and the risk are reduced, and the uploading experience of large files and oversized files is effectively improved.

The main drawbacks of this solution are:

1. in the process of uploading the object, the object cannot be visible to the user;

2. after the object is uploaded, the object data cannot be directly modified.

The patent with publication number CN109511008A provides a method for supporting content addition of video and audio files based on object storage, which relates to the technical field of video and audio processing, and the invention comprises S1: writing the source file to generate a corresponding inx file; s2: writing the video and audio files and the inx file into the object storage in an additional writing mode; s3: when the writing of the video and audio files is finished, deleting the corresponding inx files; s4: when the video and audio files are read, whether the corresponding inx file exists in the video and audio files is judged, and if the corresponding inx file does not exist, the video and audio files are completely written into the object for storage and are directly read; if the corresponding inx file exists, the video and audio file is read according to the offset and the size of each frame of the video and audio file recorded by the inx file, the video and audio file is written into the object storage in a data appending mode, so that the file is visible to a user in the acquisition process, the user can read the file in the acquisition process and edit the file, and the video and audio file is supported to be edited while being acquired in the object storage.

The main drawbacks of this solution are:

2. only audio files can be processed, and other types of files cannot use the method;

3. secondly, inx files need to be generated and need to record a large amount of information, which increases the burden of the server; the object file and the inx file need to be kept consistent, and the consistency is difficult to maintain.

The above background disclosure is only for the purpose of assisting understanding of the concept and technical solution of the present invention and does not necessarily belong to the prior art of the present patent application, and should not be used for evaluating the novelty and inventive step of the present application in the case that there is no clear evidence that the above content is disclosed at the filing date of the present patent application.

Disclosure of Invention

In order to solve the problems of invisible objects and inconvenient modification in the uploading process in the prior art, the invention provides a novel large object fragment uploading method.

The technical problem of the invention is solved by the following technical scheme:

the invention provides an object transmission method based on object storage fragment uploading, which comprises the following steps: sending an object initialization request and receiving an object identifier; dividing the object into fragments, writing data, and sending fragment uploading requests; receiving fragment data; sending a fragment uploading completion request; sending a request for modifying object data, receiving a segment interval to be modified, uploading a segment again by using the segment interval to be modified, covering old data by using new segment data, and sending a segment uploading completion request again when the uploading of the new data is completed.

In some embodiments, the following optional technical features are also included:

in some embodiments, the cutting the object into slices in the step a2 includes: and cutting the object into N fragments according to a certain size, wherein each fragment is marked with fragment metadata and a fragment number (1-N).

In some embodiments, in step a5, after receiving the segment interval to be modified, the length of the modified data is divided by the number of segments to obtain the size of each segment, and then the segments are uploaded.

In some embodiments, in step a5, when the modification target data is in a deleted form, after receiving the segment interval to be modified, using the segment interval to be modified, uploading data with a data value of 0 by a segment, and deleting the corresponding segment data in a covering manner.

The invention also provides an object collection method based on the uploading of the object storage fragments, which comprises the following steps: s1: receiving an initialization request, generating an object identifier and returning to generate hidden metadata; s2: receiving a fragment uploading request, analyzing a fragment table from hidden metadata, taking a fragment number of an uploaded fragment as a keyword of the fragment table, taking fragment metadata as a value, respectively writing the fragment number into the fragment table, updating the hidden metadata, and generating temporary metadata according to the fragment table; s3: reading temporary metadata, analyzing each fragment metadata from the temporary metadata, acquiring fragment data according to the fragment metadata, and sending the fragment data; s4: receiving a fragment uploading completion request, defining object metadata, analyzing a fragment table from hidden metadata, traversing the fragment table, merging the uploaded fragment metadata into the object metadata, deleting temporary metadata, and storing the object metadata in an object storage system; s5: receiving a request for modifying the object data, analyzing the interval of the data to be modified, acquiring the size of the object from the object metadata, determining the starting position and the ending position of the data to be modified, analyzing a fragment table from the hidden metadata, traversing the fragment table, recording the fragment numbers in the starting position and the ending position of the data to be modified to the interval of the fragment to be modified, and sending the interval.

In some embodiments, in step S1, hidden metadata is generated with the object name + object identification as a key.

In some embodiments, in step S2, the key of the parsed fragment table is the fragment number (1-N) of each fragment, and the value is the metadata of each fragment.

In some embodiments, in the step S2, generating temporary metadata according to the fragmentation table includes: defining temporary metadata of an object, traversing the fragment table, combining the fragment metadata which is uploaded completely into the temporary metadata, and storing the temporary metadata in the object storage system.

The invention also provides a terminal device based on object memory fragment uploading, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and is characterized in that the processor implements the steps of any one of the methods when executing the computer program.

The invention also proposes a computer-readable storage medium, in which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any one of the above.

Compared with the prior art, the invention has the advantages that:

1. the method can generate temporary metadata for the object in the uploading process of the object memory fragment, particularly for a large object, and the object can be visible to a user and can be operated (read operation and the like) in the uploading process due to the generation of the temporary metadata in the uploading process; meanwhile, for the uploaded object, the method can directly cover and modify the object data in an uploading new fragment mode without re-uploading the whole object to update and modify, thereby saving modification time and resources;

2. secondly, the method does not influence the original functions of asynchronous copying, multi-version and the like; the application range is wide, and the method can be used for videos, log streams and the like.

Drawings

Fig. 1 is a flowchart of an object transmission and collection method based on object memory slice upload according to an embodiment of the present invention.

Fig. 2 is a flowchart of object initialization of an object transfer and collection method based on object memory slice upload according to an embodiment of the present invention.

Fig. 3 is a flowchart of a fragment upload of an object transfer and collection method based on an object storage fragment upload according to an embodiment of the present invention.

FIG. 4 is a flowchart of generating temporary metadata according to an embodiment of the present invention.

Fig. 5 is a flowchart of object reading of an object transmission and collection method based on object memory slice uploading according to an embodiment of the present invention.

FIG. 6 is a flowchart of object metadata generation for an object delivery and collection method based on object memory slice upload according to an embodiment of the present invention.

FIG. 7 is a flowchart of modifying object data according to an embodiment of the present invention based on an object transmission and collection method for object memory slice uploading.

Detailed Description

The invention will be further described with reference to the accompanying drawings and preferred embodiments. It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.

The embodiment of the invention comprises five parts: the time sequence relations of the five parts of object initialization, fragment uploading, uploading completion, object reading and object data modification in the uploading process are shown in figure 1.

Firstly, the first step of large object uploading is that a client sends an object initialization request to a server, and the server returns a result to the client after completing the request processing.

Then, the client divides the large object into N fragments, and writes the data into the server in a fragment uploading mode. In the process, the server side can generate temporary metadata for the object, so that the object can be visible to a user in the process of uploading the large object.

Since temporary metadata is generated, the user can manipulate the object during the fragment upload process.

And finally, the client sends an uploading completion request, and the server deletes the temporary metadata and generates object metadata after receiving the request.

After the object is uploaded, if the object data needs to be modified, the client can send a request for modifying the object to the server, the server returns the interval of the data to be modified to the client, and the client rewrites the data once through fragment uploading according to the acquired interval to be modified, so that the target of modifying the data by the object can be achieved.

1. Object initialization

The object initialization is a process in which the client sends an initial request to the object storage system, and the object storage system generates a unique object identifier for the object and returns the unique object identifier to the client, and the specific flow is as shown in fig. 2.

Object initialization process:

(1) the client sends an initialization request to the server;

(2) the server generates an object identifier;

(3) and the server side uses the object name and the object identifier as keywords to generate the hidden element number.

2. Fragment uploading

The client cuts the object into N shares according to a certain size, and each share is called a fragment. And the client sends each piece of data, the object identification and the number (the number range is 1-N) of each piece to the object storage system. After the object storage system receives the request, the process is as shown in FIG. 3.

Fragment uploading process:

(1) starting to write the fragments, and firstly writing the data part;

(2) acquiring hidden metadata;

(3) parsing a fragment table from the hidden metadata;

(4) taking the fragment number of the uploaded fragment as a keyword of the fragment table, taking the fragment metadata as a value, and writing the value into the fragment table;

(5) updating the hidden metadata;

(6) and generating temporary metadata according to the fragmentation table.

After receiving the request for writing the fragments, the object storage system reads the data to be written, and then temporarily stores the metadata for the object. And if the data writing fails, returning to the client, and if the data writing succeeds, reading the hidden metadata. And analyzing a fragment table from the hidden metadata, wherein the key word of the fragment table is the fragment number of each fragment, and the value of the fragment table is the metadata of each fragment, and the metadata information of all the uploaded fragments can be acquired through the fragment table, and the object temporary metadata can be constructed through the information. The object temporary metadata is that metadata of each fragment is merged, and the positions of all fragments distributed on a disk can be known through object temporary metadata information, so that uploaded data can be read. The fragment table parsed from the hidden metadata does not contain the metadata information of the fragment being uploaded, all the fragment numbers and the fragment metadata of the fragment need to be stored in the fragment table, and then the hidden metadata is updated. And finally, generating temporary metadata for the object according to the fragment table information.

For temporary metadata, since each fragment needs to be generated once, since the temporary metadata generated each time will overwrite what existed before, many temporary metadata of the object will not be seen to the user.

The generation process of temporary metadata is shown in fig. 4.

Temporary metadata generation flow:

(1) defining temporary metadata for an object;

(2) traversing the fragment table, and merging the uploaded fragment metadata into temporary metadata;

(3) the temporary metadata is downloaded, and the temporary metadata is generated.

Firstly, defining temporary metadata of an object, then traversing the fragment table, taking out the metadata of each fragment, and merging the metadata into the temporary metadata of the object. Thus, the data of the uploaded fragment can be accessed according to the temporary metadata of the object. Then, attribute information such as an object name and an object size is set to the temporary metadata, and the temporary metadata of the object is generated together with the object name and the object size, and the temporary metadata is downloaded. The user can access the object through the temporary metadata, thereby achieving the purpose of writing data and accessing the data at the same time.

3. Reading objects in an upload process

Since temporary metadata is generated for the object during the uploading process, the object is visible to the user during the uploading process. The object reading flow is shown in fig. 5.

Reading an object in an uploading process:

(1) reading temporary metadata;

(2) analyzing the metadata of each fragment from the temporary metadata;

(3) and obtaining corresponding data according to the fragment metadata and returning the data to the client.

First, temporary metadata (temp _ obj _ meta) of the object is read. And then analyzes the metadata information of each slice from temp _ obj _ meta. And circularly reading the data of each fragment and returning the data to the client until all the fragment data are completely read. The deleted object data can be read through the flow.

4. Completion of uploading

When uploading of all fragments is completed, the client needs to send an upload completion request to the server. When the server receives the upload completion request, object metadata is generated for the object, and the temporary metadata needs to be deleted, as shown in fig. 6.

And (3) uploading to finish the flow:

(1) defining object metadata;

(2) reading the hidden metadata, and analyzing the fragment table;

(3) traversing the fragment table, and merging the uploaded fragment metadata into the object metadata;

(4) deleting temporary metadata;

(5) and downloading the object metadata.

When the server receives the fragment upload completion request, object metadata (obj _ meta) is defined. The hidden metadata is then read and the slice table information is parsed from it, and the object metadata (obj _ meta) is merged according to the slice table information. After the merging is completed, the temporary metadata needs to be deleted, and the number of object elements needs to be downloaded. At this time, the user can manipulate the object through the object metadata. Therefore, the temporary metadata only exists in the fragment uploading process, and the effect of the temporary metadata is to solve the invisible problem caused by the fact that the object does not have the metadata in the fragment uploading process. And deleting the hidden metadata so as to serve the modified object subsequently.

5. Modifying object data

The client sends a request for modifying the object data to the server, and the processing flow of the server is shown in fig. 7.

And modifying the object data flow:

(1) analyzing the interval of the data to be modified;

(2) obtaining an object size from object metadata;

(3) determining the starting position and the ending position of data to be modified;

(4) reading the hidden metadata, and analyzing a fragment table;

(5) traversing the fragment table, and recording the fragment number in the fragment table to a to-be-modified interval;

(6) the server side returns the interval to be modified to the client side;

(7) the client uses the re-uploading fragment and uses the new fragment data to cover the old data, so that the aim of modifying the object data can be fulfilled;

(8) and after the new data is uploaded, executing the uploading completion process again, and updating the object metadata.

The server receives the request for modifying the object data, can analyze the interval of the data to be modified ([ m, n ], where m and n are integers from 0 to 100, representing the percentage of the interval that the whole object data needs to be modified), then obtain the metadata of the object, and analyze the size of the whole object from the metadata information. From the size of the object and m, n, the start and end positions ofs, end where the data needs to be modified can be determined. And acquiring a fragment table through the object hidden metadata, circularly traversing the fragment table, and summing the sizes of the fragments to acquire the position (location) of the current fragment in the whole object. If the position is between [ ofs, end ], the fragment belongs to the fragment to be modified, and part _ num of the fragment is recorded into the interval [ x, y ] to be modified. And finally, returning the interval [ x, y ] to be modified to the client.

After the client acquires the interval [ x, y ] to be modified, the length of the modified data is divided by (y-x +1) to obtain the size of each fragment. And then uploading the fragments to the server side through fragment uploading.

And finally, the client sends the uploading completion request again to update the object metadata.

If the data is to be deleted, after the client acquires the interval [ x, y ] to be modified, the data with the data volume of 0 is uploaded in a fragmentation mode by using the interval [ x, y ] to be modified, and the target of deleting partial data in the uploaded object can be achieved by covering corresponding fragment metadata information.

If the client side uploads the concurrent fragments, due to the fact that updating of the hidden metadata is atomicity, updating of the temporary metadata of the concurrent object is unique. The temporary metadata that is visible to the user is always up-to-date.

The present invention can be extended to the overall upload flow. Only hidden metadata needs to be generated for the integrally uploaded object, and then the object modification function is realized according to the fragment uploading method.

The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several equivalent substitutions or obvious modifications can be made without departing from the spirit of the invention, and all the properties or uses are considered to be within the scope of the invention.

Claims

1. An object transmission method based on object memory fragmentation uploading is characterized by comprising the following steps:

a1: sending an object initialization request and receiving an object identifier;

a2: dividing the object into fragments, writing data, and sending fragment uploading requests;

a3: receiving fragment data;

a4: sending a fragment uploading completion request;

a5: sending a request for modifying object data, receiving a segment interval to be modified, uploading a segment again by using the segment interval to be modified, covering old data by using new segment data, and sending a segment uploading completion request again when the uploading of the new data is completed.

2. The object storage slice upload-based object transmission method of claim 1, wherein the step of cutting the object into slices in a2 comprises: and cutting the object into N fragments according to a certain size, wherein each fragment is marked with fragment metadata and a fragment number (1-N).

3. The object transmission method based on object storage slice uploading as claimed in claim 1, wherein in step a5, after receiving the segment interval to be modified, the length of the modified data is divided by the number of slices to obtain the size of each slice, and then the slices are uploaded.

4. The object transmission method based on object storage slice uploading as claimed in claim 1, wherein in step a5, when the modified object data is in the form of deletion, after receiving the segment interval to be modified, using the segment interval to be modified, the segment uploads data with a data value of 0, and the deletion of the corresponding segment data is covered.

5. An object collection method based on object storage fragment uploading is characterized by comprising the following steps:

s1: receiving an initialization request, generating an object identifier and returning to generate hidden metadata;

s2: receiving a fragment uploading request, analyzing a fragment table from hidden metadata, taking a fragment number of an uploaded fragment as a keyword of the fragment table, taking fragment metadata as a value, respectively writing the fragment number into the fragment table, updating the hidden metadata, and generating temporary metadata according to the fragment table;

s3: reading temporary metadata, analyzing each fragment metadata from the temporary metadata, acquiring fragment data according to the fragment metadata, and sending the fragment data;

s4: receiving a fragment uploading completion request, defining object metadata, analyzing a fragment table from hidden metadata, traversing the fragment table, merging the uploaded fragment metadata into the object metadata, deleting temporary metadata, and storing the object metadata in an object storage system;

s5: receiving a request for modifying the object data, analyzing the interval of the data to be modified, acquiring the size of the object from the object metadata, determining the starting position and the ending position of the data to be modified, analyzing a fragment table from the hidden metadata, traversing the fragment table, recording the fragment numbers in the starting position and the ending position of the data to be modified to the interval of the fragment to be modified, and sending the interval.

6. The object collection method based on object memory slice uploading according to claim 5, wherein in the step of S1, hidden metadata is generated using object name + object id as a key.

7. The method for collecting objects based on object storage slice uploading according to claim 5, wherein in step S2, the key for parsing out the slice table is the slice number (1-N) of each slice, and the value is the metadata of each slice.

8. The method for collecting objects based on object storage slice uploading according to claim 5, wherein the step of S2, generating temporary metadata according to the slice table, comprises: defining temporary metadata of an object, traversing the fragment table, combining the fragment metadata which is uploaded completely into the temporary metadata, and storing the temporary metadata in the object storage system.

9. A terminal device for object-based memory slice uploading, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any of claims 1-4 or 5-8 when executing the computer program.

10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 4 or 5 to 8.