CN107295425B - Method for rapidly splicing transcoding fragmented files - Google Patents
Method for rapidly splicing transcoding fragmented files Download PDFInfo
- Publication number
- CN107295425B CN107295425B CN201710438505.XA CN201710438505A CN107295425B CN 107295425 B CN107295425 B CN 107295425B CN 201710438505 A CN201710438505 A CN 201710438505A CN 107295425 B CN107295425 B CN 107295425B
- Authority
- CN
- China
- Prior art keywords
- file
- transcoding
- fragment
- aggregation
- distributed storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 239000012634 fragment Substances 0.000 claims abstract description 145
- 230000002776 aggregation Effects 0.000 claims description 111
- 238000004220 aggregation Methods 0.000 claims description 111
- 230000008569 process Effects 0.000 claims description 24
- 230000004048 modification Effects 0.000 claims description 5
- 238000012986 modification Methods 0.000 claims description 5
- 230000004931 aggregating effect Effects 0.000 claims description 3
- 238000001514 detection method Methods 0.000 claims description 2
- 238000005192 partition Methods 0.000 claims description 2
- 229920006395 saturated elastomer Polymers 0.000 claims description 2
- 238000012217 deletion Methods 0.000 description 5
- 230000037430 deletion Effects 0.000 description 5
- 238000004806 packaging method and process Methods 0.000 description 4
- 238000010276 construction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000006116 polymerization reaction Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/845—Structuring of content, e.g. decomposing content into time segments
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234309—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 or from Quicktime to Realvideo
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/845—Structuring of content, e.g. decomposing content into time segments
- H04N21/8455—Structuring of content, e.g. decomposing content into time segments involving pointers to the content, e.g. pointers to the I-frames of the video stream
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A method for rapidly splicing transcoding fragmented files belongs to the technical field of computers. Writing a video source file into distributed storage, providing a uniform name space for transcoding service by using a distributed storage system, reading the video source file from the distributed storage by using a plurality of transcoding servers in parallel, and acquiring data corresponding to a video source file time period according to a fragmented time period interval; and distributing the index number of the fragment data segment to the transcoding node through the transcoding scheduler, reading the time slice data corresponding to the index number from the distributed storage by the transcoding node, and transcoding the fragment data. According to the method and the device, the length of the fragment transcoding file, the file number and the offset information of the aggregated file are recorded in the index item of the aggregated information attribute of the aggregated file, so that the traditional mode of additional writing is completely abandoned, the time-consuming data reading and writing operation is changed into the increase of the attribute information, the efficiency of merging the fragment transcoding file is improved, and precious network bandwidth resources are saved.
Description
Technical Field
The invention relates to a method for rapidly splicing transcoding fragmented files, and belongs to the technical field of computers.
Background
With the requirement of the traditional broadcasting and television industry on video definition becoming higher and higher, the length of a program material file in one hour in the future 4K era can reach 180GB from the earliest code stream of 25Mbps in the standard definition era to the current code stream of 100Mbps in the high definition era and then to the future 4K era. The traditional transcoding scheme is that the fragmented transcoding files are sequentially additionally written into the final transcoding file, the additional writing scheme has reading operation on each fragmented transcoding file and writing operation on the final transcoding file, and the additional writing scheme consumes network resources and transcoding node system resources; the traditional transcoding scheme can only achieve 1.5 to 2 times of speed, and can not meet the transcoding requirement caused by video data expansion.
With the video sources becoming clearer and more television program types becoming more and more, the data volume of the radio and television industry resource system reaches the PB level, and the storage demand volume is increased suddenly; meanwhile, television station service systems are various in types, different in storage requirements, and have to adopt a plurality of sets of storage systems, but information islands are easily caused among the plurality of sets of storage systems, and meanwhile, the construction and maintenance cost is high, and the unified management is not favorable.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention provides a method for rapidly splicing transcoding fragmented files.
A method for rapidly splicing transcoding fragmented files comprises the following steps;
writing a video source file into distributed storage, providing a uniform name space for transcoding service by using a distributed storage system, reading the video source file from the distributed storage by using a plurality of transcoding servers in parallel, and acquiring data corresponding to a video source file time period according to a fragmented time period interval;
distributing index numbers of the fragment data segments to the transcoding nodes through a transcoding scheduler, reading time slice data corresponding to the index numbers from a distributed storage by the transcoding nodes, and transcoding the fragment data;
writing the transcoding data into distributed storage to generate a fragment transcoding file, wherein a uniform name space is provided for the fragment transcoding file by a distributed storage system;
calling an interface provided by distributed storage at a storage management node to generate an aggregation file;
when the fragment transcoding file meets the aggregation requirement, the unique number of the fragment transcoding file in a storage system, the length of the fragment transcoding file and the offset of the fragment transcoding file in the aggregation file are added to attribute information of the aggregation file;
deleting the name space of the fragment transcoding file in the storage system, so that the fragment transcoding file is invisible to the outside;
and reserving the attribute information and the data information of the sliced transcoding file.
The distributed storage system provides a uniform name space for the video source file, and the transcoding nodes can read the video source file in parallel;
before a video source file is transcoded, the file length is fixed, and the recorded video file is recorded according to a constant code stream, so that the length interval of the file is a corresponding time interval;
the transcoding service needs to be configured on a node together with a storage access node, and the storage access node provides a uniform name space for the transcoding service;
the transcoding scheduler is responsible for fragmenting the video file and distributing interval indexes among the fragmented partitions; then sequentially distributing interval indexes to be transcoded to the transcoding service according to the idle state of the transcoding service until the load of the transcoding service is saturated;
the transcoding service reads video source index interval data into a memory through a reading operation interface provided by distributed storage, then performs transcoding, writes the data in the memory into the distributed storage after transcoding is completed, generates a fragment transcoding file, and generates a file name according to an interval index number by naming the fragment transcoding file;
the distributed storage system provides a unified namespace for accessing the fragmented transcode file.
Sequentially adding all file index information to the aggregation information of the aggregation file attributes according to the fragment index sequence;
the process of associating the aggregate file with the fragment transcoding file is not traditional fragment file data content copying, but is the addition of aggregate information and an aggregate index in the aggregate file, and the sequence of the index addition is consistent with the sequence of the fragments of the file;
the index information corresponding to each index item comprises: the unique number of the fragment files in the distributed storage system, the length of the fragment files and the offset of the fragment files in the aggregation file;
the aggregated file only provides read permission and does not have write permission;
and converting the read aggregated file data into data of the read fragmented file through an interface provided by the distributed storage system.
The step of calling an interface provided by the distributed storage to generate the aggregation file at the storage management node comprises the following steps: calling a distributed storage interface to create an aggregate file, then pre-detecting the fragment transcoding file, and continuing an aggregate flow only after the detection is passed, wherein the pre-detecting step comprises the following steps:
judging whether the aggregate information index items in the aggregate file exceed the specification, if transcoding fails, the currently provided specification is that the upper limit of the fragment transcoding file supported in one aggregate file is 10000, and the upper limit of the specification can be adjusted according to requirements;
locking the fragment transcoding file, wherein the purpose of locking is to prevent other processes from modifying or aggregating the fragment transcoding file in the aggregation process, the simultaneous modification of the fragment transcoding file by multiple processes can cause the length of the fragment file recorded in the aggregated file to be inconsistent with the length recorded by the attribute of the fragment file, the simultaneous aggregation of the fragment transcoding files by multiple processes can cause the same fragment transcoding file to be aggregated into multiple aggregated files, the repeated locking of the fragment transcoding file can fail, and if the locking fails, the aggregation operation fails;
judging whether the fragment transcoding file is an empty file or not, if so, failing to perform aggregation operation, and unlocking the fragment transcoding file;
judging whether the fragment transcoding file is a common file or not, if not, failing the aggregation operation, and unlocking the fragment transcoding file;
and judging whether data is written into the fragment transcoding file or not, if data is written into the fragment transcoding file, the aggregation operation fails, and unlocking the fragment transcoding file.
The step of transcoding the file by the aggregation fragment comprises the following steps:
acquiring an aggregation information index item in the attribute of the aggregation file, wherein index information corresponding to the index item comprises: the unique number of the fragment transcoding file in the distributed storage system, the length of the fragment transcoding file, the offset of the fragment transcoding file in the aggregated file and the initial index item are zero;
if the file is the first sliced transcoding file, adding one to the index item, and obtaining the index information corresponding to the first index item: the unique number of the fragment transcoding file, the length of the fragment transcoding file and the offset of the fragment transcoding file in the aggregation file are zero;
if the aggregated fragmented file is not the first fragmented transcoding file, adding one to the index item, and obtaining index information corresponding to the index item of the file: the unique number of the fragmented transcoding file, the length of the fragmented transcoding file and the offset of the fragmented transcoding file in the aggregate file are the sum of the length of the last transcoding fragmented file and the offset of the last transcoding fragmented file in the aggregate file;
deleting the name space of the fragment transcoding file in distributed storage, reserving the attribute information and the data information of the fragment transcoding file, and making the fragment transcoding file invisible to the outside in a distributed storage system;
and unlocking the fragment transcoding file, and finishing the aggregation operation.
The aggregation operation process is the description of the aggregation process of a single fragmented transcoding file, and the aggregation process of a plurality of fragmented transcoding files is the repetition of the aggregation process.
The method has the advantages that the length and the file number of the fragment transcoding file and the offset information of the aggregation file are recorded in the index item of the aggregation information attribute of the aggregation file, so that the traditional mode of additional writing is completely abandoned, time-consuming data reading and writing operations are changed into the increase of the attribute information, the merging efficiency of the fragment transcoding file is improved, and precious network bandwidth resources are saved.
The invention can also be applied to the packaging application in the broadcasting and television industry, the storyboard in the packaging application can be analogized to a video source in transcoding application, the fragments of the storyboard are fragmented according to the designated segmentation points, then the packaging nodes convert the fragmented storyboard into fragment videos through editing and write the fragment videos into distributed storage, and the fragment videos are aggregated through an aggregation interface provided by the storage to realize extremely-fast packaging.
The invention can provide the transverse expansion capability for the video resource storage space and can meet the requirement of rapid increase of the data volume of the video resource; the invention can provide a uniform name space, thereby breaking through information islands among all video systems, facilitating resource management and reducing the maintenance cost of a storage system; the invention provides a function of rapidly splicing the sliced transcoding file, abandons the traditional mode of additional writing, and uses the mode of adding the extremely-fast aggregate information index item, thereby not only saving precious network bandwidth resources, but also improving the efficiency of splicing the sliced transcoding file, and the invention can improve the transcoding efficiency to 20 to 30 times.
Drawings
A more complete appreciation of the invention and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein the accompanying drawings are included to provide a further understanding of the invention and form a part of this specification, and wherein the illustrated embodiments of the invention and the description thereof are intended to illustrate and not limit the invention, as illustrated in the accompanying drawings, in which:
FIG. 1 is a flow chart of a process for rapidly splicing fragmented transcoded files according to the present invention;
FIG. 2 is a flow chart of reading the content of an aggregate file according to the present invention;
FIG. 3 is a flow chart of obtaining aggregate file attributes in accordance with the present invention;
FIG. 4 is a diagram illustrating deletion of an aggregate file according to the present invention;
the invention is further illustrated with reference to the following figures and examples.
Detailed Description
It will be apparent that those skilled in the art can make many modifications and variations based on the spirit of the present invention.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element, component or section is referred to as being "connected" to another element, component or section, it can be directly connected to the other element or section or intervening elements or sections may also be present. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.
It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
The following examples are further illustrative and should not be construed as limiting the embodiments of the present invention.
Example 1: as shown in fig. 1, a method for rapidly splicing transcoded fragmented files includes:
calling an aggregation file creating interface of the distributed storage system to create an aggregation file;
acquiring a fragment transcoding file from a distributed storage system;
calling a distributed storage system interface, and locking the fragment transcoding file;
when locking of the fragment transcoding file fails, the aggregation operation fails;
judging whether the fragment file meets a transcoding condition;
when the fragment transcoding file does not meet the condition, the aggregation operation fails;
when the fragment transcoding file meets the condition, adding one to the aggregation information index item of the aggregation file, wherein the index information corresponding to the newly added aggregation index item is the length of the fragment transcoding file, the unique serial number of the fragment transcoding file and the offset of the fragment transcoding file in the aggregation file;
deleting the name space of the fragment transcoding file, and reserving the attribute information and the data information of the fragment transcoding file;
and unlocking the fragment transcoding file, and finishing the file aggregation operation.
The deletion of the aggregated file is different from the deletion of a common file, and when the aggregated file is deleted, all the fragment transcoding files in the aggregated file need to be traversed, and the attribute information and the data information of the fragment transcoding files are deleted in sequence.
Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Example 2: as shown in fig. 1, a method for rapidly splicing transcoded fragmented files includes:
step 11, writing a video file generated by a video source into a distributed storage system, wherein the distributed storage provides a uniform name space for the video source file, and a plurality of transcoding nodes can concurrently read data of the video source;
and step 12, distributing the segmented video source data segment indexes to the transcoding nodes through the transcoding scheduler, and reading the time slice data corresponding to the indexes from the storage system by the transcoding nodes.
Wherein step 12 comprises:
step 121, fragmenting a video source according to time average through a transcoding scheduler, and sequentially allocating index numbers to the fragmented transcoding files;
step 122, the transcoding scheduler sends segment indexes to be transcoded to the idle transcoding nodes;
step 13, the transcoding node reads the fragment interval data corresponding to the index in the video source file according to the received index, then transcodes the fragment data, writes the transcoded data into a distributed storage system to generate fragment transcoding files, and each fragment data corresponds to one fragment transcoding file after being transcoded until all fragment data are transcoded;
step 14, calling an interface provided by the storage system to create an aggregation file at a management node of the storage system, and sequentially aggregating the fragment transcoding files into the created aggregation file;
step 15, calling a storage interface to lock the sliced transcoding file, and if the locking of the sliced transcoding file fails, the aggregation operation fails and the aggregation is finished;
and step 16, checking whether the sliced transcoding file meets the requirement of the aggregation operation.
Wherein step 16 comprises:
step 161, checking whether the length of the locked segment transcoding file is zero, if so, failing to aggregate the file, and ending the aggregation;
step 162, judging whether the sliced transcoding file is a common file, if not, failing the aggregation operation and ending the aggregation;
step 163, whether the sliced transcoding file is being written, if so, the aggregation operation fails, and the aggregation is finished;
and step 17, adding one to the aggregation information index item of the aggregation file, and recording the length of the fragment transcoding file, the number of the fragment transcoding file and the offset of the fragment transcoding file in the aggregation file into the index information corresponding to the index item.
Step 18, deleting the name of the fragment transcoding file from the name space in distributed storage, and reserving the attribute information and the data block information of the fragment transcoding file;
step 19, the polymerization operation is successful, and the polymerization is finished;
step 110, if the locking of the fragment transcoding file fails, the aggregation operation fails, and the aggregation is finished; or the condition of the aggregation operation is not met, the fragment transcoding file is unlocked, the aggregation operation fails, and the aggregation is finished.
In the above embodiment, the described aggregation process is an aggregation operation of a single sliced transcoding file, and the aggregation process of multiple sliced transcoding files is a repetition of the above aggregation process. In the above embodiment, the fragment transcoding file is associated to the aggregation file by adding the aggregation attribute information of the aggregation file, and no copy of the data content of the fragment transcoding file is involved, so that a new idea is provided for generating the aggregation transcoding file, and the efficiency of generating the aggregation file by adding and modifying the aggregation index information is incomparable with that of the conventional transcoding mode.
Example 3: as shown in fig. 2, a method for reading data content of an aggregate file, where sliced transcoding files in the aggregate file are uniformly described as small files in a storage client system, includes:
and step 22, inquiring the aggregation length of the aggregation file recorded by the storage client, wherein the aggregation length is the sum of the lengths of all the small files.
Wherein step 22 comprises:
step 221, if the length of the aggregate file is smaller than the sum of the read offset and the read size of the file, sending a message to the metadata to obtain the aggregate information of the aggregate file, wherein the aggregate information comprises the lengths of all the small files, the numbers of the small files and the offsets of the small files in the aggregate file;
step 222, recording the aggregation information into a client aggregation file information cache, so as to facilitate subsequent reading operations to acquire the aggregation information from the cache;
step 23, if the read operation offset is greater than or equal to the aggregate file length, the read operation is finished, and the read content is returned to be null;
step 24, if the read offset plus the read length is greater than the file length, modifying the read length to be the aggregate file length minus the read offset;
and step 25, acquiring the index information of the small files in the read length interval, wherein the index sequence is that the small files are sequentially ordered from small to large in the offset of the aggregated file, and the index information can be acquired by sequentially traversing the index numbers of the small files.
Wherein step 25 comprises:
step 251, traversing the information of the aggregated small files from the first index in sequence according to the read starting position, recording the index corresponding to the read first small file as a starting index if the read starting position falls in a closed interval from the offset of the small file in the aggregated file to the length of the small file, and exiting the search of the starting index;
step 252, if there is no index information meeting the condition, the system alarms, and the read operation is finished;
step 253, traversing the information of the aggregated small file from the initial index backwards according to the position of the end of the reading operation, namely the offset of the read aggregated file plus the read length, recording the index corresponding to the read tail small file if the read starting position falls in the closed interval from the offset of the small file to the length of the small file in the aggregated file, and exiting the searching of the tail index;
step 254, if there is no index information meeting the condition, the system alarms, and the reading operation is finished;
255, the read size is smaller than the sum of the lengths of all the small files, the system gives an alarm, and the reading operation is finished;
and step 26, constructing the offset for reading the small file, the read length and the data block for storing the read content according to the interval size for reading the aggregation file and the index information of the small file.
Wherein step 26 comprises:
step 261, obtaining an offset of the read operation of the small file: subtracting the offset of the read operation from the offset of the small file in the aggregated file to obtain the offset of the read small file;
step 262, obtain the length of the read small file.
Step 2621, obtaining the readable length of the small file: the small file length minus the read offset of the small file,
step 2622, if the length of reading the aggregate file is less than or equal to the readable length of the small file, the read length of the small file is the length of reading the aggregate file, and the construction process is finished;
step 2623, if the length of reading the aggregate file is greater than the readable length of the small file, the read length of the small file is the readable length of the small file;
step 2624, modify the offset of reading the aggregate file to the offset of reading the aggregate file plus the readable length of the small file;
step 2625, modify the length of the read aggregate file to the length of the read aggregate file minus the readable long reads of the small file.
Step 263 continues to obtain the offset and length of the read of the subsequent doclet according to the flows 261, 262.
Step 27, sending a reading request to the small file concurrently according to the offset and the length of the constructed small file reading;
and step 28, waiting for the read requests of all the small files to return, and ending the process of reading the aggregated file.
In the foregoing embodiment, in the method for reading to obtain the content of the aggregate file, the operation of reading the aggregate file is divided into the operation of reading each small file, and then the operation is concurrently sent to each small file, so that the efficiency of reading the aggregate file is improved.
Example 4: as shown in fig. 3, the method for obtaining attribute information of an aggregate file includes:
step 32, sending an acquire attribute request to the metadata, acquiring the file attributes of the aggregate file,
step 33, an aggregation length exists in the structure information of the aggregation file, the aggregation length is the sum of the lengths of all the small files, and the aggregation length is compared with the file length recorded in the attribute of the aggregation file;
step 34, if the aggregation length is smaller than the file length recorded in the attribute, sending a request for reading the aggregation file index information to the metadata to obtain the aggregation file index information;
step 35, updating the index information to the small file cache of the aggregate file record.
Wherein step 35 comprises:
step 351, acquiring index information of the small files, and then inquiring the information of the small files in the small file cache through the numbers of the small files;
and step 352, if the query is not successful, generating inode structure information of the small files at the storage client, inserting the inode structure information into a small file cache of the aggregated file, and updating the aggregation length to be the current aggregation length plus the small file length. The inode structure information of the small files is associated with the aggregated files, so that the aggregated files can be read to traverse the information of the small files conveniently;
and step 353, if the query can be obtained, continuing to query the next small file, and repeating the step 352.
Step 37, judging whether the length of the aggregated file is equal to the length of the file attribute;
and step 38, if the lengths are not equal, the storage system alarms, and the sum of the length of the aggregation file and the length of the small file conflicts.
Example 5: as shown in fig. 4, a method for deleting an aggregation includes:
step 42, sending a request for deleting the file from the storage client to the metadata;
step 43, the metadata deletes the aggregate file.
Wherein step 43 comprises:
step 431, the metadata judges whether the deleted file is an aggregated file;
in step 432, if the aggregate file is found, the namespace and attribute information of the aggregate file is deleted.
Step 44, sending a deletion success response to the client, and returning a deletion result to the application by the client;
and step 45, simultaneously, the metadata starts to delete the attribute information and the data information of the small file.
Those of ordinary skill in the art will understand that: the figures are merely schematic representations of one embodiment, and the blocks or flow diagrams in the figures are not necessarily required to practice the present invention.
From the above description of the embodiments, it is clear to those skilled in the art that the present invention can be implemented by software plus necessary general hardware platform. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which may be stored in a storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for apparatus or system embodiments, since they are substantially similar to method embodiments, they are described in relative terms, as long as they are described in partial descriptions of method embodiments. The above-described embodiments of the apparatus and system are merely illustrative, and the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
As described above, although the embodiments of the present invention have been described in detail, it will be apparent to those skilled in the art that many modifications are possible without substantially departing from the spirit and scope of the present invention. Therefore, such modifications are also all included in the scope of protection of the present invention.
Claims (5)
1. A method for rapidly splicing transcoding fragmented files is characterized by comprising the following steps:
writing a video source file into distributed storage, providing a uniform name space for transcoding service by using a distributed storage system, reading the video source file from the distributed storage by using a plurality of transcoding servers in parallel, and acquiring data corresponding to a video source file time period according to a fragmented time period interval;
distributing index numbers of the fragment data segments to the transcoding nodes through a transcoding scheduler, reading time slice data corresponding to the index numbers from a distributed storage by the transcoding nodes, and transcoding the fragment data;
writing the transcoding data into distributed storage to generate a fragment transcoding file, wherein a uniform name space is provided for the fragment transcoding file by a distributed storage system;
the step of calling an interface provided by the distributed storage to generate the aggregation file at the storage management node comprises the following steps: calling a distributed storage interface to create an aggregate file, then pre-detecting the fragment transcoding file, and continuing an aggregate flow only after the detection is passed, wherein the pre-detecting step comprises the following steps:
judging whether the aggregate information index items in the aggregate file exceed the specification, if transcoding fails, the currently provided specification is that the upper limit of the fragment transcoding file supported in one aggregate file is 10000, and the upper limit of the specification can be adjusted according to requirements;
locking the fragment transcoding file, wherein the purpose of locking is to prevent other processes from modifying or aggregating the fragment transcoding file in the aggregation process, the simultaneous modification of the fragment transcoding file by multiple processes can cause the length of the fragment file recorded in the aggregated file to be inconsistent with the length recorded by the attribute of the fragment file, the simultaneous aggregation of the fragment transcoding files by multiple processes can cause the same fragment transcoding file to be aggregated into multiple aggregated files, the repeated locking of the fragment transcoding file can fail, and if the locking fails, the aggregation operation fails;
judging whether the fragment transcoding file is an empty file or not, if so, failing to perform aggregation operation, and unlocking the fragment transcoding file;
judging whether the fragment transcoding file is a common file or not, if not, failing the aggregation operation, and unlocking the fragment transcoding file;
judging whether data are written into the fragment transcoding file or not, and if data are written into the fragment transcoding file, the aggregation operation fails, and unlocking the fragment transcoding file;
when the fragment transcoding file meets the aggregation requirement, the unique number of the fragment transcoding file in a storage system, the length of the fragment transcoding file and the offset of the fragment transcoding file in the aggregation file are added to attribute information of the aggregation file;
deleting the name space of the fragment transcoding file in the storage system, so that the fragment transcoding file is invisible to the outside;
and reserving the attribute information and the data information of the sliced transcoding file.
2. The method of claim 1, wherein the distributed storage system includes a unified namespace for video source files, and the transcoding nodes can read video source files in parallel;
before a video source file is transcoded, the file length is fixed, and the recorded video file is recorded according to a constant code stream, so that the length interval of the file is a corresponding time interval;
the transcoding service needs to be configured on a node together with a storage access node, and the storage access node provides a uniform name space for the transcoding service;
the transcoding scheduler is responsible for fragmenting the video file and distributing interval indexes among the fragmented partitions; then sequentially distributing interval indexes to be transcoded to the transcoding service according to the idle state of the transcoding service until the load of the transcoding service is saturated;
and the transcoding service reads video source index interval data into the memory through a read operation interface provided by the distributed storage, then performs transcoding, writes the data in the memory into the distributed storage after transcoding is completed, generates a fragment transcoding file, and generates a file name according to the interval index number by naming the fragment transcoding file.
3. The method for rapidly splicing the transcoding fragmented files according to claim 1, comprising:
sequentially adding all file index information to the aggregation information of the aggregation file attributes according to the fragment index sequence;
the process of associating the aggregate file with the fragment transcoding file is not traditional fragment file data content copying, but is the addition of aggregate information and an aggregate index in the aggregate file, and the sequence of the index addition is consistent with the sequence of the fragments of the file;
the index information corresponding to each index item comprises: the unique number of the fragment files in the distributed storage system, the length of the fragment files and the offset of the fragment files in the aggregation file;
the aggregated file only provides read permission and does not have write permission;
and converting the read aggregated file data into data of the read fragmented file through an interface provided by the distributed storage system.
4. The method of claim 3, wherein the step of generating the aggregate file aggregate comprises:
acquiring an aggregation information index item in the attribute of the aggregation file, wherein index information corresponding to the index item comprises: the unique number of the fragment transcoding file in the distributed storage system, the length of the fragment transcoding file, the offset of the fragment transcoding file in the aggregated file and the initial index item are zero;
if the file is the first sliced transcoding file, adding one to the index item, and obtaining the index information corresponding to the first index item: the unique number of the fragment transcoding file, the length of the fragment transcoding file and the offset of the fragment transcoding file in the aggregation file are zero;
if the aggregated fragmented file is not the first fragmented transcoding file, adding one to the index item, and obtaining index information corresponding to the index item of the file: the unique number of the fragmented transcoding file, the length of the fragmented transcoding file and the offset of the fragmented transcoding file in the aggregate file are the sum of the length of the last transcoding fragmented file and the offset of the last transcoding fragmented file in the aggregate file;
deleting the name space of the fragment transcoding file in distributed storage, reserving the attribute information and the data information of the fragment transcoding file, and making the fragment transcoding file invisible to the outside in a distributed storage system;
unlocking the fragment transcoding file, and finishing the aggregation operation;
the aggregation operation process is the description of the aggregation process of a single fragmented transcoding file, and the aggregation process of a plurality of fragmented transcoding files is the repetition of the aggregation process.
5. The method of claim 1, wherein the method for rapidly splicing the transcoding fragmented files comprises the following steps:
calling an interface provided by distributed storage at a storage management node to generate an aggregated source file;
acquiring a fragment transcoding file from a distributed storage system;
calling a distributed storage system interface, and locking the fragment transcoding file;
when locking of the fragment transcoding file fails, the aggregation operation fails;
judging whether the fragment file meets a transcoding condition;
when the fragment transcoding file does not meet the condition, the aggregation operation fails;
when the fragment transcoding file meets the condition, adding one to the aggregation information index item of the aggregation file, wherein the index information corresponding to the newly added aggregation index item is the length of the fragment transcoding file, the unique serial number of the fragment transcoding file and the offset of the fragment transcoding file in the aggregation file;
deleting the name space of the fragment transcoding file, and reserving the attribute information and the data information of the fragment transcoding file;
and unlocking the fragment transcoding file, and finishing the file aggregation operation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710438505.XA CN107295425B (en) | 2017-06-12 | 2017-06-12 | Method for rapidly splicing transcoding fragmented files |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710438505.XA CN107295425B (en) | 2017-06-12 | 2017-06-12 | Method for rapidly splicing transcoding fragmented files |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107295425A CN107295425A (en) | 2017-10-24 |
| CN107295425B true CN107295425B (en) | 2020-02-21 |
Family
ID=60096431
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710438505.XA Expired - Fee Related CN107295425B (en) | 2017-06-12 | 2017-06-12 | Method for rapidly splicing transcoding fragmented files |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107295425B (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108710702A (en) * | 2018-05-25 | 2018-10-26 | 吴卓航 | A method of data safety storage is realized based on frament reassembling technology |
| CN109391787A (en) * | 2018-09-30 | 2019-02-26 | 武汉中科通达高新技术股份有限公司 | File format, image polymerization and read method |
| CN109413487A (en) * | 2018-11-07 | 2019-03-01 | 成都索贝数码科技股份有限公司 | A method of spelling is dodged after storing fragment transcoding/synthetic video file based on object |
| CN109495752A (en) * | 2018-11-07 | 2019-03-19 | 成都索贝数码科技股份有限公司 | A method of MXF file is combined into based on object storage fragment transcoding/synthesis sudden strain of a muscle |
| CN109195012A (en) * | 2018-11-07 | 2019-01-11 | 成都索贝数码科技股份有限公司 | A method of MP4 file is combined into based on object storage fragment transcoding/synthesis sudden strain of a muscle |
| CN109511008B (en) * | 2018-11-27 | 2021-07-13 | 成都索贝数码科技股份有限公司 | Method for supporting video and audio file content addition based on object storage |
| CN110113663A (en) * | 2019-05-10 | 2019-08-09 | 深圳市网心科技有限公司 | A kind of audio-video code-transferring method, system, storage medium and distributed apparatus |
| CN112637616B (en) * | 2020-12-08 | 2024-02-23 | 网宿科技股份有限公司 | Object storage method, system and server |
| CN116137670A (en) * | 2021-11-16 | 2023-05-19 | 中兴通讯股份有限公司 | Media file aggregation method, playing method, CDN system, terminal and storage medium |
| CN114125493B (en) * | 2021-11-22 | 2023-06-09 | 威创集团股份有限公司 | Distributed storage method, device and equipment for streaming media |
| CN115421649B (en) * | 2022-08-02 | 2023-10-20 | 佳源科技股份有限公司 | Indexable and extensible parameter file slicing storage system and method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104378665A (en) * | 2014-11-24 | 2015-02-25 | 深圳市天威视讯股份有限公司 | Distributed transcoding system and method based on digital television |
| CN104731921A (en) * | 2015-03-26 | 2015-06-24 | 江苏物联网研究发展中心 | Method for storing and processing small log type files in Hadoop distributed file system |
| WO2015118835A1 (en) * | 2014-02-10 | 2015-08-13 | 日本電気株式会社 | Video encoding apparatus, video encoding method, and video encoding program |
| CN105828105A (en) * | 2015-12-10 | 2016-08-03 | 广东亿迅科技有限公司 | Distributed environment-based video transcoding system and video transcoding method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101568063B1 (en) * | 2013-11-06 | 2015-11-10 | 건국대학교 산학협력단 | Method for transcoding mutimedia, and hadoop-based mulimedia transcoding system operating the same |
-
2017
- 2017-06-12 CN CN201710438505.XA patent/CN107295425B/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015118835A1 (en) * | 2014-02-10 | 2015-08-13 | 日本電気株式会社 | Video encoding apparatus, video encoding method, and video encoding program |
| CN104378665A (en) * | 2014-11-24 | 2015-02-25 | 深圳市天威视讯股份有限公司 | Distributed transcoding system and method based on digital television |
| CN104731921A (en) * | 2015-03-26 | 2015-06-24 | 江苏物联网研究发展中心 | Method for storing and processing small log type files in Hadoop distributed file system |
| CN105828105A (en) * | 2015-12-10 | 2016-08-03 | 广东亿迅科技有限公司 | Distributed environment-based video transcoding system and video transcoding method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107295425A (en) | 2017-10-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107295425B (en) | Method for rapidly splicing transcoding fragmented files | |
| CN109254733B (en) | Method, device and system for storing data | |
| CN106326292B (en) | Data structure and file aggregation and reading method and device | |
| JP2011520189A (en) | Document synchronization via stateless protocol | |
| US20180189315A1 (en) | Method and device for calling a distributed file system | |
| WO2017117919A1 (en) | Data storage method and device | |
| CN113672175A (en) | Distributed object storage method, device and equipment and computer storage medium | |
| EP2706468A1 (en) | Data storage and query method | |
| CN108614837B (en) | File storage and retrieval method and device | |
| US20120072400A1 (en) | Directory oplock | |
| CN113297166A (en) | Data processing system, method and device | |
| CN111414424B (en) | Method, system, medium and equipment for automatically synchronizing redis of configuration data | |
| CN111177159B (en) | Data processing system and method and data updating equipment | |
| CA3039537C (en) | Techniques for generating and operating on in-memory datasets | |
| CN104021137A (en) | Method and system for opening and closing file locally through client side based on catalogue authorization | |
| CN112115206A (en) | Method and device for processing object storage metadata | |
| CN110825715B (en) | Multi-object data second combination implementation method based on Ceph object storage | |
| JP2006146615A (en) | Object-related information management program, management method and management apparatus | |
| Lu et al. | Research on Cassandra data compaction strategies for time-series data | |
| CN116820323A (en) | Data storage method, device, electronic equipment and computer readable storage medium | |
| CN108241640B (en) | Distributed file storage method | |
| CN114297196A (en) | Metadata storage method and device, electronic equipment and storage medium | |
| CN102076116B (en) | Data comparing method and device | |
| CN111796767A (en) | Distributed file system and data management method | |
| CN108989245B (en) | User data storage method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20200221 Termination date: 20200612 |