CN107295425A - A kind of method of quick splicing transcoding slicing files - Google Patents
A kind of method of quick splicing transcoding slicing files Download PDFInfo
- Publication number
- CN107295425A CN107295425A CN201710438505.XA CN201710438505A CN107295425A CN 107295425 A CN107295425 A CN 107295425A CN 201710438505 A CN201710438505 A CN 201710438505A CN 107295425 A CN107295425 A CN 107295425A
- Authority
- CN
- China
- Prior art keywords
- file
- transcoding
- burst
- aggregate
- burst transcoding
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/845—Structuring of content, e.g. decomposing content into time segments
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234309—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 or from Quicktime to Realvideo
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/845—Structuring of content, e.g. decomposing content into time segments
- H04N21/8455—Structuring of content, e.g. decomposing content into time segments involving pointers to the content, e.g. pointers to the I-frames of the video stream
Abstract
A kind of method of quick splicing transcoding slicing files, belongs to field of computer technology.Video source file is written to distributed storage, distributed memory system provides unified name space for transcoding service, the parallel data that video source file is read from distributed storage, corresponding video source document time section is obtained according to the period interval of burst of multiple transcoding servers；By the index number that transcoding scheduler is transcoding node distribution fragment data section, transcoding node reads the time sheet data of manipulative indexing numbering from distributed storage, fragment data is carried out into transcoding.The present invention is by by the length of burst transcoding file, reference number of a document, the offset information in aggregate file, record is in the index entry of the aggregation information attribute of aggregate file, the mode of traditional additional write-in has been abandoned completely, time-consuming data read-write operation is changed into the increase of attribute information, the efficiency of burst transcoding Piece file mergence is improved, the network bandwidth resources of preciousness are saved.
Description
Technical field
The present invention relates to a kind of method of quick splicing transcoding slicing files, belong to field of computer technology.
Background technology
Requirement more and more higher with traditional radio, TV and film industries to video definition, from earliest SD epoch 25Mbps code
Flow to the 100Mbps in current high definition epoch code stream, then to following 4K epoch, the program of a hour in following 4K epoch
Material file length is up to 180GB sizes.Traditional Transcoding Scheme is finally to turn burst transcoding file successively additional write
In code file, scheme is write in addition has read operation to each burst transcoding file, while having write operation to final transcoding file, chases after
Plus write scheme and both expend Internet resources, while also consuming transcoding node system resource；Traditional Transcoding Scheme can only accomplish that 1.5 arrive
2 speeds, far can not meet by video data expands brought transcoding demand.
As video source is more and more clear, TV programme species is more and more so that radio, TV and film industries resource system data volume
PB ranks are reached, cause storage demand amount to explode；Television station business system species is various simultaneously, demand of the operation system to storage
Differ, it has to using many set storage systems, but easily cause information island between set storage system more, while building and tieing up
Protect cost high, and unfavorable unified management.
The content of the invention
In order to overcome the deficiencies in the prior art, the present invention provides a kind of method of quick splicing transcoding slicing files.
A kind of method of quick splicing transcoding slicing files, contains following steps；
Video source file is written to distributed storage, it is empty that distributed memory system provides unified name for transcoding service
Between, the parallel reading video source file from distributed storage of multiple transcoding servers is obtained according to the period interval of burst
The data of correspondence video source document time section；
By the index number that transcoding scheduler is transcoding node distribution fragment data section, transcoding node is from distributed storage
The upper time sheet data for reading manipulative indexing numbering, transcoding is carried out by fragment data；
Transcoded data is write into distributed storage, burst transcoding file is generated, distributed memory system is burst transcoding text
Part provides unified name space；
The interface for calling distributed storage to provide in storage management node generates aggregate file；
Required when burst transcoding file meets polymerization, unique number, burst turn by burst transcoding file in storage system
Code file size, burst transcoding file are appended in the attribute information of aggregate file in the skew of aggregate file；
Delete the name space of burst transcoding file within the storage system so that burst transcoding file is externally invisible；
Retain the attribute information and data message of burst transcoding file.
Distributed memory system provides unified name space for video source file, and transcoding node can read video source parallel
File；
Video source file is before by transcoding, and file size has been fixed, and the video file recorded is according to constant
Stream-recording, so the length of interval of file is also corresponding time interval；
Transcoding service needs and storage access node are configured on a node, and storage access node provides for transcoding service
Unified name space；
Transcoding scheduler is responsible for video file burst, to the interval index of burst interval distribution；Then according to transcoding service
Idle condition, interval index to be transcoded is distributed to transcoding service successively, until transcoding service load saturation；
The read operation interface that transcoding service is provided by distributed storage, reads video source and indexes interval censored data to internal memory
In, transcoding is then carried out, the data in internal memory are written in distributed storage by transcoding after completing, generation burst transcoding text
Part, burst transcoding file designation generates file name according to interval index number；
Distributed memory system provides unified name space to access burst transcoding file.
In the aggregation information that All Files index information is appended to aggregate file attribute successively according to burst indexed sequential；
The process of aggregate file association burst transcoding file is not traditional slicing files data content copy, but polymerization
The addition of aggregation information and clustering index in file, indexes additional order and the burst order of file is consistent；
Each corresponding index information of index entry includes：Unique number of the slicing files in distributed memory system, burst
The length of file, slicing files are offset in aggregate file；
Aggregate file only provides read right, without write permission；
The interface conversion that reading aggregate file data are provided by distributed memory system is to read the data of slicing files.
The step of interface for calling distributed storage to provide in storage management node generates aggregate file includes：Call distribution
Formula memory interface creates aggregate file, and pre-detection is then carried out to burst transcoding file, and detection, which passes through, just may proceed to polymerization process,
Pre-detection step includes：
Judge whether the aggregation information index entry in aggregate file exceedes specification, if it exceeds specification transcoding fails, at present
The specification of offer is that the burst transcoding file upper limit supported in an aggregate file is 10000, and upper specification limit can be according to need
Ask adjustment；
To burst transcoding file locking, the purpose of locking be prevent polymerization during other processes to burst transcoding file
Operation or converging operation, multi-process modify while changing the burst that burst transcoding file can cause to record in aggregate file
The length of file and the length of slicing files attribute record are inconsistent, multi-process simultaneously polymerize burst transcoding file can cause it is same
Individual burst transcoding file is aggregated in multiple aggregate files, and repeating locking burst transcoding file can fail, if locking failure,
Converging operation fails；
Whether be empty file, if empty file if judging burst transcoding file, converging operation failure, burst transcoding file solution
Lock；
Whether be ordinary file, if not ordinary file, converging operation failure, burst transcoding if judging burst transcoding file
File is unlocked；
Judge whether burst transcoding file there are data to be currently written into, if data are currently written into burst transcoding file, gather
Closing operation fails, the unblock of burst transcoding file.
The step of polymerization burst transcoding file, includes：
The aggregation information index entry in aggregate file attribute is obtained, the corresponding index information of index entry has：Burst transcoding text
Unique number of the part in distributed memory system, burst transcoding file size, skew of the burst transcoding file in aggregate file,
Initial index is zero；
If first burst transcoding file, index entry adds one, the corresponding index information of first index entry：Burst turns
The unique number of code file, the length of burst transcoding file, burst transcoding file are zero in the skew of aggregate file；
If the slicing files being polymerize are not first burst transcoding files, index entry adds one, this document index entry pair
The index information answered：The unique number of burst transcoding file, the length of burst transcoding file, burst transcoding file are in aggregate file
The sum that is offset in aggregate file for a upper transcoding slicing files length and a upper transcoding slicing files of skew；
Burst transcoding file name space in distributed storage is deleted, retains burst transcoding file attribute information and data
Information, in distributed memory system, makes burst transcoding file externally invisible；
Burst transcoding file is unlocked, and converging operation is completed.
Above converging operation flow is all the explanation of the polymerization process of single burst transcoding file, multiple burst transcoding files
Polymerization process be above polymerization process repetition.
It is an advantage of the invention that by by the length of burst transcoding file, reference number of a document, the offset information in aggregate file,
Record has abandoned the mode of traditional additional write-in completely in the index entry of the aggregation information attribute of aggregate file, will be time-consuming
Data read-write operation be changed into the increase of attribute information, improve the efficiency of burst transcoding Piece file mergence, save the net of preciousness
Network bandwidth resources.
The present invention can also apply radio, TV and film industries packing application in, packing application in Storyboard can with analogy into turn
Video source in code application, the burst of Storyboard is to carry out burst according to specified cut-off, and node of then packing is to burst
Storyboard changes into burst video by editor and is written to distributed storage, is gathered burst video by storing the aggregation interface provided
Very fast packing can be realized altogether, and the packing manner is not related to the read-write copy of video data, the packing efficiency of very fast packing
It is that traditional packing manner is incomparable.
The present invention can provide ability extending transversely for video resource memory space, can meet video resource data volume fast
The demand that speed increases；The present invention can provide unified name space, so as to get through the information island between each video system, just
In resource management, storage system maintenance cost is reduced；The invention provides the function of quick splicing burst transcoding file, abandon
The mode that traditional addition is write, but the mode for using very fast aggregation information index entry to add, not only save the network of preciousness
Bandwidth resources, and splicing burst transcoding file detection is also improved, transcoding efficiency can be brought up to 20 to 30 times by the present invention
Speed.
Brief description of the drawings
When considered in conjunction with the accompanying drawings, by referring to following detailed description, can more completely more fully understand the present invention with
And the adjoint advantage of many of which is easily learnt, but accompanying drawing described herein is used for providing a further understanding of the present invention,
The part of the present invention is constituted, schematic description and description of the invention is used to explain the present invention, do not constituted to this hair
Bright improper restriction, such as figure are wherein：
Fig. 1 is a kind of process chart of quick splicing burst transcoding file of the present invention；
Fig. 2 is a kind of flow chart of reading aggregate file content of the present invention；
Fig. 3 is a kind of flow chart of acquisition aggregate file attribute of the present invention；
Fig. 4 is a kind of schematic diagram of deletion aggregate file of the present invention；
The present invention is further described with reference to the accompanying drawings and examples.
Embodiment
Obviously, those skilled in the art belong to the guarantor of the present invention based on many modifications and variations that spirit of the invention is done
Protect scope.
Those skilled in the art of the present technique are appreciated that unless expressly stated, singulative " one " used herein, " one
It is individual ", " described " and "the" may also comprise plural form.It is to be further understood that wording " the bag used in this specification
Include " refer to there is the feature, integer, step, operation, element and/or component, but it is not excluded that in the presence of or addition one or
Other multiple features, integer, step, operation, element, component and/or their group.It should be understood that when title element, component quilt
When ' attach ' to another element, component, it, which can be directly connected to other elements, either component or can also have cental element
Part or component.Wording "and/or" used herein includes one or more associated any cells for listing item and complete
Combine in portion.
Those skilled in the art of the present technique are appreciated that unless otherwise defined, all terms used herein (including technology art
Language and scientific terminology) with the general understanding identical meaning with the those of ordinary skill in art of the present invention.
For ease of the understanding to the embodiment of the present invention, explanation will be further explained below, and each embodiment is not
Constitute the restriction to the embodiment of the present invention.
Embodiment 1：As shown in figure 1, a kind of method of quick splicing transcoding slicing files, including：
The establishment aggregate file interface of distributed memory system is called, aggregate file is created；
Burst transcoding file is obtained from distributed memory system；
Distributed memory system interface is called, to burst transcoding file locking；
When the failure of burst transcoding file locking, converging operation failure；
Judge that burst turns whether file meets transcoding condition；
When burst transcoding file is unsatisfactory for condition, converging operation failure；
When burst transcoding file meets condition, the aggregation information index entry of aggregate file is added one, newly-increased clustering index
The corresponding index information of item is burst transcoding file size, burst transcoding file unique number, burst transcoding file are in polymerization text
The skew of part；
The name space of burst transcoding file is deleted, retains the attribute information and data message of burst transcoding file；
Burst transcoding file is unlocked, file converging operation terminates.
The deletion of aggregate file is different from the deletion of ordinary file, it is necessary to travel through in aggregate file during deletion aggregate file
All burst transcoding files, delete the attribute information and data message of burst transcoding file successively.
The additional aspect of the present invention and advantage will be set forth in part in the description, and these will become from the following description
Obtain clearly, or recognized by the practice of the present invention.
Embodiment 2：As shown in figure 1, a kind of method of quick splicing transcoding slicing files, including：
Step 11, the video file that video source is generated is written in distributed memory system, distributed storage is video
Source file provides unified name space, and multiple transcoding nodes can concurrently read the data of video source；
Step 12, it is transcoding node distribution burst video source data segment index by transcoding scheduler, transcoding node is from depositing
The time sheet data of manipulative indexing is read on storage system.
Wherein, step 12 includes：
Step 121, by transcoding scheduler by video source according to time average burst, and to burst transcoding document order
Distribution index is numbered；
Step 122, transcoding scheduler is sent to idle transcoding node needs to be indexed by the burst of transcoding；
Step 13, transcoding node reads index in the corresponding burst interval censored data of video source file according to the index received,
Then transcoding is carried out to fragment data, the data after transcoding is written to distributed memory system generation burst transcoding file,
One burst transcoding file of correspondence after each fragment data transcoding, until all fragment data transcodings are completed；
Step 14, in the management node of storage system, the interface for calling storage system to provide creates aggregate file, successively will
Burst transcoding file is aggregated in the aggregate file of establishment；
Step 15, memory interface is called by burst transcoding file locking, if burst transcoding file locking fails, polymerization behaviour
Work fails, and polymerization terminates；
Step 16, check whether burst transcoding file meets converging operation requirement.
Wherein, step 16 includes：
Step 161, check whether the burst transcoding file size being locked is zero, if zero, converging operation failure gathers
Conjunction terminates；
Step 162, whether burst transcoding file is ordinary file, if not ordinary file, converging operation failure, polymerization
Terminate；
Step 163, whether burst transcoding file is written into, if be written into, converging operation failure, polymerization knot
Beam；
Step 17, the aggregation information index entry of aggregate file adds one, by the length of burst transcoding file, burst transcoding file
Numbering, skew of the burst transcoding file in aggregate file recorded in index item manipulative indexing information.
Step 18, by the title of burst transcoding file, deleted from the name space of distributed storage, retain burst transcoding
The attribute information and data block information of file；
Step 19, converging operation success, polymerization terminates；
Step 110, if burst transcoding file locking fails, converging operation failure, polymerization terminates；Or it is unsatisfactory for polymerization
Operating condition, burst transcoding file unblock, converging operation failure, polymerization terminates.
In above-described embodiment, the polymerization process of description is the converging operation of single burst transcoding file, multiple burst transcodings
The polymerization process of file is the repetition of above-mentioned polymerization process.In above-described embodiment, pass through the Aggregate attribute information to aggregate file
Increase, burst transcoding file association is not involved with copying for burst transcoding file data content into aggregate file, completely
Shellfish, new thinking is provided for generation polymerization transcoding file, the increase and modification generation aggregate file to clustering index information
Efficiency is that traditional transcoding mode is incomparable.
Embodiment 3：As shown in Fig. 2 reading the method for aggregate file data content, the burst transcoding wherein in aggregate file
File is uniformly described as small documents in storage FTP client FTP, and reading flow includes：
Step 22, the polymerization length of the aggregate file of inquiry storage client records, polymerization length is that all small documents are long
The sum of degree.
Wherein, step 22 includes：
Step 221, if the length of aggregate file is less than the reading skew of file and reads size sum, first number is sent a message to
According to the aggregation information for obtaining aggregate file, aggregation information includes the length, the numbering of small documents, small documents of all small documents poly-
Close the skew of file；
Step 222, aggregation information recorded in client conglomerate fileinfo caching, is easy to follow-up read operation to postpone
Deposit middle acquisition aggregation information；
Step 23, if read operation skew is more than or equal to aggregate file length, read operation terminates, and returns to reading of content
For sky；
Step 24, if the skew read adds reading length to be more than file size, it is that aggregate file is long that length is read in modification
Degree subtracts the skew of reading；
Step 25, the small documents index information read in length of interval is obtained, because the order of index is that small documents are gathering
Close document misregistration to sort successively from small to large, index information can be obtained with the index number of order traversal small documents.
Wherein, step 25 includes：
Step 251, according to the starting position of reading, traversal polymerize the information of small documents successively being indexed since first,
If the starting position read falls being displaced in the closed interval of small documents length in aggregate file in small documents, what record was read
The corresponding index of first small documents exits the lookup for starting index to start index；
Step 252, if not meeting the index information of condition, ALM, read operation terminates；
Step 253, the position terminated according to read operation, that is, read length of the skew plus reading of aggregate file, from
Most start the information that index starts traversal polymerization small documents backward, if the starting position read falls in small documents in aggregate file
Be displaced in the closed interval of small documents length, record read the corresponding index of end small documents, exit end index look into
Look for；
Step 254, if not meeting the index information of condition, ALM, read operation terminates；
Step 255, the size of reading is smaller than the length sum of all small documents, and ALM, read operation terminates；
Step 26, according to the interval size and the index information of small documents for reading aggregate file, construction reads small documents
Skew, the length read, the data block of storage reading content.
Wherein, step 26 includes：
Step 261, the skew of the read operation of small documents is obtained：Small documents subtract the inclined of read operation in the skew of aggregate file
Move to read the skew of small documents；
Step 262, the length for reading small documents is obtained.
Step 2621, the readable length of small documents is obtained：Small documents length subtracts the reading skew of small documents,
Step 2622, if the length for reading aggregate file is less than or equal to the readable length of small documents, small documents
Length is read to read the length of aggregate file, construction flow terminates；
Step 2623, if the length for reading aggregate file is more than the readable length of small documents, the reading length of small documents is small
The readable length of file；
Step 2624, modification reads the skew of aggregate file to read readable length of the skew plus small documents of aggregate file；
Step 2625, the length of modification reading aggregate file is read for the readable length that the length of reading aggregate file subtracts small documents.
Step 263, continuation obtains skew and the length of the reading of follow-up small documents according to flow 261,262.
Step 27, the skew read according to the small documents constructed and length, concurrent sends read request to small documents；
Step 28, wait the read request of whole small documents to return, read aggregate file flow and terminate.
In above-described embodiment, the method to obtain the reading that the content of aggregate file is provided will read the operation point of aggregate file
The operation for reading each small documents is split into, it is then concurrent to be sent to each small documents, so as to improve the efficiency for reading aggregate file.
Embodiment 4：As shown in figure 3, the method for aggregate file attribute information is obtained, including：
Step 32, send and obtain attribute request to metadata, obtain the file attribute of aggregate file,
Step 33, there is a polymerization length in the structural information of aggregate file, polymerization length is the length of all small documents
Degree and, compare the file size that records in the attribute of polymerization length and aggregate file,；
Step 34, if polymerization length is smaller than the file size recorded in attribute, send and read aggregate file index information
Metadata is asked, aggregate file index information is obtained；
Step 35, index information is updated in the small documents caching noted down to aggregate file.
Wherein, step 35 includes：
Step 351, the index information of small documents is obtained, then inquiry should in small documents caching by the numbering of small documents
The information of small documents；
Step 352, if inquiry less than, storage client generate small documents inode structure information, be inserted into polymerization
In the small documents caching of file, while it is when pre-polymerization length adds small documents length to update polymerization length.By small documents
Inode structure information and aggregate file are associated, and are easy to aggregate file read operation to go to travel through small documents information；
Step 353, if it is possible to inquire, continues to inquire about next small documents, repeat step 352.
Step 37, judge whether the length of aggregate file is equal to file attribute length；
Step 38, if unequal, there is conflict in storage system alarm, aggregate file length and small documents length sum.
Embodiment 5：As shown in figure 4, the method for polymerization is deleted, including：
Step 42, from the request of storage client transmission deletion file to metadata；
Step 43, metadata deletes aggregate file.
Wherein, step 43 includes：
Step 431, metadata judges whether deleted file is aggregate file；
Step 432, if aggregate file, the name space and attribute information of aggregate file are deleted.
Step 44, send to delete to client and successfully respond, client returns to application by result is deleted；
Step 45, while metadata starts concurrently to delete the attribute information and data message of small documents.
One of ordinary skill in the art will appreciate that：Accompanying drawing be module in the schematic diagram of one embodiment, accompanying drawing or
Flow is not necessarily implemented necessary to the present invention.
As seen through the above description of the embodiments, those skilled in the art can be understood that the present invention can
Realized by the mode of software plus required general hardware platform.Understood based on such, technical scheme essence
On the part that is contributed in other words to prior art can be embodied in the form of software product, the computer software product
It can be stored in storage medium, such as ROM/RAM, magnetic disc, CD, including some instructions are to cause a computer equipment
(can be personal computer, server, or network equipment etc.) performs some of each of the invention embodiment or embodiment
Method described in part.
Each embodiment in this specification is described by the way of progressive, identical similar portion between each embodiment
Divide mutually referring to what each embodiment was stressed is the difference with other embodiment.Especially for device or
For system embodiment, because it is substantially similar to embodiment of the method, so describing fairly simple, related part is referring to method
The part explanation of embodiment.Apparatus and system embodiment described above is only schematical, wherein the conduct
The unit that separating component illustrates can be or may not be it is physically separate, the part shown as unit can be or
Person may not be physical location, you can with positioned at a place, or can also be distributed on multiple NEs.Can root
Some or all of module therein is factually selected to realize the purpose of this embodiment scheme the need for border.Ordinary skill
Personnel are without creative efforts, you can to understand and implement.
As described above, being explained to embodiments of the invention, as long as but essentially without this hair of disengaging
Bright inventive point and effect can have many deformations, and this will be readily apparent to persons skilled in the art.Therefore, this
The variation of sample is also integrally incorporated within protection scope of the present invention.
Claims (6)
1. a kind of method of quick splicing transcoding slicing files, it is characterised in that including containing following steps：
Video source file is written to distributed storage, distributed memory system provides unified name space for transcoding service, many
The parallel reading video source file from distributed storage of individual transcoding server, obtains correspondence according to the period interval of burst and regards
The data of frequency source file period；
By the index number that transcoding scheduler is transcoding node distribution fragment data section, transcoding node is read from distributed storage
The time sheet data for taking manipulative indexing to number, transcoding is carried out by fragment data；
Transcoded data is write into distributed storage, burst transcoding file is generated, distributed memory system is that burst transcoding file is carried
For unified name space；
The interface for calling distributed storage to provide in storage management node generates aggregate file；
Required when burst transcoding file meets polymerization, unique number, burst transcoding text by burst transcoding file in storage system
Part length, burst transcoding file are appended in the attribute information of aggregate file in the skew of aggregate file；
Delete the name space of burst transcoding file within the storage system so that burst transcoding file is externally invisible；
Retain the attribute information and data message of burst transcoding file.
2. a kind of method of quick splicing burst transcoding file according to claim 1, it is characterised in that distributed storage
System includes providing unified name space for video source file, and transcoding node can read video source file parallel；
Video source file is before by transcoding, and file size has been fixed, and the video file recorded is according to constant code stream
Record, so the length of interval of file is also corresponding time interval；
Transcoding service needs and storage access node are configured on a node, and storage access node provides unified for transcoding service
Name space；
Transcoding scheduler is responsible for video file burst, to the interval index of burst interval distribution；Then the sky serviced according to transcoding
Not busy state, distributes interval index to be transcoded, until transcoding service load saturation to transcoding service successively；
The read operation interface that transcoding service is provided by distributed storage, reads video source and indexes interval censored data into internal memory, so
After carry out transcoding, the data in internal memory are written in distributed storage by transcoding after completing, and generate burst transcoding file, burst
Transcoding file designation generates file name according to interval index number.
3. a kind of method of quick splicing burst transcoding file according to claim 1, it is characterised in that including：
In the aggregation information that All Files index information is appended to aggregate file attribute successively according to burst indexed sequential；
The process of aggregate file association burst transcoding file is not traditional slicing files data content copy, but aggregate file
The addition of middle aggregation information and clustering index, indexes additional order and the burst order of file is consistent；
Each corresponding index information of index entry includes：Unique number of the slicing files in distributed memory system, slicing files
Length, slicing files offset in aggregate file；
Aggregate file only provides read right, without write permission；
The interface conversion that reading aggregate file data are provided by distributed memory system is to read the data of slicing files.
4. the method for a kind of quick splicing burst transcoding file according to claim 3, it is characterised in that in storage management
The step of interface that node calls distributed storage to provide generates aggregate file includes：Distributed storage interface is called to create polymerization
File, then carries out pre-detection, detection is by just may proceed to polymerization process, and pre-detection step includes to burst transcoding file：
Judge whether the aggregation information index entry in aggregate file exceedes specification, if it exceeds specification transcoding fails, provide at present
Specification be that the burst transcoding file upper limit supported in an aggregate file is 10000, upper specification limit can be adjusted according to demand
It is whole；
To burst transcoding file locking, the purpose of locking is that other processes are carried out to burst transcoding file during preventing polymerization
The slicing files that burst transcoding file can cause to record in aggregate file are changed in modification operation or converging operation, multi-process simultaneously
Length and the length of slicing files attribute record it is inconsistent, multi-process simultaneously polymerize burst transcoding file can cause same point
Piece transcoding file is aggregated in multiple aggregate files, and repeating locking burst transcoding file can fail, if locking failure, polymerization
Operation failure；
Whether be empty file, if empty file if judging burst transcoding file, converging operation failure, the unblock of burst transcoding file；
Whether be ordinary file, if not ordinary file, converging operation failure, burst transcoding file if judging burst transcoding file
Unblock；
Judge whether burst transcoding file there are data to be currently written into, if data are currently written into burst transcoding file, polymerization behaviour
Work fails, the unblock of burst transcoding file.
5. a kind of method of quick splicing burst transcoding file according to claim 3, it is characterised in that generation polymerization text
The step of part polymerize includes：
The aggregation information index entry in aggregate file attribute is obtained, the corresponding index information of index entry has：Burst transcoding file exists
The unique number of distributed memory system, burst transcoding file size, skew of the burst transcoding file in aggregate file, initially
Index entry is zero；
If first burst transcoding file, index entry adds one, the corresponding index information of first index entry：Burst transcoding text
The unique number of part, the length of burst transcoding file, burst transcoding file are zero in the skew of aggregate file；
If the slicing files being polymerize are not first burst transcoding files, index entry adds one, and this document index entry is corresponding
Index information：The unique number of burst transcoding file, the length of burst transcoding file, burst transcoding file are in the inclined of aggregate file
Move the sum offset for a upper transcoding slicing files length and a upper transcoding slicing files in aggregate file；
Delete burst transcoding file name space in distributed storage, retain burst transcoding file attribute information sum it is believed that
Breath, in distributed memory system, makes burst transcoding file externally invisible；
Burst transcoding file is unlocked, and converging operation is completed；
Above converging operation flow is all the explanation of the polymerization process of single burst transcoding file, multiple burst transcoding files it is poly-
Interflow journey is the repetition of above polymerization process.
6. a kind of method of quick splicing burst transcoding file according to claim 1, its step includes：
It polymerize source file in the interface generation for calling distributed storage to provide in storage management node；
Burst transcoding file is obtained from distributed memory system；
Distributed memory system interface is called, to burst transcoding file locking；
When the failure of burst transcoding file locking, converging operation failure；
Judge that burst turns whether file meets transcoding condition；
When burst transcoding file is unsatisfactory for condition, converging operation failure；
When burst transcoding file meets condition, the aggregation information index entry of aggregate file is added one, newly-increased clustering index is right
The index information answered is burst transcoding file size, burst transcoding file unique number, burst transcoding file in aggregate file
Skew；
The name space of burst transcoding file is deleted, retains the attribute information and data message of burst transcoding file；
Burst transcoding file is unlocked, file converging operation terminates.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710438505.XA CN107295425B (en) | 2017-06-12 | 2017-06-12 | Method for rapidly splicing transcoding fragmented files |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710438505.XA CN107295425B (en) | 2017-06-12 | 2017-06-12 | Method for rapidly splicing transcoding fragmented files |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107295425A true CN107295425A (en) | 2017-10-24 |
| CN107295425B CN107295425B (en) | 2020-02-21 |
Family
ID=60096431
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710438505.XA Expired - Fee Related CN107295425B (en) | 2017-06-12 | 2017-06-12 | Method for rapidly splicing transcoding fragmented files |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107295425B (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108710702A (en) * | 2018-05-25 | 2018-10-26 | 吴卓航 | A method of data safety storage is realized based on frament reassembling technology |
| CN109195012A (en) * | 2018-11-07 | 2019-01-11 | 成都索贝数码科技股份有限公司 | A method of MP4 file is combined into based on object storage fragment transcoding/synthesis sudden strain of a muscle |
| CN109391787A (en) * | 2018-09-30 | 2019-02-26 | 武汉中科通达高新技术股份有限公司 | File format, image polymerization and read method |
| CN109413487A (en) * | 2018-11-07 | 2019-03-01 | 成都索贝数码科技股份有限公司 | A method of spelling is dodged after storing fragment transcoding/synthetic video file based on object |
| CN109495752A (en) * | 2018-11-07 | 2019-03-19 | 成都索贝数码科技股份有限公司 | A method of MXF file is combined into based on object storage fragment transcoding/synthesis sudden strain of a muscle |
| CN109511008A (en) * | 2018-11-27 | 2019-03-22 | 成都索贝数码科技股份有限公司 | A method of it is stored based on object and supports that video and audio file content is additional |
| CN110113663A (en) * | 2019-05-10 | 2019-08-09 | 深圳市网心科技有限公司 | A kind of audio-video code-transferring method, system, storage medium and distributed apparatus |
| CN112637616A (en) * | 2020-12-08 | 2021-04-09 | 网宿科技股份有限公司 | Object storage method, system and server |
| CN114125493A (en) * | 2021-11-22 | 2022-03-01 | 威创集团股份有限公司 | Distributed storage method, device and equipment for streaming media |
| CN115421649A (en) * | 2022-08-02 | 2022-12-02 | 佳源科技股份有限公司 | Indexable and extensible parameter file fragment storage system and method |
| WO2023087816A1 (en) * | 2021-11-16 | 2023-05-25 | 中兴通讯股份有限公司 | Media file aggregation method, playback method, cdn system, terminal and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104378665A (en) * | 2014-11-24 | 2015-02-25 | 深圳市天威视讯股份有限公司 | Distributed transcoding system and method based on digital television |
| US20150125133A1 (en) * | 2013-11-06 | 2015-05-07 | Konkuk University Industrial Cooperation Corp. | Method for transcoding multimedia, and hadoop-based multimedia transcoding system for performing the method |
| CN104731921A (en) * | 2015-03-26 | 2015-06-24 | 江苏物联网研究发展中心 | Method for storing and processing small log type files in Hadoop distributed file system |
| WO2015118835A1 (en) * | 2014-02-10 | 2015-08-13 | 日本電気株式会社 | Video encoding apparatus, video encoding method, and video encoding program |
| CN105828105A (en) * | 2015-12-10 | 2016-08-03 | 广东亿迅科技有限公司 | Distributed environment-based video transcoding system and video transcoding method |
-
2017
- 2017-06-12 CN CN201710438505.XA patent/CN107295425B/en not_active Expired - Fee Related
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150125133A1 (en) * | 2013-11-06 | 2015-05-07 | Konkuk University Industrial Cooperation Corp. | Method for transcoding multimedia, and hadoop-based multimedia transcoding system for performing the method |
| WO2015118835A1 (en) * | 2014-02-10 | 2015-08-13 | 日本電気株式会社 | Video encoding apparatus, video encoding method, and video encoding program |
| CN104378665A (en) * | 2014-11-24 | 2015-02-25 | 深圳市天威视讯股份有限公司 | Distributed transcoding system and method based on digital television |
| CN104731921A (en) * | 2015-03-26 | 2015-06-24 | 江苏物联网研究发展中心 | Method for storing and processing small log type files in Hadoop distributed file system |
| CN105828105A (en) * | 2015-12-10 | 2016-08-03 | 广东亿迅科技有限公司 | Distributed environment-based video transcoding system and video transcoding method |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108710702A (en) * | 2018-05-25 | 2018-10-26 | 吴卓航 | A method of data safety storage is realized based on frament reassembling technology |
| CN109391787A (en) * | 2018-09-30 | 2019-02-26 | 武汉中科通达高新技术股份有限公司 | File format, image polymerization and read method |
| CN109195012A (en) * | 2018-11-07 | 2019-01-11 | 成都索贝数码科技股份有限公司 | A method of MP4 file is combined into based on object storage fragment transcoding/synthesis sudden strain of a muscle |
| CN109413487A (en) * | 2018-11-07 | 2019-03-01 | 成都索贝数码科技股份有限公司 | A method of spelling is dodged after storing fragment transcoding/synthetic video file based on object |
| CN109495752A (en) * | 2018-11-07 | 2019-03-19 | 成都索贝数码科技股份有限公司 | A method of MXF file is combined into based on object storage fragment transcoding/synthesis sudden strain of a muscle |
| CN109511008B (en) * | 2018-11-27 | 2021-07-13 | 成都索贝数码科技股份有限公司 | Method for supporting video and audio file content addition based on object storage |
| CN109511008A (en) * | 2018-11-27 | 2019-03-22 | 成都索贝数码科技股份有限公司 | A method of it is stored based on object and supports that video and audio file content is additional |
| CN110113663A (en) * | 2019-05-10 | 2019-08-09 | 深圳市网心科技有限公司 | A kind of audio-video code-transferring method, system, storage medium and distributed apparatus |
| CN112637616A (en) * | 2020-12-08 | 2021-04-09 | 网宿科技股份有限公司 | Object storage method, system and server |
| CN112637616B (en) * | 2020-12-08 | 2024-02-23 | 网宿科技股份有限公司 | Object storage method, system and server |
| WO2023087816A1 (en) * | 2021-11-16 | 2023-05-25 | 中兴通讯股份有限公司 | Media file aggregation method, playback method, cdn system, terminal and storage medium |
| CN114125493A (en) * | 2021-11-22 | 2022-03-01 | 威创集团股份有限公司 | Distributed storage method, device and equipment for streaming media |
| CN115421649A (en) * | 2022-08-02 | 2022-12-02 | 佳源科技股份有限公司 | Indexable and extensible parameter file fragment storage system and method |
| CN115421649B (en) * | 2022-08-02 | 2023-10-20 | 佳源科技股份有限公司 | Indexable and extensible parameter file slicing storage system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107295425B (en) | 2020-02-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107295425A (en) | A kind of method of quick splicing transcoding slicing files | |
| US7860866B2 (en) | Heuristic event clustering of media using metadata | |
| US8560569B2 (en) | Method and apparatus for performing bulk file system attribute retrieval | |
| US10579595B2 (en) | Method and device for calling a distributed file system | |
| CN104794190B (en) | The method and apparatus that a kind of big data effectively stores | |
| CN105956123A (en) | Local updating software-based data processing method and apparatus | |
| CN103139302A (en) | Real-time copy scheduling method considering load balancing | |
| CN104408111A (en) | Method and device for deleting duplicate data | |
| CN110351532B (en) | Video big data cloud platform cloud storage service method | |
| CN107844524A (en) | Data processing method, data processing equipment, computer equipment and storage medium | |
| CN104348859B (en) | File synchronisation method, device, server, terminal and system | |
| CN110287201A (en) | Data access method, device, equipment and storage medium | |
| CN104750855A (en) | Method and device for optimizing big data storage | |
| CN112235396B (en) | Content processing link adjustment method, content processing link adjustment device, computer equipment and storage medium | |
| CN108228432A (en) | A kind of distributed link tracking, analysis method and server, global scheduler | |
| US20180203636A1 (en) | Likelihood of access based object storage in a cloud environment | |
| CN107181773A (en) | Data storage and data managing method, the equipment of distributed memory system | |
| CN112799588A (en) | Data storage method for loading container cluster application data by using external storage | |
| CN108182209A (en) | A kind of data index method and equipment | |
| CN110309184A (en) | A kind of caching method and system of Air Transportation Pricing data | |
| CN113873025B (en) | Data processing method and device, storage medium and electronic equipment | |
| CN112002130B (en) | Data management method, device, server and computer readable storage medium | |
| CN114741467A (en) | Full-text retrieval method and system | |
| CN105718485B (en) | A kind of method and device by data inputting database | |
| CN111143366A (en) | High-efficiency storage method for massive large object data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20200221 Termination date: 20200612 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |