CN104469396A - Distributed transcoding system and method - Google Patents
Distributed transcoding system and method Download PDFInfo
- Publication number
- CN104469396A CN104469396A CN201410812488.8A CN201410812488A CN104469396A CN 104469396 A CN104469396 A CN 104469396A CN 201410812488 A CN201410812488 A CN 201410812488A CN 104469396 A CN104469396 A CN 104469396A
- Authority
- CN
- China
- Prior art keywords
- burst
- transcoding
- file
- merging
- distributed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
- H04N21/23113—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion involving housekeeping operations for stored content, e.g. prioritizing content for deletion because of storage space restrictions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234309—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 or from Quicktime to Realvideo
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/845—Structuring of content, e.g. decomposing content into time segments
- H04N21/8456—Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments
Abstract
The invention relates to a distributed transcoding system and method. The system comprises a preprocessing unit used for acquiring a source file and judging the file type, wherein the preprocessing unit is connected with a segment computing unit used for computing the size and in/out points of segments, the segment computing unit is connected with multiple segment transcoding units, the segment transcoding units are connected with multiple merging units, and the merging units are connected with a target storage device. According to the distributed transcoding system and method, CPU and GPU computing resources are fully utilized by means of task fragmentization in a video file segment mode, traditional single-server resource utilization is expanded to more resources, and the purpose of executing a transcoding task quickly and efficiently is achieved; transcoding of the FSC DTC file type and transcoding of the ITC DTC file type are treated differently, wherein second-level quick merging is conducted on the FSC DTC file type according to the characteristics of files of the FSC DTC file type, and the ITC DTC file type is transcoded and merged simultaneously, so that the merging speed is increased, and the storage bandwidth is saved.
Description
Technical field
The present invention relates to a kind of distributed trans-coding system and method, being that a kind of computer data calculates and treatment technology, is a kind of system and method for Digital Video Processing.
Background technology
In video and audio process field, particularly there is a large amount of video and audio file format transcoding demands in fields such as TV station, Network TV Station, network company, IPTV, internet television, OTT, mobile TVs.In the urgent need to a set of efficient video and audio file code-transferring method fast, by making full use of computational resource to exchange for transcoding speed at a high speed.It is not high to there is transcoding efficiency in traditional video and audio transcoding, cannot make full use of the problem of computational resource; Or it is not comprehensive to there is the video and audio form supported, or the problem that efficiency is not high when merging treatment, becomes the bottleneck of digital video information process.
Summary of the invention
In order to overcome the problem of prior art, the present invention proposes a kind of distributed trans-coding system and method.Described system and method adopts distributed trans-coding technology, and have employed intermediate form and quick folding, carries out the process of multithreading distribution transcoding, substantially increases transcoding efficiency.
The object of the present invention is achieved like this: a kind of distributed trans-coding system, comprise: from the storage device of source, obtain source file and judge the pretreatment unit of file type, described pretreatment unit is connected with the size and the burst computing unit entered out a little calculating burst, described burst computing unit is connected with multiple burst transcoding units, described multiple burst transcoding units are connected with multiple merge cells, and described multiple merge cells is connected with the target storage storing file destination.
Further, buffer storage is provided with between described burst transcoding units and merge cells.
Use a distributed trans-coding method for said system, described method comprises the steps:
The pretreated step of transcoding: for judging the legitimacy of source file, and determine transcoding mode;
The step that burst calculates: split rule for determining, and carry out the size of burst and enter out a calculating;
The step split: for a little splitting source file with entering out according to the size of the fractionation rule determined and the burst calculated;
The step of burst transcoding: for completing the transcoding work to each burst;
The parsing of burst and the step of merging: for merging the segmental file after transcoding.
Further, the transcoding mode of described " the pretreated step of transcoding " is: " the distributed trans-coding mode based on file system " or the one in " merging distributed trans-coding mode in time ".
Further, the described fractionation rule described in " step that burst calculates " is: according to computational resource dynamic resolution or according to the one in duration fractionation mode.
Further, in described " step of burst transcoding ", carry out transcoding with " the distributed trans-coding mode based on file system ", each burst is encoded to the slicing files form of LeadFS file system.
Further, in described " step of burst transcoding ", carry out transcoding with " merge in time distributed trans-coding mode ", each burst is encoded to self-defining slicing files form.
Further, described " parsing of burst and the step of merging " comprises following sub-step:
The sub-step of document analysis: burst is resolved, determine slicing files form, if the slicing files form of LeadFS file system, then enter " sub-step based on the distributed merging mode of file system ", if self-defining slicing files form, then enter " sub-step merging distributed merging mode in time ";
Sub-step based on the distributed merging mode of file system: for revising the file index information in LeadFS file system, burst is merged into file destination;
The sub-step of the distributed merging mode of timely merging: for resolving burst and merging, comprise as follows step by step:
According to the burst group number of input, read burst one by one;
Resolve the frame information in the burst read and frame data;
Frame information and frame data are put into buffer memory;
From buffer memory, take out frame data and frame information, burst is merged into file destination.
Further, in described " sub-step based on the distributed merging mode of file system ", parsing and merging are divided into two threads to carry out, and the burst frame information obtained and frame data are carried out file destination merging simultaneously in parsing simultaneously.
The beneficial effect that the present invention produces is: the present invention is by the mode of video file burst, CPU and GPU computational resource is fully utilized by task fragmentation means, the utilization of traditional Single-Server resource is expanded to more resource, reaches the object of transcoding tasks carrying rapidly and efficiently.The transcoding of the present invention to FSC DTC and ITC DTC two kinds of file types is treated with a certain discrimination, FSC DTC file type is utilized to the system features of its file itself, the quick merging of level second can be realized, to the mode that ITC DTC file type then adopts transcoding limit, limit to merge, improve aggregate speed greatly, save memory bandwidth.
Accompanying drawing explanation
Below in conjunction with drawings and Examples, the invention will be further described.
Fig. 1 is the principle schematic of system described in embodiments of the invention one;
Fig. 2 is the principle schematic of system described in embodiments of the invention two;
Fig. 3 is the steps flow chart schematic diagram of method described in embodiments of the invention three;
Fig. 4 is the flow chart of method described in embodiments of the invention nine.
Embodiment
Embodiment one:
The present embodiment is a kind of distributed trans-coding system, as shown in Figure 1.The present embodiment comprises: from the storage device of source, obtain source file and judge the pretreatment unit of file type, described pretreatment unit is connected with the size and the burst computing unit entered out a little calculating burst, described burst computing unit is connected with multiple burst transcoding units, described multiple burst transcoding units are connected with multiple merge cells, and described multiple merge cells is connected with the target storage storing file destination.
The compositions such as pretreatment unit, burst computing unit, burst transcoding units, merge cells are comprised in system described in the present embodiment.System described in the present embodiment can realize by mode flexibly, can be embodied as a set of equipment comprising computing node cluster and dish battle array, also each software unit can be deployed in cloud computing platform and realize cloud transcoding.If described system is the equipment with server and dish battle array, can be deployed in user's building environments such as TV station, Network TV Station, network company, IPTV, internet television, OTT, mobile TV.If cloud transcoding, then provide service by privately owned cloud or publicly-owned cloud mode.System described in the present embodiment possesses very large retractility, and burst transcoding computing node can from several node to hundreds of not etc.
Source storage device is the dish battle array database storing various video file, or the hard disk of server, also can be that cloud stores.
Target storage is that various memory device and cloud store.Can be physically together with the storage device of source, or independent memory device.
The function of pretreatment unit judges the legitimacy of source file, and determine to adopt which kind of transcoding mode.Adopt which kind of mode can be specified by upper strata, also can by the file system type automatic decision according to source file, if source file is LeadFS file system, adopt fast distributed transcoding (the File System Combination Distributed Transcode:FSC DTC) mode based on LeadFS file system, if not then by distributed trans-coding (the In Time Combination Distributed Transcode:ITC DTC) mode based on universal document system.
The function of burst computing unit determines to split rule, and carry out the size of burst and enter out a calculating.Split rule and have two kinds of modes: split mode according to computational resource dynamic resolution with according to duration.The basic ideas of the present embodiment source file are split as multiple fragment (fragment being called " burst " in the present embodiment), utilizes multiple thread to carry out transcoding simultaneously, to make full use of computational resource and to improve the efficiency of transcoding.
Foundation computational resource dynamic resolution is as fractionation reference frame by available computational resource quantity, this mode can ensure that number of fragments can be mated with computational resource, simultaneously in order to ensure degree of parallelism, tear the multiple that sheet quantity can be computational resource (idle thread) open.
Determine that distributed fractionation number is used as splitting reference frame according to clip durations according to fixing burst duration, just can obtain splitting quantity divided by clip durations with total duration.
If there is residue frame number during fractionation, also need to do registration process.Consider actual efficiency and the effect of transcoding, clip durations can not too small can not be long.Too small then dispatching efficiency declines, long, does not reach enough degree of parallelisms.
If split according to computational resource, split number of fragments and be generally the multiple of computational resource quantity, can realize the fully parallel of burst transcoding and merging like this, that is after first burst transcoding completes, merging work has started to carry out; Meanwhile, the burst transcoding of second batch also can synchronously carry out.
If when computing node quantity is a lot, then segmentation can not based on computational resource quantity, otherwise can cause burst quantity too much, causes dispatching efficiency to decline.In this case fixed length burst is directly used just can to realize good overall efficiency.
The function of burst transcoding units has been the transcoding work to each burst.The transcoding of burst uses multi-process to complete, and can dispatch different computational resource nodes and carry out simultaneously, reach maximum process throughput.First call decoder to decode to source clip file according to the position entered out a little, then the clip file that encoder encodes is object format is called, concrete burst transcoding has two kinds of different modes, one is " distributed trans-coding (FSC DTC) mode based on file system ", and another kind is " merging distributed trans-coding (ITC DTC) mode in time ".Under different modes, transcoding process is different:
Based under distributed trans-coding (FSC DTC) mode of file system, each point of sector-meeting is encoded to the fragment format of LeadFS file system, and these fragments cannot normally use or play before merging.This method relies on the file system of LeadFS.
Merging in time under distributed trans-coding (ITC DTC) mode, each burst wants formation sheet segment file.Clip file adopts self-defining file format, so that whether the merge cells of subsequent step can complete required packing process by detection lug segment file, carries out union operation if complete namely can start to read clip file and be written to file destination.The effect of this method is by burst is divided refinement, burst division numbers is allowed to be greater than the quantity of transcoding server, the situation utilizing the machine of server cluster can complete at part burst just can start to merge work, burst transcoding burst is merged and realizes walking abreast, reduce the time overhead that transcoding is overall.This method is unrestricted to file system, and can realize burst transcoding and merge synchronous.
The function of merge cells is responsible for merging the burst after transcoding.In order to file destination is complete available, after burst transcoding, just need to resolve burst and complete final merging work.Merge module and resolve slicing files type, the processing mode merged for different burst types is different.
Based under distributed trans-coding (FSC DTC) mode of file system, the parsing of burst is completed by the rapid polymerization function of LeadFS file system self with merging.Can realize merging very fast (1-2 second), and the memory space not outside occupying volume.The rapid polymerization function of LeadFS file system self is the file index information directly revised in LeadFS file system, and unactual data copy process, so can reach high efficiency.
Merging in time under distributed trans-coding (ITC DTC) mode, when starting the slicing files after generating transcoding, merge cells brings into operation, and can perform union operation while clip file generates.Merge cells according to the burst array of input, can read clip file one by one.
In order to improve concurrency, merge cells is divided into two separate threads: clip file resolves thread and file destination generates thread.
Clip file is resolved thread primary responsibility and is resolved all clip file, and by the frame information (through serializing) of resolving and frame data stored in buffer memory.File destination generates thread and is then responsible for constantly from buffer memory, taking out frame information and frame data, then pushes out generating object file.
Embodiment two:
The present embodiment is the improvement of embodiment one, is embodiment one about the refinement of burst transcoding units and merge cells.Buffer storage is provided with, as shown in Figure 2 between burst transcoding units described in the present embodiment and merge cells.
Clip file caching process when distributed trans-coding merges.FSC DTC, for the file after burst, directly uses the merging of file system, so there is not the problem needing buffer memory; ITC DTC then needs, first by distributed fragment buffer, then to merge.Buffer memory can use following several mode:
The higher dish battle array of serviceability is as buffer memory, and this storage can provide very high I/O performance, can meet the bandwidth demand that multiple clip file reads while write.
Use the storage of computing node this locality, SSD/SAS/SATA hard disk can be adopted, during burst transcoding, each burst can be stored into computing node this locality and store.Readwrite bandwidth can be distributed in whole computing network by this mode, and to storage not special requirement.
Embodiment three:
The present embodiment is a kind of distributed trans-coding method using said system, as shown in Figure 3.
The concrete steps of the method described in the present embodiment are as follows:
The pretreated step of transcoding: for judging the legitimacy of source file, and determine transcoding mode.This step mainly judges source file, and whether its file is supported for existing transcoder.Originally be that transcoding major part popular coding can be set in the system that uses of embodiment, but always have some minority's coded systems to be left in the basket, therefore, also need to carry out validity judgement to source file.After validity judgement, source file is divided into two type FSC DTC types and ITC DTC type, so that subsequent step processes respectively.
The step that burst calculates: split rule for determining, and carry out the size of burst and enter out a calculating.The information etc. of burst number, each burst is decided according to the major parameter of source file input.The major parameter of decision-making foundation comprises: computing node quantity, file duration, acquiescence burst duration etc.
Fractionation rule is: dynamically determine to split number or split number according to duration determination distributed trans-coding according to computational resource.What these two kinds rules solved is all according to the allocation optimization problems between resource and task.
The step split: for a little splitting source file with entering out according to the size of the fractionation rule determined and the burst calculated.The process split also is a parallel process, can split on limit, and limit forms slicing files.This step, by adding unique slicing files form in burst, makes slicing files can realize the distributed trans-coding of various video and audio form, and can support various file system, ensure that the availability of distributed method.User-defined format is structurally convenient to read-while-write compared with common format, can improved sheet segment file efficiency of immediately merging when generating.Adopt specific clip file form automatically to detect desirable tablet segment file at merging phase whether to complete, just start in the completed read clip file and be written to file destination.By fragment is divided further refinement, utilizing the computational resource in cluster to complete the packing of whole source front portion as early as possible, just can start to merge completed clip file when carrying out rear portion packing.
Distributed trans-coding time and Piece file mergence time suitably can be walked abreast like this, reduce the most latter incorporated time overhead.The advantage of this mode be to destination file format and file system unrestricted.
The step of burst transcoding: for completing the transcoding work to each burst.Source file, after splitting, forms multiple burst, namely starts each burst carry out transcoding simultaneously in multiple transcoding units, improve the efficiency of transcoding with this.
The parsing of burst and the step of merging: for merging the slicing files after transcoding.Can start after slicing files transcoding to merge.Merging can perform union operation while file generated after transcoding.Merge cells according to the filename array of the burst of input, can read slicing files one by one.
Resolve and merge can carry out two separate threads simultaneously: slicing files resolves thread and file destination generates thread.
Slicing files is resolved thread primary responsibility and is resolved all slicing files, and by the frame information (through serializing) of parsing and the buffer memory of frame data constantly stored in fixing frame number.The process of resolving first judges whether this slicing files exists, and whether has data, generate end? if do not satisfied condition, then continue to wait for, until normally data can be read, or cannot wait until always and think locked and exit.
File destination generates thread and is then responsible for constantly from buffer memory, taking out frame information and frame data, then pushes out generating object file.
Embodiment four:
The present embodiment is the improvement of embodiment three, is the refinement of embodiment three about " the pretreated step of transcoding ".The transcoding mode of " the pretreated step of transcoding " described in the present embodiment is: " the distributed trans-coding mode based on file system " or the one in " merging distributed trans-coding mode in time ".
Method described in the present embodiment can be achieved with two kinds of distributed trans-codings.A kind of is fast distributed code-transferring method (FSC DTC) based on LeadFS file system, and a kind of is distributed trans-coding method (ITC DTC) based on universal document system.
Implementation method (FSC DTC) based on file system:
Source file is split as multiple fragment according to the decision-making of fragment burst by FSC DTC, (each fragment cannot independently decode broadcasting), these clip file are directly merged into file destination at file system layer by the pooled function using LeadFS file system to provide after all fragments generate.Piece file mergence can instantaneously complete (1-2 second), and the memory space not outside occupying volume.The method is the rapid polymerization function based on LeadFS file system, fundamentally can solve the performance bottleneck of distributed trans-coding at merging phase.But this mode must use specific file system, and only support the distributed trans-coding of AVI and MXF two kinds of file formats.
Timely merging distributed trans-coding (ITC DTC):
ITC DTC is the distributed trans-coding mode that transcoding limit, a kind of limit merges.Adopt self-defining clip file form, automatically whether can complete required packing process by detection lug segment file, carry out union operation if complete namely can start to read clip file and be written to file destination.The ideal effect of this method is by fragment is divided refinement, fragment division numbers is allowed to be greater than the quantity of transcoding server, utilizing the machine of server array to complete the transcoding of front portion fragment as early as possible, just can start when carrying out rear portion fragment transcoding to merge completed clip file.Fragment Transcode Clip can be merged like this and realize walking abreast, reduce the time overhead that transcoding is overall.
This method is unrestricted to destination file format and file system, and can realize fragment transcoding and merge synchronous.
Embodiment five:
The present embodiment is the improvement of embodiment four, is the refinement of embodiment four about " step that burst calculates ".The fractionation rule described in " step that burst calculates " described in the present embodiment is: according to computational resource dynamic resolution or according to the one in duration fractionation mode.
No matter be FSC or ITC, the fractionation rule of burst task is similar.There are two kinds of modes:
Dynamically determine to split number according to computational resource: by available computational resource quantity as fractionation reference frame, this mode can ensure that number of fragments can be mated with computational resource, simultaneously in order to ensure degree of parallelism, tears the multiple that sheet quantity can be computational resource open.
Split number according to duration determination distributed trans-coding: be used as splitting reference frame according to burst duration, just can obtain splitting quantity divided by burst duration with total duration.According to the principle that burst quantity is moderate, burst duration is traditionally arranged to be 1-2 minute.
When the aliquant problem of burst information, bottom layer treatment method: according to upper strata suggestion duration, bottom carries out GOP alignment to suggestion frame number, and then judging whether can decile, if can decile, directly divides burst, as can not decile, need judgement to divide exactly residue frame number, as residue frame number is greater than the half of suggestion frame number, then will remain frame number as last burst, otherwise in each burst be evenly added to residue frame number above, thus avoid last burst to divide uneven problem.
Consider actual efficiency and the effect of transcoding, burst duration can not too small can not be long.Too small then dispatching efficiency declines, long, does not reach enough degree of parallelisms.
Embodiment six:
The present embodiment is the improvement of embodiment five, is the refinement of embodiment five about " step of burst transcoding ".In " step of burst transcoding " described in the present embodiment, carry out transcoding with " the distributed trans-coding mode based on file system ", each burst is encoded to the slicing files form of LeadFS file system.
The file system of LeadFS file system to be a kind of can be file rapid polymerization, (Chinese patent application that detail is shown in " a kind of method of shared-file system multifile rapid polymerization and reading ", the applying date: on October 3rd, 2014, application number 2014106000039), for file merges interface fast, can support that the file of segmentation does quick merging when not making the copy of data.Quick merging is that the metadata information by directly revising file data in file system block has come.
Embodiment seven:
The present embodiment is the improvement of embodiment five, is the refinement of embodiment five about " step of burst transcoding ".In " step of burst transcoding " described in the present embodiment, carry out transcoding with " merge in time distributed trans-coding mode ", each burst is encoded to self-defining slicing files form.
User-defined format is structurally convenient to read-while-write compared with common format, can improved sheet segment file efficiency of immediately merging when generating.Adopt specific clip file form automatically to detect desirable tablet segment file at merging phase whether to complete, just start in the completed read clip file and be written to file destination.By fragment is divided further refinement, utilizing the computational resource in cluster to complete the packing of whole source front portion as early as possible, just can start to merge completed clip file when carrying out rear portion packing.
Distributed trans-coding time and Piece file mergence time suitably can be walked abreast like this, reduce the most latter incorporated time overhead.The advantage of this mode be to destination file format and file system unrestricted.
Embodiment eight:
The present embodiment is the improvement of embodiment six, seven, is the refinement of embodiment six, seven about " parsing of burst and the step of merging "." parsing of burst and the step of merging " described in the present embodiment comprises following sub-step:
The sub-step of document analysis: burst is resolved, determine slicing files form, if the slicing files form of LeadFS file system, then enter " sub-step based on the distributed merging mode of file system ", if self-defining slicing files form, then enter " sub-step merging distributed merging mode in time ".
Sub-step based on the distributed merging mode of file system: for revising the file index information in LeadFS file system, burst is merged into file destination.
The sub-step of the distributed merging mode of timely merging: for resolving burst and merging, comprise as follows step by step:
According to the burst group number of input, read burst one by one.
Resolve the frame information in the burst read and frame data.
Frame information and frame data are put into buffer memory.
From buffer memory, take out frame data and frame information, burst is merged into file destination.
Clip file is resolved primary responsibility and is resolved all clip file, and by the frame information (through serializing) of parsing and the buffer memory of frame data constantly stored in fixing frame number.The process of resolving first judges whether this fragment exists, and whether has data, generate end? if do not satisfied condition, then continue to wait for, until normally data can be read, or cannot wait until always and think locked and exit.
File destination generates then to be responsible for constantly from buffer memory, taking out frame information and frame data, then pushes out generating object file.
Parsing and merging can be divided into two threads, at any time file destination merging is carried out to the frame information of burst obtained and frame data.
Embodiment nine:
The present embodiment is the improvement of embodiment eight, is the refinement about " sub-step based on the distributed merging mode of file system " in embodiment eight.In " sub-step based on the distributed merging mode of file system " described in the present embodiment, parsing and merging are divided into two threads to carry out, and the burst frame information obtained and frame data are carried out file destination merging simultaneously, as shown in Figure 4 in parsing simultaneously.
Fig. 4 is the schematic diagram of the handling process of resolving and merging, and thread A represents that clip file resolves thread, and thread B represents that file destination generates thread, is built into asynchronous read and write buffer memory like this, improves combined efficiency.
Finally it should be noted that, below only in order to technical scheme of the present invention to be described and unrestricted, although with reference to preferred arrangement scheme to invention has been detailed description, those of ordinary skill in the art is to be understood that, can modify to technical scheme of the present invention (sequencing etc. of the connected mode of such as system, each annexation, step) or equivalently to replace, and not departing from the spirit and scope of technical solution of the present invention.
Claims (9)
1. a distributed trans-coding system, it is characterized in that, comprise: from the storage device of source, obtain source file and judge the pretreatment unit of file type, described pretreatment unit is connected with the size and the burst computing unit entered out a little calculating burst, described burst computing unit is connected with multiple burst transcoding units, described multiple burst transcoding units are connected with multiple merge cells, and described multiple merge cells is connected with the target storage storing file destination.
2. system according to claim 1, is characterized in that, is provided with buffer storage between described burst transcoding units and merge cells.
3. use a distributed trans-coding method for system described in claim 2, it is characterized in that, described method comprises the steps:
The pretreated step of transcoding: for judging the legitimacy of source file, and determine transcoding mode;
The step that burst calculates: split rule for determining, and carry out the size of burst and enter out a calculating;
The step split: for a little splitting source file with entering out according to the size of the fractionation rule determined and the burst calculated;
The step of burst transcoding: for completing the transcoding work to each burst;
The parsing of burst and the step of merging: for merging the segmental file after transcoding.
4. method according to claim 3, is characterized in that, the transcoding mode of described " the pretreated step of transcoding " is: " the distributed trans-coding mode based on file system " or the one in " merging distributed trans-coding mode in time ".
5. method according to claim 4, is characterized in that, the described fractionation rule described in " step that burst calculates " is: according to computational resource dynamic resolution or according to the one in duration fractionation mode.
6. method according to claim 5, is characterized in that, in described " step of burst transcoding ", carry out transcoding with " the distributed trans-coding mode based on file system ", each burst is encoded to the slicing files form of LeadFS file system.
7. method according to claim 5, is characterized in that, in described " step of burst transcoding ", carry out transcoding with " merge in time distributed trans-coding mode ", each burst is encoded to self-defining slicing files form.
8. the method according to claim 6 or 7, is characterized in that, described " parsing of burst and the step of merging " comprises following sub-step:
The sub-step of document analysis: burst is resolved, determine slicing files form, if the slicing files form of LeadFS file system, then enter " sub-step based on the distributed merging mode of file system ", if self-defining slicing files form, then enter " sub-step merging distributed merging mode in time ";
Sub-step based on the distributed merging mode of file system: for revising the file index information in LeadFS file system, burst is merged into file destination;
The sub-step of the distributed merging mode of timely merging: for resolving burst and merging, comprise as follows step by step:
According to the burst group number of input, read burst one by one;
Resolve the frame information in the burst read and frame data;
Frame information and frame data are put into buffer memory;
From buffer memory, take out frame data and frame information, burst is merged into file destination.
9. method according to claim 8, it is characterized in that, in described " sub-step based on the distributed merging mode of file system ", parsing and merging are divided into two threads to carry out, and the burst frame information obtained and frame data are carried out file destination merging simultaneously in parsing simultaneously.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410812488.8A CN104469396B (en) | 2014-12-24 | 2014-12-24 | A kind of distributed trans-coding system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410812488.8A CN104469396B (en) | 2014-12-24 | 2014-12-24 | A kind of distributed trans-coding system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104469396A true CN104469396A (en) | 2015-03-25 |
| CN104469396B CN104469396B (en) | 2018-01-05 |
Family
ID=52914658
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410812488.8A Active CN104469396B (en) | 2014-12-24 | 2014-12-24 | A kind of distributed trans-coding system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104469396B (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105338371A (en) * | 2015-10-29 | 2016-02-17 | 南京秦杜明视信息技术有限公司 | Multimedia transcoding scheduling method and apparatus |
| CN106791932A (en) * | 2017-01-06 | 2017-05-31 | 广州亦云信息技术股份有限公司 | Distributed trans-coding system, method and its device |
| CN108268432A (en) * | 2017-12-14 | 2018-07-10 | 中央电视台 | A kind of conversion method and device of program project file |
| CN110740345A (en) * | 2019-09-20 | 2020-01-31 | 北京旷视科技有限公司 | Offline video file speed doubling analysis method and device |
| CN110856018A (en) * | 2019-11-14 | 2020-02-28 | 武汉珞佳伟业科技有限公司 | Rapid transcoding method and system in monitoring system based on cloud computing |
| CN110996172A (en) * | 2019-12-17 | 2020-04-10 | 杭州当虹科技股份有限公司 | Method for quickly synthesizing 4K MXF file |
| CN111163335A (en) * | 2019-12-31 | 2020-05-15 | 深圳市瑞讯云技术有限公司 | Video processing method and device for distributed server |
| CN111432243A (en) * | 2020-04-01 | 2020-07-17 | 山东汇贸电子口岸有限公司 | Video transcoding method and device |
| CN111752946A (en) * | 2020-06-22 | 2020-10-09 | 上海众言网络科技有限公司 | Method and device for preprocessing research data based on fragmentation mode |
| CN113852840A (en) * | 2021-09-18 | 2021-12-28 | 北京百度网讯科技有限公司 | Video rendering method and device, electronic equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101945278A (en) * | 2010-09-15 | 2011-01-12 | 中国科学院深圳先进技术研究院 | Video self-adaptive transcoding method and system |
| CN102055965A (en) * | 2009-10-28 | 2011-05-11 | 腾讯科技（深圳）有限公司 | Media file transcoding system and method |
| CN102123279A (en) * | 2010-12-28 | 2011-07-13 | 乐视网信息技术（北京）股份有限公司 | Distributed real-time transcoding method and system |
| CN102780918A (en) * | 2012-08-15 | 2012-11-14 | 华数传媒网络有限公司 | Video distributed coding format converting method |
| CN103297807A (en) * | 2013-06-21 | 2013-09-11 | 哈尔滨工业大学深圳研究生院 | Hadoop-platform-based method for improving video transcoding efficiency |
| US20130279602A1 (en) * | 2012-04-24 | 2013-10-24 | Vixs Systems, Inc. | Configurable transcoder and methods for use therewith |
| CN103838779A (en) * | 2012-11-27 | 2014-06-04 | 深圳市腾讯计算机系统有限公司 | Idle computing resource multiplexing type cloud transcoding method and system and distributed file device |
-
2014
- 2014-12-24 CN CN201410812488.8A patent/CN104469396B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102055965A (en) * | 2009-10-28 | 2011-05-11 | 腾讯科技（深圳）有限公司 | Media file transcoding system and method |
| CN101945278A (en) * | 2010-09-15 | 2011-01-12 | 中国科学院深圳先进技术研究院 | Video self-adaptive transcoding method and system |
| CN102123279A (en) * | 2010-12-28 | 2011-07-13 | 乐视网信息技术（北京）股份有限公司 | Distributed real-time transcoding method and system |
| US20130279602A1 (en) * | 2012-04-24 | 2013-10-24 | Vixs Systems, Inc. | Configurable transcoder and methods for use therewith |
| CN102780918A (en) * | 2012-08-15 | 2012-11-14 | 华数传媒网络有限公司 | Video distributed coding format converting method |
| CN103838779A (en) * | 2012-11-27 | 2014-06-04 | 深圳市腾讯计算机系统有限公司 | Idle computing resource multiplexing type cloud transcoding method and system and distributed file device |
| CN103297807A (en) * | 2013-06-21 | 2013-09-11 | 哈尔滨工业大学深圳研究生院 | Hadoop-platform-based method for improving video transcoding efficiency |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105338371A (en) * | 2015-10-29 | 2016-02-17 | 南京秦杜明视信息技术有限公司 | Multimedia transcoding scheduling method and apparatus |
| CN105338371B (en) * | 2015-10-29 | 2019-05-17 | 四川奇迹云科技有限公司 | A kind of multi-media transcoding dispatching method and device |
| CN106791932A (en) * | 2017-01-06 | 2017-05-31 | 广州亦云信息技术股份有限公司 | Distributed trans-coding system, method and its device |
| CN106791932B (en) * | 2017-01-06 | 2019-08-30 | 广州亦云信息技术股份有限公司 | Distributed trans-coding system, method and device thereof |
| CN108268432A (en) * | 2017-12-14 | 2018-07-10 | 中央电视台 | A kind of conversion method and device of program project file |
| CN110740345A (en) * | 2019-09-20 | 2020-01-31 | 北京旷视科技有限公司 | Offline video file speed doubling analysis method and device |
| CN110856018A (en) * | 2019-11-14 | 2020-02-28 | 武汉珞佳伟业科技有限公司 | Rapid transcoding method and system in monitoring system based on cloud computing |
| CN110996172A (en) * | 2019-12-17 | 2020-04-10 | 杭州当虹科技股份有限公司 | Method for quickly synthesizing 4K MXF file |
| CN111163335A (en) * | 2019-12-31 | 2020-05-15 | 深圳市瑞讯云技术有限公司 | Video processing method and device for distributed server |
| CN111163335B (en) * | 2019-12-31 | 2022-04-01 | 深圳市瑞讯云技术有限公司 | Video processing method and device for distributed server |
| CN111432243A (en) * | 2020-04-01 | 2020-07-17 | 山东汇贸电子口岸有限公司 | Video transcoding method and device |
| CN111752946A (en) * | 2020-06-22 | 2020-10-09 | 上海众言网络科技有限公司 | Method and device for preprocessing research data based on fragmentation mode |
| CN111752946B (en) * | 2020-06-22 | 2021-04-30 | 上海众言网络科技有限公司 | Method and device for preprocessing research data based on fragmentation mode |
| CN113852840A (en) * | 2021-09-18 | 2021-12-28 | 北京百度网讯科技有限公司 | Video rendering method and device, electronic equipment and storage medium |
| CN113852840B (en) * | 2021-09-18 | 2023-08-22 | 北京百度网讯科技有限公司 | Video rendering method, device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104469396B (en) | 2018-01-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104469396A (en) | Distributed transcoding system and method | |
| CN105828105A (en) | Distributed environment-based video transcoding system and video transcoding method | |
| CN102123279B (en) | Distributed real-time transcoding method and system | |
| CN109213593B (en) | Resource allocation method, device and equipment for panoramic video transcoding | |
| CN103838779A (en) | Idle computing resource multiplexing type cloud transcoding method and system and distributed file device | |
| CN103279521A (en) | Video big data distributed decoding method based on Hadoop | |
| CN103297807A (en) | Hadoop-platform-based method for improving video transcoding efficiency | |
| WO2013078583A1 (en) | Method and apparatus for optimizing data access, method and apparatus for optimizing data storage | |
| CN103188521A (en) | Method and device for transcoding distribution, method and device for transcoding | |
| CN107370726B (en) | Virtual slicing method and system for distributed media file transcoding system | |
| CN103440290A (en) | Big data loading system and method | |
| CN103974081A (en) | HEVC coding method based on multi-core processor Tilera | |
| CN105187845A (en) | Video data decoding device and method | |
| CN102857533B (en) | A kind of long-distance interactive system based on cloud computing | |
| CN105657450A (en) | Distributed video transcoding method | |
| US20160248834A1 (en) | Algorithmic transcoding | |
| CN114968953A (en) | Log storage and retrieval method, system, terminal equipment and medium | |
| CN107066205B (en) | Data storage system | |
| CN104811646A (en) | Multi-video streaming data concurrent modulation and buffer storage method based on continuous storage model | |
| US11250001B2 (en) | Accurate partition sizing for memory efficient reduction operations | |
| JP7026227B2 (en) | Distributed transcoding method and distributed transcoding system | |
| CN102857535B (en) | A kind of computer processing unit, computer gateway, interactive system | |
| KR101617550B1 (en) | Method for transcoding mutimedia, and cloud mulimedia transcoding system operating the same | |
| CN103309742B (en) | Cloud storage system data efficient coded method | |
| CN114238213A (en) | Multithreading file analysis method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |