CN113747248A - Rapid synthesis method, device and system based on live stream editing - Google Patents
Rapid synthesis method, device and system based on live stream editing Download PDFInfo
- Publication number
- CN113747248A CN113747248A CN202010469247.3A CN202010469247A CN113747248A CN 113747248 A CN113747248 A CN 113747248A CN 202010469247 A CN202010469247 A CN 202010469247A CN 113747248 A CN113747248 A CN 113747248A
- Authority
- CN
- China
- Prior art keywords
- live
- live stream
- file
- synthesis
- parameter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001308 synthesis method Methods 0.000 title claims abstract description 14
- 230000015572 biosynthetic process Effects 0.000 claims abstract description 98
- 238000003786 synthesis reaction Methods 0.000 claims abstract description 97
- 239000012634 fragment Substances 0.000 claims abstract description 96
- 238000000034 method Methods 0.000 claims description 41
- 230000002194 synthesizing effect Effects 0.000 claims description 19
- 102100037812 Medium-wave-sensitive opsin 1 Human genes 0.000 claims description 16
- 238000013507 mapping Methods 0.000 claims description 11
- 238000003860 storage Methods 0.000 claims description 10
- 241000023320 Luma <angiosperm> Species 0.000 claims description 2
- OSWPMRLSEDHDFF-UHFFFAOYSA-N methyl salicylate Chemical compound COC(=O)C1=CC=CC=C1O OSWPMRLSEDHDFF-UHFFFAOYSA-N 0.000 claims description 2
- 230000005540 biological transmission Effects 0.000 abstract description 11
- 230000008569 process Effects 0.000 description 23
- 238000010586 diagram Methods 0.000 description 21
- 238000012545 processing Methods 0.000 description 15
- 238000004590 computer program Methods 0.000 description 7
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 238000013459 approach Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 230000008672 reprogramming Effects 0.000 description 3
- 101100476639 Caenorhabditis elegans gop-3 gene Proteins 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000011265 semifinished product Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
- H04N21/47205—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/23424—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
Abstract
A fast synthesis method, a device and a system based on live stream clips comprise the following steps: receiving and recording the fragment files of the live stream; locally storing the live streaming fragment file and uploading the live streaming fragment file to a cloud terminal; finishing the editing of the live stream file according to the locally stored live stream fragment file; the live stream fragment file comprises a live stream fragment file; generating a synthesis task after finishing the editing and sending the synthesis task to the cloud end; the synthesis task comprises access point information of live files. According to the scheme, a space time-changing mode is adopted, the fragmented files are transmitted firstly, and then the fragmented files are synthesized at the cloud end, so that the transmission time is saved, and the overall synthesis publishing efficiency is improved.
Description
Technical Field
The present application relates to video editing technologies, and in particular, to a method, an apparatus, and a system for fast synthesizing live stream clips.
Background
With the continuous development of fusion media, the traditional television station carries out unified strategic deployment on a station and a network, even part of the television stations propose strategic layout of a network background, which puts new requirements on the traditional television station technology, especially in the links of making and synthesizing live programs, the synthesis in the traditional mode is to package and transcode to finally form programs, the whole process is time-consuming and has very low efficiency; the internet-oriented program form requires the processes of light production, fast synthesis and re-distribution, so that the fast synthesis of the program is very important.
The traditional non-coding synthesis adopts a synthesis process of fully decoding and fully coding video and audio by adopting a high-code file, and the quality loss can be reduced as much as possible and the synthesized frame precision can be ensured simultaneously due to the adoption of the high-code synthesis in the whole process. But significant disadvantages are: because of the full decoding and full encoding processes, especially when multiple code rates are required, the synthesis efficiency is very low, the resource consumption is very serious, and even resource assistance such as a GPU (Graphics Processing Unit) is required. This severely limits the user's use.
Problems existing in the prior art:
the program synthesis efficiency of the traditional television station is very low, and the requirement of the internet program form cannot be met.
Disclosure of Invention
The embodiment of the application provides a fast synthesis method, a device and a system based on live stream editing, so as to solve the technical problems.
According to a first aspect of the embodiments of the present application, there is provided a fast composition method based on live stream clips, including the following steps:
receiving and recording the fragment files of the live stream;
locally storing the live streaming fragment file and uploading the live streaming fragment file to a cloud terminal;
finishing the editing of the live stream file according to the locally stored live stream fragment file; the live stream fragment file comprises a live stream fragment file;
generating a synthesis task after finishing the editing and sending the synthesis task to the cloud end; the synthesis task comprises access point information of live files.
According to a second aspect of the embodiments of the present application, there is provided a fast composition method based on live stream clips, including:
receiving and storing the fragment files of the live stream;
receiving a synthesis task of a fragment file of a live stream; the synthesis task comprises access point information of a live broadcast file; the live files comprise live stream fragment files;
and synthesizing the stored live streaming fragment files according to the synthesis task.
According to a third aspect of the embodiments of the present application, there is provided a fast composition apparatus based on live stream clips, including:
the receiving and recording module is used for receiving and recording the fragment files of the live stream;
the local storage module is used for locally storing the fragment files of the live streaming;
the first sending module is used for uploading the live streaming fragment file to a cloud;
the editing module is used for finishing editing of the live stream file according to the locally stored live stream fragment file; the live stream fragment file comprises a live stream fragment file;
the task generation module is used for generating a synthesis task after finishing the editing; the synthesis task comprises access point information of a live broadcast file;
and the second sending module is used for sending the synthesis task to a cloud.
According to a fourth aspect of the embodiments of the present application, there is provided a fast composition apparatus based on live stream clips, including:
the first receiving module is used for receiving the fragment file of the live stream;
the cloud storage module is used for storing the live streaming fragment file;
the second receiving module is used for receiving a synthesis task of the live broadcast file; the synthesis task comprises access point information of a live broadcast file; the live stream fragment file comprises a live stream fragment file;
and the synthesis module is used for synthesizing the stored live streaming file according to the synthesis task.
In a fifth aspect of the embodiments of the present application, a fast composition system based on live stream clips is provided, including:
a first fast composition means, which is a fast composition means based on live stream clips as described in the third aspect;
and the number of the first and second groups,
a plurality of second fast composing means, the second fast composing means being the live-stream-clip-based fast composing means according to the fourth aspect;
and the second sending module is used for scheduling resources after receiving the synthesis tasks and sending the synthesis tasks to relatively idle devices in the plurality of second quick synthesis devices.
According to the fast synthesis method, device and system based on live stream editing, a space time-switching mode is adopted, fragment files are transmitted firstly, and then synthesis is carried out at the cloud end, so that the transmission time is saved, and the overall synthesis and release efficiency is improved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
fig. 1 shows a flow chart of an implementation of a fast composition method based on live stream clips in an embodiment of the present application;
fig. 2 is a schematic structural diagram illustrating a fast composition apparatus based on live stream editing in an embodiment of the present application;
fig. 3 is a flowchart illustrating an implementation of a fast synthesis method based on live stream clips in a third embodiment of the present application;
fig. 4 is a schematic structural diagram illustrating a fast composition apparatus based on live stream editing in a fourth embodiment of the present application;
fig. 5 shows a schematic structural diagram of a fast composition system based on live stream editing in the fifth embodiment of the present application;
fig. 6 is a schematic diagram illustrating a fast composition architecture based on live stream editing in an embodiment of the present application;
FIG. 7 is a schematic diagram illustrating an implementation of a fast composition engine in the sixth embodiment of the present application;
FIG. 8 is an enlarged view of the structure of a GOP in the sixth embodiment of the present application;
FIG. 9 shows a schematic diagram of the process of rapid synthesis in example six of the present application.
Detailed Description
In the process of implementing the present application, the inventors found that:
the traditional video clip is based on a material clipping mode, and the clip of the new media content is mostly based on a semi-finished product or a live stream clipping mode, so that in the process of clipping the video, an important point is to point out and point in, valuable partial content is extracted, and the number of complex special effects is very small.
Some internet companies also start to perform a Live editing process of related objects, and the adopted mode of uniformly recording and recording HLS (HTTP Live Streaming based on HTT) is uniformly stored according to a Group of pictures (GOP) unit, and the editing mode of a complete GOP unit is used for performing a COPY splicing and synthesizing process at a GOP level aiming at multiple access points. The synthesis method can ensure the synthesis efficiency, but can not ensure the frame precision, once the in-point and the out-point of the user are in the middle of the GOP, the content is intercepted in a complete GOP mode, which is equivalent to that the content is stored in a large amount. This approach is a GOP-accurate synthesis approach, not a frame-accurate synthesis approach. And simultaneously, the multi-code rate synchronous synthesis is not supported. This also causes inconvenience to the end user.
In addition, both conventional synthesis engines and fast synthesis engines of internet companies currently have certain drawbacks to h.265 and dolby Atmos panoramic sonification. Even if the traditional non-compiler supports fast production composition of new media with a 4K composition engine, it would be a significant waste of resources. Meanwhile, no traditional non-editorial manufacturer and internet company in the industry can process the rapid synthesis processing of dolby Atmos by using a synthesis engine at present.
Aiming at the technical problems in the prior art, the embodiment of the application provides a rapid synthesis method, which is used for realizing frame precision and rapid release of live stream clips.
In order to make the technical solutions and advantages of the embodiments of the present application more apparent, the following further detailed description of the exemplary embodiments of the present application with reference to the accompanying drawings makes it clear that the described embodiments are only a part of the embodiments of the present application, and are not exhaustive of all embodiments. It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.
Example one
Fig. 1 shows a flow chart of an implementation of a fast composition method based on live stream clips in an embodiment of the present application.
As shown in the figure, the fast composition method based on live stream clips includes:
102, locally storing a live streaming fragment file and uploading the live streaming fragment file to a cloud;
103, finishing the editing of the live broadcast file according to the locally stored live broadcast stream fragment file; the live stream fragment file comprises a live stream fragment file;
104, generating a synthesis task after finishing editing and sending the synthesis task to a cloud end; the synthesis task comprises access point information of live files.
In specific implementation, according to the requirement of live clip, the file fragments recorded in live may be in the minute level or in the second level, which is not limited in this application.
And after the recorded fragment files are stored locally, uploading the fragment files to a cloud terminal. Editing personnel clips contents, sets multiple pairs of access points of the whole live broadcast file are completed, the live broadcast file is submitted to the cloud after the editing is completed, the live broadcast stream fragment file at the cloud is rapidly synthesized by the synthesis node at the cloud, the synthesized file is stored in an object storage at the cloud, and finally the user is provided to browse and play.
In the prior art, fragment files are generally collected and then synthesized by a DMA (direct memory access) intranet, and the synthesized files are transmitted to a cloud. For a few GB large files, the transmission time is long, requiring more time to transmit the file to the cloud. In order to save the transmission time, the fragment file is transmitted to the cloud end, and the large file is synthesized at the cloud end after the live broadcast is finished, so that the overall synthesis processing efficiency is improved.
The fast synthesis method based on live stream editing provided by the embodiment of the application adopts a space time-changing mode for a local processing process, and firstly transmits the fragment files and then synthesizes the fragment files at the cloud, so that the transmission time is saved, and the overall synthesis publishing efficiency is improved.
In one embodiment, before the recording the live stream fragment file, the method further comprises:
performing one-in-multiple-out live broadcast coding on the live broadcast stream by using an encoder according to the multi-code-rate requirement of live broadcast arrangement to obtain a multi-code-rate live broadcast stream;
and starting a multi-code-rate live broadcast receiving and recording task.
Example two
Based on the same inventive concept, the embodiment of the present application provides a fast synthesis device based on live stream editing, and the principle of the device for solving the technical problem is similar to that of the fast synthesis method based on live stream editing in the embodiment, and repeated parts are not described again.
Fig. 2 shows a schematic structural diagram of a fast composition apparatus based on live stream clips in the second embodiment of the present application.
As shown, the fast composition device based on live stream clips includes:
a receiving and recording module 201, configured to receive and record a fragment file of a live stream;
a local saving module 202, configured to save the live stream fragment file locally;
the first sending module 203 is configured to upload the live streaming fragment file to the cloud;
the clipping module 204 is configured to complete clipping of a live stream file according to a locally stored live stream fragment file; the live stream fragment file comprises a live stream fragment file;
a task generating module 205, configured to generate a composition task after the editing is completed; the synthesis task comprises access point information of a live broadcast file;
a second sending module 206, configured to send the composition task to the cloud.
The fast synthesis device based on live stream editing provided in the embodiment of the application adopts a space time-changing mode for local end equipment, firstly transmits the fragment files, and then synthesizes the fragment files at the cloud end, so that the transmission time is saved, and the overall synthesis publishing efficiency is improved.
In one embodiment, further comprising:
the encoding module is used for carrying out one-in and multiple-out live broadcast encoding on the live broadcast stream according to the multi-code-rate requirement of live broadcast editing by using an encoder before the live broadcast stream fragment file is recorded and received, so as to obtain the multi-code-rate live broadcast stream;
and the receiving and recording starting module is used for starting the multi-code-rate live broadcasting receiving and recording task.
EXAMPLE III
Fig. 3 shows a flow chart of an implementation of a fast composition method based on live stream clips in the third embodiment of the present application.
As shown in the figure, the fast composition method based on live stream clips includes:
and 303, synthesizing the stored live streaming fragment files according to the synthesis task.
In the prior art, fragment files are generally collected and then synthesized by a DMA (direct memory access) intranet, and the synthesized files are transmitted to a cloud. For a few GB large files, the transmission time is long, requiring more time to transmit the file to the cloud. In order to save the transmission time, the cloud receives the fragment files first, and the received fragment files are synthesized into the large file at the cloud according to the received synthesis task after the live broadcast is finished, so that the overall synthesis processing efficiency is improved.
The fast synthesis method based on live stream editing provided by the embodiment of the application is a cloud processing process, adopts a space time-changing mode, transmits fragment files firstly, and then synthesizes the fragment files at the cloud, so that the transmission time is saved, and the overall synthesis publishing efficiency is improved.
The inventor finds out in the invention process that:
the general audio and video files are stored by taking GOPs as units for related contents, and each GOP unit comprises contents with different forms, such as IPBBBPBBB (3B frame examples), and the like. Each GOP begins with an I-frame and ends with a B-frame if included. Usually, the GOPs in the video file are all of Close type, that is, data of each GOP can be decoded independently after being taken out, the picture is not mosaic, that is, the first I frame does not need to depend on other frames when decoding the picture, the P frame needs to depend on the frame before the P frame for reference, and the B frame depends on both the frame before the B frame and the frame after the B frame for bidirectional reference. So long as the GOP is not corrupted, the data within the GOP can be used in its entirety.
Therefore, the embodiments of the present application can also be implemented as follows.
In one embodiment, the synthesizing the saved live stream fragment file includes:
determining an in-point for each clip;
when an entry point of a clip is positioned in a P frame or a B frame of a GOP, decoding is started from an initial I frame of the GOP where the entry point is positioned until the current GOP is finished, and data obtained by decoding is re-encoded;
copying and splicing other GOPs of the clip according to the GOP where the entry point is located;
and synthesizing the plurality of clip segments into an edited output.
In specific implementation, if the entry point is located in the middle of the GOP (P frame or B frame), the embodiment of the present application performs complete decoding on the GOP, then performs SEEK to the location of the entry point, starts to encode the content of the latter part, and performs COPY splicing with other GOPs. Therefore, only the GOP of the entry point is subjected to re-encoding processing, and the other GOPs are directly COPY, so that the synthesis efficiency is very high.
The embodiment of the application realizes the single GOP reprogramming with frame precision, the whole rapid synthesis process is frame precision, the partial reprogramming is carried out on the first access point GOP, other GOPs adopt a Copy mode, the current rapid synthesis engine of the internet takes the GOP as a unit for processing, the length of the general GOP is 2 seconds, and therefore the precision of the GOPs is in units of seconds instead of frames.
In one embodiment, the synthesizing the saved live stream fragment file further includes:
when an in-point of a clip is placed in an I-frame of a GOP, the clip does not need to be re-encoded.
The embodiment of the application only re-encodes partial video frames in one GOP, but not encodes all frames, and does not need to re-encode the video frames of the I frames of the GOP with the entry points.
In one embodiment, before the re-encoding the decoded data, the method includes:
analyzing and acquiring decoding parameters from network abstraction layer NAL data obtained by decoding;
comparing the decoding parameters with the decoding parameters of the last file;
and if the decoding parameters are consistent with the decoding parameters, obtaining the coding parameters according to the decoding parameter reverse mapping.
Although only the GOP of the entry point is subjected to re-encoding and decoding, the difference between the decoded parameters and the encoded parameters in h.264 and h.265 is very large, and it is very difficult to establish the corresponding relationship between the decoded parameters and the encoded parameters.
In one embodiment, the mapping relationship between the decoding parameters and the encoding parameters is as follows:
if the clip is in H264 format, the mapping relationship between the decoding parameters and the encoding parameters is as follows:
the decoding parameters chroma _ format _ idc of the sequence Parameter set sps (sequence Parameter set) or the picture Parameter set pps (picture Parameter set) correspond to the color space internalCsp Parameter of x264_ param;
the decoding parameter bit _ depth _ luma _ minus8 of the SPS or PPS corresponds to the bit depth i _ bit parameter of x264_ param;
the decoding parameter time _ scale of SPS or PPS corresponds to the frame rate i _ fps _ num parameter of x264_ param;
the decoding parameter pic _ width _ in _ mbs _ minus1 of SPS or PPS corresponds to the wide i _ width parameter of resolution x264_ param;
the decoding parameter pic _ height _ in _ mbs _ minus1 of SPS or PPS corresponds to the high i _ height parameter of the resolution of x264_ param;
the details are shown in the following table:
| SPS/PPS parameter set | x264_ param parameter |
| chroma_format_idc | InternalCsp color space |
| bit_depth_luma_minus8 | i _ bitdepth bit depth |
| time_scale | i _ fps _ num frame rate |
| pic_width_in_mbs_minus1 | Wide i _ width resolution |
| pic_height_in_mbs_minus1 | i _ height resolution is high |
If the clip is in H265 format, the mapping relationship between the decoding parameters and the encoding parameters is as follows:
the decoding Parameter bit _ depth _ chroma of the video Parameter set vps (video Parameter set), SPS or PPS corresponds to the bit depth internalBitDepth Parameter of the encoder;
the decoding parameter width of VPS, SPS or PPS corresponds to the wide sourceWidth parameter of the resolution of the encoder;
the decoding parameter height of VPS, SPS or PPS corresponds to the high sourceHeight parameter of the resolution of the encoder;
the decoding parameter chroma _ format _ idc of the VPS, SPS or PPS corresponds to the color space internalCsp parameter of the encoder;
the decoding parameters fpsDenom of VPS, SPS or PPS correspond to the frame rate numerator vui _ num _ units _ in _ tick parameter of the encoder;
the decoding parameter fpsNum of VPS, SPS or PPS corresponds to the frame rate denominator vui _ time _ scale parameter of the encoder;
the details are shown in the following table:
| VPS/SPS/PPS resolution parameters | Encoder initialization parameters |
| bit_depth_chroma | InternalBitDepth bit depth 8bit/10bit |
| width | sourceWidth resolution is wide |
| height | sourceHeight resolution is high |
| chroma_format_idc | InternalCsp color space ID |
| fpsDenom | vui _ num _ units _ in _ tick frame rate molecule |
| fpsNum | vui _ time _ scale frame rate denominator |
In the embodiment of the application, the initialization parameters of the encoder are parsed and reversely mapped from NAL data packets of VPS/SPS/PPS and the like of the file. The code stream formats of H264 and H265 are basically the same, and H265 has more VPSs than H264, which is specifically as follows:
VPS Video Parameter Set (h265)
SPS Sequence Parameter Set (h264/h265)
PPS Picture Parameter Set Picture Parameter Set (h264/h265)
Through years of accumulation, test and summary, the embodiment of the application provides the mapping relation between the decoding parameters and the coding parameters, and all rapidly synthesized files ensure that each terminal player can play normally.
Example four
Based on the same inventive concept, the embodiment of the present application provides a fast synthesis device based on live stream editing, and the principle of the device for solving the technical problem is similar to that of the fast synthesis method based on live stream editing in the third embodiment, and repeated parts are not described again.
Fig. 4 shows a schematic structural diagram of a fast composition apparatus based on live stream clips in the fourth embodiment of the present application.
As shown, the fast composition device based on live stream clips includes:
a first receiving module 401, configured to receive a live stream fragment file;
a cloud storage module 402, configured to store the live stream fragment file;
a second receiving module 403, configured to receive a composition task of the live broadcast file; the synthesis task comprises access point information of a live broadcast file; the live stream fragment file comprises a live stream fragment file;
and a synthesizing module 404, configured to synthesize the stored live stream file according to the synthesizing task.
The fast synthesis device based on live streaming clip provided in the embodiment of the application adopts a space time-changing mode for a cloud processing process, transmits fragment files firstly, and then synthesizes at the cloud, so that the transmission time is saved, and the overall synthesis issuing efficiency is improved.
In one embodiment, the synthesis module comprises:
an in-point determining unit for determining an in-point of each clip;
the encoding unit is used for decoding from the initial I frame of the GOP where the entry point is located until the current GOP is finished when the entry point of the clip is shot in the P frame or the B frame of the GOP, and re-encoding the data obtained by decoding;
the splicing unit is used for copying and splicing other GOPs of the clip according to the GOP where the entry point is located;
and a synthesizing unit for synthesizing the plurality of clip segments into an edited output.
In one embodiment, further comprising:
the analysis module is used for analyzing and acquiring decoding parameters from network abstraction layer NAL data obtained by decoding before the data obtained by decoding is recoded;
the mapping module is used for comparing the decoding parameters with the decoding parameters of the previous file; and if the decoding parameters are consistent with the decoding parameters, obtaining the coding parameters according to the decoding parameter reverse mapping.
EXAMPLE five
Based on the same inventive concept, the embodiment of the application provides a fast synthesis system based on live stream editing.
Fig. 5 shows a schematic structural diagram of a fast composition system based on live stream clips in the fifth embodiment of the present application.
As shown, the fast composition system based on live stream clips includes:
a first fast composition means, which is the fast composition means based on live stream editing as described in embodiment two;
and the number of the first and second groups,
a plurality of second fast composing means, said second fast composing means being the live-stream-clip-based fast composing means as described in embodiment four;
and the second sending module is used for scheduling resources after receiving the synthesis tasks and sending the synthesis tasks to relatively idle devices in the plurality of second quick synthesis devices.
The fast synthesis system based on live stream editing provided in the embodiment of the application adopts a space time-changing mode, transmits the fragment files to the cloud terminal by the local terminal, and then synthesizes the fragment files at the cloud terminal, so that the transmission time is saved, and the overall synthesis issuing efficiency is improved.
EXAMPLE six
In order to facilitate the implementation of the present application, the embodiments of the present application are described with a specific example.
Fig. 6 shows a schematic diagram of a fast composition architecture based on live stream clips in an embodiment of the present application.
The whole process starts from the live broadcast stream, after relevant live broadcast lists are arranged on the integrated release platform by editors, live broadcast tasks are issued to the encoder, one-in-more-out live broadcast encoding can be carried out on the encoder according to multi-code rate requirements required by the live broadcast arrangement, meanwhile, multi-code rate live broadcast receiving and recording tasks are started, and the multi-code rate live broadcast stream is received and recorded. And after the recorded fragment files are stored locally, uploading the fragment files to a cloud terminal. The editing personnel can see fragment files automatically added to the online recorded time in real time through a fast-pass interface of the integrated publishing platform, edit the contents, complete setting of multiple pairs of input and output points of the whole content through selection of a blue area, submit the edited tasks to the synthesis system after editing is completed, perform dynamic resource scheduling after the synthesis system receives the synthesis tasks, distribute the tasks to cloud synthesis nodes which are as free as possible, perform rapid synthesis processing on the multi-code-rate materials of the cloud by the synthesis nodes of the cloud, store the synthesized files in object storage of the cloud, and finally provide browsing and playing for users.
Live stream clip composition is intended to quickly clip a plurality of segments of one or more files and compose them into one file.
Synthesizing functional options:
1. one file or a plurality of files (each file has a plurality of fragments) are synthesized and edited to be output as one file.
2. One file or a plurality of files (each file having a plurality of fragments) are synthesized into an output multi-file (in units of original files).
Fig. 7 shows an implementation diagram of a fast composition engine in the sixth embodiment of the present application.
As shown in the figure, the in-point of the clip is in the middle of the GOP3, and the embodiment of the application decodes the GOP3 completely, then SEEK is positioned at the in-point, the content of the later part is coded, and then COPY splicing is carried out with other GOPs. Therefore, only the GOP of the entry point is subjected to the re-encoding process, and other GOPs are directly coded by COPY.
Fig. 8 is an enlarged schematic diagram showing the structure of a GOP in the sixth embodiment of the present application.
As shown, a complete GOP is one that can completely decode and render a picture:
i frame: can be decoded independently, i.e. without reference to other frames.
P/B frame: instead of decoding the frames independently, reference must be made to the previous I-frame and other P/B-frames.
In the embodiment of the present application, when an in-point of a segment of a fast clip is marked in an I-frame, i.e., the start of a GOP, all frame data of the segment does not need to be re-encoded.
When the entry point of the segment is printed on the P/B frame, decoding must be started from the initial I frame of the GOP where the entry point is positioned, and the decoded YUV original data is re-encoded from the entry point to the end of the GOP (the exit point is not in the current GOP) and sent to a multiplexer (muxer). Since the subsequent data starts with an I-frame, it can be sent directly to a multiplexer (muxer) without encoding.
FIG. 9 shows a schematic diagram of the process of rapid synthesis in example six of the present application.
Firstly, opening a file, carrying out demux demultiplexing, acquiring NAL UNIT packets such as VPS/SPS/PPS and the like, analyzing and acquiring decoding parameters, comparing the decoding parameters with the above stable decoding parameters, and judging whether the two file parameters are consistent;
if the two codes are consistent, the coding parameters are mapped reversely, and the coder is initialized by utilizing the coding parameters;
if not, the program is exited.
Then, seek the access point of segment current segment GOP, fetch the frame, judge whether need to reprogram (the concrete judging process is whether to play in P/B frame according to the access point), if need to reprogram, encode after decoding; if the reprogramming is not needed, directly carrying out the next step;
and then multiplexing the mux write file, judging whether a segment exit point is reached, namely whether the GOP of the current segment is traversed completely, and if the segment exit point of the current GOP is reached, further judging whether the current segment of the current live stream file is traversed completely.
And finally, processing the next file after traversing the current file segment. And exiting the program after traversing all the files.
The embodiment of the application provides a rapid synthesis engine. The method processes a plurality of pairs of out-in points, only partially re-encodes the damaged GOP, and completes the whole synthesis splicing by adopting Copy mode for other GOP content parts. The synthesis supports various media container formats such as AVI, WMV, MPG, MP5, RM, FLV and the like, supports the current popular H.265 video format and 4k high-rate video, and supports the Dolby Atoms panoramic audio coding format. The fast synthesis technology of multi-code rate frame synchronization is supported, the high-speed synthesis process of frame precision is finally achieved, and the fast synthesis engine only needs about 20 seconds to generate a finished program for one hour.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein. The scheme in the embodiment of the application can be implemented by adopting various computer languages, such as object-oriented programming language Java and transliterated scripting language JavaScript.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the application.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (10)
1. A fast synthesis method based on live stream clips is characterized by comprising the following steps:
receiving and recording the fragment files of the live stream;
locally storing the live streaming fragment file and uploading the live streaming fragment file to a cloud terminal;
finishing the editing of the live stream file according to the locally stored live stream fragment file; the live stream fragment file comprises a live stream fragment file;
generating a synthesis task after finishing the editing and sending the synthesis task to the cloud end; the synthesis task comprises access point information of live files.
2. The method of claim 1, further comprising, prior to the receiving the listing of the live stream shard file:
performing one-in-multiple-out live broadcast coding on the live broadcast stream by using an encoder according to the multi-code-rate requirement of live broadcast arrangement to obtain a multi-code-rate live broadcast stream;
and starting a multi-code-rate live broadcast receiving and recording task.
3. A fast synthesis method based on live stream clips is characterized by comprising the following steps:
receiving and storing the fragment files of the live stream;
receiving a synthesis task of a fragment file of a live stream; the synthesis task comprises access point information of a live broadcast file; the live files comprise live stream fragment files;
and synthesizing the stored live streaming fragment files according to the synthesis task.
4. The method of claim 3, wherein the synthesizing the saved live stream fragment file comprises:
determining an in-point for each clip;
when an entry point of a clip is positioned in a P frame or a B frame of a GOP, decoding is started from an initial I frame of the GOP where the entry point is positioned until the current GOP is finished, and data obtained by decoding is re-encoded;
copying and splicing other GOPs of the clip according to the GOP where the entry point is located;
and synthesizing the plurality of clip segments into an edited output.
5. The method of claim 3, wherein the synthesizing the saved live stream fragment file further comprises:
when an in-point of a clip is placed in an I-frame of a GOP, the clip does not need to be re-encoded.
6. The method of claim 4, wherein prior to said re-encoding the decoded data, comprising:
analyzing and acquiring decoding parameters from network abstraction layer NAL data obtained by decoding;
comparing the decoding parameters with the decoding parameters of the last file;
and if the decoding parameters are consistent with the decoding parameters, obtaining the coding parameters according to the decoding parameter reverse mapping.
7. The method of claim 6, wherein the mapping relationship between the decoding parameters and the encoding parameters is as follows:
if the clip is in H264 format, the mapping relationship between the decoding parameters and the encoding parameters is as follows:
the decoding parameter chroma _ format _ idc of the sequence parameter set SPS or the picture parameter set PPS corresponds to the color space internalCsp parameter of x264_ param;
the decoding parameter bit _ depth _ luma _ minus8 of the SPS or PPS corresponds to the bit depth i _ bit parameter of x264_ param;
the decoding parameter time _ scale of SPS or PPS corresponds to the frame rate i _ fps _ num parameter of x264_ param;
the decoding parameter pic _ width _ in _ mbs _ minus1 of SPS or PPS corresponds to the wide i _ width parameter of resolution x264_ param;
the decoding parameter pic _ height _ in _ mbs _ minus1 of SPS or PPS corresponds to the high i _ height parameter of the resolution of x264_ param;
if the clip is in H265 format, the mapping relationship between the decoding parameters and the encoding parameters is as follows:
the decoding parameter bit _ depth _ chroma of the video parameter set VPS, SPS or PPS corresponds to the bit depth internalBitDepth parameter of the encoder;
the decoding parameter width of VPS, SPS or PPS corresponds to the wide sourceWidth parameter of the resolution of the encoder;
the decoding parameter height of VPS, SPS or PPS corresponds to the high sourceHeight parameter of the resolution of the encoder;
the decoding parameter chroma _ format _ idc of the VPS, SPS or PPS corresponds to the color space internalCsp parameter of the encoder;
the decoding parameters fpsDenom of VPS, SPS or PPS correspond to the frame rate numerator vui _ num _ units _ in _ tick parameter of the encoder;
the decoding parameter fpsNum of the VPS, SPS or PPS corresponds to the frame rate denominator vui _ time _ scale parameter of the encoder.
8. A fast composition apparatus based on live stream editing, comprising:
the receiving and recording module is used for receiving and recording the fragment files of the live stream;
the local storage module is used for locally storing the fragment files of the live streaming;
the first sending module is used for uploading the live streaming fragment file to a cloud;
the editing module is used for finishing editing of the live stream file according to the locally stored live stream fragment file; the live stream fragment file comprises a live stream fragment file;
the task generation module is used for generating a synthesis task after finishing the editing; the synthesis task comprises access point information of a live broadcast file;
and the second sending module is used for sending the synthesis task to a cloud.
9. A fast composition apparatus based on live stream editing, comprising:
the first receiving module is used for receiving the fragment file of the live stream;
the cloud storage module is used for storing the live streaming fragment file;
the second receiving module is used for receiving a synthesis task of the live broadcast file; the synthesis task comprises access point information of a live broadcast file; the live stream fragment file comprises a live stream fragment file;
and the synthesis module is used for synthesizing the stored live streaming file according to the synthesis task.
10. A fast composition system based on live stream editing, comprising:
first fast compositing means, said first fast compositing means being live-stream clip-based fast compositing means as claimed in claim 8;
and the number of the first and second groups,
a plurality of second fast composing means, said second fast composing means being live-stream-clip-based fast composing means as claimed in claim 9;
and the second sending module is used for scheduling resources after receiving the synthesis tasks and sending the synthesis tasks to relatively idle devices in the plurality of second quick synthesis devices.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010469247.3A CN113747248A (en) | 2020-05-28 | 2020-05-28 | Rapid synthesis method, device and system based on live stream editing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010469247.3A CN113747248A (en) | 2020-05-28 | 2020-05-28 | Rapid synthesis method, device and system based on live stream editing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113747248A true CN113747248A (en) | 2021-12-03 |
Family
ID=78724199
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010469247.3A Pending CN113747248A (en) | 2020-05-28 | 2020-05-28 | Rapid synthesis method, device and system based on live stream editing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113747248A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103531218A (en) * | 2013-04-17 | 2014-01-22 | Tcl集团股份有限公司 | Online multimedia file editing method and system |
| CN106803992A (en) * | 2017-02-14 | 2017-06-06 | 北京时间股份有限公司 | Video clipping method and device |
| CN106937129A (en) * | 2015-12-31 | 2017-07-07 | 广州华多网络科技有限公司 | A kind of live real-time clipping method of Online Video and device |
| CN107734354A (en) * | 2017-10-19 | 2018-02-23 | 中视亚广（北京）文化传播有限公司 | It is a kind of that real-time clipping method and device are carried out to live stream video |
-
2020
- 2020-05-28 CN CN202010469247.3A patent/CN113747248A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103531218A (en) * | 2013-04-17 | 2014-01-22 | Tcl集团股份有限公司 | Online multimedia file editing method and system |
| CN106937129A (en) * | 2015-12-31 | 2017-07-07 | 广州华多网络科技有限公司 | A kind of live real-time clipping method of Online Video and device |
| CN106803992A (en) * | 2017-02-14 | 2017-06-06 | 北京时间股份有限公司 | Video clipping method and device |
| CN107734354A (en) * | 2017-10-19 | 2018-02-23 | 中视亚广（北京）文化传播有限公司 | It is a kind of that real-time clipping method and device are carried out to live stream video |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP3662129B2 (en) | Multimedia information editing device | |
| CN104602117B (en) | A kind of speed plays the method and device of video | |
| US20170309310A1 (en) | Method and device for compressed-domain video editing | |
| CN103024603B (en) | A kind of for solving playing network video time the device and method that pauses in short-term | |
| WO2017092340A1 (en) | Method and device for compressing and playing video | |
| CN102984465B (en) | A kind of program synthesis system and method being applicable to network-enabled intelligent Digital Media | |
| CN1264120A (en) | Digital recording and replaying device | |
| JP5011017B2 (en) | Image decoding device | |
| CN101859585A (en) | System and method for frame-accuracy cutting of video material | |
| CN112653904B (en) | Rapid video clipping method based on PTS and DTS modification | |
| CN105791938A (en) | Multimedia file splicing method and apparatus | |
| JPH10271446A (en) | Method and device for sticking audiovisual segments, method for joining audiovisual segments, method for copying segment from audiovisual file, device for cutting segment from audiovisual file, and computer readable medium | |
| US20090232470A1 (en) | Video contents editing apparatus, controller, and method | |
| KR101142379B1 (en) | Method and Apparatus of playing digital broadcasting and Method of recording digital broadcasting | |
| JP6269734B2 (en) | Movie data editing device, movie data editing method, playback device, and program | |
| CN113747248A (en) | Rapid synthesis method, device and system based on live stream editing | |
| JP5269063B2 (en) | Video server and seamless playback method | |
| Kikuchi et al. | Development of full-featured 8K recorder | |
| JP3880597B2 (en) | Multimedia information editing device | |
| JP5066557B2 (en) | Video decoding method, video decoding apparatus, and video decoding program | |
| JP3880596B2 (en) | Multimedia information editing device | |
| Takeuchi et al. | A program production system using ID and file-data over IP networks | |
| JP2023040447A (en) | Video sending system and video editing device | |
| US20110019973A1 (en) | Editing apparatus, editing method and program | |
| CN117354533A (en) | Transparent video processing method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |