WO2012034442A1

WO2012034442A1 - System and method for realizing synchronous transmission and reception of scalable video coding service

Info

Publication number: WO2012034442A1
Application number: PCT/CN2011/076622
Authority: WO
Inventors: 童登金; 谢文军; 戴志军
Original assignee: 中兴通讯股份有限公司
Priority date: 2010-09-17
Filing date: 2011-06-30
Publication date: 2012-03-22
Also published as: CN101951506A; CN101951506B

Abstract

A transmission method is provided in the present invention. The method includes that: a video service is coded to generate multi-path media streams which are respectively encapsulated in different multiplex sub-frames in a broadcast channel frame by taking a media unit as a unit; a mobile multi-media broadcast timestamp of each media unit encapsulated in the multiplex sub-frames is encapsulated; the mobile multi-media broadcast timestamps of the media streams at the same sampling time are synchronized; the position information of the multiplex sub-frames in which the media streams are located is encapsulated in the broadcast channel frame and is transmitted to a terminal; the media streams comprise a base layer code stream and enhancement layer code streams corresponding to the base layer code stream generated by coding video stream of the video service, and may also comprise an audio stream of the video service; and thereby synchronous transmission of a scalable video coding service is realized in the mobile multi-media broadcast. A corresponding transmission system is also provided in the present invention. A reception method and a device of the scalable video coding service, which align the received media data according to the mobile multi-media broadcast timestamp, are also provided in the present invention.

Description

System and method for realizing synchronous transmission and reception of scalable video coding service

Technical field

The present invention relates to a transmission technology in a mobile multimedia broadcasting system, and more particularly to a scalable mobile video coding service (SVC, Scalable Video Coding) service in a China Mobile Multimedia Broadcasting System (CMMB), which implements a scalable video coding service. Systems and methods for simultaneous transmission and reception. Background technique

With the development of communication technology, the application of mobile multimedia broadcasting technology has become wider and wider. At present, the China Mobile Multimedia Broadcasting System Standard specifies the frame structure, channel coding and modulation of the broadcast channel transmission signal of the mobile multimedia broadcasting system within the frequency range of the broadcasting service. The CMMB standard "Mobile Multicast Broadcasting Part 2: Multiplexing" specifies that multiplexed sub-frames are used to encapsulate streaming media such as video and audio for transmission.

Scalable Video Coding (SVC) is a video grading coding method. The encoder encodes the video content source to generate a plurality of layers of code streams, and the base layer code stream can be separately decoded. The enhancement layer code stream includes additional information for improving the quality of the lower layer code stream, and needs a low layer including the base layer. Decode together. SVC technology can provide scalable and scalable services, differentiate services with different quality of service, and adapt to the capabilities of various terminals. It has many advantages. Therefore, it is necessary to transmit SVC services in the CMMB system.

The patent application number is 200910088679.3, "Classification Transmission and Reception Method and Apparatus in Mobile Multimedia Broadcasting System", and a method for implementing SVC transmission in CMMB is given. According to the method, the base layer code stream in the SVC video service can be used. Layered transmission with the enhancement layer code stream, and the base layer code stream and the enhancement layer code stream are respectively encapsulated in different multiplex subframes in the broadcast channel frame according to the layer to which they belong, and the code stream of each layer of the video stream is simultaneously The location information of the multiplex subframe in which it is located is encapsulated in the broadcast channel frame and transmitted to the receiving end. The terminal monitors location information of a multiplex subframe in which each layer of the video stream in the video service in the broadcast channel frame is located, and the receiving terminal receives the base layer code stream according to the video stream processing capability of the broadcast channel, or receives the base layer code stream and Corresponding enhancement layer code stream, for base layer code stream, Or decoding the base layer code stream and the corresponding enhancement layer code stream, and outputting the video data of the base layer code stream or the video data of the base layer code stream and the enhancement layer code stream.

The SVCs are separated and transmitted in different multiplex subframes. The synchronization and coordination of each layer is a problem that must be solved. When the multiplexing frames to which the multiplex subframes in which the layer streams are located belong to different multiplex frames, how to perform synchronization is a problem that must be solved.

In an SVC video, an access unit at a specific time point, such as a video frame, may be encoded into multiple layers, which are separately transmitted in different multiplex subframes; on the terminal side, it is required to pass different complexes. The coding units of different layers transmitted by the subframe are combined, such as combining multiple layers of the same video frame, and then performing video decoding presentation. When merging, each layered coding unit must be synchronized to ensure that the coding unit of the same access unit is involved in the merging, and the success of the merging is ensured, and the merging success can ensure the success of the subsequent decoding.

What the present invention needs to solve is how to ensure synchronization between the multiple SVC layers when transmitting SVC layered services in the CMMB. Summary of the invention

The technical problem to be solved by the present invention is to provide a system and method for implementing SVC video synchronous transmission and reception when transmitting SVC layered services in a CMMB, and to ensure separate SVC layered data transmitted in different multiplex subframes. Synchronization between the two, thus ensuring the normal implementation of the SVC service in the CMMB.

In order to solve the above problems, the present invention provides a sending method, including:

The video service is encoded to generate a multi-channel media stream, and the multiple media streams are respectively encapsulated in different multiplex subframes in a broadcast channel frame in units of media units, and the multiplexer is also carried in the multiplex subframe. a mobile multimedia broadcast timestamp of each media unit encapsulated in the frame, and a mobile multimedia broadcast timestamp of the media stream at the same time instant is synchronized, and the location information of the multiplexed subframe in which the media stream is located is encapsulated in the broadcast In the channel frame, the broadcast channel frame is sent to the receiving terminal, where the media stream includes a basic layer code stream generated by the video stream coding of the video service and its corresponding enhancement layer code stream, or includes audio of a video service. The basic layer code stream generated by the video stream coding of the stream and video service and its corresponding enhancement layer code stream; thereby implementing the scalable video coding industry in the mobile multimedia broadcast Send synchronously.

Preferably, the foregoing method may further have the following feature: the media stream is encapsulated into the multiplex subframe according to the following manner:

Encapsulating the multi-channel media stream into a multi-channel real-time transport protocol (RTP) code stream, where each RTP code stream is accompanied by a real-time transport control protocol (RTCP) code stream, and the RTCP code stream ensures the same sample time Network Time Protocol (NTP) time synchronization of media streams;

Extracting the media stream encapsulated in the RTP code stream, and encapsulating the media stream in different multiplex subframes in a broadcast channel frame in units of media units, and converting the RTP timestamp of the media stream into an NTP time And converting the NTP time into a mobile multimedia broadcast timestamp under the unified time reference, and packaging the mobile multimedia broadcast timestamp into a multiplex subframe in which the corresponding media unit is located.

Preferably, the foregoing method may further have the following feature: converting the RTP timestamp into a mobile multimedia broadcast timestamp as follows: For each media unit, extracting an RTP timestamp of the RTP packet in which it is located, and corresponding to the RTP code stream corresponding thereto Calculating the time information of the RTCP packet transmitted by the RTCP code stream, and calculating the NTP time of the media unit;

Multiplying the NTP time of the media unit by the mobile multimedia broadcast time scale to obtain a mobile multimedia broadcast time stamp of the media unit.

Preferably, the foregoing method may further have the following feature: the mobile multimedia broadcast timestamp synchronization of the media stream at the same sampling time refers to: the difference between the mobile multimedia broadcast timestamp values of the media streams at the same sampling time is The preset time stamp tolerance value range. The present invention also provides a transmitting system, where the system includes an encoding device and a front-end transmitting device, where:

The encoding device is configured to: encode a video service to generate a multi-media media stream; the media stream includes a base layer code stream generated by the video stream coding of the video service and a corresponding enhancement layer code stream, or include the An audio stream of a video service, a base layer code stream generated by the video stream coding of the video service, and a corresponding enhancement layer code stream;

The front-end transmitting device is configured to: separately package the media stream in units of media units In a different multiplex subframe in the broadcast channel frame, the mobile multimedia broadcast timestamp of each media unit encapsulated in the multiplex subframe is encapsulated in the multiplex subframe, and the media stream moves at the same time The multimedia broadcast timestamp is synchronized, and the location information of the multiplex subframe in which the media stream is located is encapsulated in the broadcast channel frame, and the broadcast channel frame is sent to the receiving terminal; thereby implementing scalability in the mobile multimedia broadcast The video encoding service is sent synchronously.

Preferably, the above system may further have the following features, the encoding device comprising an encoding unit and a packaging unit, wherein:

The encoding unit is configured to: encode a video service to generate a multi-channel media stream;

The encapsulating unit is configured to: encapsulate the multi-path media stream into a multi-channel real-time transport protocol (RTP) code stream, where each RTP code stream is accompanied by a real-time transport control protocol (RTCP) code stream, and the RTCP The code stream guarantees network time protocol (NTP) time synchronization of the media stream at the same time;

The front-end transmitting device includes a first encapsulating unit, a second encapsulating unit, a converting unit, a third encapsulating unit, and a sending unit, where:

The first encapsulating unit is configured to: extract media streams encapsulated in the RTP code stream, and encapsulate the media streams in different multiplex sub-frames in a broadcast channel frame in units of media units;

The second encapsulating unit is configured to: encapsulate location information of a multiplex subframe in which the media stream is located in the broadcast channel frame;

The converting unit is configured to: convert the RTP timestamp of the media stream into an NTP time, and then convert the NTP time into a mobile multimedia broadcast timestamp under the unified time reference;

The third encapsulating unit is configured to: encapsulate the mobile multimedia broadcast timestamp into a multiplex subframe in which the corresponding media unit is located; and the sending unit is configured to: send the broadcast channel frame to the receiving terminal.

Preferably, the above system may further have the following features, the conversion unit includes a first conversion unit and a second conversion unit, wherein:

The first converting unit is configured to: for each media unit, take out an RTP timestamp of the RTP packet in which it is located, and calculate time information of the RTCP packet transmitted by the RTCP code stream corresponding to the RTP stream in which the RTP stream is located, and calculate the media unit. NTP time; The second converting unit is configured to: multiply the NTP time of the media unit by a mobile multimedia broadcast time scale to obtain a mobile multimedia broadcast time stamp of the media unit.

Preferably, the above system may also have the following features: the mobile multimedia broadcast timestamp synchronization of the media stream at the same sampling time refers to: the difference between the mobile multimedia broadcast timestamp values of the media streams at the same sampling time is The preset time stamp tolerance value range. The invention also provides a receiving method, comprising:

Receiving, by the receiving terminal, location information of a multiplex sub-frame in which each layer of the video stream in the video service in the broadcast channel frame is located;

The receiving terminal receives the basic layer code stream according to its own video stream processing capability, or receives the basic layer code stream and the corresponding enhancement layer code stream;

When receiving the base layer code stream and the corresponding enhancement layer code stream, the layer code streams are aligned and combined according to the mobile multimedia broadcast timestamp, and then the base layer code stream and the corresponding enhancement layer code stream are decoded, and the basic layer code stream is output. The video data combined with the enhancement layer code stream; thereby implementing scalable video coding service reception in the mobile multimedia broadcast.

Preferably, the foregoing method may further have the following feature: the receiving the base layer code stream and the corresponding enhanced layer code stream, and aligning the layer code streams according to the mobile multimedia broadcast timestamp comprises:

The receiving terminal will receive the base layer code stream and the enhancement layer code stream and store them in the buffer;

Extracting base layer code stream data belonging to a video access unit from a buffer, and extracting a mobile multimedia broadcast time with the base layer code stream data based on a mobile multimedia broadcast timestamp of the base layer code stream data The synchronized enhancement layer code stream data is stamped and combined as data of the same video access unit.

Preferably, the foregoing method may further have the following feature: the enhanced layer code stream data synchronized with the mobile multimedia broadcast timestamp of the base layer code stream data refers to a mobile multimedia broadcast time with the base layer code stream data. The enhancement layer code stream data within the preset time stamp tolerance value difference.

The invention also provides a receiving device, comprising:

a monitoring unit, configured to: monitor location information of a multiplex subframe in which each layer of the video stream in the video service in the broadcast channel frame is located; a receiving unit, configured to: receive a base layer code stream according to its own video stream processing capability, or receive a base layer code stream and a corresponding enhancement layer code stream;

Aligning the merging unit, which is set to: align each layer code stream according to the mobile multimedia broadcast time stamp;

a decoding unit, configured to: decode the base layer code stream and the corresponding enhancement layer code stream, and output video data combined by the base layer code stream and the enhancement layer code stream; thereby implementing scalable video coding service reception in the mobile multimedia broadcast .

Preferably, the foregoing apparatus may further have the following feature, the receiving unit is further configured to: store the received base layer code stream and the enhancement layer code stream in a buffer;

The alignment merging unit is configured to: extract base layer code stream data belonging to a video access unit from a buffer, and extract the base layer based on a mobile multimedia broadcast timestamp of the base layer code stream data. The mobile multimedia broadcast time-stamped enhanced layer code stream data of the code stream data is combined as data of the same video access unit.

Preferably, the foregoing apparatus may further have the following feature, the extracting the enhanced layer code stream data synchronized with the mobile multimedia broadcast timestamp of the base layer code stream data, and extracting the mobile multimedia with the base layer code stream data The enhancement layer code stream data whose difference between the broadcast time stamps is within a preset time stamp tolerance value range.

In the method of the embodiment of the present invention, synchronous CMMB broadcast time stamps are set in different layered SVC data to ensure synchronization between different hierarchical data. BRIEF abstract

1 is a schematic structural diagram of a CMMB channel frame;

2 is a schematic diagram of the system of the present invention;

3 is a schematic structural diagram of a structure of a broadcast channel frame according to the present invention;

FIG. 4 is a schematic diagram of a terminal processing function according to the present invention. In the SVC video frame synthesis, the base layer video unit and the enhancement layer video unit participating in the synthesis must be synchronized. Preferred embodiment of the invention The basic idea of the embodiment of the present invention is to perform synchronous SCM service synchronization by applying synchronized CMMB timestamps on the audio stream, the base layer code stream, and the corresponding enhancement layer code stream at the same time.

An embodiment of the present invention provides a method for implementing synchronous transmission of a scalable video coding service in a mobile multimedia broadcast, including:

The video service is encoded to generate a multi-channel media stream, and the multiple media streams are respectively encapsulated in different multiplex subframes in a broadcast channel frame in units of media units, and the multiplexer is also carried in the multiplex subframe. a mobile multimedia broadcast timestamp of each media unit encapsulated in the frame, and a mobile multimedia broadcast timestamp of the media stream at the same time instant is synchronized, and the location information of the multiplexed subframe in which the media stream is located is encapsulated in the broadcast In the channel frame, the broadcast channel frame is sent to the receiving terminal, where the media stream includes a base layer code stream generated by the video stream coding and its corresponding enhancement layer code stream, or includes a basic generation of the audio stream and the video stream coding. Layer code stream and its corresponding enhancement layer code stream.

The media stream is encapsulated into the multiplex subframe according to the following manner:

The RTP timestamp is converted into a mobile multimedia broadcast timestamp as follows: For each media unit, the RTP timestamp of the RTP packet in which it is located is extracted, and the RTCP packet transmitted by the RTCP code stream corresponding to the RTP stream in which it is located is related. Time information, calculating an NTP time of the media unit;

The mobile multimedia broadcast timestamp of the media unit includes two parts: a start play time and a relative play time corresponding to each media unit. Start of each media unit within the same multiplex sub-frame Play time is the same.

The mobile multimedia broadcast timestamp synchronization of the media stream at the same sampling time refers to: the difference between the mobile multimedia broadcast timestamp values of the media streams at the same sampling time is within a preset time stamp tolerance range Inside.

An embodiment of the present invention further provides a system for implementing synchronous transmission of a scalable video coding service in a mobile multimedia broadcast, where the system includes an encoding device and a front-end transmitting device, where:

The encoding device is configured to encode a video service to generate a multi-path media stream; the media stream includes a base layer code stream generated by the video stream coding and a corresponding enhancement layer code stream, or includes an audio stream and a video stream coding generation Base layer code stream and its corresponding enhancement layer code stream;

The front-end transmitting device is configured to encapsulate the media stream in different multiplex subframes in a broadcast channel frame in units of media units, and move multimedia broadcast time of each media unit encapsulated in the multiplex subframe Stamping is encapsulated in the multiplexed subframe, and the mobile multimedia broadcast timestamp of the media stream at the same time instant is synchronized, and the location information of the multiplexed subframe in which the media stream is located is encapsulated in the broadcast channel frame. And transmitting the broadcast channel frame to the receiving terminal.

The encoding device includes an encoding unit and a packaging unit, where:

The coding unit is configured to encode a video service to generate a multi-channel media stream;

The encapsulating unit is configured to encapsulate the multi-path media stream into a multi-channel real-time transport protocol (RTP) code stream, where each RTP code stream is accompanied by a real-time transport control protocol (RTCP) code stream, and the RTCP The code stream guarantees network time protocol (NTP) time synchronization of the media stream at the same time;

a first encapsulating unit, configured to extract a media stream encapsulated in the RTP code stream, and encapsulate the media stream in different multiplex subframes in a broadcast channel frame in units of media units; the second encapsulating unit And for encapsulating location information of the multiplex subframe in which the media stream is located in the broadcast channel frame;

The converting unit is configured to convert an RTP timestamp of the media stream into an NTP time, and then convert the NTP time into a mobile multimedia broadcast timestamp under a unified time reference; The third encapsulating unit is configured to encapsulate the mobile multimedia broadcast timestamp into a multiplex subframe in which the corresponding media unit is located, and the sending unit is configured to send the broadcast channel frame to the receiving terminal.

The converting unit includes a first converting unit and a second converting unit, where: the first converting unit is configured to: for each media unit, take out an RTP timestamp of an RTP packet in which it is located, and combine the RTP stream in which the RTP stream is located. Corresponding time information of the RTCP packet transmitted by the corresponding RTCP code stream, and calculating an NTP time of the media unit;

The second converting unit is configured to multiply the NTP time of the media unit by a mobile multimedia broadcast time scale to obtain a mobile multimedia broadcast timestamp of the media unit.

An embodiment of the present invention further provides a method for implementing a scalable video coding service in a mobile multimedia broadcast, including:

When receiving the base layer code stream and the corresponding enhancement layer code stream, the layer code streams are aligned and combined according to the mobile multimedia broadcast timestamp, and then the base layer code stream and the corresponding enhancement layer code stream are decoded, and the basic layer code stream is output. Video data combined with the enhancement layer stream. The receiving the base layer code stream and the corresponding enhancement layer code stream, and aligning the layer code streams according to the mobile multimedia broadcast timestamp includes:

The enhanced layer code stream data synchronized with the mobile multimedia broadcast timestamp of the base layer code stream data refers to a mobile multimedia broadcast timestamp of the enhancement layer code stream data and the mobile multimedia of the base layer code stream data. The difference between the broadcast timestamps is within the preset timestamp tolerance value range. An embodiment of the present invention further provides a device for implementing a scalable video coding service in a mobile multimedia broadcast, including:

a monitoring unit, which monitors location information of a multiplex subframe in which each layer of the video stream in the video service in the broadcast channel frame is located;

The receiving unit receives the basic layer code stream according to its own video stream processing capability, or receives the basic layer code stream and the corresponding enhancement layer code stream;

The aligning unit is configured to align the code streams of each layer according to the time limit of the mobile multimedia broadcast; the decoding unit decodes the base layer code stream and the corresponding enhancement layer code stream, and combines the output base layer code stream and the enhancement layer code stream. Video data.

The receiving unit is further configured to store the received base layer code stream and the enhancement layer code stream in a buffer;

The alignment merging unit is configured to extract base layer code stream data belonging to a video access unit from a buffer, and extract the base layer based on a mobile multimedia broadcast timestamp of the base layer code stream data. The mobile multimedia broadcast time-stamped enhanced layer code stream data of the code stream data is combined as data of the same video access unit.

The extracting the enhanced layer code stream data synchronized with the mobile multimedia broadcast timestamp of the base layer code stream data, the difference between the mobile multimedia broadcast timestamp and the mobile layer broadcast timestamp of the base layer code stream data is The enhancement layer code stream data within the preset time stamp tolerance value range.

An embodiment of the present invention provides a system for implementing multiple hierarchical synchronization of a scalable video coding service, including an encoding device, a front-end transmitting device, and a terminal, where:

The encoding device is configured to encode a video source, generate an SVC code stream including a base layer and a plurality of enhancement layers, and the plurality of layered SVC code streams are encapsulated into multiple RTP (Real Time Transport Protocol) code streams, and Each RTP code stream is accompanied by an RTCP (Real Time Transmission Control Protocol) code stream, which is used to ensure synchronization of audio and video services in the base layer over NTP (Network Time Protocol), and multiple layered SVC services. Synchronization at NTP (Network Time Protocol) time, that is, to ensure synchronization of the base layer code stream and the corresponding enhancement layer code stream at the same time of NTP time, if there is an audio stream, the audio of the same time is also guaranteed. The stream is synchronized with the base layer code stream and the corresponding enhancement layer code stream at NTP time. The front-end transmitting device is configured to receive an RTP code stream and an RTCP code stream sent by the encoding device, extract an SVC base layer code stream and an enhancement layer code stream from the RTP code stream, and use the SVC base layer code stream and the enhancement layer code stream. The CMMB broadcast timestamps of the video units encapsulated in the multiplexed sub-frames are also carried in the multiplexed subframes of the broadcast channel frame. And storing location information of the multiplex subframe in which each layer of the video stream is located in the broadcast channel frame. On the device, the RTP timestamp of the media data is converted into a CMMB broadcast timestamp under the unified time reference, and the synchronization of the SVC layered services on the CMMB broadcast timestamp is controlled, and the media data to be sent together with the media data is controlled. The CMMB broadcast timestamp is broadcasted together.

The terminal is configured to listen to location information of a multiplex subframe where each layer of the video stream in the video service in the broadcast channel frame is located, and receive the base layer code stream according to the video stream processing capability of the broadcast channel, or receive the base layer code. Stream and corresponding enhancement layer code stream. The coding units of different layers are synchronized according to the CMMB broadcast time stamp, and then video decoding is performed.

The embodiment of the invention further provides a method for implementing multiple hierarchical synchronization of a scalable video coding service, including:

The encoding device encapsulates the plurality of layered SVC code streams generated by the encoding into a plurality of RTP code streams, and sends an RTCP code stream for each RTP code stream. The RTCP code stream ensures synchronization of the base layer SVC data, enhancement layer SVC data, and audio data at the same time instant at the NTP time;

The front-end transmitting device receives the RTP code stream and the RTCP code stream sent by the encoding device, extracts the encapsulated SVC service data from the RTP code stream, and encapsulates the SVC base layer code stream and the enhancement layer code stream into the broadcast channel according to the layer to which they belong. In the different multiplex subframes in the frame, the location information of the multiplex subframe in which the layer stream of each layer of the video stream is located is encapsulated in the broadcast channel frame. On the device, the RTP timestamp of the media data is converted into a CMMB broadcast timestamp under the unified time reference, and the synchronization of the SVC hierarchical services on the CMMB broadcast timestamp is controlled, and the media data to be sent together with the media data is controlled. The CMMB broadcast timestamp is broadcasted together.

The terminal monitors the location information of the multiplex subframe in which the layer streams of the video stream in the video service in the broadcast channel frame are located, and receives the base layer code stream according to the video stream processing capability of the broadcast channel, or receives the base layer code stream and the corresponding enhancement. Layer code stream, aligning coding units of different layers by CMMB broadcast timestamp, After the synchronization, the merge is performed, and then the video decoding is performed.

On the front-end sending device, the SVC layered services are synchronized on the CMMB broadcast timestamp, as follows:

For each multiplex subframe corresponding to each layer of the SVC,

A1. The received RTP and RTCP streams are calculated according to the time information of the RTCP packet transmitted by the RTP packet corresponding to the RTP packet corresponding to the RTP packet, and the NTP time corresponding to the video unit is calculated.

Bl. For each video unit, multiply the CMMB time scale by the above NTP time, and round up the result according to the maximum number of CMMB broadcast timestamps to obtain the CMMB broadcast timestamp of the video unit.

For each multiplex subframe, a start time is taken for all time-stamped media units to be encapsulated into the multiplex subframe, and the CMMB broadcast time stamp of each video unit is decomposed into multiplex subframes. The start play time and the relative play time corresponding to each unit are two parts, and the start play time and the relative play time corresponding to each unit are encapsulated into the multiplex subframe. The CMMB time scale described in step B 1 indicates the number of CMMB time units per second. According to the CMMB standard, Mobile Multimedia Broadcasting Part 2: Multiplexing, the CMMB time scale is 22,500. The maximum number of CMMB broadcast timestamps specified by the standard is 32 bits.

The CMMB broadcast timestamp is similar to the RTP timestamp and is a relative timestamp. The initial play time in the multiplex subframe indicates a play time reference of each audio unit or video unit in the multiplex subframe, and the relative play time of each audio unit or video unit indicates each of the multiplex subframes. The offset of the playing time of the audio unit or the video unit relative to the starting playing time, and the relative time relationship of playing of each audio unit or video unit in one multiplexed sub-frame may be determined according to the initial playing time and the relative playing time. However, when a service such as SVC is transmitted with a multiplexed sub-frame stream, since the CMMB broadcast timestamp is a relative timestamp suitable for use in a single-way multiplexed sub-frame stream, the multiplexed sub-frame stream Inter-subframes have a common reference for their timestamps, lacking a starting reference for the timestamp of the SVC traffic transmitted in the multiplexed sub-frame stream, so that synchronization between the multiplexed sub-frame streams is not possible .

In the embodiment of the present invention, the CMMB broadcast timestamp corresponding to the video unit is converted by directly multiplying the CMMB time scale by the NTP time, because the NTP time of the video unit at the same time is the same Steps, thereby ensuring that the video units of the same sample time are synchronized with their CMMB broadcast timestamps. The CMMB broadcast timestamp can be jointly represented by the initial play time and the relative play time. The start play time in the same multiplex sub-frame is the same, and the start play time of different multiplex sub-frames can be the same or different.

In the embodiment of the present invention, the CMMB broadcast timestamp start value of the SVC service may be determined by using the NTP time of the first video unit in the processed first multiplex subframe (other time may also be used as the CMMB broadcast timestamp). The starting value), and the CMMB broadcast timestamp of the subsequent video unit is determined by the conversion method.

In the embodiment of the present invention, in such an easy-to-implement method, a common timestamp reference of the SVC service transmitted by the multiplexed sub-frame stream is established, and each video in the multiplex sub-frame of the same service is established. The unit determines the play time offset of the unit relative to the CMMB broadcast timestamp start value relative to the deviation of the CMMB broadcast timestamp start value according to the CMMB broadcast timestamp converted according to the above method. The invention can ensure the unification of the SVC hierarchical service data at the same NTP time point in the CMMB broadcast timestamp, thereby providing a guarantee means for synchronizing between different layers of the SVC.

According to the method, each multiplex subframe corresponding to each layer of the SVC can be processed independently, and there is no coupling dependency between them.

For the above method, when the data transmitted in the SVC layered stream contains audio data in addition to the video data, the time stamp of the audio data is processed in the same manner as the video data.

On the terminal, the synchronization of each layered service of the SVC is performed as follows:

A2, the received SVC layered service data together with its corresponding time stamp is put into the buffer, the buffer may be one, or one for each layer, the buffer size should be able to accommodate each The transmission time difference of hierarchical data. For some reasons, the SVC layered data at the same time cannot be completely guaranteed to be received at the same time. The threshold of the receiving time difference of each layered service data is Td, and the buffer should be able to accommodate all points of the SVC in the Td time range. Layer data

B2. At a specific moment, such as the decoding time of a specific interval, the merged data is taken from the buffer, and the merge is aligned. First, the base layer data belonging to a video access unit is taken, and the enhancement layer data corresponding to the base layer data timestamp is taken as the reference based on the timestamp of the base layer data, and is used as the same video. Data merge of access units.

In step B2, for some reason, the time information carried by the RTP and RTCP sent by the encoding device may not ensure that the basic layer data, the enhancement layer data, and the audio data at the same sampling time are completely consistent in the NTP time. In this way, the CMMB broadcast timestamp converted on the front-end device cannot be completely consistent, and there may be a small deviation. When the terminal finds the corresponding enhancement layer data according to the timestamp of the base layer data, a timestamp tolerance is also set. value.

In order to make the objects, technical solutions and advantages of the present invention more comprehensible, the present invention will be further described in detail.

2 is a schematic diagram of the system of the present invention.

The encoding device encodes the SVC video into a base layer code stream and at least one enhancement layer code stream, and the base layer code stream can be separately decoded, and the enhancement layer code stream includes additional information for improving the quality of the lower layer code stream, and needs and includes the base layer. The lower layers are decoded together.

On the timeline, the video data is based on the video access unit, and a typical video access unit is a video frame. In the SVC mode, a video access unit at a specific time point can be encoded into multiple layers of data, and the data of the multiple layers can be divided into multiple channels for transmission; the receiving terminal can receive multiple channels simultaneously according to needs. The data of multiple layers transmitted is combined and decoded by the associated video access unit. In this embodiment, it is assumed that the encoding device is an H264 SVC encoder, and the basic unit of the encoded output video data is a Network Abstraction Layer Unit (NALU), and a plurality of NALUs at the same time point form a video access unit.

For convenience of description, in this embodiment, it is assumed that the broadcast service includes SVC video and one channel audio.

SVC video uses spatial layering mode, encoded as a QVGA (Quad VGA, 320 240 pixel image) elementary stream and a VGA (Video Graphic Array, 640x480 pixel image) enhanced video stream, the audio is encoded as an audio stream. It should be noted that the method of the present invention is equally applicable to the case of multiple enhanced streams and other SVC coding modes.

The encoding device sends the generated base layer code stream, the enhancement layer code stream, and the audio stream into an RTP code stream. For each RTP stream, a RTCP stream is sent along with it. The time stamps of each RTP stream may not depend on each other, and have independent RTP time scales and initial time stamps, and the time scale indicates the number of time units of the medium in 1 second. Guarantee the same through RTCP Basic layer SVC data, enhancement layer SVC data, and audio data synchronization at NTP time. For example, the audio has a time scale of 48000 and the video has a time scale of 90000. For a certain video access unit, after the basic layer data is encapsulated in the RTP packet of the basic stream, the RTP timestamp may be Tbase, and the enhancement layer data is encapsulated in the RTP packet of the enhanced stream, and the RTP timestamp may be Text, synchronized with it. The audio RTP packet timestamp at the moment can be Raudio. However, the RTP streams corresponding to the RTP streams are respectively carried. The SR (sender report) in the RTCP packet carries the reference NTP time and the corresponding reference RTP timestamp, and the base layer SVC at the same time should be guaranteed. The data, the enhancement layer SVC data, and the synchronization of the audio data in the NTP time, that is, the NTP times corresponding to the above Tbase, Text, and Taudio should be consistent. In order to achieve flexibility, the "NTP time should be consistent" here, the corresponding NTP time of Tbase, Text, Taudio is approximately equal, and a deviation value is allowed, which can be determined according to needs.

The front-end transmitting device receives the RTP code stream and the RTCP code stream sent by the encoding device, extracts the encapsulated SVC service data from the RTP code stream, and encapsulates the base layer code stream and the enhancement layer code stream into the broadcast channel frame according to the layer to which the layer is belonged. In the different multiplex subframes, the location information of the multiplex subframe in which the code stream of each layer of the video stream is located is encapsulated in the broadcast channel frame.

FIG. 3 is a schematic structural diagram of a structure of a broadcast channel frame according to the present invention. As shown in Fig. 3, there are 40 time slots on a certain frequency point F, wherein time slot 0 (multiplex frame 0) is used to transmit control information, and time slot 1 to time slot 39 are used to transmit service information. The front-end transmitting device configures the video base layer stream, audio, and data information in the video service S into the multiplex frame 1, occupying slot 1 to slot 4, and multiplexing the subframe number to 1. The video enhancement layer code stream is configured to multiplex frame 2, occupying slot 5 to slot 6, and having a multiplex subframe number of 1. No other services are transmitted in multiplex frame 1 and multiplex frame 2. Descriptive information of a multiplexing frame position carrying each layer code stream (the base layer code stream and its corresponding enhancement layer code stream) is added to the service control information and the ESG (Electronic Service Guide) information, and the description is performed. The information indicates that the video service S includes two multiplexed subframes: the multiplex subframe 1 of the multiplex frame 1 is the service base layer code stream data, and the multiplex subframe 1 of the multiplex frame 2 is the enhancement layer code stream data.

The front-end transmitting device obtains the basic layer code stream VI of the video service S, encapsulates it into the multiplex subframe 1 of the multiplex frame 1, acquires the enhancement layer code stream V2 of the video service S, and encapsulates the multiplex subframe of the multiplex frame 2. 1 in. The front-end transmitting device acquires the audio code stream and the data segment information of the video service S, and adds it to the multiplex subframe 1 of the multiplex frame 1, that is, is carried in the same multiplex subframe together with the base layer code stream. Layer code The description information of the multiplexed frame position of the stream is carried in the multiplex frame 0 in the broadcast channel frame to indicate the multiplexed frame position of each layer of the code stream of the receiving terminal, so as to facilitate receiving the video service s.

When audio and video data is encapsulated into a multiplexed sub-frame, it is encapsulated in units of audio units or video units. For each audio unit or video unit, there is a corresponding CMMB broadcast time stamp. According to the CMMB standard, the time stamp is a combination of the initial play time in the multiplex subframe and the relative play time corresponding to each unit. The initial play time and the relative play time corresponding to each unit are also encapsulated in the multiplex subframe and transmitted together with the media data.

On the front-end transmitting device, the RTP timestamp of the received media data needs to be converted into a CMMB broadcast timestamp, and in this process, the synchronization of the SVC hierarchical services on the CMMB broadcast timestamp is controlled at the same time.

For each multiplex subframe corresponding to each layer of SVC:

A. Receive the input RTP and RTCP streams, for each video unit, take the RTP timestamp of the RTP packet in which it is located, and calculate the video according to the relevant time information of the RTCP packet transmitted by the RTCP code stream corresponding to the RTP stream in which it is located. The NTP time corresponding to the unit;

B. Multiply the CMMB time scale by the NTP time to obtain the CMMB broadcast timestamp of the video unit. For the multiplex subframe, take a starting play time, and decompose the CMMB broadcast time stamp of the video unit into two parts: the start play time and the relative play time corresponding to each unit, and the start play time and each unit corresponding to each unit. The relative play time is encapsulated into the multiplex subframe.

The calculation of the NTP time corresponding to the video unit in step A can use the following methods:

(1) For each video unit, take the RTP timestamp of the RTP packet in which it is located, and take the reference NTP time carried in the SR (sender report) and the corresponding reference RTP timestamp from the corresponding RTCP packet;

(2) Using the RTP timestamp in the RTP packet and the reference RTP timestamp in the corresponding RTCP codestream to make a difference, and then dividing by timescale (timescale) to obtain a reference absolute time difference, using this difference plus reference At NTP time, the NTP time corresponding to the RTP packet is obtained, that is, the NTP time corresponding to the video unit. RTP timestamp - reference RTP timestamp, , ,

NTP time = J ¹ - + reference NTP time formula ( 1 ) timescale

The number of time units of the media in 1 second, the media can be different in different time units, such as RTP video, the clock of 90000Hz is commonly used as the time unit, the number of clocks in 1 second is 90000 is the timescale of the video; the common sampling rate of audio is the time unit. The timescale is the number of samples in 1 second. If the sample rate is 48000/second, the number of samples in 1 second is the timescale of the audio.

The CMMB time scale described in step B indicates the number of CMMB time units per second, according to the CMMB standard, Mobile Multimedia Broadcasting Part 2: Multiplexing, which is 22,500.

According to the method, each multiplex subframe corresponding to each layer of the SVC can be processed independently, and there is no coupling dependency between them. The CMMB standard does not specify how the starting value of the CMMB broadcast timestamp of the service is determined (the starting play time in the multiplex subframe is for each multiplex subframe rather than for the service), through the multiplexer Frame transmission between SVC layered data does not have an existing method of guaranteeing synchronization. The method converts the CMMB broadcast timestamp corresponding to the SVC video unit by directly multiplying the CMMB time scale by the NTP time, and determines the service by using the NTP time of the first SVC video unit in the first multiplexed subframe processed. The CMMB broadcasts a timestamp start value and determines the CMMB broadcast timestamp of the subsequent SVC video unit by the conversion method. For this embodiment, for a certain video access unit, the base layer data RTP timestamp is Tbase, the enhancement layer data RTP timestamp is Text, and the audio RTP timestamp Raudio at the synchronization time, the RTCP message sent by the encoder, Ensure that the NTP times of the respective Tbase, Text, and Taudio are consistent (the NTP times corresponding to each are equal or approximately equal, and a small offset value is allowed). After processing by the method of the present invention, the basic layer data of the video stream, The enhancement layer data and the audio data on the synchronization point are corresponding to the CMMB broadcast timestamps when they are encapsulated into the multiplexed sub-frames (equal or approximately equal, allowing a small offset value) even though they Encapsulated in different multiplex subframes, the multiplex subframes are processed independently. The method can ensure that the SVC service data at the same time point is unified in the CMMB broadcast time stamp, thereby providing a guarantee means for synchronizing between different layers of the SVC. As shown in FIG. 4, when receiving on the terminal, the terminal monitors the multiplex frame 0 in the broadcast channel frame, and receives the control information and the ESG information to correctly receive the multimedia broadcast service, and simultaneously monitors, and also includes the layer codes in the SVC service. The location information of the multiplex subframe in which the stream is located. The terminal according to its own needs, such as its The video stream processing capability or the network transmission status determines whether the base layer code stream is received, or the base layer code stream and the corresponding enhancement layer code stream are received. In this embodiment, the terminal is a netbook that can process VGA video, and needs to receive the basic stream of QVGA and the enhanced stream of VGA for processing, and decode and display VGA video. After receiving the elementary stream and the enhancement stream transmitted from different multiplex subframes, the terminal extracts the video unit from the multiplex subframe, parses out the H264 basic coding unit NALU, and broadcasts the NALUs belonging to different layers according to the CMMB broadcast time. The stamps are aligned, synchronized, and then the NALUs belonging to the same video access unit are merged, and then video decoding is performed.

A. Each video unit has a corresponding CMMB broadcast timestamp, which is a combination of the initial play time in the multiplex subframe and the relative play time corresponding to each video unit. The NALU data contained in the video unit is placed in a buffer along with its corresponding time stamp. The buffer can be independent of the base layer and the enhancement layer. For some reasons, the SVC layered data at the same time cannot be completely guaranteed to be received at the same time. The threshold value of the receiving time difference of each layered service data is Td, that is, the data belonging to the same video access unit is allowed to be received at the earliest. The time difference between the data and the latest received data is Td, then the buffer should be able to accommodate all layers of SVC data in the Td time range;

B. At every fixed time interval, the terminal takes a video access unit from the foregoing buffer, aligns and merges the NALU data of different layers of the video access unit, and sends the data to the decoder. In order to ensure the smoothing of the decoder, the first data fetching operation can be started after the buffer data is not less than the tolerance data of the aforementioned Td time. First, the base layer data belonging to a video access unit is taken, and the enhancement layer data corresponding to the base layer data timestamp is taken as the reference, and the data is used as the data of the same video access unit, and the base layer data. Merging, the combined data is sent to the decoder for decoding as a complete video access unit.

In step B, for some reason, the time information carried by the RTP and RTCP sent by the encoding device may not ensure that the basic layer data, the enhancement layer data, and the audio data at the same sampling time are completely consistent in the NTP time. In this way, the CMMB broadcast timestamp converted on the front-end device cannot be completely consistent, and there may be a small deviation. When the terminal finds the corresponding enhancement layer data according to the timestamp of the base layer data, a timestamp tolerance is also set. value.

The embodiments of the present invention are not only applicable to the CMMB system, but also to other mobile multimedia broadcast systems. The above is only the preferred embodiment of the present invention and is not intended to limit the scope of the present invention.

Industrial applicability

In the method of the embodiment of the present invention, synchronous CMMB broadcast time stamps are set in different layered SVC data to ensure synchronization between different hierarchical data.

Claims

Claim

1. A method of transmitting, comprising:

The video service is encoded to generate a multi-channel media stream, and the multiple media streams are respectively encapsulated in different multiplex subframes in a broadcast channel frame in units of media units, and the multiplexer is also carried in the multiplex subframe. a mobile multimedia broadcast timestamp of each media unit encapsulated in the frame, and a mobile multimedia broadcast timestamp of the media stream at the same time instant is synchronized, and the location information of the multiplexed subframe in which the media stream is located is encapsulated in the broadcast And transmitting, in the channel frame, the broadcast channel frame to the receiving terminal, where the media stream includes a base layer code stream generated by the video stream coding of the video service and a corresponding enhancement layer code stream, or an audio stream including a video service. And the basic layer code stream generated by the video stream of the video service and the corresponding enhancement layer code stream; thereby implementing synchronous transmission of the scalable video coding service in the mobile multimedia broadcast.

2. The method of claim 1, wherein the media stream is encapsulated into the multiplex subframe as follows:

3. The method according to claim 2, wherein the RTP timestamp is converted into a mobile multimedia broadcast timestamp as follows: For each media unit, the RTP timestamp of the RTP packet is taken out, and is combined with the corresponding RTP code stream. Calculating the time information of the RTCP packet transmitted by the RTCP code stream, and calculating the NTP time of the media unit;

4. The method according to claim 1, wherein the mobile multimedia broadcast timestamp synchronization of the media stream at the same sample time refers to: a difference between mobile multimedia broadcast timestamp values of media streams at the same sample time The value is within the preset time stamp tolerance value range.

A transmission system, where the system includes an encoding device and a front-end transmitting device, where: the encoding device is configured to: encode a video service to generate a multi-channel media stream; and the media stream includes a video stream encoding of the video service. a generated base layer code stream and a corresponding enhancement layer code stream, or an audio stream including the video service, a base layer code stream generated by the video stream coding of the video service, and a corresponding enhancement layer code stream;

The front-end transmitting device is configured to: encapsulate the media stream in different multiplex subframes in a broadcast channel frame in units of media units, and move the multimedia broadcast time of each media unit encapsulated in the multiplex subframe Stamping is encapsulated in the multiplexed subframe, and the mobile multimedia broadcast timestamp of the media stream at the same time instant is synchronized, and the location information of the multiplexed subframe in which the media stream is located is encapsulated in the broadcast channel frame. Transmitting the broadcast channel frame to the receiving terminal;

Thereby, synchronous transmission of the scalable video coding service is implemented in the mobile multimedia broadcast.

6. The system of claim 5, wherein

The encoding device includes an encoding unit and a packaging unit, where:

The converting unit is configured to: convert an RTP timestamp of the media stream into an NTP time, and then Converting the NTP time to a mobile multimedia broadcast timestamp under the unified time reference; the third encapsulating unit is configured to: encapsulate the mobile multimedia broadcast timestamp into a multiplex subframe in which the corresponding media unit is located; The transmitting unit is configured to: send the broadcast channel frame to the receiving terminal.

7. The system of claim 6 wherein

The conversion unit includes a first conversion unit and a second conversion unit, wherein:

The first converting unit is configured to: take out an RTP timestamp of the RTP packet, and calculate an NTP time of the media unit according to the time information of the RTCP packet transmitted by the RTCP code stream corresponding to the RTP stream for each media unit. ;

The second converting unit is configured to: multiply the NTP time of the media unit by a mobile multimedia broadcast time scale to obtain a mobile multimedia broadcast timestamp of the media unit.

8. The system according to claim 5, 6 or 7, wherein the mobile multimedia broadcast timestamp synchronization of the media stream at the same sample time refers to: a mobile multimedia broadcast timestamp value of the media stream at the same sample time The difference between the preset time stamp tolerance values.

9. A receiving method, comprising:

10. The method according to claim 9, wherein the receiving the base layer code stream and the corresponding enhanced layer code stream, and aligning the layer code streams according to the mobile multimedia broadcast timestamp comprises:

Extracting base layer code stream data belonging to a video access unit from the buffer, to the basic The mobile multimedia broadcast timestamp of the layer code stream data is used as a reference, and the enhancement layer code stream data synchronized with the mobile multimedia broadcast timestamp of the base layer code stream data is taken out as data merge of the same video access unit.

11. The method of claim 10, wherein

The enhancement layer code stream data synchronized with the mobile multimedia broadcast timestamp of the base layer code stream data refers to a difference between a mobile multimedia broadcast timestamp and the base layer code stream data at a preset timestamp. Enhancement layer code stream data within the difference range.

12. A receiving device, comprising:

a monitoring unit, configured to: monitor location information of a multiplex subframe in which each layer of the video stream in the video service in the broadcast channel frame is located;

a receiving unit, configured to: receive a base layer code stream according to its own video stream processing capability, or receive a base layer code stream and a corresponding enhancement layer code stream;

a decoding unit, configured to: decode the base layer code stream and the corresponding enhancement layer code stream, and output the combined video data of the base layer code stream and the enhancement layer code stream;

Thereby, scalable video coding service reception is implemented in mobile multimedia broadcasting.

13. The apparatus according to claim 12, wherein

The receiving unit is further configured to: store the received base layer code stream and the enhancement layer code stream into the buffer area;

The alignment merging unit is configured to: extract base layer code stream data belonging to a video access unit from a buffer, and extract and base the mobile multimedia broadcast timestamp of the base layer code stream data The enhanced layer code stream data of the mobile multimedia broadcast timestamp synchronization of the layer code stream data is merged as data of the same video access unit.

14. The apparatus according to claim 13, wherein

And extracting the enhancement layer code stream data synchronized with the mobile multimedia broadcast timestamp of the base layer code stream data, where the difference between the mobile multimedia broadcast timestamp and the base layer code stream data is taken out at a preset The enhancement layer code stream data within the time stamp tolerance range.