WO2011113315A1

WO2011113315A1 - Stream media live service system and implementation method thereof

Info

Publication number: WO2011113315A1
Application number: PCT/CN2011/071090
Authority: WO
Inventors: 陈晨
Original assignee: 中兴通讯股份有限公司
Priority date: 2010-03-17
Filing date: 2011-02-18
Publication date: 2011-09-22
Also published as: CN102196319A

Abstract

The invention provides a stream media live service system and implementation method thereof, which serve to implement synchronous playback of audio, video and subtitles in a stream media service using real-time transport Protocol (RTP). The present invention is used for a stream media live service using RTP as a data transport protocol. A stream media live server end calculates sampling time of input audio, video and subtitle contents, wherein the sampling time is accurate to each sampling sample. On a RTP transport channel of each media, each RTP packet comprising media samples carries the sampling time of the media samples. A stream media client end sets playback time of audio samples as a reference, calculates, according to the sampling time of video samples and subtitle samples, playback time of the video samples and subtitle samples. Thus the present invention enables synchronous playback control of audio, video and subtitles in time sequence, thereby enabling accurately restoring the effect of live area.

Description

Streaming media live broadcast service system and implementation method thereof

The present invention relates to the field of communications and multimedia applications, and in particular, to a streaming media live broadcast service system and an implementation method thereof. Background technique

In recent years, wireless data network technology has developed rapidly. As the bandwidth of wireless networks continues to increase, the application of dense data services in wireless environments becomes possible. Wireless Streaming Live broadcasting is an emerging multimedia service where users can watch ongoing games, concerts, news sites and more on a wireless network on a handheld mobile terminal. The Live Streaming Service (RTSP) implements control (start, pause, and end) between the streaming client and the streaming media server through Real Time Streaming Protocol (RTSP), and real-time transport protocol (RTP). ) Realizing the transmission of live content data from the collector to the receiver.

Live streaming is a typical real-time sending/receiving system with the ability to transmit images, sounds, and text. The time at which the live content is presented to the user at the receiving end is substantially the same as the time actually generated at the collecting end. Since the receiving terminal is convenient to carry and uses the wireless network as a physical bearer, the user can easily understand the situation of the live broadcast site in the first time. Therefore, the streaming media direct broadcast service has broad prospects in the broadband wireless era.

The streaming media service uses the real-time transport protocol RTP to transmit images, sounds, and subtitle data. The RTP protocol stipulates that only one type of multimedia data can be transmitted per RTP channel on the network, so images, sounds, and subtitles must be implemented. The transmission of data requires three RTP channels as bearers. Since different types of multimedia data samples, data sizes, network transmission speeds, etc. are different, it is necessary to input images, sounds, and subtitles on the receiving terminal. Synchronous control over the line time to accurately reproduce the live live content. Otherwise, images, sounds, and subtitles can cause serious inconsistencies in time, affecting the user's visual and audible experience. Summary of the invention

In view of this, the main object of the present invention is to provide a streaming media live broadcast service system and a method for implementing the same. Synchronous playback of audio, video, and subtitles for streaming services using the Real-Time Transport Protocol.

To achieve the above objective, an aspect of an embodiment of the present invention provides a method for implementing a live streaming service, the method comprising the following steps:

The streaming media server obtains the sampling time of the collected samples while collecting audio samples, video samples and subtitle samples;

Encapsulating audio samples, video samples and subtitle samples and sample sampling time into network packets in the respective transmission channels of audio, video and subtitles, and sending them to the streaming client;

The streaming client restores the audio samples, video samples, and subtitle sample sequences with the same time from the network packets of each channel, and reproduces the live content according to the time of occurrence according to the sampling time of the samples.

Further, the streaming media client reproduces the live content according to the time of the sample according to the sampling time of the sample, which is specifically:

Playing the audio sample, the video sample, and the subtitle sample sequence with the play time of the audio sample as a reference time on the play time axis; the streaming media client plays the standard play according to the sample time of the audio sample and the reference time Time deviation; For video samples and subtitle samples, the ideal playback time is first calculated based on the sampling time of the sample and the standard playback time deviation, and then the waiting time is calculated by the ideal playback time.

Further, the standard playing time deviation is equal to the reference time minus the sampling time of the audio sample; the ideal playing time of the video sample and the subtitle sample is equal to the standard playing time offset The sampling time of the difference sample, the waiting time of the video sample and the subtitle sample is equal to its ideal playing time minus the current system time; if the waiting time of the video sample and the subtitle sample is less than zero, the sample is discarded.

Further, a real-time transport protocol (RTP) is used between the streaming media server and the streaming client to implement network transmission of media data, where the sampling time of the sample is located in a network packet containing sample content. In the header.

Based on the embodiment of the present invention, the present invention further provides a streaming media live broadcast service system, where the system includes:

The live streaming server is configured to collect audio samples, video samples, and subtitle samples, and simultaneously obtain the sampling time of the samples; respectively, the audio samples, the video samples, the subtitle samples, and the sampling time of the collected samples are encapsulated into a network. Packets are sent to the streaming client through their respective transmission channels;

The audio samples, video samples, and subtitle sample sequences of the sample time, and the live content is reproduced in the order of occurrence according to the sampling time of the audio samples, the video samples, and the subtitle samples.

Further, in the system, the streaming media client plays the audio sample, the video sample and the subtitle sample sequence with the playing time of the audio sample as a reference time on the playing time axis; the streaming media client according to the audio Calculating the standard playback time deviation of the sampling time of the sample and the reference time; for the video sample and the subtitle sample, first calculating the ideal playing time according to the sampling time and the standard playing time deviation, and then calculating the waiting time by the ideal playing time time.

The standard play time deviation is equal to the reference time minus the sample time of the audio sample; the ideal play time of the video sample and the caption sample is equal to the standard play time deviation plus the sample sample time, the video sample and the caption sample The waiting time is equal to its ideal playing time minus the current system time; When the streaming media client plays the video sample and the subtitle sample sequence, if the waiting time of the sample is determined to be less than zero, the streaming device discards the sample.

Further, in the system, the streaming media live server end and the streaming media client use RTP to implement network transmission of media data, and the sampling time of the sample is located in the packet header of the network data packet containing the sample content. in.

The present invention is directed to a streaming media live broadcast service using RTP as a data transmission protocol, and calculates a sampling time for input audio, video and subtitle content on the streaming media live server side, and the sample time is accurate to each sample sample in each medium. In the RTP transmission channel, each RTP packet containing the media sample carries the sample time of the media sample, and in the streaming client, based on the playing time of the audio sample, and according to the sampling time of the video sample and the subtitle sample Calculate the playing time of the video samples and subtitle samples, so as to achieve synchronous control playback of audio, video and subtitles in time, and achieve accurate restoration of the live scene. DRAWINGS

1 is a schematic diagram of a streaming media live broadcast system architecture according to an embodiment of the present invention;

2 is a schematic diagram showing the form of a video data packet represented by video data transmission in a video channel according to an embodiment of the present invention;

3 is a sequence of media sample samples output by a streaming media client after processing an RTP channel according to an embodiment of the present invention;

4 is a schematic diagram showing deviations of a video sample with an audio sample sample as a time reference and a subtitle sample with respect to a time reference according to an embodiment of the present invention;

Fig. 5 is a flow chart showing temporal synchronization control of sample playback by sample sampling time according to an embodiment of the present invention. detailed description

In order to make the objects, technical solutions and advantages of the present invention more clear, the following embodiments are The present invention is further described in detail with reference to the accompanying drawings.

In the embodiment of the present invention, the streaming media live broadcast system architecture is mainly composed of the following network elements: a streaming media live server, which is set as a collection, production, and publishing device for live content. The streaming media client, which is set as the receiving device of the live content, receives the content posted by the streaming live broadcast server through the network, and processes and plays the content.

FIG. 1 is a schematic diagram of a streaming media live broadcast system architecture according to an embodiment of the present invention. As shown in FIG. 1 , the streaming media live server uses the audio, video collection device, and subtitle generating device as input, and converts the scene of the live scene into audio samples, video samples, and subtitles through the data collection module, and then, Through the network processing module, the samples of the audio, video and subtitles and the corresponding sampling time are encapsulated into an RTP network packet for network transmission. Finally, the RTP network packet is transmitted over the wireless network. The present invention does not limit the real-time transmission protocol used, and those skilled in the art can extend to other real-time transmission protocols according to the technical solutions disclosed in the present invention.

In the streaming media client, the RTP network packet received from the network is processed by the network processing module and the playback control module, and then restored to a sequence of audio, video, and subtitle samples, and the scene of the live scene is restored to the user by playing these sample sequences. . The RTP network packet transmission between the streaming media client and the streaming media broadcast server uses the RTP protocol. The control protocol between the streaming client and the streaming server uses the RTSP protocol (control information such as the start and end of the live broadcast).

FIG. 2 illustrates an RTP network packet sequence form including a sample sample in an RTP network transmission channel according to an embodiment of the present invention, and a structure of an RTP network packet. More importantly, it shows how a multimedia sample sample can be packaged with the corresponding sample time in the form of an RTP network packet. Through this implementation, it is very convenient to obtain the sampling time of a sample sample in the streaming media client for subsequent synchronization processing of audio, video and subtitles.

As shown in Figure 1 and Figure 2, the streaming media live server needs to go through the data collected on the site. Processing:

Step S102: Sample data of the input video data, audio data, and subtitles by the data collection module, and record the sampling time for each of the audio, video, and subtitle samples recorded by the collection device. The sample time needs to be accurate enough to reflect the order in which the samples occur.

Step S104: The network processing module encapsulates the sample content and the sample sampling time into the same RTP network data packet according to the RFC 1889 protocol. The sample content constitutes the payload of the RTP network packet, and the sample sample time is filled in the Time Stamp field position in the header of the RTP network packet.

RTP network packet header format

On the server side, the processed media samples become the sequence of RTP network packets transmitted on the network as shown in Figure 2.

The inventive idea and implementation process of the streaming media client in the embodiment of the present invention will be described in detail below with reference to FIG. 3, FIG. 4 and FIG. Figure 3 depicts the sequence of audio, video, and subtitle samples generated by the streaming media client after processing the received RTP packets. Figure 4 depicts the deviation of the video samples, the sample sequence of the subtitles, and the reference time of the time axis in the case where the sample time of the audio sample is taken as the time axis reference time in the embodiment of the present invention.

The processing of receiving the RTP network packet by the streaming media client mainly includes the following steps: Step S106: cyclically processing the RTP audio channel, the video channel, and the subtitle channel, and reading the RTP network data packet therefrom. The RTP network packet is parsed according to the RFC 1889 protocol, the sample time is read in the RTP header, and the sample content is read in the RTP payload.

FIG. 5 is a flow chart of synchronizing audio, video, and subtitle samples according to the sampling time in the streaming media client according to the embodiment of the present invention, and the flow description of step S108 is performed by the playback control module of the streaming media client. The flow diagram shows how to synchronize the audio, video and subtitle samples according to the sample time to reproduce the time sequence they generated on the live scene. It mainly includes calculating the "play time deviation" of each sample; when playing the sample, the "play time deviation" of the audio sample is used as the time reference; and the video and subtitle samples advanced in the audio time are processed. The specific steps are as follows:

Step 501: The playback control module cyclically reads the media sample transmitted from the RTP channel from the network processing module, and determines the end of transmission flag. If the transmission ends, the process ends; otherwise, step 502 is performed;

The first processing reads an audio sample, a video sample and a subtitle sample. At least one audio sample needs to be read in the subsequent loop processing. For the video and subtitle samples, if it is still in the sleep waiting state (see step 4, step 6), it is not read in from the RTP channel.

Step 502: Processing the audio sample, reading the sample time of the audio sample, playing the audio sample, and using the playing time of the audio sample as the "base time" on the time axis. The "standard playback time deviation" is calculated based on the audio sample sampling time and playback time. The calculation method is:

Standard Play Time Deviation = Base Time - Sample Time of Audio Sample ( 1 ) Step 503: Process the video sample, read the sample time of the video sample, and calculate the ideal playback time of the video sample based on the current "standard play time deviation". And further calculating the video sample waiting time, if the video sample waiting time is less than 0, step 504 is performed, otherwise step 505 is performed;

According to formula (1), the ideal playback time of the video sample can be derived, and the calculation method is: ideal playback time of the video sample = standard playback time deviation + sample time of the video sample

(2) According to the ideal playback time of the video sample and the current system time, the video sample waiting time can be calculated by the following method:

Video sample waiting time = ideal playing time of the video sample - current system time (3) Step 504: If the video sample waiting time is less than zero, indicating that the video sample lags behind the audio sample in time, the video sample is discarded (hop If the video sample waiting time is greater than zero, indicating that the video sample is ahead of the audio sample in time, the video sample is postponed to play, and the delayed playback time is equal to the video sample. Waiting time; if the video sample waiting time is equal to zero, indicating that the video sample and the audio sample are synchronized in time, the video sample is played directly. Then performing step 506;

Step 506: Processing the subtitle sample, reading the sampling time of the subtitle sample, calculating the ideal playing time of the subtitle sample according to the current "standard playing time deviation", and further calculating the subtitle sample waiting time, if the subtitle sample waiting time is less than 0 Step 507 is performed, otherwise step 508; ideal playing time of the subtitle sample = standard playing time deviation + sampling time of the subtitle sample subtitle sample waiting time = ideal playing time of the subtitle sample - current system time step 507: if the subtitle sample waiting time If the subtitle sample is later than the audio sample, the subtitle sample is discarded (the skip is not played), and then step 501 is performed; step 508: if the subtitle sample waiting time is greater than zero, the subtitle sample is in time In advance of the audio sample, the subtitle sample is postponed to play, and the delayed playback time is equal to the subtitle sample waiting time; if the subtitle sample waiting time is equal to zero, indicating that the subtitle sample and the audio sample are synchronized in time, the subtitle is directly played. Sample; then execute the step 501;

The invention requires that the time on the sample device or the sample system is sufficiently accurate, and the sampling time of the audio sample, the video sample and the subtitle sample needs to be able to truly reflect the order of generation of each sample in time, and the sample time should be one. A variable that grows in threads. For synchronized playback, the system time of the streaming client should be accurate enough for the playback speed of the audio device.

The present invention is directed to a streaming media live broadcast service using RTP as a data transmission protocol, in streaming media The live server side calculates the sampling time for the input audio, video and subtitle content, and the sampling time is accurate to each sample sample. In the RTP transmission channel of each medium, each RTP packet containing the media sample carries the media. The sampling time of the sample, in this way, provides conditions for accurate reproduction of live live content on the streaming client. Since the data bytes occupied by the sample time field are negligible compared to the data bytes occupied by the media samples, there is no impact on the real-time performance of the media data when transmitted over the network. In the streaming media client, the playing time of the audio, video, and subtitle samples is calculated by the method used in the present invention, thereby achieving synchronous control playback of audio, video, and subtitles in time, and accurately reducing the effect of the live broadcast site.

The above is only the preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes can be made to the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the present invention.

Claims

Claim

A streaming media live broadcast service implementation method, the method comprising: the streaming media live server receiving the sampled time of the collected sample while collecting the audio sample, the video sample and the subtitle sample;

The method according to claim 1, wherein the streaming media client reproduces the live content according to the time of occurrence of the sample according to the sampling time of the sample:

3. The method of claim 2, wherein

The standard play time deviation is equal to the reference time minus the sample time of the audio sample; the ideal play time of the video sample and the caption sample is equal to the standard play time deviation plus the sample sample time, the video sample and the caption sample The waiting time is equal to its ideal playing time minus the current system time;

The sample is discarded if the latency of the video sample and the caption sample is less than zero.

The method according to claim 1, wherein the real-time transport protocol (RTP) is used between the streaming media server and the streaming client to implement network transmission of media data.

5. The method according to claim 4, wherein the sampling time of the sample is in a header of a network data packet containing sample content.

6. A streaming media live broadcast service system, characterized in that the system comprises:

The system according to claim 6, wherein the streaming media client plays the audio sample, the video sample and the subtitle sample sequence with the playing time of the audio sample as a reference time on the playing time axis; The streaming media client calculates a standard playback time deviation according to the sampling time of the audio sample and the reference time; for the video sample and the subtitle sample, first calculating the ideal playing time according to the sampling time and the standard playing time deviation, Then calculate its waiting time by the ideal playing time.

8. The system of claim 7 wherein:

When the streaming media client plays the video sample and the subtitle sample sequence, if the waiting time of the sample is determined to be less than zero, the streaming device discards the sample.

The system according to claim 6, wherein the streaming media live server end and the streaming media client use RTP to implement network transmission of media data.

10. The system of claim 9, wherein the sample time of the sample is in a header of a network packet containing sample content.