WO2013152562A1

WO2013152562A1 - Method and system for synchronizing mobile multimedia broadcasting and subtitles

Info

Publication number: WO2013152562A1
Application number: PCT/CN2012/077420
Authority: WO
Inventors: 夏智海; 黄泽武; 陈志兵
Original assignee: 中兴通讯股份有限公司
Priority date: 2012-04-10
Filing date: 2012-06-25
Publication date: 2013-10-17
Also published as: CN102630017B; CN102630017A

Abstract

A method and a system for synchronizing mobile multimedia broadcasting and subtitles. The method comprises: receiving a subtitle data packet sent by a subtitle server and an audio subframe sent by an audio coder; obtaining the start time of audio play from the audio subframe; obtaining a subtitle from the subtitle data packet; determining the time offset of the subtitle relative to the start time of audio play; when encapsulation time is up, encapsulating character strings of the subtitle, the start time of audio play and the time offset of the subtitle relative to the start time of audio play into a data segment of the audio subframe corresponding to the subtitle; and sending the data segment to a mobile multimedia broadcasting terminal.

Description

Method and system for synchronizing mobile multimedia broadcast subtitles

Technical field

The present invention relates to the field of mobile multimedia broadcasting, and in particular, to a method and system for synchronizing mobile multimedia broadcast subtitles.

Background technique

As the number of commercial and subscribers of China Mobile Multimedia Broadcasting (CMMB) increases, the user community and user needs tend to diversify. Providing a variety of flexible services is one of the means to improve user physical examination and increase user loyalty.

In the existing CMMB, there are generally the following ways to provide subtitles for pure audio programs: The subtitles are embedded in pure audio data, and the subtitle form is fixed in a single manner;

The subtitle information is transmitted through the mobile multimedia broadcast data segment to solve the problem that the subtitles are fixed single, but this method cannot guarantee the synchronization of the subtitles and audio broadcasted, and the subtitles cannot be flexibly inserted, for example, the programs A and B are sequentially played, only Program B provides subtitles;

The subtitles are sent to the terminal, and the user who uses the terminal manually adjusts the playing time of the subtitles to synchronize the subtitles with the audio. This way the user experience is poor.

After the subtitle is sent to the terminal, the synchronization information of the subtitle and the audio is sent to the terminal, and the terminal adjusts the playing time of the subtitle according to the synchronization information. In this manner, the terminal reserves a large storage space to buffer the entire subtitle file, and needs Subtitles are sent for a long time in advance, resulting in poor real-time performance.

Summary of the invention

Embodiments of the present invention provide a mobile multimedia subtitle synchronization method and system to solve the technical problem of how to implement subtitle and audio synchronization.

The method for synchronizing mobile multimedia broadcast captions in the embodiment of the present invention includes:

Receiving a subtitle data packet sent by the subtitle server and an audio sub-frame sent by the audio encoder; acquiring audio from the audio sub-frame to start playing time; Obtaining a subtitle from the subtitle data packet;

Determining the time offset of the subtitle relative to the start time of the audio;

When the encapsulation time arrives, the character string of the subtitle, the audio start playing time, and the time offset of the subtitle relative to the audio start playing time are encapsulated into the data segment of the audio sub-frame corresponding to the subtitle;

The data segment is transmitted to a mobile multimedia broadcast terminal.

Optionally, the step of determining a time offset of the subtitle relative to an audio start playing time includes:

Obtaining information about a relationship between a subtitle and an audio sub-frame from a data stream including the subtitle data packet; determining an audio sub-frame corresponding to the subtitle according to the information of the subtitle and the audio sub-frame relationship; according to the subtitle in a local time zone a start time and a time stamp of the audio sub-frame corresponding to the subtitle setting a synchronization mark of the subtitle and the audio sub-frame;

A time offset of the subtitle relative to the start time of the audio playback is determined based on the synchronization flag.

Optionally, the method further includes determining, according to the start time of the subtitle in a local time zone, the encapsulation time, where:

The start time of the subtitle in the local time zone is M, the current time (TOD) provided by the local GPS is N, and the transmission time difference from receiving the subtitle data packet to transmitting the data segment is R, and the encapsulation time is S; M, N, R And the unit of S is seconds;

The time difference between M and N is / = floor(\M - N|) , floor ) means rounding, and 卜| means taking the absolute value;

J = floor ;

Then, S=I+J.

Optionally, the subtitle obtained according to the subtitle data packet is composed of multiple subtitles;

Determining, according to the synchronization mark, a time offset of the subtitle piece relative to an audio start playing time;

When the encapsulation time arrives, the character string of the subtitle piece, the start playing time of the audio, and the time offset of the subtitle piece relative to the audio start playing time are encapsulated into an audio sub-frame corresponding to the subtitle. Data segment.

Optionally, the step of determining, according to the synchronization mark, a time offset of the subtitle piece relative to an audio start playing time includes:

Turn on a timer, set the timing length of the timer and the value of the synchronization flag;

When the timing length is reached, the time offset of the subtitle piece relative to the audio start playing time is calculated.

The system for moving multimedia broadcast subtitle synchronization in the embodiment of the present invention includes:

a receiving module, configured to receive a subtitle data packet sent by the subtitle server and an audio sub-frame sent by the audio encoder;

An obtaining module, configured to acquire a subtitle from the subtitle data packet, and obtain an audio starting time from the audio sub-frame;

a time offset determination module configured to determine a time offset of the subtitle relative to an audio start playing time;

a packaging module, configured to encapsulate a character string of the subtitle, an audio start playing time, and a time offset of the subtitle relative to an audio start playing time to an audio sub-frame corresponding to the subtitle when the encapsulation time arrives Data segment;

And a sending module, configured to send the data segment to the mobile multimedia broadcast terminal.

Optionally, the time offset determining module is configured to determine a time offset of the subtitle relative to an audio start playing time in the following manner:

Acquiring information of a relationship between a subtitle and an audio sub-frame from a data stream that includes the subtitle data packet; determining an audio sub-frame corresponding to the subtitle according to the information of the subtitle and the audio sub-frame relationship; according to the start time of the subtitle in the local time zone And the time stamp of the audio sub-frame corresponding to the subtitle sets a synchronization mark of the subtitle and the audio sub-frame; and determining, according to the synchronization mark, a time offset of the sub-title relative to the audio start playing time.

Optionally, the encapsulating module is further configured to determine the encapsulation time according to a start time of the subtitle in a local time zone in the following manner: The start time of the subtitle in the local time zone is M, the current time (TOD) provided by the local GPS is N, and the transmission time difference from receiving the subtitle data packet to transmitting the data segment is R, and the encapsulation time is S; M, N, R And the unit of S is seconds;

Let the time difference between M and N be / = floor(\M - N|) , floor ) denote rounding, and b| denote absolute value;

J = floor ; Then, S=I+J.

Optionally, the acquiring module is configured to obtain a subtitle composed of a plurality of subtitles from the subtitle data packet;

The time offset determining module is configured to determine, according to the synchronization flag, a time offset of the title film relative to the audio start playing time;

The encapsulating module is configured to, when the encapsulation time arrives, encapsulate a character string of the subtitle piece, a start playing time of the audio, and a time offset of the subtitle piece relative to the audio start playing time to the audio subtitle corresponding to the subtitle The data segment of the frame.

Optionally, the time offset determining module is configured to determine, according to the synchronization flag, a time offset of the beginning time of the movie with respect to the audio according to the synchronization flag:

In the above technical solution, the data segment of the encapsulated audio sub-frame includes a time start of audio start time and a time offset of the subtitle relative to the audio start play time, so that the mobile multimedia broadcast terminal that receives the audio sub-frame can The time offset of the audio start playing time determines the playing time of the subtitle, achieving the technical effect of synchronizing with the audio.

BRIEF abstract

1 is a flowchart of a mobile multimedia broadcast caption synchronization method according to an embodiment of the present invention; 2 is a schematic diagram of a package for transmitting data streams to a multiplexer by an application example caption server; FIG. 3 is a schematic diagram of a format of a data segment of an audio sub-frame in which caption information is encapsulated according to an application example; FIG. 4 is a mobile multimedia broadcast of the present embodiment. The subtitle synchronization system forms a module diagram. Preferred embodiment of the invention

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that, in the case of no conflict, the features in the embodiments and the embodiments in the present application may be arbitrarily combined with each other.

FIG. 1 is a flowchart of a mobile multimedia broadcast caption synchronization method according to an embodiment of the present invention.

S101: receiving a subtitle data packet sent by the subtitle server and an audio sub-frame sent by the audio encoder; S102: acquiring audio from the audio sub-frame to start playing time;

S103: acquiring a subtitle from the subtitle data packet;

In practical applications, the subtitle data packet is also allowed to include information indicating the time-lapse data of the subtitle data packet (such as Coordinated Universal Time UTC time system), information indicating the time zone in which the subtitle data packet is located, and the start time of the subtitle in the time zone in which it is located. After receiving the data packet, correcting the subtitle in the time zone of the time zone in which the subtitle data packet is located according to the information indicating the time slot system of the subtitle data packet, the information indicating the time zone in which the subtitle data packet is located, and the time zone in which the receiving end is locally located. The start time of the local time zone. For example, the information indicating the time capping system of the caption data is UTC, indicating that the time zone of the caption data packet is the East 5 zone, the start time of the caption in the time zone where the caption is located is 0 o'clock, and the time zone of the local end of the receiving end is In the East 8th district, the revised subtitles start at 3 o'clock in the local time zone.

In addition to the information described above, the subtitle data packet may further include other information according to different application requirements, for example, in order to clearly distinguish from the audio sub-frame, add a representation in the subtitle data packet. The data packet is information of the subtitle; in order to enable the terminal to select the correct character set, the subtitle string encoding mode is added in the subtitle data packet; in order for the terminal to recognize the subtitle display duration and end the subtitle display at an appropriate time, the subtitle Increase the duration of subtitles in the packet.

S104 determines a time offset of the subtitle relative to the start time of the audio playing time;

Specifically: Obtaining information about a relationship between a subtitle and an audio sub-frame from a data stream including the subtitle data packet; determining an audio sub-frame corresponding to the subtitle according to the information of the subtitle and the audio sub-frame relationship; according to the subtitle in a local time zone a start time and a time stamp of the audio sub-frame corresponding to the subtitle setting a synchronization mark of the subtitle and the audio sub-frame; wherein a start time of the sub-title in the local time zone is obtainable from the subtitle data packet;

Determining, according to the synchronization mark, a time offset of the subtitle relative to the start time of playing of the audio;

S105, when the encapsulation time arrives, encapsulating a character string of the subtitle, an audio start playing time, and a time offset of the subtitle relative to the audio start playing time into a data segment of the audio sub-frame corresponding to the subtitle;

The encapsulation time is determined according to a start time of the subtitle in a local time zone, and the determining step includes: setting a start time of the subtitle in the local time zone to be M, and a current time TOD (Time of Day) provided by the local global positioning system (GPS) is N The transmission time difference from receiving the subtitle data packet to the transmission data segment is R, the encapsulation time is S; the units of M, N, R, and S are seconds;

J = floor ;

Then, S=I+J.

S106 sends the data segment to the mobile multimedia broadcast terminal.

The caption string parsed from the caption data packet may also be composed of a plurality of captions to facilitate long caption data transmission or to be applied to multi-lingual subtitles. At this time, the caption may be determined according to the synchronization mark. a time offset of the audio start playing time; when the encapsulation time arrives, the start play time of the audio, the time offset of the subtitle piece relative to the audio start play time, and the character string of the subtitle piece are encapsulated to correspond to the subtitle The data segment of the audio sub-frame.

After receiving the data segment of the audio sub-frame, the mobile multimedia broadcast terminal performs an operation of playing a character string of the subtitle according to a start time of the audio included in the data segment and a time offset of the subtitle relative to the start time of the audio; Encapsulated in the segment is the subtitle piece information, and the character string of the subtitle piece is played according to the start play time of the audio contained in the data segment and the time offset of the subtitle piece relative to the start time of the audio start time. The mobile multimedia broadcast caption synchronization method of the above embodiment is further described below with a specific application example. The application example relates to a subtitle server, an audio encoder and a multiplexer, wherein the data stream transmitted between the subtitle server and the multiplexer is encapsulated in UDP, and the encapsulation format is as shown in FIG. 2.

The subtitle synchronization method of this application example includes the following steps:

Step 1 : The subtitle server sends a UDP encapsulated data stream to the multiplexer, where each UDP message body part is used to form a subtitle data packet; the audio encoder sends an audio sub-frame to the multiplexer; the subtitle data packet includes an indication The time zone information of the subtitle data packet, the subtitle update sequence number, the start time of the subtitle in the time zone in which it is located, the encoding mode, the number of subtitles, the subtitle slice 1 parameter, the subtitle slice 2 parameter subtitle slice n parameter, and the subtitle content, wherein

Information indicating the time zone in which the subtitle packet is located (4bit);

The subtitle update sequence number (4bit), when the subtitle is updated, the sequence number is incremented by 1. The multiplexer can compare whether the received sequence number is carried in the subtitle data packet and whether the serial number carried in the received subtitle data packet is consistent, and whether the re-determination is re-determined. The start time of the synchronized subtitles in the local time zone;

The start time of the subtitle in its time zone (32bit);

Subtitle coding method (bit);

Number of subtitles ( 8bit ) ;

The subtitle piece n parameter (80bit), which further contains three fields, respectively: subtitle piece n play time (32bit), subtitle piece n end time (32bit) and subtitle piece n length (16bit);

Subtitle content, according to the number of subtitles included in the subtitle data packet and the length of the subtitle piece n, the corresponding subtitle piece content can be extracted from the subtitle content.

Step 2: After receiving the UDP encapsulated data stream, the multiplexer extracts the message body from the UDP to form the subtitle data packet; after receiving the audio sub-frame, the multiplexer extracts the audio starting time from the audio sub-frame;

Step 3: The multiplexer parses the information identifying the relationship between the subtitle and the subframe from the UDP encapsulated data stream, and determines a subframe corresponding to the subtitle according to the information; Step 4: The multiplexer obtains, from the subtitle data packet, information indicating a time zone in which the subtitle data packet is located and a start time of the subtitle in a time zone in which the subtitle is located, and the multiplexer compares information about the time zone in which the subtitle data packet is located The time zone in which the multiplexer is located (ie, the local time zone) is compared. If they are consistent, the start time of the subtitle contained in the subtitle packet is the start time of the subtitle in the local time zone; if not, the subtitle is The start time of the time zone in which it is located is corrected to the start time of the subtitle in the local time zone;

Step 5: The multiplexer sets a synchronization mark of the subtitle and the audio sub-frame according to the start time of the subtitle in the local time zone and the time stamp of the audio sub-frame corresponding to the subtitle;

Step 6, the multiplexer determines, according to the synchronization mark, a time offset of the time when the subtitle piece starts playing with respect to the audio;

Step 7: The multiplexer obtains the current time TOD from the local GPS as N, and the multiplexer obtains the time difference of the received and sent data according to the debugging statistics as R, and the units of N and R are seconds;

Step 8. The multiplexer starts a timer. When the timing time reaches S=S=I+J, the subtitle information is encapsulated into the data segment of the audio sub-frame corresponding to the subtitle. The format of the data segment is as shown in FIG. 3. Where; / = _ 3⁄4or(|M- N|) , J = floor(R) , ΟΟΓ(·) means round, |·| means absolute value, Μ means the start time of the local time zone of the subtitle, The unit is seconds;

The data segment is composed of a data segment header, a data unit 1, and a data unit 2 data unit η; wherein, the data segment header is further composed of a data unit number, a data unit 1 parameter, a data unit 2 parameter, ..., a data unit η parameter, CRC composition; the data unit η parameter includes two fields of data unit type and data unit length;

The data unit n is sequentially divided by the service ID (16bit), the reserved field (16bit), the encoding mode (32bit), the reserved field, the audio start playing time (32bit), and the time offset of the subtitle piece n relative to the audio start playing time (16bit). , subtitle piece n continuous play time (32bit), subtitle piece n length (16bit) and subtitle content;

Start time of the first subtitle of the first package to be packaged = audio start playback time + - N\ -I ) + ( R _J ) + time offset of the subtitle slice 1 relative to the start time of the audio] *22.5; 1 second n-th subtitle start time = audio start time +

- N\ -I )+

( R _J ) + time offset of the subtitle piece n relative to the start time of the audio *1000] * 22.5; Start time of the nth subtitle of the Hth second = audio start time + [( \M - N\ -I ) + ( R _ J ) + (time offset of the subtitle n relative to the audio start time -H *1000]*22.5; Step 9. The multiplexer encapsulates the subframe into a multiframe, and converts the multiframe into a PMS stream and sends it to the mobile multimedia broadcast terminal.

FIG. 4 is a block diagram showing the components of the mobile multimedia broadcast caption synchronization system of the embodiment.

The system includes:

Obtaining a module, configured to acquire a subtitle according to the subtitle data packet, and obtain an audio start playback time according to the audio sub-frame;

a time offset determining module, configured to determine a time offset of the subtitle relative to an audio start playing time;

The time offset determining module may be configured to: obtain information about a relationship between a subtitle and an audio sub-frame from a data stream that includes the subtitle data packet; and determine an audio sub-subtitle corresponding to the subtitle according to the information of the subtitle and the audio sub-frame relationship a frame; setting a synchronization flag of the subtitle and the audio sub-frame according to a start time of the subtitle in a local time zone and a time stamp of the audio sub-frame corresponding to the subtitle; determining, according to the synchronization flag, a subtitle relative to an audio start playing time Time offset

The encapsulating module is configured to, when the encapsulation time arrives, encapsulate a character string of the subtitle, an audio start playing time, and a time offset of the subtitle relative to the audio start playing time to the data of the audio sub-frame corresponding to the subtitle Paragraph

The start time of the subtitle in the local time zone is M, the current time TOD provided by the local GPS is N, the transmission time difference from receiving the subtitle data packet to the transmission data segment is R, and the encapsulation time is S; units of M, N, R and S In seconds;

J = floor(R); Then, S=I+J.

The subtitle string obtained by the obtaining module according to the subtitle data packet in the foregoing embodiment may be composed of a plurality of subtitle chips;

At this time, the time offset determining module is configured to determine a time offset of the subtitle piece relative to the audio start playing time according to the synchronization mark;

The time offset determining module can start a timer, set a timing length of the timer and a synchronization mark equivalent; when the timing length is reached, calculate a time offset of the subtitle piece relative to the audio start playing time.

a packaging module, configured to: when the encapsulation time arrives, time start playback time of the audio, time offset of the subtitle piece relative to the audio start play time, and a character string of the subtitle piece to the audio sub-frame corresponding to the subtitle Data segment.

One of ordinary skill in the art will appreciate that all or a portion of the above steps may be accomplished by a program instructing the associated hardware, such as a read-only memory, a magnetic disk, or an optical disk. Optionally, all or part of the steps of the foregoing embodiments may also be implemented by using one or more integrated circuits. Accordingly, each module/unit in the foregoing embodiment may be implemented in the form of hardware, or may use software functions. The form of the module is implemented. The invention is not limited to any specific form of combination of hardware and software.

It is to be understood that the invention may be embodied in other forms and modifications without departing from the spirit and scope of the invention.

INDUSTRIAL APPLICABILITY In the above technical solution, the data segment of the encapsulated audio sub-frame includes a time start of audio start time and a time offset of the subtitle relative to the audio start play time, so that the mobile multimedia broadcast terminal receiving the audio sub-frame can The time offset of the subtitle relative to the start time of the audio determines the playback time of the subtitle, achieving the technical effect of synchronizing with the audio.

Claims

Claim

A method for synchronizing mobile multimedia broadcast captions, the method comprising:

Receiving a subtitle data packet sent by the subtitle server and an audio sub-frame sent by the audio encoder; acquiring audio from the audio sub-frame to start playing time;

Obtaining a subtitle from the subtitle data packet;

The data segment is transmitted to a mobile multimedia broadcast terminal.

2. The method of claim 1 wherein

Determining, by the data stream containing the subtitle data packet, information about a relationship between the subtitle and the audio sub-frame; and determining the relationship between the subtitle and the audio sub-frame The information determines an audio sub-frame corresponding to the subtitle; and sets a synchronization mark of the subtitle and the audio sub-frame according to a start time of the subtitle in the local time zone and a time stamp of the audio sub-frame corresponding to the subtitle;

3. The method of claim 2, further comprising:

Determining the encapsulation time according to the start time of the subtitle in the local time zone, wherein: the start time of the subtitle in the local time zone is M, and the current time (TOD) provided by the local global positioning system (GPS) is N, and the subtitle data is received from the subtitle data. The time difference between the packet transmission and the transmission of the data segment is R, and the encapsulation time is S; the units of M, N, R, and S are seconds;

J = floor (K);

Then, S=I+J.

4. The method of claim 3, wherein

In the step of acquiring a subtitle from the subtitle data packet, the subtitle obtained according to the subtitle data packet is composed of a plurality of subtitle sheets;

In the step of determining a time offset of the subtitle relative to the audio start playing time, determining, according to the synchronization flag, a time offset of the subtitle piece relative to the audio start playing time;

In the step of encapsulating, a character string of the subtitle piece, a start playing time of the audio, and a time offset of the subtitle piece with respect to the audio start playing time are encapsulated into a data segment of the audio sub-frame corresponding to the subtitle.

5. The method of claim 4, wherein

The step of determining, according to the synchronization mark, a time offset of the subtitle piece relative to an audio start playing time includes:

6. A system for synchronizing mobile multimedia broadcast captions, the system comprising:

a receiving module, configured to: receive a subtitle data packet sent by the subtitle server and an audio sub-frame sent by the audio encoder;

An obtaining module, configured to: acquire a subtitle from the subtitle data packet, and obtain an audio start time from the audio sub-frame;

a time offset determining module, configured to: determine a time offset of the subtitle relative to an audio start playing time;

a packaging module, configured to: when the encapsulation time arrives, encapsulate a character string of the subtitle, an audio start playing time, and a time offset of the subtitle relative to an audio start playing time into an audio sub-frame corresponding to the subtitle Data segment; and

And a sending module, configured to: send the data segment to a mobile multimedia broadcast terminal.

7. The system according to claim 6, wherein the time offset determining module is configured to determine a time offset of the subtitle relative to an audio start playing time in the following manner: Acquiring information of a relationship between a subtitle and an audio sub-frame from a data stream that includes the subtitle data packet; determining an audio sub-frame corresponding to the subtitle according to the information of the subtitle and the audio sub-frame relationship; according to the start time of the subtitle in the local time zone And the time stamp of the audio sub-frame corresponding to the subtitle sets a synchronization mark of the subtitle and the audio sub-frame; and determining, according to the synchronization mark, a time offset of the sub-title relative to the audio start playing time.

8. The system of claim 7, wherein

The encapsulating module is further configured to determine the encapsulation time according to a start time of the subtitle in a local time zone in the following manner:

The start time of the local time zone is M, the current time (TOD) provided by the local global positioning system (GPS) is N, and the time difference from receiving the subtitle data packet to transmitting the data segment is R, and the encapsulation time is S; The units of M, N, R, and S are seconds;

J = floor{R); Then, S=I+J.

9. The method of claim 8 wherein

The obtaining module is configured to acquire a word composed of a plurality of subtitle sheets from the subtitle data packet;

The encapsulating module is configured to: when the encapsulation time arrives, encapsulate a character string of the subtitle piece, a start playing time of the audio, and a time offset of the subtitle piece relative to the audio start playing time to the audio corresponding to the subtitle The data segment of the sub-frame.

10. The system of claim 9, wherein

The time offset determination module is configured to determine a time offset of the slide show relative to the start time of the audio according to the synchronization flag in the following manner: