CN106454547B - A kind of real-time subtitle broadcasting method and system - Google Patents

A kind of real-time subtitle broadcasting method and system Download PDF

Info

Publication number
CN106454547B
CN106454547B CN201510491214.8A CN201510491214A CN106454547B CN 106454547 B CN106454547 B CN 106454547B CN 201510491214 A CN201510491214 A CN 201510491214A CN 106454547 B CN106454547 B CN 106454547B
Authority
CN
China
Prior art keywords
subtitle
media stream
segment
stream
media
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201510491214.8A
Other languages
Chinese (zh)
Other versions
CN106454547A (en
Inventor
朱小勇
耿立宏
郭志川
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhengzhou Xinrand Network Technology Co ltd
Original Assignee
Institute of Acoustics CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Institute of Acoustics CAS filed Critical Institute of Acoustics CAS
Priority to CN201510491214.8A priority Critical patent/CN106454547B/en
Publication of CN106454547A publication Critical patent/CN106454547A/en
Application granted granted Critical
Publication of CN106454547B publication Critical patent/CN106454547B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/488—Data services, e.g. news ticker
    • H04N21/4884—Data services, e.g. news ticker for displaying subtitles
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/845—Structuring of content, e.g. decomposing content into time segments
    • H04N21/8456—Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

The invention discloses real-time subtitle playing method and system, the method comprises the steps of segmenting a media stream to generate a plurality of media stream segments, extracting audio information of each of the plurality of media stream segments, generating subtitle streams according to the audio information, storing the plurality of subtitle streams, receiving a playing request sent by a terminal, wherein the playing request is used for searching a subtitle stream corresponding to a streaming media segment, and sending a th subtitle stream and a th streaming media segment to the terminal when determining that the subtitle stream is stored so that the terminal can analyze an th subtitle stream to obtain a th subtitle content and enable the th subtitle content and the th media stream segment to be played synchronously.

Description

一种实时字幕播出方法及系统A kind of real-time subtitle broadcasting method and system

技术领域technical field

本发明涉及多媒体通信领域，尤其涉及一种实时字幕播出方法及系统。The invention relates to the field of multimedia communication, in particular to a real-time subtitle broadcasting method and system.

背景技术Background technique

近年来，随着我国宽带网络建设和媒体技术的飞速发展，流媒体业务已逐渐成为互联网中最具代表性的应用。基于分段思想的媒体分发方法具有快速启动、自适应码率切换、用户体验好等优点，得到广泛应用。In recent years, with the rapid development of my country's broadband network construction and media technology, streaming media services have gradually become the most representative applications in the Internet. The media distribution method based on segmentation idea has the advantages of fast startup, adaptive bit rate switching, and good user experience, and is widely used.

目前，流媒体技术已经是比较成熟的媒体技术，广泛应用于电子商务、新闻发布、视频直播、视频点播、视频会议和即时通信等互联网信息服务上。为了给用户带来更丰富的体验，除了向用户提供优质的视频服务外，一般还提供字幕服务。对于点播媒体，工作人员有足够的时间对视频做后期处理，加上字幕；对于字幕显示位置固定或变化不大的直播，比如体彩号、比赛比分等，可以基于字幕模板实时更新固定位置的字幕信息；但对于更为广泛的直播，字幕无法实时生成。对于直播现场比较混乱的情况或者有听力障碍的观众，没有字幕的直播，将导致观众无法获取到准确信息，用户体验满意度将大大降低。At present, streaming media technology is a relatively mature media technology, which is widely used in Internet information services such as e-commerce, news release, live video, video on demand, video conferencing and instant messaging. In order to bring a richer experience to users, in addition to providing users with high-quality video services, subtitle services are also generally provided. For on-demand media, the staff has enough time to post-process the video and add subtitles; for live broadcasts where the subtitle display position is fixed or does not change much, such as sports lottery numbers, game scores, etc., the subtitles at the fixed position can be updated in real time based on the subtitle template information; however, for more extensive live broadcasts, subtitles cannot be generated in real-time. For audiences with chaotic live broadcasts or hearing-impaired audiences, live broadcasts without subtitles will result in the inability of audiences to obtain accurate information, and user experience satisfaction will be greatly reduced.

发明内容SUMMARY OF THE INVENTION

本发明的目的是为了解决现有技术存在的上述不足，提供一种通过对直播过程中的流媒体片段进行音频信息提取，从而生成字幕文件，使终端同步播放字幕内容和流媒体片段的方法。The purpose of the present invention is to solve the above-mentioned deficiencies in the prior art, and provide a method for generating subtitle files by extracting audio information from the streaming media segments in the live broadcast process, so that the terminal can synchronously play the subtitle content and the streaming media segments.

为实现上述目的，第一方面，本发明提供了一种实时字幕播出方法，该方法包括以下步骤：In order to achieve the above object, in the first aspect, the present invention provides a real-time subtitle broadcasting method, the method includes the following steps:

将媒体流分段生成多个媒体流片段；segment the media stream to generate multiple media stream segments;

提取多个媒体流片段中的每个媒体流片段的音频信息；extracting audio information for each of the multiple media stream segments;

根据音频信息生成字幕流，其中，字幕流包括，字幕文件和字幕文件对应的索引信息；Generate a subtitle stream according to the audio information, wherein the subtitle stream includes a subtitle file and index information corresponding to the subtitle file;

存储多个字幕流；Store multiple subtitle streams;

接收终端发送的播放请求，播放请求用于查找第一流媒体片段对应的第一字幕流；receiving a playback request sent by the terminal, where the playback request is used to find the first subtitle stream corresponding to the first streaming media segment;

当确定第一字幕流已存储，向终端发送第一字幕流和第一流媒体片段，以便终端解析第一字幕流，得到第一字幕内容，使第一字幕内容和第一媒体流片段同步播放。When it is determined that the first subtitle stream has been stored, the first subtitle stream and the first streaming media segment are sent to the terminal, so that the terminal can parse the first subtitle stream to obtain the first subtitle content and play the first subtitle content and the first media stream segment synchronously.

优选地，该方法还包括：当第一字幕流发送完毕，将存储的第一字幕流进行删除。Preferably, the method further includes: when the first subtitle stream is sent, deleting the stored first subtitle stream.

优选地，将媒体流分段生成多个媒体流片段，还包括：Preferably, segmenting the media stream to generate multiple media stream segments also includes:

确定多个媒体流片段中的一个媒体流片段为第一段媒体流片段；Determining that one media stream segment among the multiple media stream segments is the first segment of the media stream segment;

当多个媒体流片段中的一个媒体流片段为第一段媒体流片段时，生成第一段媒体流片段对应的索引信息。When one media stream segment among the multiple media stream segments is the first segment of the media stream segment, index information corresponding to the first segment of the media stream segment is generated.

优选地，将媒体流分段生成多个媒体流片段，还包括：Preferably, segmenting the media stream to generate multiple media stream segments also includes:

当多个媒体流片段中的一个媒体流片段为第N段媒体流片段时，确定第N-1段媒体流片段对应的字幕流已存储，其中，N为大于1的整数；When one of the multiple media stream segments is the Nth media stream segment, it is determined that the subtitle stream corresponding to the N-1th media stream segment has been stored, where N is an integer greater than 1;

当第N-1段媒体流片段对应的字幕流已存储时，生成第N段媒体流片段对应的索引信息。When the subtitle stream corresponding to the media stream segment of the N-1th segment has been stored, the index information corresponding to the media stream segment of the Nth segment is generated.

第二方面，本发明提供了一种实时字幕播出方法，该方法包括以下步骤：In a second aspect, the present invention provides a method for broadcasting real-time subtitles, the method comprising the following steps:

向服务器发送播放请求，播放请求用于查找第一流媒体片段对应的第一字幕流；Sending a playback request to the server, where the playback request is used to find the first subtitle stream corresponding to the first streaming media segment;

接收服务器发送的第一流媒体片段和第一媒体流片段对应的第一字幕流；receiving the first streaming media segment sent by the server and the first subtitle stream corresponding to the first media streaming segment;

解析第一字幕流，得到第一字幕内容，使第一字幕内容和第一媒体流片段同步播放。The first subtitle stream is parsed to obtain the first subtitle content, so that the first subtitle content and the first media stream segment are played synchronously.

第三方面，本发明提供了一种服务器，该服务器包括：编码模块、字幕生成模块、处理模块；In a third aspect, the present invention provides a server, the server comprising: an encoding module, a subtitle generating module, and a processing module;

编码模块用于将媒体流分段生成多个媒体流片段；The encoding module is used for segmenting the media stream to generate multiple media stream segments;

字幕生成模块用于提取多个媒体流片段中的每个媒体流片段的音频信息；并且，根据音频信息生成字幕流，其中，字幕流包括，字幕文件和字幕文件对应的索引信息；The subtitle generation module is used to extract the audio information of each media stream segment in the plurality of media stream segments; and, generate a subtitle stream according to the audio information, wherein the subtitle stream includes the subtitle file and the index information corresponding to the subtitle file;

处理模块用于存储多个字幕流；The processing module is used to store multiple subtitle streams;

处理模块还用于接收终端发送的播放请求，并根据播放请求，将第一媒体流片段和第一媒体流片段对应的第一字幕流发送至终端。The processing module is further configured to receive a play request sent by the terminal, and according to the play request, send the first media stream segment and the first subtitle stream corresponding to the first media stream segment to the terminal.

优选地，处理模块还用于，当第一流字幕流发送完毕，将存储的第一字幕流进行删除。Preferably, the processing module is further configured to delete the stored first subtitle stream when the first subtitle stream is sent.

优选地，编码模块还用于：Preferably, the encoding module is also used for:

确定多个媒体流片段中的一个媒体流片段为第一段媒体流片段；Determining that one media stream segment among the multiple media stream segments is the first segment of the media stream segment;

当多个媒体流片段中的一个媒体流片段为第一段媒体流片段时，生成第一段媒体流片段对应的索引信息。When one media stream segment among the multiple media stream segments is the first segment of the media stream segment, index information corresponding to the first segment of the media stream segment is generated.

优选地，编码模块还用于：Preferably, the encoding module is also used for:

当多个媒体流片段中的一个媒体流片段为第N段媒体流片段时，确定第N-1段媒体流片段对应的字幕流已存储，其中，N为大于1的整数；When one of the multiple media stream segments is the Nth media stream segment, it is determined that the subtitle stream corresponding to the N-1th media stream segment has been stored, where N is an integer greater than 1;

当第N-1段媒体流片段对应的字幕流已存储时，生成第N段媒体流片段对应的索引信息。When the subtitle stream corresponding to the media stream segment of the N-1th segment has been stored, the index information corresponding to the media stream segment of the Nth segment is generated.

第四方面，本发明提供了一种系统，该系统包括：终端和服务器；In a fourth aspect, the present invention provides a system, the system comprising: a terminal and a server;

该终端包括：发送模块、接收模块、解析模块和显示模块；The terminal includes: a sending module, a receiving module, a parsing module and a display module;

发送模块用于向服务器发送播放请求，播放请求用于查找第一流媒体片段对应的第一字幕流；The sending module is used to send a play request to the server, and the play request is used to find the first subtitle stream corresponding to the first streaming media segment;

接收模块用于接收服务器发送的第一流媒体片段和第一媒体流片段对应的第一字幕流；The receiving module is configured to receive the first streaming media segment sent by the server and the first subtitle stream corresponding to the first media streaming segment;

解析模块用于解析第一字幕流，得到第一字幕内容；The parsing module is used to parse the first subtitle stream to obtain the first subtitle content;

显示模块用于同步播放第一字幕内容和第一媒体流片段。The display module is used for synchronously playing the first subtitle content and the first media stream segment.

本发明通过对直播过程中的流媒体片段进行音频信息提取，实时生成每个流媒体片段对应的字幕流，从而使终端同步播放字幕内容和流媒体片段。同时，本发明将字幕流与流媒体片段分开传输，使字幕流只需以简单文本格式即可进行传输，简单高效；并且字幕的结构不受限制，可以轻易扩展，极大地提升了用户体验度。The present invention generates the subtitle stream corresponding to each streaming media segment in real time by extracting the audio information of the streaming media segment in the live broadcast process, so that the terminal can play the subtitle content and the streaming media segment synchronously. At the same time, the present invention transmits the subtitle stream and the streaming media segment separately, so that the subtitle stream can be transmitted only in a simple text format, which is simple and efficient; and the structure of the subtitle is not limited and can be easily expanded, which greatly improves the user experience. .

附图说明Description of drawings

图1为本发明实施例提供的一种实时字幕播出方法的示意过程图；1 is a schematic process diagram of a method for broadcasting real-time subtitles according to an embodiment of the present invention;

图2为本发明实施例提供的一种生成多个媒体流片段的方法流程图；2 is a flowchart of a method for generating multiple media stream segments provided by an embodiment of the present invention;

图3为本发明实施例提供的一种安装应用程序的方法流程图；3 is a flowchart of a method for installing an application program provided by an embodiment of the present invention;

图4为本发明实施例提供的一种实时字幕播出系统的结构性框图。FIG. 4 is a structural block diagram of a real-time subtitle broadcasting system according to an embodiment of the present invention.

具体实施方式Detailed ways

为使本发明实施例的技术方案以及优点表达的更清楚，下面通过附图和实施例，对本发明的技术方案做进一步的详细描述。In order to express the technical solutions and advantages of the embodiments of the present invention more clearly, the technical solutions of the present invention are further described in detail below through the accompanying drawings and embodiments.

图1为本发明实施例提供的一种实时字幕播出方法流程图，如图1所示，该方法包括：FIG. 1 is a flowchart of a method for broadcasting real-time subtitles provided by an embodiment of the present invention. As shown in FIG. 1 , the method includes:

步骤110，将媒体流分段生成多个媒体流片段。Step 110, segment the media stream to generate multiple media stream segments.

具体地，如图2所示，步骤110，将媒体流分段生成多个媒体流片段，还包括：Specifically, as shown in FIG. 2, step 110, segmenting the media stream to generate multiple media stream segments, further comprising:

步骤111，确定多个媒体流片段中的一个媒体流片段为第一段媒体流片段。Step 111: Determine one media stream segment among the multiple media stream segments as the first media stream segment.

当多个媒体流片段中的一个媒体流片段为第一段媒体流片段时，执行步骤113；When one of the multiple media stream segments is the first media stream segment, perform step 113;

当所述多个媒体流片段中的一个媒体流片段为第N段媒体流片段时，其中，N为大于1的整数，执行步骤112。When one of the multiple media stream segments is the Nth media stream segment, where N is an integer greater than 1, step 112 is performed.

步骤112，确定第N-1段媒体流片段对应的字幕流已存储，其中，N为大于1的整数。Step 112: Determine that the subtitle stream corresponding to the N-1th media stream segment has been stored, where N is an integer greater than 1.

当第N-1段媒体流片段对应的字幕流已存储完毕时，执行步骤113；When the subtitle stream corresponding to the N-1 media stream segment has been stored, step 113 is performed;

若第N-1段媒体流片段对应的字幕流未存储完毕，则挂起等待，直至第N-1段媒体流片段对应的字幕流存储完毕，再执行步骤113。If the subtitle stream corresponding to the N-1 segment of the media stream segment has not been stored, the process suspends and waits until the subtitle stream corresponding to the N-1 segment of the media stream segment is stored, and then executes step 113.

步骤113，生成第N段媒体流片段对应的索引信息。Step 113: Generate index information corresponding to the Nth media stream segment.

步骤120，提取多个媒体流片段中的每个媒体流片段的音频信息。Step 120: Extract audio information of each media stream segment in the multiple media stream segments.

需要说明的是，若无媒体流片段生成时，则挂起等待。It should be noted that, if no media stream segment is generated, it will suspend and wait.

步骤130，根据音频信息生成字幕流，其中，字幕流包括，字幕文件和字幕文件对应的索引信息。Step 130: Generate a subtitle stream according to the audio information, wherein the subtitle stream includes subtitle files and index information corresponding to the subtitle files.

具体地，生成字幕时需记录音频相对时间与当前字幕内容的对应关系以及当前音频片段在整个直播媒体片段中的序号，并以该媒体片段序号为区分特征为该字幕文件命名。索引信息指向字幕文件的存储位置，使终端能正确地请求到该字幕文件。Specifically, when generating subtitles, it is necessary to record the corresponding relationship between the audio relative time and the current subtitle content and the serial number of the current audio clip in the entire live media clip, and use the media clip serial number as a distinguishing feature to name the subtitle file. The index information points to the storage location of the subtitle file, so that the terminal can correctly request the subtitle file.

步骤140，存储多个字幕流。Step 140, store multiple subtitle streams.

具体地，当存储第一段字幕流时，新建一个索引文件，并添加索引信息。当存储第N段字幕流时，将新生成的索引信息添加到原索引文件末尾，同时删除索引文件中最前端、已被请求过的片段索引信息。同时，将字幕文件存到索引指定的位置，并删除索引文件中最前端索引对应的字幕文件。Specifically, when the first subtitle stream is stored, a new index file is created, and index information is added. When storing the subtitle stream of the Nth segment, the newly generated index information is added to the end of the original index file, and at the same time, the segment index information that has been requested at the front end of the index file is deleted. At the same time, the subtitle file is saved to the position specified by the index, and the subtitle file corresponding to the foremost index in the index file is deleted.

步骤145，终端向服务器发送播放请求，播放请求用于查找第一流媒体片段对应的第一字幕流。Step 145: The terminal sends a play request to the server, where the play request is used to search for the first subtitle stream corresponding to the first streaming media segment.

可选地，播放请求可以包括：请求第一索引文件，根据第一索引文件信息请求第一字幕文件。Optionally, the play request may include: requesting a first index file, and requesting a first subtitle file according to the information of the first index file.

步骤150，接收终端发送的播放请求，播放请求用于查找第一流媒体片段对应的第一字幕流。Step 150: Receive a play request sent by the terminal, where the play request is used to search for the first subtitle stream corresponding to the first streaming media segment.

步骤160，当确定第一字幕流已存储。Step 160, when it is determined that the first subtitle stream has been stored.

需要说明的是，当第一字幕流未存储，则拒绝该请求。终端收到请求失败信息后，将增大向服务器请求字幕索引文件的频率，继续请求该索引文件，直至请求成功或超时后进入下一段的请求过程。It should be noted that when the first subtitle stream is not stored, the request is rejected. After receiving the request failure information, the terminal will increase the frequency of requesting the subtitle index file from the server, and continue to request the index file until the request succeeds or the next stage of the request process is entered after timeout.

步骤165，向终端发送第一字幕流和第一流媒体片段，以便终端解析第一字幕流，得到第一字幕内容，使第一字幕内容和第一媒体流片段同步播放。Step 165: Send the first subtitle stream and the first streaming media segment to the terminal, so that the terminal parses the first subtitle stream, obtains the first subtitle content, and causes the first subtitle content and the first media stream segment to be played synchronously.

步骤170，接收服务器发送的第一流媒体片段和第一媒体流片段对应的第一字幕流。Step 170: Receive the first streaming media segment sent by the server and the first subtitle stream corresponding to the first media streaming segment.

步骤180，解析第一字幕流，得到第一字幕内容。Step 180: Parse the first subtitle stream to obtain first subtitle content.

具体地，如图3所示，解析第一字幕流具体步骤如下：Specifically, as shown in Figure 3, the specific steps of parsing the first subtitle stream are as follows:

步骤181，解析字幕格式。Step 181, parse the subtitle format.

若字幕格式正确，则执行步骤182；若字幕格式不正确，停止操作。If the subtitle format is correct, go to step 182; if the subtitle format is incorrect, stop the operation.

步骤182，解析字幕文件。Step 182, parse the subtitle file.

具体地，解析字幕文件名，提取段序号信息n；解析字幕文件序号，该序号为字幕文件中当前字幕的排序；解析字幕的相对显示时间，设某个序号字幕的相对显示时间为t1和t2。Specifically, parse the subtitle file name to extract the segment serial number information n; parse the subtitle file serial number, which is the order of the current subtitles in the subtitle file; parse the relative display time of the subtitle, and set the relative display time of a certain serial number subtitle to be t1 and t2 .

步骤183，生成绝对显示时间戳。Step 183, generate an absolute display time stamp.

绝对显示时间T1＝(n-1)·d+t1，同理，T2＝(n-1)·d+t2，其中，d为媒体流片段大小，单位秒。The absolute display time T 1 =(n-1)·d+t 1 , and similarly, T 2 =(n-1)·d+t 2 , where d is the size of the media stream segment, in seconds.

若步骤182中解析的信息不足以生成绝对时间戳，停止操作。If the information parsed in step 182 is insufficient to generate an absolute timestamp, the operation is stopped.

步骤184，更新字幕显示时间。Step 184, update the subtitle display time.

具体地，将相对显示时间替换为绝对显示时间。Specifically, the relative display time is replaced with the absolute display time.

步骤185，生成字幕内容。Step 185, generating subtitle content.

步骤190，同步播放第一字幕内容和第一媒体流片段。Step 190: Play the first subtitle content and the first media stream segment synchronously.

首先获取第一媒体流片段的时间位置信息，将此位置信息转为以秒为单位的播放时间值，然后用当前播放时间值与字幕文件中绝对显示时间值作比较，当该播放时间值落入某个序号绝对显示时间内时，显示当前序号下字幕内容。First, obtain the time position information of the first media stream segment, convert this position information into the playback time value in seconds, and then compare the current playback time value with the absolute display time value in the subtitle file. When the playback time value falls When the absolute display time of a certain serial number is entered, the subtitle content under the current serial number is displayed.

步骤195，当第一流字幕流发送完毕，将存储的第一字幕流进行删除。Step 195, when the first subtitle stream is sent, delete the stored first subtitle stream.

本发明实施例通过对直播过程中的流媒体片段进行音频信息提取，实时生成每个流媒体片段对应的字幕流，从而使终端同步播放字幕内容和流媒体片段。同时，本发明实施例将字幕流与流媒体片段分开传输，使字幕流只需以简单文本格式即可进行传输，简单高效；并且字幕的结构不受限制，可以轻易扩展，极大地提升了用户体验度The embodiment of the present invention generates a subtitle stream corresponding to each streaming media segment in real time by extracting audio information from the streaming media segment in the live broadcast process, so that the terminal can play the subtitle content and the streaming media segment synchronously. At the same time, the embodiment of the present invention transmits the subtitle stream and the streaming media segment separately, so that the subtitle stream can be transmitted only in a simple text format, which is simple and efficient; and the structure of the subtitle is not limited and can be easily expanded, which greatly improves the user experience. Experience

第二方面，图4为本发明实施例提供的一种实时字幕播出系统的结构性框图，如图4所示，该系统包括：服务器40和终端50。In the second aspect, FIG. 4 is a structural block diagram of a real-time subtitle broadcasting system provided by an embodiment of the present invention. As shown in FIG. 4 , the system includes: a server 40 and a terminal 50 .

该服务器40包括：编码模块41、字幕生成模块42、处理模块43。The server 40 includes: an encoding module 41 , a subtitle generation module 42 , and a processing module 43 .

编码模块41用于将媒体流分段生成多个媒体流片段。The encoding module 41 is used for segmenting the media stream to generate a plurality of media stream segments.

字幕生成模块42用于提取多个媒体流片段中的每个媒体流片段的音频信息；并且，根据音频信息生成字幕流，其中，字幕流包括，字幕文件和字幕文件对应的索引信息。The subtitle generation module 42 is configured to extract audio information of each media stream segment in the multiple media stream segments; and generate a subtitle stream according to the audio information, wherein the subtitle stream includes subtitle files and index information corresponding to the subtitle files.

处理模块43用于存储多个字幕流。The processing module 43 is used for storing multiple subtitle streams.

处理模块43还用于接收终端50发送的播放请求，并根据播放请求，将第一媒体流片段和第一媒体流片段对应的第一字幕流发送至终端50。The processing module 43 is further configured to receive a playback request sent by the terminal 50 , and send the first media stream segment and the first subtitle stream corresponding to the first media stream segment to the terminal 50 according to the playback request.

可选地，处理模块43还用于，当第一流字幕流发送完毕，将存储的第一字幕流进行删除。Optionally, the processing module 43 is further configured to delete the stored first subtitle stream when the first subtitle stream is sent.

具体地，编码模块41还用于：Specifically, the encoding module 41 is also used for:

确定多个媒体流片段中的一个媒体流片段为第一段媒体流片段；Determining that one media stream segment among the multiple media stream segments is the first segment of the media stream segment;

当多个媒体流片段中的一个媒体流片段为第一段媒体流片段时，生成第一段媒体流片段对应的索引信息。When one media stream segment among the multiple media stream segments is the first segment of the media stream segment, index information corresponding to the first segment of the media stream segment is generated.

具体地，编码模块41还用于：Specifically, the encoding module 41 is also used for:

当多个媒体流片段中的一个媒体流片段为第N段媒体流片段时，确定第N-1段媒体流片段对应的字幕流已存储，其中，N为大于1的整数；When one of the multiple media stream segments is the Nth media stream segment, it is determined that the subtitle stream corresponding to the N-1th media stream segment has been stored, where N is an integer greater than 1;

当第N-1段媒体流片段对应的字幕流已存储时，生成第N段媒体流片段对应的索引信息。When the subtitle stream corresponding to the media stream segment of the N-1th segment has been stored, the index information corresponding to the media stream segment of the Nth segment is generated.

该终端50包括：发送模块51、接收模块52、解析模块53和显示模块54。The terminal 50 includes: a sending module 51 , a receiving module 52 , a parsing module 53 and a display module 54 .

发送模块51用于向服务器40发送播放请求，播放请求用于查找第一流媒体片段对应的第一字幕流。The sending module 51 is configured to send a play request to the server 40, where the play request is used to search for the first subtitle stream corresponding to the first streaming media segment.

接收模块52用于接收服务器40发送的第一流媒体片段和第一媒体流片段对应的第一字幕流。The receiving module 52 is configured to receive the first streaming media segment sent by the server 40 and the first subtitle stream corresponding to the first media streaming segment.

解析模块53用于解析第一字幕流，得到第一字幕内容。The parsing module 53 is configured to parse the first subtitle stream to obtain the first subtitle content.

显示模块54用于同步播放第一字幕内容和第一媒体流片段。The display module 54 is configured to play the first subtitle content and the first media stream segment synchronously.

本发明实施例通过对直播过程中的流媒体片段进行音频信息提取，实时生成每个流媒体片段对应的字幕流，从而使终端同步播放字幕内容和流媒体片段。同时，本发明实施例将字幕流与流媒体片段分开传输，使字幕流只需以简单文本格式即可进行传输，简单高效；并且字幕的结构不受限制，可以轻易扩展，极大地提升了用户体验度。The embodiment of the present invention generates a subtitle stream corresponding to each streaming media segment in real time by extracting audio information from the streaming media segment in the live broadcast process, so that the terminal can play the subtitle content and the streaming media segment synchronously. At the same time, the embodiment of the present invention transmits the subtitle stream and the streaming media segment separately, so that the subtitle stream can be transmitted only in a simple text format, which is simple and efficient; and the structure of the subtitle is not limited and can be easily expanded, which greatly improves the user experience. experience.

专业人员应该还可以进一步意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、计算机软件或者二者的结合来实现，为了清楚地说明硬件和软件的可互换性，在上述说明中已经按照功能一般性地描述了各示例的组成及步骤。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本发明的范围。Professionals should be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. Interchangeability, the above description has generally described the components and steps of each example in terms of function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of the present invention.

结合本文中所公开的实施例描述的方法或算法的步骤可以用硬件、处理器执行的软件模块，或者二者的结合来实施。软件模块可以置于随机存储器(RAM)、内存、只读存储器(ROM)、电可编程ROM、电可擦除可编程ROM、寄存器、硬盘、可移动磁盘、CD-ROM、或技术领域内所公知的任意其它形式的存储介质中。The steps of a method or algorithm described in connection with the embodiments disclosed herein may be implemented in hardware, a software module executed by a processor, or a combination of the two. A software module can be placed in random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other in the technical field. in any other known form of storage medium.

以上所述的具体实施方式，对本发明的目的、技术方案和有益效果进行了进一步详细说明，所应理解的是，以上所述仅为本发明的具体实施方式而已，并不用于限定本发明的保护范围，凡在本发明的精神和原则之内，所做的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。The specific embodiments described above further describe the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above descriptions are only specific embodiments of the present invention, and are not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (8)

1.一种实时字幕播出方法，其特征在于，该方法包括：1. a real-time subtitle broadcasting method, is characterized in that, the method comprises: 将媒体流分段生成多个媒体流片段；其中，确定所述多个媒体流片段中的一个媒体流片段为第一媒体流片段；当所述多个媒体流片段中的一个媒体流片段为第一媒体流片段时，生成所述第一媒体流片段对应的索引信息；当所述多个媒体流片段中的一个媒体流片段为第N段媒体流片段时，确定第N-1段媒体流片段对应的字幕流已存储，其中，N为大于1的整数；当所述第N-1段媒体流片段对应的字幕流已存储时，生成所述第N段媒体流片段对应的索引信息；The media stream is segmented to generate multiple media stream segments; wherein, one media stream segment in the multiple media stream segments is determined as the first media stream segment; when one media stream segment in the multiple media stream segments is When the first media stream segment is the first media stream segment, index information corresponding to the first media stream segment is generated; when one media stream segment in the multiple media stream segments is the Nth segment of the media stream segment, determine the N-1th segment of media The subtitle stream corresponding to the stream segment has been stored, where N is an integer greater than 1; when the subtitle stream corresponding to the N-1th media stream segment has been stored, generate index information corresponding to the Nth segment of the media stream segment ; 提取所述多个媒体流片段中的每个媒体流片段的音频信息；extracting audio information of each media stream segment in the plurality of media stream segments; 根据所述音频信息生成字幕流，其中，所述字幕流包括，字幕文件和所述字幕文件对应的索引信息；Generating a subtitle stream according to the audio information, wherein the subtitle stream includes a subtitle file and index information corresponding to the subtitle file; 存储多个所述字幕流；其中，当存储第一字幕流时，新建一个索引文件，并添加索引信息；当存储第N段字幕流时，将新生成的索引信息添加到原索引文件末尾，同时删除索引文件中最前端、已被请求过的片段索引信息；同时，将字幕文件存到索引指定的位置，并删除索引文件中最前端索引对应的字幕文件；Store a plurality of described subtitle streams; wherein, when storing the first subtitle stream, create a new index file and add index information; when storing the Nth subtitle stream, add the newly generated index information to the end of the original index file, At the same time, delete the index information of the most front-end and requested segment in the index file; at the same time, save the subtitle file to the position specified by the index, and delete the subtitle file corresponding to the foremost index in the index file; 接收终端发送的播放请求，所述播放请求用于查找第一媒体流片段对应的第一字幕流；receiving a playback request sent by the terminal, where the playback request is used to search for the first subtitle stream corresponding to the first media stream segment; 当确定所述第一字幕流已存储，向所述终端发送所述第一字幕流和所述第一媒体流片段，以便所述终端解析所述第一字幕流，得到第一字幕内容，使所述第一字幕内容和所述第一媒体流片段同步播放。When it is determined that the first subtitle stream has been stored, send the first subtitle stream and the first media stream segment to the terminal, so that the terminal can parse the first subtitle stream and obtain the first subtitle content, so that the The first subtitle content and the first media stream segment are played synchronously. 2.根据权利要求1所述的方法，其特征在于，所述方法还包括：当所述第一字幕流发送完毕，将存储的所述第一字幕流进行删除。2 . The method according to claim 1 , further comprising: deleting the stored first subtitle stream when the first subtitle stream is sent. 3 . 3.一种实时字幕播出方法，其特征在于，所述方法包括：3. a real-time subtitle broadcasting method, is characterized in that, described method comprises: 向服务器发送播放请求，所述播放请求用于查找第一媒体流片段对应的第一字幕流；其中，所述第一媒体流片段为多个媒体流片段中的一个媒体流片段；当所述多个媒体流片段中的一个媒体流片段为第一媒体流片段时，生成所述第一媒体流片段对应的索引信息；当所述多个媒体流片段中的一个媒体流片段为第N段媒体流片段时，确定第N-1段媒体流片段对应的字幕流已存储，其中，N为大于1的整数；当所述第N-1段媒体流片段对应的字幕流已存储时，生成所述第N段媒体流片段对应的索引信息；Send a playback request to the server, where the playback request is used to find the first subtitle stream corresponding to the first media stream segment; wherein, the first media stream segment is a media stream segment among multiple media stream segments; when the When one of the multiple media stream segments is the first media stream segment, index information corresponding to the first media stream segment is generated; when one of the multiple media stream segments is the Nth segment When a media stream fragment is used, it is determined that the subtitle stream corresponding to the N-1st media stream fragment has been stored, where N is an integer greater than 1; when the subtitle stream corresponding to the N-1th media stream fragment has been stored, generate index information corresponding to the Nth media stream segment; 接收所述服务器发送的所述第一媒体流片段和所述第一媒体流片段对应的第一字幕流；其中，当服务器存储第一字幕流时，新建一个索引文件，并添加索引信息；当服务器存储第N段字幕流时，将新生成的索引信息添加到原索引文件末尾，同时删除索引文件中最前端、已被请求过的片段索引信息；同时，将字幕文件存到索引指定的位置，并删除索引文件中最前端索引对应的字幕文件；Receive the first media stream segment sent by the server and the first subtitle stream corresponding to the first media stream segment; wherein, when the server stores the first subtitle stream, a new index file is created, and index information is added; when When the server stores the Nth subtitle stream, it adds the newly generated index information to the end of the original index file, and deletes the index information of the segment that has been requested at the forefront of the index file; at the same time, saves the subtitle file to the location specified by the index , and delete the subtitle file corresponding to the front-end index in the index file; 解析所述第一字幕流，得到第一字幕内容，使所述第一字幕内容和所述第一媒体流片段同步播放。The first subtitle stream is parsed to obtain first subtitle content, and the first subtitle content and the first media stream segment are played synchronously. 4.一种服务器，其特征在于，所述服务器包括：编码模块、字幕生成模块、处理模块；4. A server, characterized in that the server comprises: an encoding module, a subtitle generation module, and a processing module; 所述编码模块用于将媒体流分段生成多个媒体流片段；其中，确定所述多个媒体流片段中的一个媒体流片段为第一媒体流片段；当所述多个媒体流片段中的一个媒体流片段为第一媒体流片段时，生成所述第一媒体流片段对应的索引信息；当所述多个媒体流片段中的一个媒体流片段为第N段媒体流片段时，确定第N-1段媒体流片段对应的字幕流已存储，其中，N为大于1的整数；当所述第N-1段媒体流片段对应的字幕流已存储时，生成所述第N段媒体流片段对应的索引信息；The encoding module is configured to segment the media stream to generate multiple media stream segments; wherein, one media stream segment in the multiple media stream segments is determined as the first media stream segment; when the multiple media stream segments are When one of the media stream segments is the first media stream segment, generate index information corresponding to the first media stream segment; when one media stream segment in the multiple media stream segments is the Nth media stream segment, determine The subtitle stream corresponding to the N-1 media stream segment has been stored, where N is an integer greater than 1; when the subtitle stream corresponding to the N-1 media stream segment has been stored, the N-th media segment is generated. The index information corresponding to the stream segment; 所述字幕生成模块用于提取所述多个媒体流片段中的每个媒体流片段的音频信息；并且，根据所述音频信息生成字幕流，其中，所述字幕流包括，字幕文件和所述字幕文件对应的索引信息；The subtitle generation module is configured to extract audio information of each media stream segment in the multiple media stream segments; and generate a subtitle stream according to the audio information, wherein the subtitle stream includes a subtitle file and the Index information corresponding to the subtitle file; 所述处理模块用于存储多个所述字幕流；其中，当存储第一字幕流时，新建一个索引文件，并添加索引信息；当存储第N段字幕流时，将新生成的索引信息添加到原索引文件末尾，同时删除索引文件中最前端、已被请求过的片段索引信息；同时，将字幕文件存到索引指定的位置，并删除索引文件中最前端索引对应的字幕文件；The processing module is used to store a plurality of the subtitle streams; wherein, when the first subtitle stream is stored, a new index file is created and index information is added; when the Nth subtitle stream is stored, the newly generated index information is added To the end of the original index file, at the same time delete the index information of the segment that has been requested in the index file; at the same time, save the subtitle file to the location specified by the index, and delete the subtitle file corresponding to the foremost index in the index file; 所述处理模块还用于接收终端发送的播放请求，并根据所述播放请求，将所述第一媒体流片段和所述第一媒体流片段对应的第一字幕流发送至终端。The processing module is further configured to receive a play request sent by the terminal, and send the first media stream segment and the first subtitle stream corresponding to the first media stream segment to the terminal according to the play request. 5.根据权利要求4所述的服务器，其特征在于，5. The server according to claim 4, characterized in that, 所述处理模块还用于，当所述第一字幕流发送完毕，将存储的所述第一字幕流进行删除。The processing module is further configured to delete the stored first subtitle stream when the first subtitle stream is sent. 6.根据权利要求4所述的服务器，其特征在于，所述编码模块还用于：6. The server according to claim 4, wherein the encoding module is also used for: 确定所述多个媒体流片段中的一个媒体流片段为第一段媒体流片段；determining that a media stream segment in the plurality of media stream segments is the first media stream segment; 当所述多个媒体流片段中的一个媒体流片段为第一段媒体流片段时，生成所述第一段媒体流片段对应的索引信息。When one media stream segment among the plurality of media stream segments is the first segment of the media stream segment, index information corresponding to the first segment of the media stream segment is generated. 7.根据权利要求4所述的服务器，其特征在于，所述编码模块还用于：7. The server according to claim 4, wherein the encoding module is also used for: 当所述多个媒体流片段中的一个媒体流片段为第N段媒体流片段时，确定第N-1段媒体流片段对应的字幕流已存储，其中，N为大于1的整数；When a media stream segment in the plurality of media stream segments is the Nth media stream segment, it is determined that the subtitle stream corresponding to the N-1th media stream segment has been stored, where N is an integer greater than 1; 当所述第N-1段媒体流片段对应的字幕流已存储时，生成所述第N段媒体流片段对应的索引信息。When the subtitle stream corresponding to the N-1th media stream segment has been stored, the index information corresponding to the Nth segment of the media stream segment is generated. 8.一种实时字幕播出系统，其特征在于，所述系统包括：终端和如权利要求4-7任一所述的服务器；8. A system for broadcasting real-time subtitles, wherein the system comprises: a terminal and the server according to any one of claims 4-7; 所述终端包括：发送模块、接收模块、解析模块和显示模块；The terminal includes: a sending module, a receiving module, a parsing module and a display module; 所述发送模块用于向服务器发送播放请求，所述播放请求用于查找第一媒体流片段对应的第一字幕流；The sending module is configured to send a playback request to the server, where the playback request is used to find the first subtitle stream corresponding to the first media stream segment; 所述接收模块用于接收所述服务器发送的所述第一媒体流片段和所述第一媒体流片段对应的第一字幕流；The receiving module is configured to receive the first media stream segment sent by the server and the first subtitle stream corresponding to the first media stream segment; 所述解析模块用于解析所述第一字幕流，得到第一字幕内容；The parsing module is configured to parse the first subtitle stream to obtain first subtitle content; 所述显示模块用于同步播放所述第一字幕内容和所述第一媒体流片段。The display module is configured to play the first subtitle content and the first media stream segment synchronously.
CN201510491214.8A 2015-08-11 2015-08-11 A kind of real-time subtitle broadcasting method and system Active CN106454547B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201510491214.8A CN106454547B (en) 2015-08-11 2015-08-11 A kind of real-time subtitle broadcasting method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201510491214.8A CN106454547B (en) 2015-08-11 2015-08-11 A kind of real-time subtitle broadcasting method and system

Publications (2)

Publication Number Publication Date
CN106454547A CN106454547A (en) 2017-02-22
CN106454547B true CN106454547B (en) 2020-01-31

Family

ID=58093718

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201510491214.8A Active CN106454547B (en) 2015-08-11 2015-08-11 A kind of real-time subtitle broadcasting method and system

Country Status (1)

Country Link
CN (1) CN106454547B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019194742A1 (en) * 2018-04-04 2019-10-10 Nooggi Pte Ltd A method and system for promoting interaction during live streaming events

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101382937A (en) * 2008-07-01 2009-03-11 深圳先进技术研究院 Speech recognition-based multimedia resource processing method and its online teaching system
CN101540847A (en) * 2008-03-21 2009-09-23 株式会社康巴思 Caption creation system and caption creation method
CN102802044A (en) * 2012-06-29 2012-11-28 华为终端有限公司 Video processing method, terminal and subtitle server
CN103297709A (en) * 2013-06-19 2013-09-11 江苏华音信息科技有限公司 Device for adding Chinese subtitles to Chinese audio video data
CN103544978A (en) * 2013-11-07 2014-01-29 上海斐讯数据通信技术有限公司 Multimedia file manufacturing and playing method and intelligent terminal
CN103561217A (en) * 2013-10-14 2014-02-05 深圳创维数字技术股份有限公司 Method and terminal for generating captions

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8914534B2 (en) * 2011-01-05 2014-12-16 Sonic Ip, Inc. Systems and methods for adaptive bitrate streaming of media stored in matroska container files using hypertext transfer protocol

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101540847A (en) * 2008-03-21 2009-09-23 株式会社康巴思 Caption creation system and caption creation method
CN101382937A (en) * 2008-07-01 2009-03-11 深圳先进技术研究院 Speech recognition-based multimedia resource processing method and its online teaching system
CN102802044A (en) * 2012-06-29 2012-11-28 华为终端有限公司 Video processing method, terminal and subtitle server
CN103297709A (en) * 2013-06-19 2013-09-11 江苏华音信息科技有限公司 Device for adding Chinese subtitles to Chinese audio video data
CN103561217A (en) * 2013-10-14 2014-02-05 深圳创维数字技术股份有限公司 Method and terminal for generating captions
CN103544978A (en) * 2013-11-07 2014-01-29 上海斐讯数据通信技术有限公司 Multimedia file manufacturing and playing method and intelligent terminal

Also Published As

Publication number Publication date
CN106454547A (en) 2017-02-22

Similar Documents

Publication Publication Date Title
KR102091414B1 (en) Enriching broadcast media related electronic messaging
US9961398B2 (en) Method and device for switching video streams
US20160316233A1 (en) System and method for inserting, delivering and tracking advertisements in a media program
CN104023250B (en) Based on the real-time interactive method and system of Streaming Media
WO2015090095A1 (en) Information pushing method, device, and system
CN104902343B (en) A kind of method, server and the terminal of transmission and playing audio-video and message
JP2002297630A (en) Method and device for index generation, index addition system, program, and storage medium
US12206961B2 (en) Systems and methods for summarizing missed portions of storylines
KR101991188B1 (en) Promotion information processing method, device, and apparatus, and non-volatile computer storage medium
CN112954434B (en) Subtitle processing method, system, electronic device and storage medium
US11540028B2 (en) Information presenting method, terminal device, server and system
TWI802797B (en) Method for detecting and responding to a fingerprint mismatch detected after a previously detected fingerprint match, non-transitory computer-readable storage medium, and computing system
CN109168020A (en) Method for processing video frequency, device, calculating equipment and storage medium based on live streaming
CN103179450A (en) Method, device and system for playing video, and audio track server
US20220321970A1 (en) Dynamic Real-Time Audio-Visual Search Result Assembly
US20170180445A1 (en) Advertisement data acquisition method and electronic equipment
CN111512635A (en) Method and system for selectively skipping media content
CN116346794A (en) Method, device and non-volatile computer-readable storage medium for receiving media data
CN109842804A (en) Processing method and server, the computer storage medium of audio, video data
CN115623264A (en) Live stream subtitle processing method and device and live stream playing method and device
WO2012071931A1 (en) Method and system for dynamically inserting cut-in program into segmented program
US20120150990A1 (en) System and method for synchronizing with multimedia broadcast program and computer program product thereof
CN106454547B (en) A kind of real-time subtitle broadcasting method and system
CN104602046A (en) HLS protocol based information publishing method, device and system
CN109587517A (en) Multimedia file playing method and device, server and storage medium

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right

Effective date of registration: 20210804

Address after: Room 1601, 16th floor, East Tower, Ximei building, No. 6, Changchun Road, high tech Industrial Development Zone, Zhengzhou, Henan 450001

Patentee after: Zhengzhou xinrand Network Technology Co.,Ltd.

Address before: 100190, No. 21 West Fourth Ring Road, Beijing, Haidian District

Patentee before: INSTITUTE OF ACOUSTICS, CHINESE ACADEMY OF SCIENCES

TR01 Transfer of patent right