CN106993239B

CN106993239B - Information display method in live broadcast process

Info

Publication number: CN106993239B
Application number: CN201710198538.1A
Authority: CN
Inventors: 吕现广
Original assignee: Guangzhou Kugou Computer Technology Co Ltd
Current assignee: Guangzhou Kugou Computer Technology Co Ltd
Priority date: 2017-03-29
Filing date: 2017-03-29
Publication date: 2019-12-10
Anticipated expiration: 2037-03-29
Also published as: CN106993239A

Abstract

the invention discloses an information display method and device in a live broadcast process, and belongs to the field of live broadcast. The method comprises the following steps: when the stream pushing client plays the multimedia file in the live broadcasting process, the video frame for live broadcasting is sent to the stream media server, the file identification of the multimedia file and the playing progress of the multimedia file are encapsulated to obtain a pseudo video frame, and the pseudo video frame is sent. That is, the stream pushing client sends the video frame and the pseudo video frame respectively, instead of converting the prompt information corresponding to the multimedia file at the current playing progress into an image and synthesizing the image with the current video frame, so that the stream pulling player end can display the video frame and simultaneously independently display the prompt information corresponding to the multimedia file at the current playing progress, a user of the stream pulling player end can conveniently and independently perform operations such as moving and closing on the prompt information corresponding to the multimedia file, and the flexibility of information display in the live broadcasting process is improved.

Description

Information display method in live broadcast process

Technical Field

the invention relates to the field of live broadcasting, in particular to an information display method and device in a live broadcasting process.

Background

Live broadcasting, that is, a terminal acquires a video in real time and sends the video to other terminals through a streaming media server in real time, so that other users can watch the video through the other terminals in real time. When playing multimedia files such as songs or movies in the live broadcasting process, in order to enrich the display content of the live broadcasting, other terminals can also display prompt information corresponding to the multimedia files, such as lyrics or subtitles, on the current playing interface when playing the multimedia files, and the displayed prompt information needs to be kept synchronous with the playing of the multimedia files.

for convenience of description, a terminal performing live video is referred to as a push streaming client, and a terminal playing live video is referred to as a pull streaming player. Currently, live video broadcasting is mainly completed based on an RTMP (Real Time Messaging Protocol) Protocol, but the RTMP Protocol does not provide direct support for transmission of prompt information corresponding to the multimedia file, so that in order to achieve synchronization between prompt information displayed at a stream player end and a corresponding multimedia file during live video broadcasting, a stream push client converts the prompt information corresponding to the current playing progress of the multimedia file into an image during live video broadcasting, synthesizes the converted image with a first video frame generated by the current live broadcasting to obtain a second video frame, and sends the second video frame to the stream player end through a stream media server. And when receiving the second video frame, the pull stream player terminal displays a live broadcast picture and prompt information based on the second video frame and keeps the second video frame and the corresponding audio frame synchronous through an RTMP protocol. The second video frame comprises the prompt information corresponding to the multimedia file at the current playing progress, so that the displayed prompt information and the corresponding multimedia file are kept synchronous. For example, it may be achieved that songs played and lyrics displayed in a live video remain synchronized.

the second video frame received by the pull stream player end is obtained by synthesizing the first video frame and the image of the multimedia file after the prompt information corresponding to the current playing progress is converted, that is, in the current interface displayed by the pull stream player end, the first video frame and the prompt information corresponding to the multimedia file at the current playing progress are integrated, so that other users cannot independently perform any operation on the prompt information corresponding to the multimedia file at the current playing progress, and the flexibility of displaying information in the live broadcasting process is seriously reduced.

disclosure of Invention

In order to solve the problem that other users cannot independently perform any operation on the prompt information of the multimedia file in the current playing progress in the prior art, the embodiment of the invention provides an information display method in the live broadcasting process. The technical scheme is as follows:

In a first aspect, an information display method in a live broadcast process is provided, where the information display method in the live broadcast process is applied to a push streaming client, and the method includes:

acquiring a video frame generated by current live broadcasting, and sending the video frame to a streaming media server so that the streaming media server sends the video frame to a pull stream player end;

When a multimedia file is played in a live broadcasting process, acquiring a file identifier of the multimedia file and a playing progress of the multimedia file every other preset time period;

packaging the file identification of the multimedia file and the playing progress of the multimedia file to obtain a pseudo video frame;

and sending the pseudo video frame to the streaming media server so that the streaming media server sends the pseudo video frame to the pull stream player end, and displaying corresponding prompt information of the multimedia file in the playing progress according to the pseudo video frame by the pull stream player end in the process of playing the live video through the video frame.

optionally, before sending the video frame to the streaming media server, the method further includes:

Determining system time when the video frame is collected, and determining acquisition time when the plug flow client acquires the video frame;

converting the system time into a first time stamp, and converting the acquisition time of the video frame acquired by the stream pushing client into a second time stamp;

packaging and coding the video frame, the first timestamp and the second timestamp to obtain a coded video frame;

accordingly, the sending the video frame to the streaming media server includes:

and sending the encoded video frame to the streaming media server.

optionally, after the file identifier of the multimedia file and the playing progress of the multimedia file are encapsulated to obtain the pseudo video frame, the method further includes:

determining a first timestamp and a second timestamp sent when a video frame is sent to the streaming media server before the current time and the last time from the current time;

determining the file identification of the multimedia file and the acquisition time of the playing progress of the multimedia file, and converting the file identification of the multimedia file and the acquisition time of the playing progress of the multimedia file into a third timestamp;

adding the difference value between the second timestamp and the third timestamp with the first timestamp to obtain a timestamp of the pseudo video frame, wherein the timestamp of the pseudo video is used for indicating the acquisition time of the playing progress of the multimedia file;

Accordingly, the sending the pseudo video frame to the streaming media server includes:

And sending the pseudo video frame and the timestamp of the pseudo video frame to the streaming media server.

optionally, the acquiring the file identifier of the multimedia file and the playing progress of the multimedia file every preset time period includes:

When the multimedia file is not a file pre-stored in the plug-flow client, acquiring a preset number of audio frames from the received audio frames of the multimedia file every a preset time period according to the sequence of the receiving time of the audio frames from late to early;

Synthesizing the obtained audio frames to obtain audio information;

Sending an audio query request to an audio identification server, wherein the audio query request carries the audio information;

And receiving the file identification and the playing progress of the multimedia file determined by the audio recognition server based on the audio information.

in a second aspect, another information display method in a live broadcasting process is provided, where the information display method in the live broadcasting process is applied to a pull stream player end, and the method includes:

Receiving a video frame sent by a streaming media server, and displaying a live broadcast picture based on the video frame, wherein the video frame is obtained by a stream pushing client when live broadcast is currently carried out and is sent to the streaming media server;

receiving a pseudo video frame sent by the streaming media server in the display process of the live broadcast picture, wherein the pseudo video frame is obtained by encapsulating a file identifier of the multimedia file and the playing progress of the multimedia file every other preset time period when the multimedia file is played by the push streaming client in the current live broadcast process;

and acquiring prompt information corresponding to the multimedia file in the playing progress according to the pseudo video frame, and displaying the prompt information corresponding to the multimedia file in the playing progress above the layer of the live broadcast picture.

optionally, the obtaining, according to the pseudo video frame, prompt information corresponding to the multimedia file at the play progress includes:

analyzing the pseudo video frame to obtain a file identifier of the multimedia file and a playing progress of the multimedia file;

judging whether a prompt file of the multimedia file is stored in the pull stream player end or not according to the file identifier of the multimedia file;

And when the prompt file of the multimedia file is stored in the pull stream player terminal, acquiring the prompt information corresponding to the multimedia file in the playing progress from the prompt file of the multimedia file.

Optionally, after determining whether the prompt file of the multimedia file is stored in the pull stream player end according to the file identifier of the multimedia file, the method further includes:

When the prompt file of the multimedia file is not stored in the pull stream player terminal, sending a file downloading request to a file downloading server according to the file identifier of the multimedia file, wherein the file downloading request carries the file identifier of the multimedia file;

receiving a prompt file sent by the file download server based on the file identification of the multimedia file;

and acquiring prompt information corresponding to the multimedia file in the playing progress from the received prompt file.

Optionally, after receiving the pseudo video frame sent by the streaming media server, the method further includes:

storing the pseudo video frame into a pseudo video frame list, wherein the pseudo video frame list is used for caching the received pseudo video frame;

Correspondingly, the obtaining of the prompt information corresponding to the multimedia file in the playing progress according to the pseudo video frame includes:

searching a pseudo video frame synchronous with the video frame from the pseudo video frame list;

And acquiring prompt information corresponding to the multimedia file in the playing progress according to the searched pseudo video frame.

Optionally, a first timestamp is also received when the video frame is received, where the first timestamp is obtained by converting system time when the stream pushing client acquires the video frame;

when the pseudo video frame is received, a timestamp of the pseudo video frame is also received, and the timestamp of the pseudo video is used for indicating the acquisition time of the playing progress of the multimedia file;

correspondingly, the searching for the dummy video frame synchronized with the video frame from the dummy video frame list includes:

acquiring a pseudo video frame with a smallest difference value between a timestamp of the pseudo video frame and the first timestamp received when the video frame is received from the pseudo video frame list;

and determining the obtained pseudo video frame as a pseudo video frame synchronous with the video frame.

in a third aspect, an information display apparatus in a live broadcast process is provided, which is applied to a push streaming client, and the apparatus includes:

the first sending module is used for obtaining a video frame generated by current live broadcasting and sending the video frame to a streaming media server so that the streaming media server sends the video frame to a pull stream player end;

The system comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring a file identifier of a multimedia file and the playing progress of the multimedia file every preset time period when the multimedia file is played in a live broadcasting process;

the packaging module is used for packaging the file identification of the multimedia file and the playing progress of the multimedia file to obtain a pseudo video frame;

and the second sending module is used for sending the pseudo video frame to the streaming media server so that the streaming media server sends the pseudo video frame to the pull stream player end, and the pull stream player end displays corresponding prompt information of the multimedia file in the playing progress according to the pseudo video frame in the process of playing the live video through the video frame.

optionally, the apparatus comprises:

the first determining module is used for determining the system time when the video frame is acquired and determining the acquisition time when the plug flow client acquires the video frame;

the conversion module is used for converting the system time into a first time stamp and converting the acquisition time of the video frame acquired by the plug flow client into a second time stamp;

the encoding module is used for packaging and encoding the video frame, the first timestamp and the second timestamp to obtain an encoded video frame;

correspondingly, the first sending module is further configured to:

and sending the encoded video frame to the streaming media server.

Optionally, the apparatus further comprises:

a second determining module, configured to determine a first timestamp and a second timestamp that are sent when a video frame is sent to the streaming media server before a current time and last time from the current time;

a third determining module, configured to determine a file identifier of the multimedia file and an acquisition time of the playing progress of the multimedia file, and convert the file identifier of the multimedia file and the acquisition time of the playing progress of the multimedia file into a third timestamp;

an adding module, configured to add the difference between the second timestamp and the third timestamp to the first timestamp to obtain a timestamp of the pseudo video frame, where the timestamp of the pseudo video is used to indicate an acquisition time of a playing progress of the multimedia file;

Correspondingly, the second sending module is further configured to:

optionally, the obtaining module includes:

The acquisition unit is used for acquiring a preset number of audio frames from the received audio frames of the multimedia file according to the sequence of the receiving time of the audio frames from late to early every preset time period when the multimedia file is not a file pre-stored in the plug-flow client;

the synthesis unit is used for synthesizing the acquired audio frames to obtain audio information;

the sending unit is used for sending an audio query request to an audio identification server, wherein the audio query request carries the audio information;

And the receiving unit is used for receiving the file identification and the playing progress of the multimedia file determined by the audio recognition server based on the audio information.

in a fourth aspect, there is provided another information display apparatus in a live broadcast process, applied to a pull stream player, the apparatus comprising:

the first receiving module is used for receiving a video frame sent by a streaming media server and displaying a live broadcast picture based on the video frame, wherein the video frame is obtained by a stream pushing client when live broadcast is currently carried out and is sent to the streaming media server;

The second receiving module is used for receiving a pseudo video frame sent by the streaming media server in the display process of the live broadcast picture, wherein the pseudo video frame is obtained by encapsulating a file identifier of the multimedia file and the playing progress of the multimedia file every other preset time period when the multimedia file is played by the stream pushing client in the current live broadcast process;

and the acquisition module is used for acquiring prompt information corresponding to the multimedia file in the playing progress according to the pseudo video frame and displaying the prompt information corresponding to the multimedia file in the playing progress above the layer of the live broadcast picture.

Optionally, the obtaining module includes:

the analysis unit is used for analyzing the pseudo video frame to obtain a file identifier of the multimedia file and the playing progress of the multimedia file;

the judging unit is used for judging whether a prompt file of the multimedia file is stored in the pull stream player end or not according to the file identifier of the multimedia file;

And the first obtaining unit is used for obtaining the corresponding prompt information of the multimedia file in the playing progress from the prompt file of the multimedia file when the prompt file of the multimedia file is stored in the pull stream player terminal.

optionally, the obtaining module further includes:

a sending unit, configured to send a file download request to a file download server according to a file identifier of the multimedia file when the multimedia file prompt file is not stored in the pull stream player terminal, where the file download request carries the file identifier of the multimedia file;

the receiving unit is used for receiving a prompt file sent by the file downloading server based on the file identifier of the multimedia file;

and the second acquisition unit is used for acquiring prompt information corresponding to the multimedia file in the playing progress from the received prompt file.

Optionally, the apparatus further comprises:

the storage module is used for storing the pseudo video frame into a pseudo video frame list, and the pseudo video frame list is used for caching the received pseudo video frame;

accordingly, the acquisition module comprises:

the searching unit is used for searching a pseudo video frame synchronous with the video frame from the pseudo video frame list;

And the third obtaining unit is used for obtaining the corresponding prompt information of the multimedia file in the playing progress according to the searched pseudo video frame.

correspondingly, the lookup unit is further configured to:

The technical scheme provided by the embodiment of the invention has the following beneficial effects: when the stream pushing client plays the multimedia file in the live broadcasting process, the video frame currently live broadcasting is sent to the stream media server, the file identification of the multimedia file and the playing progress of the multimedia file are encapsulated to obtain a pseudo video frame, and the pseudo video frame is sent to the stream media server. That is, the stream pushing client sends the video frame and the pseudo video frame respectively, instead of converting the prompt information corresponding to the multimedia file at the current playing progress into an image and synthesizing the image with the current video frame, so that the stream pulling player end can display the video frame and simultaneously independently display the prompt information corresponding to the multimedia file at the current playing progress, a user of the stream pulling player end can conveniently and independently perform operations such as moving and closing on the prompt information corresponding to the multimedia file, and the flexibility of information display in the live broadcasting process is improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

in order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

fig. 1 is a schematic diagram of a video live broadcast system provided in an embodiment of the present invention;

Fig. 2 is a flowchart of an information display method in a live broadcast process according to an embodiment of the present invention;

fig. 3 is a flowchart of an information display method in another live broadcasting process according to an embodiment of the present invention;

Fig. 4 is a flowchart of an information display method in another live broadcasting process according to an embodiment of the present invention;

Fig. 5 is a block diagram of an information display apparatus in a live broadcast process according to an embodiment of the present invention;

fig. 6 is a block diagram of an information display apparatus in another live broadcasting process according to an embodiment of the present invention.

Detailed Description

in order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

before explaining the embodiments of the present invention in detail, an application scenario of the embodiments of the present invention will be described. In the live broadcasting process, when multimedia files such as songs or movies are also played in the current live broadcasting process, in order to enrich the display content of the live broadcasting, when the current live broadcasting picture is displayed, the pull stream player end can also display prompt information corresponding to the multimedia files in the current playing progress. For example, when a user live broadcasts a singing video through the push streaming client, the pull streaming player can also display the corresponding lyrics of the songs played in the singing video at the current playing progress, so as to improve the user experience of watching the singing video by the audience. The information display method in the live broadcasting process provided by the embodiment of the invention is applied to scenes with multimedia files in the live broadcasting process.

Fig. 1 is a schematic view of a video live broadcasting system 100 according to an embodiment of the present invention, and as shown in fig. 1, the video live broadcasting system 100 includes a push streaming client 101, a streaming media server 102, and a pull streaming player 103. The push streaming client 101 and the streaming media server 102, and the streaming media server 102 and the pull streaming player 103 may communicate with each other through a wireless network or a wired network.

the push streaming client 101 is configured to obtain a video frame currently being live and send the video frame to the streaming media server 102, the streaming media server 102 is configured to receive the video frame sent by the push streaming client 101 and send the video frame to the pull streaming player 103, and the pull streaming player 103 is configured to receive the video frame sent by the streaming media server 102 and display a current live frame based on the video frame. In addition, the stream pushing client 101 is further configured to obtain an audio frame currently performing live broadcasting and send the audio frame to the streaming media server 102, the streaming media server 102 is configured to receive the audio frame sent by the stream pushing client 101 and send the audio frame to the stream player 103, and the stream player 103 is configured to receive the audio frame sent by the streaming media server 102 and play the audio frame when displaying a current live broadcasting picture.

Optionally, when the push streaming client 101 plays the multimedia file during the live broadcast, the video live broadcast system 100 may further include an audio recognition server 104 and a file download server 105, and communication between the push streaming client 101 and the audio recognition server 104 and between the pull streaming player 103 and the file download server 105 may also be performed by way of a wireless network or a wired network. The audio recognition server 104 is configured to determine a file identifier of the multimedia file and a playing progress of the multimedia file according to an audio query request sent by the plug-flow client 101, and the file download server 105 is configured to determine a prompt text of the multimedia file according to a file download request sent by the pull-flow player 103. For example, the audio recognition server 104 may be a song listening recognition server, and the file download server 105 may be a lyrics server.

it should be noted that, when the multimedia file played by the push streaming client 101 in the live broadcasting process is the multimedia file stored in the push streaming client 101, the live video broadcasting system 100 may not include the audio recognition server 104, that is, the push streaming client 101 may directly determine the file identifier of the multimedia file and the playing progress of the multimedia file according to the stored multimedia file. In addition, when the pull stream player end stores the prompt file of the multimedia file, the live video broadcast system 100 may not include the file download server 105, that is, the pull stream player end may directly determine the prompt information of the multimedia file at the playing progress according to the stored prompt file of the multimedia file.

fig. 2 is a flowchart of an information display method in a live broadcasting process according to an embodiment of the present invention, where the method is applied to the stream pushing client 101 shown in fig. 1, and as shown in fig. 2, the information display method in the live broadcasting process includes the following steps.

Step 201: and acquiring a video frame generated by the current live broadcast, and sending the video frame to a streaming media server so that the streaming media server sends the video frame to a pull stream player.

step 202: when a multimedia file is played in a live broadcasting process, the file identification of the multimedia file and the playing progress of the multimedia file are obtained every other preset time period.

step 203: and encapsulating the file identification of the multimedia file and the playing progress of the multimedia file to obtain a pseudo video frame.

step 204: and sending the pseudo video frame to a streaming media server so that the streaming media server sends the pseudo video frame to a pull stream player end, and the pull stream player end displays corresponding prompt information of the multimedia file in the playing progress according to the pseudo video frame in the process of playing the live video through the video frame.

When the stream pushing client plays the multimedia file in the live broadcasting process, the video frame currently live broadcasting is sent to the stream media server, the file identification of the multimedia file and the playing progress of the multimedia file are encapsulated to obtain a pseudo video frame, and the pseudo video frame is sent to the stream media server. That is, the stream pushing client sends the video frame and the pseudo video frame respectively, instead of converting the prompt information corresponding to the multimedia file at the current playing progress into an image and synthesizing the image with the current video frame, so that the stream pulling player end can display the video frame and simultaneously independently display the prompt information corresponding to the multimedia file at the current playing progress, a user of the stream pulling player end can conveniently and independently perform operations such as moving and closing on the prompt information corresponding to the multimedia file, and the flexibility of information display in the live broadcasting process is improved.

determining the system time when the video frame is acquired, and determining the acquisition time when the plug flow client acquires the video frame;

converting the system time into a first time stamp, and converting the acquisition time of the plug flow client for acquiring the video frame into a second time stamp;

Packaging and coding the video frame, the first time stamp and the second time stamp to obtain a coded video frame;

accordingly, the video frame is sent to the streaming media server, and the method comprises the following steps:

and sending the encoded video frame to a streaming media server.

Determining a first timestamp and a second timestamp which are sent when a video frame is sent to a streaming media server before the current time and is the latest time away from the current time;

Determining the file identifier of the multimedia file and the acquisition time of the playing progress of the multimedia file, and converting the file identifier of the multimedia file and the acquisition time of the playing progress of the multimedia file into a third timestamp;

adding the difference value between the second time stamp and the third time stamp to the first time stamp to obtain the time stamp of the pseudo video frame, wherein the time stamp of the pseudo video is used for indicating the acquisition time of the playing progress of the multimedia file;

Accordingly, the sending the pseudo video frame to the streaming media server comprises:

And sending the pseudo video frame and the timestamp of the pseudo video frame to a streaming media server.

optionally, the obtaining the file identifier of the multimedia file and the playing progress of the multimedia file every preset time period includes:

When the multimedia file is not a file pre-stored in the plug-flow client, acquiring a preset number of audio frames from the received audio frames of the multimedia file at intervals of a preset time period according to the sequence of the receiving time of the audio frames from late to early;

synthesizing the obtained audio frames to obtain audio information;

all the above optional technical solutions can be combined arbitrarily to form an optional embodiment of the present invention, which is not described in detail herein.

Fig. 3 is a flowchart of another information display method in a live broadcasting process according to an embodiment of the present invention, where the method is applied to the pull stream player terminal 103 shown in fig. 1, and as shown in fig. 3, the information display method in the live broadcasting process includes the following steps.

Step 301: and receiving a video frame sent by the streaming media server, and displaying a live broadcast picture based on the video frame, wherein the video frame is obtained by the stream pushing client during live broadcast and sent to the streaming media server.

step 302: and receiving a pseudo video frame sent by a streaming media server in the display process of the live broadcast picture, wherein the pseudo video frame is obtained by encapsulating the file identifier of the multimedia file and the playing progress of the multimedia file every other preset time period when the multimedia file is played by the stream pushing client in the current live broadcast process.

Step 303: and acquiring prompt information corresponding to the multimedia file in the playing progress according to the pseudo video frame, and displaying the prompt information corresponding to the multimedia file in the playing progress above the layer of the live broadcast picture.

When the push stream client plays the multimedia file in the current live broadcast process, the pull stream player end receives the video frame and also receives the pseudo video frame sent by the push stream client through the stream media server, and the pseudo video frame is obtained by encapsulating the file identifier of the multimedia file and the play progress of the multimedia file at intervals of a preset time period for the push stream client. That is, the pull stream player end can independently display the video frame in the current live broadcast picture and the prompt information corresponding to the multimedia file at the playing progress, so that the user of the pull stream player end can independently perform operations such as moving and closing on the prompt information corresponding to the multimedia file, and the flexibility of information display in the live broadcast process is improved.

optionally, obtaining, according to the pseudo video frame, prompt information corresponding to the multimedia file at the play progress, where the prompt information includes:

When the prompt file of the multimedia file is stored in the pull stream player terminal, the prompt information corresponding to the multimedia file in the playing progress is obtained from the prompt file of the multimedia file.

Receiving a prompt file sent by a file download server based on the file identification of the multimedia file;

correspondingly, according to the pseudo video frame, obtaining the prompt information corresponding to the multimedia file in the playing progress, including:

searching a pseudo video frame synchronous with the video frame from a pseudo video frame list;

Optionally, a first timestamp is also received when the video frame is received, and the first timestamp is obtained by converting the system time when the video frame is acquired by the stream pushing client;

Accordingly, searching the pseudo video frame synchronized with the video frame from the pseudo video frame list comprises:

acquiring a pseudo video frame with the smallest difference value between the timestamp of the pseudo video frame and a first timestamp received when the video frame is received from a pseudo video frame list;

and determining the acquired pseudo video frame as a pseudo video frame synchronous with the video frame.

Fig. 4 is a flowchart of another information display method in a live broadcasting process according to an embodiment of the present invention, where the method is applied to the video live broadcasting system 100 shown in fig. 1, and as shown in fig. 4, the information display method in the live broadcasting process includes the following steps.

Step 401: and the stream pushing client acquires a video frame generated by current live broadcasting and sends the video frame to the stream media server, so that the stream media server sends the video frame to the stream pulling player.

The video frames are single images forming a video, and when a plurality of video frames are played continuously, a dynamic image animation, namely the video, is formed. Therefore, the push stream client live broadcasts the video to the pull stream player end through the stream media server, that is, the push stream client obtains the video frame generated by the current live broadcast in real time and sends the video frame to the pull stream player end through the stream media server, and when the pull stream player end receives a plurality of video frames and continuously displays the video frames, the pull stream player end plays the video live broadcast by the push stream player end.

When the stream pushing client acquires a video frame generated by currently live broadcasting, the video frame generated by currently live broadcasting can be directly acquired by the stream pushing client, or the video frame generated by currently live broadcasting can be acquired by other terminals and transmitted to the stream pushing client, for example, the video frame can be acquired by a terminal which is currently provided with the stream pushing client, and the acquired video frame is transmitted to the stream pushing client.

it should be noted that the video frame is only a single still image and does not include audio information currently in live broadcasting, so that when the stream push client live broadcasts a video to the stream pull player through the stream media server, the stream push client needs to acquire not only a video frame generated by the current live broadcasting but also an audio frame generated by the current live broadcasting and send the audio frame to the stream media server, so that the stream media server sends the audio frame to the stream pull player.

In addition, in order to ensure that the video frames and the audio frames played by the pull-streaming player end after receiving the video frames and the audio frames are synchronous, the push-streaming client end can stamp each captured video frame and each captured audio frame with a time stamp when capturing the video frames and the audio frames, wherein the time stamp of the video frame is used for indicating the time when the push-streaming client end captures the video frame, and the time stamp of the audio frame is used for indicating the time when the push-streaming client end captures the audio frame.

it should be noted that the time stamp has two representation modes, one is a time stamp taking a system clock as reference time, and at this time, the time stamp can directly represent the current system time; the timestamp is obtained by converting the current system time according to the reference clock as a time zero point, for example, the reference clock is 2:00, the current time is 2:01, and the timestamp of the current time under the reference clock is 60 s. In order to facilitate the mathematical calculation between the timestamps, in the embodiment of the present invention, the timestamps of the video frames and the timestamps of the audio frames both adopt the second representation manner.

In addition, after the stream pushing client accesses the network, the control plane of the stream pushing client comprises a physical layer and an application layer, wherein the physical layer is used for collecting information and receiving instructions from the application layer, and the application layer is used for receiving the information sent by the physical layer and issuing the instructions to the physical layer. And the reference clocks of the physical layer and the application layer are usually not the same reference clock, that is, the timestamp corresponding to the physical layer and the timestamp corresponding to the application layer at the same time are not the same. Therefore, in the embodiment of the present invention, the time stamp of each video frame and the time stamp of each audio frame each include two different time stamps, that is, the time stamp of each video frame includes a time stamp under the reference clock of the physical layer and a time stamp under the reference clock of the application layer, and the time stamp of each audio frame also includes a time stamp under the reference clock of the physical layer and a time stamp under the reference clock of the application layer.

in one possible implementation, determining the timestamp of each video frame may be: determining the system time when the video frame is acquired and the acquisition time when the push streaming client acquires the video frame, converting the system time into a first time stamp under a reference clock of a physical layer, and converting the acquisition time when the push streaming client acquires the video frame into a second time stamp under the reference clock of an application layer. The first timestamp and the second timestamp are two timestamps of the video frame, when the stream pushing client determines the two timestamps of the video frame, the first timestamp and the second timestamp are packaged and encoded to obtain an encoded video frame, and the encoded video frame is sent to the streaming media server.

It is worth noting that when the video frame generated by the live broadcast at present is obtained by the stream pushing client, the system time for the physical layer of the stream pushing client to collect the video frame and the obtaining time for the application layer to obtain the video frame are basically the same time, but because the reference clock of the physical layer and the reference clock of the application layer are not the same, the timestamp of the same time under the reference clock of the physical layer and the timestamp of the same time under the reference clock of the application layer are not the same. For example, the reference clock of the physical layer is 2:00, the reference clock of the application layer is 2:01, when the system time for acquiring the video frame by the physical layer of the plug flow client and the acquisition time for acquiring the video frame by the application layer are the same time 2:02, the timestamp of the same time 2:02 under the reference clock of the physical layer is 60s, and the timestamp of the same time 2:02 under the reference clock of the application layer is 120 s.

Similarly, two timestamps of each audio frame are obtained according to the method, the audio frame and the two timestamps of the audio frame are packaged and encoded to obtain an encoded audio frame, and the encoded audio frame is sent to the streaming media server.

In addition, when the stream pushing client obtains the encoded video frame and the encoded audio frame, the video frame and the audio frame that are transmitted last time may not be successfully transmitted due to network congestion and the like, so the stream pushing client further includes a video frame list and an audio frame list, the video frame list is used for temporarily caching the video frame to be transmitted, and the audio frame list is used for temporarily caching the audio frame to be transmitted. At this time, for the video frames in the video frame list, the stream pushing client sequentially sends the video frames in the video frame list to the streaming media server according to the descending order of the first timestamps of the video frames. For the audio frames in the audio frame list, the audio frames in the audio frame list are also transmitted according to the method.

for example, the acquisition system for acquiring the video frame and the audio frame is a DirectShow acquisition system, when the acquisition system acquires a video frame currently performing live broadcasting, the video frame is marked as Vn, and because the acquisition system is located in a physical layer, the acquisition system can determine a first time stamp according to a system time for acquiring the video frame and a reference clock of the physical layer, mark the first time stamp as TDVn, and upwardly send the video frame and the first time stamp to an application layer of the push streaming client. When the application layer receives the video frame, because there is almost no difference between the system time of the acquisition system collecting the video frame and the system time of the application layer receiving the video frame, the application determines a second timestamp, which is marked as TAVn, according to the system time of receiving the video frame, that is, the time of the acquisition system acquiring the video frame and the reference clock of the application layer. At this time, the stream pushing client determines the video frame Vn to be sent currently and two timestamps of the video frame Vn, that is, the first timestamp TDVn and the second timestamp are TAVn, encodes the video frame Vn, TDVn and TAVn, and adds the encoded video frame to the video frame list to wait for sending. Similarly, when An audio frame currently being live-broadcasted is acquired by the acquisition system, the audio frame is marked as An, a timestamp of the audio frame An under a reference clock of a physical layer and a timestamp of the audio frame An under a reference clock of An application layer are determined according to a method for determining two timestamps of a video frame Vn and are respectively marked as TDAn and TAAn, the audio frame An, the TDAn and the TAAn are encoded, and the encoded audio frame is added to An audio frame list to be transmitted.

the DirectShow acquisition system is a streaming media processing development kit proposed by microsoft corporation, and is used for acquiring data and performing corresponding post-processing on the acquired data.

Step 402: and the pull stream player end receives the video frame sent by the streaming media server and displays a live broadcast picture based on the video frame, wherein the video frame is a video frame which is acquired by the push stream client end when live broadcast is currently carried out and is sent to the streaming media server.

Step 401 shows that, when determining a video frame generated by live broadcasting currently, the stream pushing client may further determine two timestamps of the video frame, that is, a first timestamp and a second timestamp, according to the system time for acquiring the video frame, and encapsulate and encode the video frame, the first timestamp and the second timestamp, and send the encoded video frame to the streaming media server. Optionally, the stream-pulling player may further receive a second timestamp, where the second timestamp is obtained by converting the acquisition time when the stream-pushing client acquires the video frame.

in addition, as shown in step 401, the push streaming client sends the video frame to the streaming media server and also sends the audio frame generated by the current live broadcast to the streaming media server, so that the pull streaming player can also receive the audio frame generated by the current live broadcast. In addition, when the push streaming client sends the two timestamps of the audio frame to the streaming media server, the pull streaming player may also receive the two timestamps of the audio frame, and for convenience of description in the following sequence, of the two timestamps of the audio frame, the timestamp below the reference clock of the physical layer is referred to as a fourth timestamp, and the timestamp below the reference clock of the application layer is referred to as a fifth timestamp.

In the embodiment of the present invention, in order to implement synchronization between a currently displayed video frame and a played audio frame, when a stream player receives a video frame sent by a streaming server, a fourth time stamp of each audio frame in the received audio frames is determined to obtain at least one fourth time stamp, a time stamp identical to a first time stamp of the video frame is searched from the at least one fourth time stamp, and an audio frame corresponding to the searched fourth time stamp is determined as the audio frame synchronized with the video frame, at this time, the audio frame synchronized with the video frame is played while a live broadcast picture is displayed based on the video frame, thereby implementing synchronous playing of the video frame and the audio frame in the live broadcast picture. Specifically, when the timestamp identical to the first timestamp of the video frame is not searched from the at least one fourth timestamp, in the process of displaying the video frame, waiting to receive the audio frame with the fourth timestamp identical to the first timestamp of the video frame until the audio frame is received, and then playing the audio frame, so that synchronous playing of the video frame and the audio frame is realized.

Step 403: when a multimedia file is played in a live broadcasting process, the stream pushing client acquires the file identification of the multimedia file and the playing progress of the multimedia file every other preset time period.

When a multimedia file is played in a live broadcast process, because the existing RTMP protocol does not provide direct support for the transmission of the prompt information corresponding to the multimedia file, in order to enable a stream pulling player end to display the prompt information corresponding to the multimedia file in the process of playing a live video, a stream pushing client end can send a file identifier of the multimedia file and the playing progress of the multimedia file to the stream pulling player end through a stream media server, so that the stream pulling player end can obtain the prompt information corresponding to the multimedia file.

the multimedia file can be a song or a movie, when the multimedia file is a song, the prompt information corresponding to the multimedia file is lyrics, and when the multimedia file is a movie, the prompt information corresponding to the multimedia file is subtitles. The file identifier of the multimedia file is used to uniquely identify the multimedia file, and may be a name of the multimedia file, such as a song name or a movie name. The playing progress of the multimedia file is used for indicating the playing progress of the multimedia file played in the current live broadcasting process, if the time length of the multimedia file is 5 minutes, the playing progress of the multimedia file played in the current live broadcasting process is 3 minutes and 20 seconds, and the playing progress of the multimedia file is 3 minutes and 20 seconds at the moment. In addition, the preset time period is a preset time period, and the preset time period may be 10s, 20s, 30s, or the like.

specifically, when a multimedia file is played in a live broadcast process, the multimedia file may be a multimedia file played by a stream pushing client, or may also be a multimedia file pushed by a stream media server in the live broadcast process, that is, the multimedia file may be a file pre-stored in the stream pushing client, or may not be a multimedia file pre-stored in the stream pushing client, so that two implementation manners may be provided for acquiring a file identifier of the multimedia file and a playing progress of the multimedia file every preset time period.

in a first implementation manner, when the multimedia file is a file pre-stored in the plug-flow client, the file identifier of the multimedia file and the playing progress of the multimedia file are directly obtained every preset time period. For example, the preset time period is 10s, and the song played in the current live broadcast process is a song pre-stored in the stream pushing client, at this time, the stream pushing client may directly obtain the name of the song, which is marked as MNcurrent, every 10s in the process of playing the song, and obtain the playing progress of the song when the song is currently played, which is marked as TPcurrent.

In a second implementation manner, when the multimedia file is not a file pre-stored in the stream pushing client, the stream pushing client acquires a preset number of audio frames from the received audio frames of the multimedia file according to the sequence of the receiving time of the audio frames from late to early every preset time period; and synthesizing the acquired audio frames to obtain audio information. The stream pushing client sends an audio query request to an audio identification server, wherein the audio query request carries the audio information; when the audio recognition server receives the audio query request, determining the file identification of the multimedia file corresponding to the audio information and the playing progress of the multimedia file according to the audio information, and sending the file identification of the multimedia file and the playing progress of the multimedia file to the stream pushing client. And the stream pushing client receives the file identification and the playing progress of the multimedia file determined by the audio recognition server based on the audio information, takes the file identification of the received multimedia file as the file identification of the multimedia file played in the current live broadcasting process, and takes the playing progress of the received multimedia file as the playing progress of the multimedia file played in the current live broadcasting process.

Since the time length of each audio frame is fixed, when a plurality of preset audio frames are acquired and the acquired audio frames are synthesized, the audio information is acquired as the audio information with the preset time length. And because the audio frames are acquired from late to early according to the receiving time of the audio frames, the obtained audio information can represent the audio information which is played in the current live broadcasting process. In addition, the audio recognition server is used for determining the file identification and the playing progress of the multimedia file corresponding to the audio information according to the audio information. For example, the audio recognition server may be a song listening recognition server.

For example, the stream pushing client obtains a preset number of audio frames from the audio frame received the last time in the order from late to early according to the receiving time of the audio frames, and combines the preset number of audio frames to obtain audio information with preset time length information of 200 ms. The method comprises the steps that an audio query request is sent to a song listening and music identifying server by a stream pushing client, the audio query request carries 200ms of audio information, when the song listening and music identifying server receives the audio query request, according to the 200ms of audio information, the name of a multimedia file corresponding to the audio information is SMname, the playing progress of the multimedia file corresponding to the audio information is determined to be TPtime, and the name of the multimedia file corresponding to the audio information SMname and the playing progress TPtime of the multimedia file corresponding to the audio information are sent to the stream pushing client. And when the stream pushing client receives that the name of the multimedia file corresponding to the audio information is the SMname and the playing progress of the multimedia file corresponding to the audio information is TPtime, taking the SMname as the MNcurrent name of the song played in the current live broadcast process, and taking the TPtime as the TPcurrent playing progress of the song played in the current live broadcast process.

step 404: and the plug-flow client encapsulates the file identification of the multimedia file and the playing progress of the multimedia file to obtain a pseudo video frame.

in order to facilitate the transmission of information such as the file identifier of the multimedia file and the playing progress of the multimedia file, which can be supported by the existing RTMP protocol, the stream push client needs to encapsulate the file identifier of the multimedia file and the playing progress of the multimedia file, and encode the file identifier and the playing progress according to a preset encoding type to obtain a pseudo video frame.

In a possible implementation manner, in existing H264 (a high-performance video coding technology), data obtained after a video frame is coded is data in a NAL (network abstraction Layer) format. And a plurality of different NAL types are included in H264, wherein some of the NAL types are already used to define different types of video frames, and there is still some of the NAL types that are undefined, and at this time, one NAL type can be selected from the undefined NAL types to define the pseudo video frame provided by the embodiment of the present invention. Therefore, the above-mentioned predetermined coding type is also a NAL type selected from the undefined NAL types.

In addition, in order to enable the stream pulling player end to synchronously display the prompt information corresponding to the multimedia file when displaying the video frame, the stream pushing client end needs to stamp a timestamp for the pseudo video frame when obtaining the pseudo video frame, that is, to determine the timestamp of the pseudo video frame, where the timestamp of the pseudo video is used for the stream pushing client end to indicate the acquisition time of the playing progress of the multimedia file. It should be noted that the stream pushing client acquires the playing progress of the multimedia file as a behavior of the application layer, so that when the stream pushing client determines the file identifier of the multimedia file and the acquisition time of the playing progress of the multimedia file, the stream pushing client converts the acquisition time into a third timestamp under the reference clock of the physical layer and records the third timestamp, that is, the acquisition time of the progress of the multimedia file recorded by the stream pushing client is the time under the reference clock of the application layer. However, for the stream-pulling player end, the acquisition time of the video frame corresponding to the physical layer and the acquisition time of the multimedia file corresponding to the multimedia file at the physical layer need to be synchronized, so the stream-pushing client end needs to convert the third timestamp into a timestamp under the reference clock of the physical layer, and convert the third timestamp into a timestamp under the reference clock of the physical layer as the timestamp of the pseudo video frame. Specifically, the stream pushing client determines a first timestamp and a second timestamp which are sent when a video frame is sent to the streaming media server before the current time and the video frame is sent to the streaming media server the last time from the current time; determining the file identification of the multimedia file and the acquisition time of the playing progress of the multimedia file, and converting the acquisition time into a third timestamp; and adding the difference value between the second time stamp and the third time stamp to the first time stamp to obtain the time stamp of the pseudo video frame.

for example, the video frame sent to the streaming media server before the current time and the last time from the current time is Vnlast, and the first timestamp sent when Vnlast is sent is TDVnlast and the second timestamp TAVnlast, that is, the first timestamp TDVnlast and the second timestamp TAVnlast of Vnlast. When the stream pushing client determines the name of the song played in the current live broadcasting process and the playing progress acquiring time of the song, the acquiring time is converted into a third timestamp under a reference clock of a physical layer and is recorded as TAMcurrent, and the timestamp of the pseudo video frame can be represented as: TDVnlast + (TAMcurrent-tamvnlast), i.e., converting TAMcurrent to a timestamp at the reference clock of the physical layer.

Step 405: and the stream pushing client sends the pseudo video frame to the stream media server so that the stream media server sends the pseudo video frame to the stream pulling player, and the stream pulling player displays corresponding prompt information of the multimedia file in the playing progress according to the pseudo video frame in the process of playing the live video through the video frame.

It is noted that, according to step 404, when the streaming client determines the pseudo video frame, the timestamp of the pseudo video frame may also be determined, and therefore, the step of sending the pseudo video frame to the streaming server by the streaming client may also be: and the stream pushing client sends the pseudo video frame and the timestamp of the pseudo video frame to the streaming media server. And when the streaming media server receives the pseudo video frame and the timestamp of the pseudo video frame, sending the pseudo video frame and the timestamp of the pseudo video frame to a pull stream player end.

Step 406: the method comprises the steps that a pull stream player end receives a pseudo video frame sent by a streaming media server in the display process of a live broadcast picture, and the pseudo video frame is obtained by encapsulating a file identifier of a multimedia file and the playing progress of the multimedia file every other preset time period when a push stream client plays the multimedia file in the current live broadcast process.

optionally, when the stream pushing client sends the pseudo video frame and the timestamp of the pseudo video frame to the stream player end through the streaming media server at the same time, the stream player end also receives the timestamp of the pseudo video frame when receiving the pseudo video frame.

in addition, in order to facilitate the pull stream player end to realize synchronous display of the current video frame and the prompt information corresponding to the multimedia file, after the pull stream player end receives the pseudo video frame sent by the streaming media server, the pseudo video frame and the timestamp of the pseudo video frame can also be stored in a pseudo video frame list, and the pseudo video frame list is used for caching the received pseudo video frame and the timestamp of the pseudo video frame. For example, the pseudo video frame list is a lyric queue, a song is being played in the current live broadcasting process, the stream pushing client sends a pseudo video frame and a timestamp of the pseudo video frame to the stream pulling player through the streaming media server based on the song, and when the stream pulling player receives the timestamps of the pseudo video frame and the pseudo video frame, the pseudo video frame and the timestamp of the pseudo video frame are stored in the lyric queue.

Step 407: and the pull stream player end acquires corresponding prompt information of the multimedia file in the playing progress according to the pseudo video frame, and displays the corresponding prompt information of the multimedia file in the playing progress above the layer of the live broadcast picture.

specifically, the stream player end analyzes the pseudo video frame to obtain a file identifier of the multimedia file and a playing progress of the multimedia file; judging whether a prompt file of the multimedia file is stored in the pull stream player end or not according to the file identifier of the multimedia file; when the prompt file of the multimedia file is stored in the pull stream player terminal, the prompt information corresponding to the multimedia file in the playing progress is directly obtained from the prompt file of the multimedia file. When the prompt file of the multimedia file is not stored in the pull stream player terminal, sending a file downloading request to a file downloading server according to the file identifier of the multimedia file, wherein the file downloading request carries the file identifier of the multimedia file; when the file download server receives the file download request, determining a prompt file of the multimedia file according to the file identifier of the multimedia file, and sending the prompt file of the multimedia file to the pull stream player terminal, wherein the pull stream player terminal receives the prompt file sent by the file download server based on the file identifier of the multimedia file; and acquiring prompt information corresponding to the multimedia file in the playing progress from the received prompt file.

the file download server is used for determining a prompt file of the multimedia file according to the file identifier of the multimedia file, and if the file download server is a lyric server, the lyric server is used for determining a lyric file of the song according to the name of the song.

It should be noted that, when the multimedia file prompt file is not stored in the pull stream player terminal, the pull stream player terminal needs to download the multimedia file prompt file from the file download server, and since it takes a certain time to download the multimedia file prompt file, when the pull stream player acquires the multimedia file prompt file, the play progress of the multimedia file needs to be relocated, that is, the play progress of the multimedia file is determined by the sum of the play progress of the multimedia file and the time of downloading the multimedia file prompt file.

for example, when a song is played in the current live broadcast process, the pull stream player analyzes the pseudo video frame to obtain the name MNcurrent of the song and the playing progress TPcurrent of the song. According to the name MNcurrent of the song, the pull stream player end judges whether a prompt file corresponding to the song, namely a lyric file, is stored at the home end, when the lyric file of the song is stored at the home end, the lyric corresponding to the song when the playing progress TPcurrent is located in the lyric file directly according to the playing progress TPcurrent, namely, prompt information corresponding to the multimedia file when the playing progress TPcurrent is obtained. When the lyric file is not stored in the local terminal, a lyric file downloading request is sent to a lyric server, when the lyric server receives the lyric file downloading request, the lyric file of the song is determined according to the name of the song carried by the lyric file downloading request, the lyric file of the song is sent to a pull stream player terminal, when the pull stream player terminal receives the lyric file of the song, the time required for downloading the lyric file is determined as TDtime, and according to the playing progress TPcurrent, the lyric corresponding to the song when the playing progress is TDtime + TPcurrent is positioned in the lyric file.

Optionally, when the multimedia file prompt file is not stored in the pull stream player end, if the multimedia file audio file is stored in the pull stream player end, the pull stream player end may perform text processing on the multimedia file audio file to obtain the multimedia file prompt file, and then determine the prompt information of the multimedia file at the playing progress according to the multimedia file prompt file and the playing progress.

In addition, in order to improve the user experience of watching the live video, when the pull stream player end displays the prompt information corresponding to the multimedia file, the video frame and the prompt information corresponding to the multimedia file can be synchronously displayed. Specifically, when a live frame is displayed on the basis of a video frame at a pull stream player end, a pseudo video frame synchronous with the video frame is searched from a pseudo video frame list; and acquiring prompt information corresponding to the multimedia file at the playing progress according to the searched pseudo video frame, and displaying the prompt information corresponding to the multimedia file at the playing progress while displaying the video frame, thereby realizing synchronous display of the video frame and the prompt information corresponding to the multimedia file. As shown in step 402, the pull stream player end can synchronize the currently played video frame and audio frame according to the timestamp of the video frame and the timestamp of the audio frame, so that the pull stream player end can synchronize the currently played audio frame, video frame, and the prompt information corresponding to the multimedia file.

specifically, the first time stamp of the video frame is determined, the pseudo video frame with the smallest difference between the time stamp of the pseudo video frame and the first time stamp received when the video frame is received is obtained from a pseudo video frame list, and the obtained pseudo video frame is determined as the pseudo video frame synchronous with the video frame.

optionally, in order to reduce the storage pressure at the end of the pull stream player, when a pseudo video frame synchronized with the video frame is found from the pseudo video frame list and the prompt information corresponding to the multimedia file is displayed according to the pseudo video frame, the pseudo video frame may be deleted from the pseudo video frame list. In addition, because the pseudo video frames are sent to the pull stream player end by the stream media server at intervals of a preset time period, the push stream client may send a plurality of video frames to the pull stream player end by the stream media server within the preset time period, when the pull stream player end receives the video frames, the number of pseudo videos in the pseudo video list may be 0, and at this time, the pull stream player end does not perform any operation.

in addition, after the end of the pull stream player displays the prompt information of the multimedia file in the playing progress, when the current time changes, the playing progress of the multimedia file can be re-determined according to the changed current time, and the prompt information of the multiple pairs of multimedia files can be updated according to the re-determined playing progress of the multimedia file and the determined prompt file of the multimedia file.

In the embodiment of the invention, when the stream pushing client plays the multimedia file in the live broadcasting process, in addition to sending the video frame and the audio frame which are currently live broadcasting to the streaming media server, the file identifier of the multimedia file and the playing progress of the multimedia file are encapsulated to obtain the pseudo video frame, and the pseudo video frame is sent to the streaming media server, that is, the stream pushing client sends the video frame and the pseudo video frame respectively instead of converting the corresponding prompt information of the multimedia file in the current playing progress into an image and synthesizing the image and the current video frame, so that the stream pulling player end can independently display the corresponding prompt information of the multimedia file in the current playing progress while displaying the video frame, so that a user of the stream pulling player end can independently perform operations such as moving, closing and the like on the prompt information corresponding to the multimedia file, the flexibility of information display in the live broadcast process is improved.

Fig. 5 is an information display apparatus 500 in a live broadcast process according to an embodiment of the present invention, where the apparatus 500 is applied to the stream pushing client 101 shown in fig. 1, and as shown in fig. 5, the apparatus 500 includes a first sending module 501, an obtaining module 502, an encapsulating module 503, and a second sending module 504:

A first sending module 501, configured to obtain a video frame generated by currently performing live broadcast, and send the video frame to a streaming media server, so that the streaming media server sends the video frame to a stream player;

An obtaining module 502, configured to obtain a file identifier of a multimedia file and a playing progress of the multimedia file every preset time period when the multimedia file is played in a live broadcast process;

the encapsulating module 503 is configured to encapsulate the file identifier of the multimedia file and the playing progress of the multimedia file to obtain a pseudo video frame;

a second sending module 504, configured to send the pseudo video frame to a streaming media server, so that the streaming media server sends the pseudo video frame to a pull stream player end, and the pull stream player end displays, according to the pseudo video frame, a prompt message corresponding to the multimedia file in the playing progress in a process of playing a live video through the video frame.

Optionally, the apparatus 500 further comprises:

the conversion module is used for converting the system time into a first time stamp and converting the acquisition time into a second time stamp;

the encoding module is used for packaging and encoding the video frame, the first time stamp and the second time stamp to obtain an encoded video frame;

correspondingly, the first sending module 501 is further configured to:

and sending the encoded video frame to a streaming media server.

optionally, the apparatus 500 further comprises:

The second determining module is used for determining a first timestamp and a second timestamp which are sent when the video frame is sent to the streaming media server before the current time and is the latest time away from the current time;

accordingly, the second sending module 504 is further configured to:

optionally, the obtaining module 502 includes:

The acquisition unit is used for acquiring a preset number of audio frames from the received audio frames of the multimedia file according to the sequence of the receiving time of the audio frames from late to early every a preset time period when the multimedia file is not a file pre-stored in the plug-flow client;

the sending unit is used for sending an audio query request to the audio identification server, wherein the audio query request carries the audio information;

it should be noted that: in the information display device in the live broadcasting process provided in the above embodiment, when displaying information, only the division of the above functional modules is taken as an example, and in practical applications, the above function distribution may be completed by different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the information display device in the live broadcasting process and the information display method in the live broadcasting process provided by the above embodiments belong to the same concept, and specific implementation processes thereof are described in detail in the method embodiments and are not described herein again.

Fig. 6 is an information display apparatus 600 in a live broadcast process according to an embodiment of the present invention, where the apparatus 600 is applied to the pull stream player end 103 shown in fig. 1, and as shown in fig. 6, the apparatus 600 includes a first receiving module 601, a second receiving module 602, and an obtaining module 603:

a first receiving module 601, configured to receive a video frame sent by a streaming media server, and display a live broadcast picture based on the video frame, where the video frame is a video frame obtained by a stream pushing client during live broadcast currently and sent to the streaming media server;

a second receiving module 602, configured to receive a pseudo video frame sent by a streaming media server in a display process of the live broadcast frame, where the pseudo video frame is obtained by encapsulating, by a stream pushing client, a file identifier of a multimedia file and a playing progress of the multimedia file every preset time period when the multimedia file is played in a current live broadcast process;

The obtaining module 603 is configured to obtain, according to the pseudo video frame, a prompt message corresponding to the multimedia file at the playing progress, and display the prompt message corresponding to the multimedia file at the playing progress above the layer of the live broadcast frame.

optionally, the obtaining module 603 includes:

The analysis unit is used for analyzing the pseudo video frame to obtain the file identification of the multimedia file and the playing progress of the multimedia file;

The first obtaining unit is used for obtaining the prompt information corresponding to the multimedia file in the playing progress from the prompt file of the multimedia file when the prompt file of the multimedia file is stored in the pull stream player terminal.

optionally, the obtaining module 603 further includes:

the sending unit is used for sending a file downloading request to a file downloading server according to the file identifier of the multimedia file when the prompt file of the multimedia file is not stored in the pull stream player terminal, wherein the file downloading request carries the file identifier of the multimedia file;

optionally, the apparatus 600 further comprises:

accordingly, the obtaining module 603 includes:

accordingly, the lookup unit is further configured to:

when the push stream client plays the multimedia file in the live broadcast process, the pull stream player end receives the video frame and also receives the pseudo video frame sent by the push stream client through the stream media server, and the pseudo video frame is obtained by encapsulating the file identifier of the multimedia file and the playing progress of the multimedia file every other preset time period when the push stream client plays the multimedia file in the current live broadcast process. That is, the pull stream player end can independently display the video frame in the current live broadcast picture and the prompt information corresponding to the multimedia file at the playing progress, so that the user of the pull stream player end can independently perform operations such as moving and closing on the prompt information corresponding to the multimedia file, and the flexibility of information display in the live broadcast process is improved.

it will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

the above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.

Claims

1. An information display method in a live broadcast process is applied to a stream pushing client, and is characterized by comprising the following steps:

2. the method of claim 1, wherein prior to sending the video frames to a streaming server, further comprising:

and sending the encoded video frame to the streaming media server.

3. the method of claim 2, wherein after encapsulating the file identifier of the multimedia file and the playing progress of the multimedia file to obtain the pseudo video frame, the method further comprises:

4. The method according to any one of claims 1 to 3, wherein the obtaining the file identifier of the multimedia file and the playing progress of the multimedia file every preset time period comprises:

synthesizing the obtained audio frames to obtain audio information;

5. An information display method in a live broadcast process is applied to a pull stream player end, and is characterized by comprising the following steps:

receiving a pseudo video frame sent by the streaming media server in the display process of the live broadcast picture, wherein the pseudo video frame is obtained by encapsulating a file identifier of a multimedia file and the playing progress of the multimedia file every other preset time period when the multimedia file is played by the stream pushing client in the current live broadcast process;

6. the method of claim 5, wherein the obtaining, according to the pseudo video frame, prompt information corresponding to the multimedia file in the playing progress comprises:

7. the method as claimed in claim 6, wherein after determining whether the multimedia file hint file is stored in the pull stream player according to the file identifier of the multimedia file, the method further comprises:

8. the method of claim 5, wherein after receiving the pseudo video frame sent by the streaming media server, further comprising:

9. The method of claim 8, wherein a first timestamp is also received when the video frame is received, and the first timestamp is converted from a system time when the video frame is to be captured by the stream pushing client;

10. an information display device in a live broadcast process is applied to a stream pushing client, and is characterized in that the device comprises:

11. The apparatus of claim 10, wherein the apparatus further comprises:

correspondingly, the first sending module is further configured to:

And sending the encoded video frame to the streaming media server.

12. The apparatus of claim 11, wherein the apparatus further comprises:

Correspondingly, the second sending module is further configured to:

13. The apparatus of any of claims 10-12, wherein the obtaining module comprises:

14. the utility model provides an information display device in live broadcast process, is applied to and draws stream broadcast ware end, its characterized in that, the device includes:

the second receiving module is used for receiving a pseudo video frame sent by the streaming media server in the display process of the live broadcast picture, wherein the pseudo video frame is obtained by encapsulating a file identifier of a multimedia file and the playing progress of the multimedia file every other preset time period when the multimedia file is played by the stream pushing client in the current live broadcast process;

15. the apparatus of claim 14, wherein the acquisition module comprises:

16. the apparatus of claim 15, wherein the acquisition module further comprises:

17. the apparatus of claim 14, wherein the apparatus further comprises:

accordingly, the acquisition module comprises:

18. the apparatus of claim 17, wherein a first timestamp is also received when the video frame is received, the first timestamp being converted from a system time when the video frame is to be captured by the stream pushing client;

correspondingly, the lookup unit is further configured to: