CN110166788B

CN110166788B - Information synchronous playing method, device and storage medium

Info

Publication number: CN110166788B
Application number: CN201810868257.7A
Authority: CN
Inventors: 彭勇; 陈爱平; 王威威; 李帅; 魏亮
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2018-08-02
Filing date: 2018-08-02
Publication date: 2021-11-02
Anticipated expiration: 2038-08-02
Also published as: CN110166788A

Abstract

The embodiment of the invention discloses a method, a device and a storage medium for synchronously playing information, wherein the method for synchronously playing the information comprises the following steps: receiving video data sent by a first server, and playing the video data, wherein the video data comprises a video frame carrying a timestamp; when receiving information to be synchronously played, which is sent by a second server and needs to be synchronously played with the video data, extracting a timestamp of the information to be synchronously played; extracting a timestamp of a latest video frame, wherein the latest video frame is a video frame which is closest to the current time and carries the timestamp in the currently received video frame; acquiring the time difference between the timestamp of the latest video frame and the timestamp of the information to be synchronously played; and determining the playing time of the information to be played synchronously according to the time difference, and playing the information to be played synchronously according to the playing time. The scheme can improve the playing synchronism of the video and the information to be synchronously played.

Description

Information synchronous playing method, device and storage medium

Technical Field

The invention relates to the technical field of communication, in particular to a method and a device for synchronously playing information and a storage medium.

Background

With the development of the internet and the continuous expansion of live broadcast users, domestic live broadcast enters a full-fledged period, and the live broadcast industry is about to be smooth in recent years and faces the problem of user loss.

In order to solve the problem of user loss, various live broadcast interactive playing methods are successively introduced by various live broadcast platforms, but the problems are obvious, and due to the fact that video streams need to be distributed through a video stream distribution server, certain time delay exists, and the live broadcast videos and the interactive information are not played synchronously.

For example, in a live interactive playing method for rewarding answering, the questions are not consistent with the questions played by the host, the questions are already played, but the host does not play the questions, and although the influence on the answers is not large, synchronization between the videos and the questions cannot be guaranteed, so that the user experience is seriously reduced.

Disclosure of Invention

The embodiment of the invention provides a method, a device and a storage medium for synchronously playing information, which can improve the playing synchronism of videos and information.

The embodiment of the invention provides an information synchronous playing method, which comprises the following steps:

receiving video data sent by a first server, and playing the video data, wherein the video data comprises a video frame carrying a timestamp;

when receiving information to be synchronously played, which is sent by a second server and needs to be synchronously played with the video data, extracting a timestamp of the information to be synchronously played;

extracting a timestamp of a latest video frame, wherein the latest video frame is a video frame which is closest to the current time and carries the timestamp in the currently received video frame;

acquiring the time difference between the timestamp of the latest video frame and the timestamp of the information to be synchronously played;

and determining the playing time of the information to be played synchronously according to the time difference, and playing the information to be played synchronously according to the playing time.

The embodiment of the invention provides an information synchronous playing device, which comprises:

the video receiving unit is used for receiving video data sent by a first server and playing the video data, and the video data comprises video frames carrying timestamps;

the first extraction unit is used for extracting a timestamp of the information to be synchronously played when receiving the information to be synchronously played, which is sent by a second server and needs to be synchronously played with the video data;

the second extraction unit is used for extracting the timestamp of the latest video frame, wherein the latest video frame is the video frame which is closest to the current time in the currently received video frames and carries the timestamp;

the time difference acquisition unit is used for acquiring the time difference between the timestamp of the latest video frame and the timestamp of the information to be synchronously played;

and the information playing unit is used for determining the playing time of the information to be synchronously played according to the time difference and playing the information to be synchronously played according to the playing time.

The embodiment of the invention also provides a storage medium, wherein the storage device is used for storing a plurality of instructions, and the instructions are suitable for being loaded by the processor and executing the information synchronous playing method provided by the embodiment of the invention.

The embodiment of the invention receives video data sent by a first server and plays the video data, wherein the video data comprises a video frame carrying a timestamp; when receiving information to be synchronously played, which is sent by a second server and needs to be synchronously played with the video data, extracting a timestamp of the information to be synchronously played; extracting a timestamp of a latest video frame, wherein the latest video frame is a video frame which is closest to the current time and carries the timestamp in the currently received video frame; acquiring the time difference between the timestamp of the latest video frame and the timestamp of the information to be synchronously played; and determining the playing time of the information to be played synchronously according to the time difference, and playing the information to be played synchronously according to the playing time. According to the scheme, the video data and the information to be synchronously played are transmitted separately, the playing time of the information to be synchronously played is determined according to the time difference between the video frame timestamp in the video data and the timestamp of the information to be synchronously played, and the information to be synchronously played is played according to the playing time, so that the playing synchronism of the video and the information to be synchronously played is improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1a is a schematic view of a scene of an information synchronous playing system according to an embodiment of the present invention.

Fig. 1b is a schematic view of another scene of the information synchronous playing system according to the embodiment of the present invention.

Fig. 1c is a schematic flow chart of the method for synchronously playing information according to the embodiment of the present invention.

Fig. 2 is a schematic flowchart of another method for synchronously playing information according to an embodiment of the present invention.

Fig. 3a is a schematic structural diagram of an information synchronous playing device according to an embodiment of the present invention.

Fig. 3b is a schematic structural diagram of an information synchronous playing device according to an embodiment of the present invention.

Fig. 4 is a schematic structural diagram of another information synchronous playing device according to an embodiment of the present invention.

Fig. 5 is a schematic structural diagram of a server according to an embodiment of the present invention.

Detailed Description

Referring to the drawings, wherein like reference numbers refer to like elements, the principles of the present application are illustrated as being implemented in a suitable computing environment. The following description is based on illustrated embodiments of the application and should not be taken as limiting the application with respect to other embodiments that are not detailed herein.

In the description that follows, specific embodiments of the present application will be described with reference to steps and symbols executed by one or more computers, unless otherwise indicated. Accordingly, these steps and operations will be referred to, several times, as being performed by a computer, the computer performing operations involving a processing unit of the computer in electronic signals representing data in a structured form. This operation transforms the data or maintains it at locations in the computer's memory system, which may be reconfigured or otherwise altered in a manner well known to those skilled in the art. The data maintains a data structure that is a physical location of the memory that has particular characteristics defined by the data format. However, while the principles of the application have been described in language specific to above, it is not intended to be limited to the specific form set forth herein, and it will be recognized by those of ordinary skill in the art that various of the steps and operations described below may be implemented in hardware.

The term module, as used herein, may be considered a software object executing on the computing system. The various components, modules, engines, and services described herein may be viewed as objects implemented on the computing system. The apparatus and method described herein may be implemented in software, but may also be implemented in hardware, and are within the scope of the present application.

The terms "first", "second", and "third", etc. in this application are used to distinguish between different objects and not to describe a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or modules is not limited to only those steps or modules listed, but rather, some embodiments may include other steps or modules not listed or inherent to such process, method, article, or apparatus.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.

The embodiment of the invention provides an information synchronous playing system, which can comprise a first information synchronous playing device and a second information synchronous playing device provided by the embodiment of the invention.

For example, live webcasting may be a process in which a user at the audience site views live audiovisual scenes, such as events, meetings, teaching, surgery, etc., that the user at the live webcasting site is performing over the web. In some live scenes, the audience plays not only the video data of the main broadcasting end but also other information to be played synchronously. For example, in a scene of a live answer, the playing information to be synchronized may include a question. The live question-answering mode can be that a user logs in a live broadcast room within a specified time, online answering is carried out under the guidance of a host, bonus can be carried out on specified number (such as 12) of questions, and bonus set by each period of melon scores is carried out. At the audience end, the answer frame containing the questions can be played on the live video, so that when the audience utilizes the terminal to hear the questions broadcasted by the host in the live video, the audience can see the answer frame from the screen of the terminal and answer the questions. As another example, in a live-broadcast first-purchase scenario, the to-be-synchronized playing information may include information of a product to be sold. The live broadcast shopping mode can be that a user logs in a live broadcast room within a specified time and carries out online shopping under the guidance of a host. At the audience, the commodity information to be sold can be played on the live broadcast video, the audience can listen to the live broadcast video and host broadcast commodity information, and meanwhile, the audience can also operate a screen, select the preferred commodities and perform order placing/purchasing operation.

It can be understood that the disclosed embodiments can be applied in the following scenarios: in the live broadcast process, a spectator end can receive two paths of data, wherein one path of data can be live broadcast video data, and the other path of data can be information to be synchronously played. And video frames in the live video data, which are associated with the information to be synchronously played, need to be synchronously played with the information to be synchronously played. The information to be synchronously played and at least one frame of video frame in the live video data have an association relationship. For example, the association relationship may be that a video frame broadcasting information to be played synchronously exists in live video data. In addition, the information to be played synchronously may be data such as question, answer statistics, information of goods to be sold, information to be memorized, etc., which are not listed herein.

In an embodiment, the same server can realize live video data distribution and information distribution to be synchronously played. As shown in fig. 1a, fig. 1a is a schematic view of a scene of an information synchronous playing system according to an embodiment of the present invention. In the scene, a main broadcasting end, a server and a spectator end can be included, the first information synchronous playing device is integrated at the spectator end, and the second information synchronous playing device is integrated at the server. It is understood that the scene is illustrated by way of example at one spectator site, and in practice, there may be one or more spectator sites.

The anchor side can be a video data initiating side, the user side can be a video data receiving side, in some scenes, the anchor side can be called a host side, and the audience side can be called a user side. The anchor side and the viewer side can be software installed on the electronic equipment, and can also be terminal equipment. The anchor terminal can call a camera to record videos, take photos and other methods to make live broadcast data, and then send the live broadcast data to the server through the network; both the audience end and the main broadcast end can send the input interactive messages (such as barrage messages) to the server, and the interactive messages can be played at all ends of the same live broadcast room. For the information to be synchronously played, the information is often sent to the server by the main broadcasting terminal and sent to each audience terminal through the server. The server is used for providing background services of internet live broadcast, such as storing the corresponding relation between a main broadcast end and a spectator end, distributing live broadcast data, distributing interactive messages, distributing information to be synchronously played and the like. In one embodiment, in order to adapt the received video data to various live platforms, various protocols, the received video data may be transcoded at the server into a format supporting PTMP, HLS, FLV, etc. for streaming to the corresponding viewer end.

In another embodiment, in order to distinguish different services and improve the timeliness of information processing, the server may include a video distribution server and a live platform service server, and the like, the video distribution server may be used to distribute video streams, and the live platform service server is used to distribute data, in particular to distribute information to be synchronously played. It can be understood that, according to actual requirements, the server may further include a bullet screen server for distributing bullet screens, a gift server for distributing gifts, and the like, which are not described herein in detail.

For example, referring to fig. 1b, fig. 1b is a schematic view of another scene of the information synchronous playing system according to the embodiment of the present invention. In the scene, a main broadcasting end, a first server (video distribution server), a second server (live platform service server) and a spectator end can be included, the first information synchronous playing device is integrated at the spectator end, and the second information synchronous playing device is integrated in the first server and the second server.

When video data need to be sent to a first server, a main broadcasting end adds corresponding timestamps to collected video frames according to the frame rate of the video, sends the timestamps to the first server, and sends the video data to a spectator end through the first server; when a sending instruction of the information to be synchronously played is triggered, the anchor terminal adds a corresponding timestamp for the information to be synchronously played, sends the information to the second server, and sends the information to be synchronously played with the timestamp to the audience terminal by the second server. When the information to be synchronously played is issued to the second server, the currently generated video frame is determined as the associated video frame. Since the video frame associated with the information to be synchronously played is being generated when the information to be synchronously played is transmitted, that is, the timestamp added to the information to be synchronously played at this time corresponds to the timestamp added to the captured video frame at this time. Therefore, the currently generated video frame is a video frame associated with the to-be-synchronized play information. Therefore, when the spectator terminal performs the synchronization operation, the spectator terminal can compare the time stamp of the video frame with the time stamp of the information to be played synchronously, and when the time difference between the two reaches the minimum threshold value, the information to be played synchronously is played in the playing picture of the video, so that the playing synchronization of the associated video frame and the information to be played synchronously is improved.

The following description is made separately.

In this embodiment, the first information synchronous playing device may be integrated at the viewer end, as shown in fig. 1c, and the specific flow of the method of this embodiment may be as follows:

101. and receiving video data sent by a first server, and playing the video data, wherein the video data comprises a video frame carrying a timestamp.

The video data can be recorded and shot by the anchor terminal by calling the camera, and can also be directly obtained from the audio and video acquisition terminal, and the like, and the video data can be manufactured and obtained. In an embodiment, in order to achieve audio and video synchronization, the anchor terminal may synthesize the audio data and the image data after obtaining the audio data and the image data, and ensure that the audio data and the image data are synchronized, so as to avoid the condition that the transmission process causes the picture and the audio to be unsynchronized.

The video data is composed of a plurality of video frames, and the time stamp is added according to the frame rate of the video, for example, the time stamp is added every two frames.

The mode of receiving and playing can be adopted, and when the video frame is received, the video frame can be played.

102. And when receiving to-be-synchronized playing information which is sent by a second server and needs to be played synchronously with the video data, extracting a timestamp of the to-be-synchronized playing information.

The information to be played synchronously carries a timestamp corresponding to the information, and the timestamp of the information to be played synchronously corresponds to a timestamp carried by at least one frame of associated video frame in the video data.

Specifically, when receiving the information to be played synchronously sent by the second server, the timestamp of the information to be played synchronously can be extracted, so that the timestamp of the information to be played synchronously can be compared with the timestamp of the video frame in the following process, and the information to be played synchronously can be played according to the comparison result.

The information to be synchronized may be information associated with at least one frame of video frame in the video data. For example, the association relationship may be that a video frame broadcasting the information to be played synchronously exists in the video data. The information to be played synchronously is divided according to types, and the information to be played synchronously can be character information, image information, audio information, video information and the like. And dividing the information to be synchronously played according to the content, wherein the information to be synchronously played can be question questions, answer statistical results, information of commodities to be sold and the like.

For example, during the live broadcast, the host can broadcast the information to be played synchronously. If the information to be played synchronously is character information or image information, the broadcast of the information to be played synchronously can be reading the specific content of the information to be played synchronously or reading indication information for indicating the information to be played synchronously; if the information to be played synchronously is character information, image information, audio information or video information, broadcasting the information to be played synchronously can be playing the information to be played synchronously, and meanwhile, introducing the information to be played synchronously. The playing is not limited to playing audio and video, and can also include playing information. Taking the information to be played synchronously as the question, in one example, the host may broadcast the specific content of the question, for example: the national treasures in china are what animals, and even answer options can be broadcasted, as follows: A. kangaroo; B. pandas; C. penguin.

103. And extracting the timestamp of the latest video frame, wherein the latest video frame is the video frame which is closest to the current time and carries the timestamp in the currently received video frames.

And the latest video frame is the video frame which is closest to the current time in the currently received video frame and carries the timestamp. For example, the current time is 10:00, 5 video frames are received before the current time, the receiving times are 9:51, 9:53, 9:55, 9:57, and 9:59, where only the video frames received by 9:53 and 9:57 carry timestamps, and then the latest video frame is the video frame received by 9: 57.

When the timestamp of the latest video frame needs to be extracted, the video frame which is closest to the current time and carries the timestamp can be selected from the currently received video frames to be used as the latest video frame, and then the timestamp of the latest video frame is extracted so as to be compared with the timestamp of the information to be synchronously played, so that the information to be synchronously played can be played according to the comparison result.

104. And acquiring the time difference between the timestamp of the latest video frame and the timestamp of the information to be synchronously played.

The information to be synchronized is transmitted through signaling, and the real-time performance of the signaling transmission is higher than that of the video data transmission. Therefore, the video frame and the information to be synchronously played which are sent at the same time can arrive first, and the timestamp of the information to be synchronously played is larger than that of the video frame, namely the timestamp of the information to be synchronously played is larger than that of the latest video frame.

For example, the time stamp is added according to the transmission time. For example, 9:51 sends information to be played and associated video frames at the same time, and since the real-time property of the information to be played transmitted through signaling is higher than that of video data transmission, the information to be played synchronously arrives earlier than the associated video frames. Therefore, the video frame arriving at the same time as the information to be synchronously played is not an associated video frame, that is, the timestamp of the video frame arriving at the same time is smaller than the timestamp of the information to be synchronously played, that is, the timestamps of the video frames arriving at the current time are all smaller than the timestamp of the information to be synchronously played.

Specifically, the time stamp of the information to be played synchronously may be subtracted from the time stamp of the latest video frame to obtain a time difference between the time stamp of the information to be played synchronously and the time stamp of the latest video frame.

105. And determining the playing time of the information to be played synchronously according to the time difference, and playing the information to be played synchronously according to the playing time.

The information to be synchronously played sent at the same time arrives earlier than the video frames sent at the same time, so that the key for aligning the associated video frames and the information to be synchronously played is to wait for the associated video frames for a period of time after the information to be synchronously played arrives, and the time difference between the two is ensured to be small enough. Therefore, the information to be played synchronously can be played by setting a minimum threshold, but the time difference between the timestamps is infinite because the video data transmission is blocked due to network difference, so that the timestamps of the video frames cannot be updated in time, and the information to be played synchronously only waits for the duration of the maximum threshold when the time difference is greater than the maximum threshold.

Specifically, the time difference may be compared with a first preset time duration and a second preset time duration, the playing time of the to-be-synchronized playing information is determined according to the comparison result, and then the to-be-synchronized playing information is played according to the playing time.

The first preset time is a maximum threshold, the second preset time is a minimum threshold, and the first preset time and the second preset time can be obtained according to current network preview data.

In practical applications, for example, in a live question answering scene, a host is prohibited from submitting answers before publishing answers, and considering the fairness of the time for presenting questions among users and the network, a question answering deadline needs to be set in the background to give the remaining question answering time of the user. And when the theme arrives, subtracting the time needing to wait for audio and video synchronization from the residual answering time given by the background to obtain the playing time of the theme. Due to network problems, it may happen that a title arrives late, video data transmission is blocked or cut off, if the title is played after waiting for a maximum threshold, it may happen that the user's remaining answering time is left for only a very short time, such as 3s or even 1s, and the user experience is affected. Therefore, between the user's answering time and synchronization, it is necessary to preferentially guarantee the user's answering time. In a specific implementation process, the average time for each user to answer one question can be obtained according to current network preview data, the average time for each user to answer one question is set as the minimum answer time left for the user, and the real waiting time is the residual answer time given by the background minus the minimum answer time left for the user. The remaining answer time given by the background starts counting after the audience receives the questions, and can be set according to the actual situation.

Therefore, when the time difference is greater than the maximum threshold, the playing time length of the information to be synchronized at the viewer side needs to be considered preferentially. At this time, the waiting time of the information to be synchronously played is not the maximum threshold. At this time, the waiting duration of the to-be-synchronized playing information is the maximum waiting duration, and can be obtained by subtracting the preset minimum playing duration from the preset playing duration given by the background.

Specifically, the comparison result can be divided into three cases:

when the time difference is greater than or equal to the first preset time length, the video data transmission is blocked due to network abnormality, at this time, the information to be played synchronously is played after directly waiting for the maximum waiting time length, and the minimum playing time length is played, that is, the current time after waiting for the maximum waiting time length is the playing time of the information to be played synchronously.

When the time difference is less than or equal to the second preset time length, the information to be played synchronously arrives late due to network abnormality, and at this time, the information to be played synchronously does not wait for the normal playing time length to be directly played, that is, the current time is the playing time of the information to be played synchronously.

When the time difference is smaller than the first preset time and longer than the second preset time, the normal condition is met, and at this time, a timer can be set to wait for the maximum waiting time at regular time. In the waiting process, if a new video frame carrying a timestamp is received, the timestamp of the new video frame can be extracted to update the timestamp of the latest video frame, so that the time difference between the timestamp of the information to be synchronously played and the timestamp of the latest video frame is updated, and the timer is reset to wait for the current waiting time. And when the timestamp of the latest video frame is updated, repeating the step until the time difference between the timestamp of the information to be synchronously played and the timestamp of the latest video frame is less than a second preset time length or the timing time before the timestamp of the latest video frame is updated is up, and ending the cyclic process.

The current waiting time length can be obtained by subtracting the preset minimum playing time length from the preset playing time length given by the background and then subtracting the current waiting time length of the information to be synchronously played, that is, subtracting the current waiting time length of the information to be synchronously played from the maximum waiting time length, and when the timestamp of the latest video frame is updated, the current waiting time length is also updated. The preset playing time, the preset minimum playing time and the normal playing time can be obtained from current network preview data, and can also be set according to actual conditions.

That is, the step of "determining the playing time of the playing information to be synchronized according to the time difference" may include:

when the time difference is not less than a first preset time length, waiting according to the first waiting time length, and determining the current time as the playing time of the information to be synchronously played when waiting is finished;

and when the time difference is not greater than a second preset time length, determining that the current time is the playing time of the information to be synchronously played, wherein the second preset time length is less than the first preset time length.

The step of determining the playing time of the playing information to be synchronized according to the time difference may further include:

when the time difference is smaller than the first preset time and larger than the second preset time, waiting according to the first waiting time, and detecting whether a new video frame carrying a timestamp is received or not in the waiting process;

if so, updating the time difference based on the timestamp of the new video frame, and determining the playing time of the information to be synchronously played according to the updated time difference;

if not, determining the current time after waiting for the first waiting time length as the playing time of the information to be synchronously played.

For example, the first waiting duration to be waited is 10 seconds, the waiting time to start waiting is 9:40:45 (hour/minute/second), and then the current time after waiting for the first waiting duration is 9:40: 55. The first waiting time at this time is the maximum waiting time, and can be obtained by subtracting the preset playing time given by the background from the preset minimum playing time.

For example, the preset playback time is 20 seconds, the preset minimum playback time is 7 seconds, and the first waiting time to be waited at this time is 20 seconds to 7 seconds to 13 seconds.

In an embodiment, after receiving a new video frame carrying a timestamp in the waiting process, the timestamp of the latest video frame may be updated according to the timestamp of the new video frame, and then the time difference is updated based on the timestamp of the to-be-synchronized playing information and the timestamp of the updated latest video frame. That is, the step of "updating the time difference based on the timestamp of the new video frame" may include:

updating the timestamp of the latest video frame according to the timestamp of the new video frame;

acquiring a new time difference between the time stamp of the information to be synchronously played and the time stamp of the updated latest video frame;

and updating the time difference according to the new time difference.

Specifically, the timestamp of the original latest video frame may be replaced with the timestamp of the new video frame, that is, the updated timestamp of the latest video frame is the timestamp of the new video frame. And then, subtracting the timestamp of the information to be synchronously played from the timestamp of the new video frame to obtain a new time difference, and replacing the original time difference with the new time difference.

After the time difference between the time stamp of the information to be synchronously played and the time stamp of the latest video frame is updated, the updated time difference can be compared with the first preset time length and the second preset time length, and then the playing time of the information to be synchronously played is determined according to the comparison result.

In an embodiment, when the updated time difference is not greater than the second preset time, the current time may be directly determined as the playing time of the information to be synchronously played. And when the updated time difference is between the first preset time and the second preset time, acquiring the current time needing to wait, and then waiting, wherein if a new video frame carrying a time stamp is received in the waiting process, the time difference can be updated according to the time stamp of the new video frame. That is, the step of "determining the target video frame required to be synchronized with the to-be-played information according to the updated time difference" may include:

when the updated time difference is not greater than the second preset time length, determining the current time as the playing time of the information to be synchronously played;

when the updated time difference is smaller than the first preset time length and larger than the second preset time length, acquiring a second waiting time length;

and updating the first waiting duration according to the second waiting duration, and returning to execute the step of waiting according to the first waiting duration.

Specifically, after the second waiting duration is obtained, the first waiting duration may be replaced by the second waiting duration, and the updated first waiting duration is not the maximum waiting duration.

The second waiting duration may be obtained by subtracting the preset minimum playing duration from the preset playing duration given by the background and then subtracting the currently waiting duration of the to-be-synchronized playing information, that is, by subtracting the currently waiting duration of the to-be-synchronized playing information from the maximum waiting duration.

The currently-waiting time length of the to-be-synchronized playing information may be a time length from arrival of the to-be-synchronized playing information to the currently-waiting time length. For example, the time length of the currently waiting time of the to-be-synchronized playing information is 10 seconds if the to-be-synchronized playing information has already waited for 10 seconds from the arrival time to the current time. For example, if the current waiting time is 10 seconds, the maximum waiting time is 20 seconds, and the second waiting time is 20 seconds to 10 seconds.

In an embodiment, when the information to be played synchronously is display information, the information to be played synchronously may be played on a picture of the target video frame. During display, the information to be synchronously played can be directly displayed, or the information to be synchronously played can be processed and then displayed, so that a control which can be operated by audiences is provided. Taking the information to be played synchronously as the question, the answer box containing the question can be displayed on the video frame picture. In one embodiment, the question content is contained in the answer box, and an answer input control is provided so that the viewer can input an answer through the answer input control. In another embodiment, to improve the efficiency of answering, each option may be a trigger control, and when the viewer clicks or touches the option, the triggered option serves as an answer submitted by the viewer.

The embodiment adopts the steps of receiving video data sent by a first server and playing the video data, wherein the video data comprises a video frame carrying a timestamp; when information to be synchronously played, which is sent by a second server and needs to be synchronously played with the video data, is received, extracting a timestamp of the information to be synchronously played, and extracting a timestamp of a latest video frame, wherein the latest video frame is a video frame which is closest to the current time and carries the timestamp in the currently received video frame; acquiring the time difference between the timestamp of the latest video frame and the timestamp of the information to be synchronously played; and determining the playing time of the information to be played synchronously according to the time difference, and playing the information to be played synchronously according to the playing time. According to the scheme, the video data and the information to be synchronously played are transmitted separately, the playing time of the information to be synchronously played is determined according to the time difference between the video frame timestamp in the video data and the timestamp of the information to be synchronously played, and the information to be synchronously played is played according to the playing time, so that the playing synchronism of the video and the information to be synchronously played is improved.

In order to better implement the above method, the present invention further provides another information synchronous playing method, which is implemented by integrating the second information synchronous playing device into one entity device.

In this embodiment, the second information synchronous playing device will be described, and with reference to fig. 2, a specific flow of the information synchronous playing method may be as follows:

201. video data is acquired, the video data including a plurality of video frames.

202. And when the video data needs to be sent, adding a time stamp to the video frame and then sending the video frame to the first server.

Specifically, when the video sending instruction is received, a corresponding timestamp may be added to a video frame in the video data according to the frame rate of the video, for example, a timestamp may be added every 2 frames. And then, the video data after the time stamp is added is coded and pushed to a first server. Wherein the timestamp may be added according to the current time.

203. And acquiring to-be-synchronized playing information which needs to be played synchronously with the video data.

204. And when the information to be synchronously played needs to be sent, adding a time stamp to the information to be synchronously played and then sending the information to a second server.

Specifically, when the currently acquired audio/video is associated with the information to be synchronously played, the issuing instruction of the information to be synchronously played is triggered and executed. For example, the acquisition end shoots a picture containing a host and acquires corresponding audio, and when the host broadcasts the information to be synchronously played, the information to be synchronously played is issued, so that the audience end can receive video data and the information to be synchronously played, and plays the information to be synchronously played while playing the main video frame.

When the information to be synchronously played is issued to the second server, the currently generated video frame can be determined as the associated video frame. Since the video frame associated with the information to be synchronously played is being generated when the information to be synchronously played is transmitted, that is, the timestamp added to the information to be synchronously played at this time corresponds to the timestamp added to the captured video frame at this time. Therefore, the currently generated video frame is a video frame associated with the to-be-synchronized play information. Therefore, when the spectator terminal performs the synchronization operation, the spectator terminal can compare the time stamp of the video frame with the time stamp of the information to be played synchronously, and when the time difference between the two reaches the minimum threshold value, the information to be played synchronously is played in the playing picture of the video, so that the playing synchronization of the associated video frame and the information to be played synchronously is improved.

The issuing instruction is an instruction for sending information to be played synchronously. The condition for issuing the instruction can be an instruction generated by triggering at the time of generating the associated video frame and playing the information to be synchronously played in advance. For example, the issuing instruction may be an instruction triggered and generated when the information to be synchronously played starts to be broadcast. The embodiment takes the start of broadcasting the information to be synchronously played as a condition for triggering generation of the issuing instruction, and can realize synchronous playing of the information to be synchronously played and video frames when the information to be synchronously played is broadcasted at a spectator.

In an embodiment, the issuing instruction may be an instruction generated by triggering a key by a specified object when the to-be-synchronized playing information starts to be broadcasted. The keys may be physical keys or touch-sensitive virtual keys. For example, when the host starts broadcasting the information to be played synchronously, such as: when the host recites that the first topic is … …, a worker matched with the host clicks a key on the equipment to trigger and generate an issuing instruction of the information to be synchronously played. Therefore, the sending time of the issued instruction is manually controlled, and the method is easy to realize.

In an embodiment, the issuing instruction may be generated by triggering, when the device recognizes that the live video frame includes a preset keyword or a preset key audio. For example, when the host starts broadcasting the information to be played synchronously, such as: when the host recites that the first topic is … …, the director identifies the voice to obtain a preset key voice, and then triggers to generate an issuing instruction of the information to be synchronously played. Therefore, the issuing instruction can be automatically triggered and generated through voice recognition or image recognition, and labor cost is saved.

It can be understood that the issuing instruction may also be generated by triggering other conditions, which is not described herein.

As can be seen from the above, the present embodiment obtains video data; when the video data needs to be sent, adding a timestamp for a video frame in the video data and then sending the video frame to a first server; acquiring to-be-synchronized playing information to be played synchronously with the video data; and when the information to be synchronously played needs to be sent, adding a time stamp to the information to be synchronously played and then sending the information to a second server. Therefore, the terminal can determine the playing time of the information to be synchronously played by comparing the time stamp of the video frame with the time stamp of the information to be synchronously played, and the playing synchronism of the associated video frame and the information to be synchronously played is improved.

In order to better implement the above method, an embodiment of the present invention provides an information synchronous playing apparatus, as shown in fig. 3a, the information synchronous playing apparatus includes: the video receiving unit 301, the first extracting unit 302, the second extracting unit 303, the duration obtaining unit 304, and the information playing unit 305 are as follows:

a video receiving unit 301, configured to receive video data sent by a first server, and play the video data, where the video data includes a video frame carrying a timestamp;

a first extracting unit 302, configured to, when receiving to-be-synchronized playing information that needs to be played synchronously with the video data and is sent by a second server, extract a timestamp of the to-be-synchronized playing information;

a second extracting unit 303, configured to extract a timestamp of a latest video frame, where the latest video frame is a video frame that is closest to a current time and carries the timestamp, in a currently received video frame;

a time difference obtaining unit 304, configured to obtain a time difference between a timestamp of the latest video frame and a timestamp of the to-be-synchronized playing information;

an information playing unit 305, configured to determine a playing time of the information to be played synchronously according to the time difference, and play the information to be played synchronously according to the playing time.

In some embodiments, referring to fig. 3b, the information playing unit 305 may include:

the first playing sub-unit 3051, configured to, when the time difference is not smaller than a first preset time, obtain a target time length that needs to be waited, determine a current time after waiting for the target time length as a playing time of the to-be-synchronized playing information, and play the to-be-synchronized playing information according to the playing time;

the second determining subunit 3052 is configured to determine, when the time difference is not greater than a second preset time length, that the current time is the playing time of the to-be-synchronized playing information, and play the to-be-synchronized playing information according to the playing time, where the second preset time length is smaller than the first preset time length.

In some embodiments, the information playing unit 305 may further include a third playing sub-unit 3053, configured to:

when the time difference is smaller than the first preset time and larger than the second preset time, acquiring a target time to be waited;

waiting according to the target duration, and detecting whether a new video frame carrying a timestamp is received or not in the waiting process;

if so, updating the time difference based on the timestamp of the new video frame, determining the playing time of the information to be synchronously played according to the updated time difference, and playing the information to be synchronously played according to the playing time.

In some embodiments, the playing subunit 3053 may be specifically configured to:

if so, updating the timestamp of the latest video frame according to the timestamp of the new video frame;

updating the time difference according to the new time difference;

determining the playing time of the information to be synchronously played according to the updated time difference;

if not, determining the current time as the playing time of the information to be synchronously played after waiting for the first waiting time;

and playing the information to be synchronously played according to the playing time.

The information synchronous playing device provided by this embodiment receives video data sent by a first server through a video receiving unit 301, and plays the video data, where the video data includes a video frame carrying a timestamp; when receiving information to be played synchronously with the video data, which is sent by a second server and needs to be played synchronously, a first extraction unit 302 extracts a timestamp of the information to be played synchronously; the second extraction unit 303 extracts a timestamp of a latest video frame, where the latest video frame is a video frame that is closest to a current time and carries the timestamp in the currently received video frame; the time difference obtaining unit 304 obtains a time difference between the timestamp of the latest video frame and the timestamp of the to-be-synchronized playing information; the information playing unit 305 determines the playing time of the information to be played synchronously according to the time difference, and plays the information to be played synchronously according to the playing time. According to the scheme, the video data and the information to be synchronously played are transmitted separately, and the playing time of the information to be synchronously played is determined according to the time difference between the time stamp of the video frame in the video data and the time stamp of the information to be synchronously played, so that the playing synchronism of the video and the information to be synchronously played is improved.

In order to better implement the above method, another information synchronous playing apparatus is further provided in an embodiment of the present invention, as shown in fig. 4, the information synchronous playing apparatus may include: the video acquisition unit 401, the video transmission unit 402, the information acquisition unit 403, and the information transmission unit 404 are as follows:

a video acquisition unit 401 configured to acquire video data, where the video data includes a plurality of video frames;

a video sending unit 402, configured to add a timestamp to the video frame and send the video frame to a first server when the video data needs to be sent;

an information obtaining unit 403, configured to obtain to-be-synchronized playing information that needs to be played synchronously with the video data;

an information sending unit 404, configured to, when the to-be-synchronized playing information needs to be sent, add a timestamp to the to-be-synchronized playing information and send the to-be-synchronized playing information to a second server.

The information synchronous playing device provided by this embodiment acquires video data through the video acquiring unit 401, where the video data includes a plurality of video frames; when the video data needs to be sent, the video sending unit 402 adds a timestamp to the video frame and sends the video frame to the first server; the information obtaining unit 403 obtains to-be-synchronized playing information that needs to be played synchronously with the video data; when the to-be-synchronized playing information needs to be sent, the information sending unit 404 adds a timestamp to the to-be-synchronized playing information and sends the to-be-synchronized playing information to the second server. According to the scheme, the corresponding time stamp is added to the video frame while the push stream is collected, and when the sending instruction of the information to be synchronously played is triggered, the corresponding time stamp is added to the information to be synchronously played. Therefore, the terminal can determine the playing time of the information to be synchronously played by comparing the video frames with the time stamps of the information to be synchronously played so as to improve the playing synchronism of the associated video frames and the information to be synchronously played.

It should be noted that, when the information synchronous playing apparatus provided in the foregoing embodiment plays information, only the division of the functional modules is illustrated, and in practical applications, the above functions may be distributed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the above described functions. In addition, the information synchronous playing device and the information synchronous playing method provided by the above embodiments belong to the same concept, and specific implementation processes thereof are detailed in the method embodiments and are not described herein again.

An embodiment of the present invention further provides a server, as shown in fig. 5, which shows a schematic structural diagram of the server according to the embodiment of the present invention, specifically:

the server may include components such as a processor 501 of one or more processing cores, memory 502 of one or more computer-readable storage media, Radio Frequency (RF) circuitry 503, a power supply 505, an input unit 505, and a display unit 506. Those skilled in the art will appreciate that the server architecture shown in FIG. 5 is not meant to be limiting, and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components. Wherein:

the processor 501 is a control center of the server, connects various parts of the entire server by various interfaces and lines, and performs various functions of the server and processes data by running or executing software programs and/or modules stored in the memory 502 and calling data stored in the memory 502, thereby performing overall monitoring of the server. Optionally, processor 501 may include one or more processing cores; preferably, the processor 501 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 501.

The memory 502 may be used to store software programs and modules, and the processor 501 executes various functional applications and data processing by operating the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data created according to the use of the server, and the like. Further, the memory 502 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

The RF circuit 503 may be used for receiving and transmitting signals during information transmission and reception, and in particular, for receiving downlink information of a base station and then processing the received downlink information by one or more processors 501; in addition, data relating to uplink is transmitted to the base station. In general, the RF circuitry 503 includes, but is not limited to, an antenna, at least one Amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, and the like. In addition, the RF circuitry 503 may also communicate with networks and other devices via wireless communications. The wireless communication may use any communication standard or protocol, including but not limited to Global System for Mobile communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Message Service (SMS), and the like.

The server also includes a power supply 504 (e.g., a battery) for powering the various components, and preferably, the power supply 504 is logically connected to the processor 501 via a power management system, such that functions of managing charging, discharging, and power consumption are performed via the power management system. The power supply 504 may also include any component of one or more dc or ac power sources, recharging systems, power failure detection circuitry, power converters or inverters, power status indicators, and the like.

The server may further include an input unit 505, and the input unit 505 may be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control. In particular, in one particular embodiment, input unit 505 may include a touch-sensitive surface as well as other input devices.

The server may also include a display unit 506, and the display unit 506 may be used to display information input by the user or provided to the user, as well as various graphical user interfaces of the server, which may be made up of graphics, text, icons, video, and any combination thereof.

Although not shown, the server may further include other modules such as a bluetooth module, which will not be described herein. Specifically, in this embodiment, the processor 501 in the server loads the executable file corresponding to the process of one or more application programs into the memory 502 according to the following instructions, and the processor 501 runs the application program stored in the memory 502, thereby implementing various functions as follows:

determining the playing time of the information to be played synchronously according to the time difference, and playing the information to be played synchronously according to the playing time;

in an embodiment, when determining the playing time of the to-be-synchronized playing information according to the time difference, the processor 501 is specifically configured to execute the following steps:

when the time difference is not less than a first preset time length, waiting according to the first waiting time length, and determining the current time as the playing time of the information to be synchronously played after the waiting is finished;

if not, determining that the current time is the playing time of the information to be synchronously played after waiting for the first waiting time.

In an embodiment, when determining the playing time of the to-be-synchronized playing information according to the updated time difference, the processor 501 is specifically configured to execute the following steps:

In an embodiment, when the time difference is updated based on the timestamp of the new video frame, the processor 501 is specifically configured to perform the following steps:

and updating the time difference according to the new time difference.

The server provided by this embodiment receives video data sent by a first server, and plays the video data, where the video data includes a video frame carrying a timestamp; when receiving information to be synchronously played, which is sent by a second server and needs to be synchronously played with the video data, extracting a timestamp of the information to be synchronously played; extracting a timestamp of a latest video, wherein the latest video frame is a video frame which is closest to the current time and carries the timestamp in the currently received video frames; acquiring the time difference between the timestamp of the latest video frame and the timestamp of the information to be synchronously played; and determining the playing time of the information to be played synchronously according to the time difference, and playing the information to be played synchronously according to the playing time. According to the scheme, the video data and the information to be synchronously played are transmitted separately, and the playing time of the information to be synchronously played is determined according to the time difference between the video frame time stamp in the video data and the time stamp of the information to be synchronously played, so that the playing synchronism of the video and the information to be synchronously played is improved.

An embodiment of the present application further provides a storage device, where the storage device stores a computer program, and when the computer program runs on a computer, the computer is enabled to execute the information synchronous playing method in any of the above embodiments, for example: receiving video data sent by a first server, and playing the video data, wherein the video data comprises a video frame carrying a timestamp; when receiving information to be synchronously played, which is sent by a second server and needs to be synchronously played with the video data, extracting a timestamp of the information to be synchronously played; extracting a timestamp of a latest video, wherein the latest video frame is a video frame which is closest to the current time and carries the timestamp in the currently received video frames; acquiring the time difference between the timestamp of the latest video frame and the timestamp of the information to be synchronously played; and determining the playing time of the information to be played synchronously according to the time difference, and playing the information to be played synchronously according to the playing time.

In the embodiment of the present application, the storage device may be a magnetic disk, an optical disk, a Read Only Memory (ROM), a Random Access Memory (RAM), or the like.

In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

It should be noted that, for the information synchronous playing method in the embodiment of the present application, a person having ordinary skill in the art may understand that all or part of the process for implementing the information synchronous playing method in the embodiment of the present application may be completed by controlling the relevant hardware through a computer program, where the computer program may be stored in a computer readable storage medium, such as a memory of the electronic device, and executed by at least one processor in the electronic device, and the process of executing the computer program may include, for example, the process of the embodiment of the information synchronous playing method. The storage medium may be a magnetic disk, an optical disk, a read-only memory, a random access memory, etc.

For the information synchronous playing device in the embodiment of the present application, each functional module may be integrated in one processing chip, or each module may exist alone physically, or two or more modules are integrated in one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium, such as a read-only memory, a magnetic or optical disk, or the like.

The foregoing describes in detail an information synchronized playing method, terminal, and storage medium provided in the embodiments of the present application, and a specific example is applied in the present application to explain the principle and implementation of the present application, and the description of the foregoing embodiments is only used to help understand the method and core ideas of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

1. An information synchronous playing method is characterized by comprising the following steps:

wherein, the determining the playing time of the information to be played synchronously according to the time difference comprises:

when the time difference is smaller than a first preset time and larger than a second preset time, waiting according to the first preset time, and detecting whether a new video frame carrying a timestamp is received or not in the waiting process, wherein the second preset time is smaller than the first preset time;

2. The method for synchronously playing information according to claim 1, wherein the determining the playing time of the information to be synchronously played according to the time difference further comprises:

and when the time difference is not greater than a second preset time length, determining the current time as the playing time of the information to be synchronously played.

3. The method for synchronously playing information according to claim 1, wherein the determining the playing time of the information to be synchronously played according to the updated time difference comprises:

4. The method for synchronously playing information according to claim 1, wherein the updating the time difference based on the time stamp of the new video frame comprises:

and updating the time difference according to the new time difference.

5. An information synchronous playing device, comprising:

the information playing unit is used for determining the playing time of the information to be synchronously played according to the time difference and playing the information to be synchronously played according to the playing time;

the information playing unit further comprises a third playing subunit, configured to:

6. The information synchronized playback device according to claim 5, wherein the information playback unit further includes:

the first playing subunit is used for waiting according to a first waiting duration when the time difference is not less than the first preset duration, determining the current time as the playing time of the information to be synchronously played when the waiting is finished, and playing the information to be synchronously played according to the playing time;

and the second playing subunit is configured to determine, when the time difference is not greater than a second preset time length, that the current time is the playing time of the to-be-synchronized playing information, and play the to-be-synchronized playing information according to the playing time, where the second preset time length is smaller than the first preset time length.

7. The information synchronous playback device according to claim 5, wherein the third playback sub-unit is specifically configured to:

updating the time difference according to the new time difference;

8. A storage medium for storing a plurality of instructions adapted to be loaded by a processor and to perform the method according to any one of claims 1 to 4.