WO2019105239A1 - 视频流发送方法、播放方法、装置、设备及存储介质 - Google Patents

视频流发送方法、播放方法、装置、设备及存储介质 Download PDF

Info

Publication number
WO2019105239A1
WO2019105239A1 PCT/CN2018/115745 CN2018115745W WO2019105239A1 WO 2019105239 A1 WO2019105239 A1 WO 2019105239A1 CN 2018115745 W CN2018115745 W CN 2018115745W WO 2019105239 A1 WO2019105239 A1 WO 2019105239A1
Authority
WO
WIPO (PCT)
Prior art keywords
video
frame
data
live
video stream
Prior art date
Application number
PCT/CN2018/115745
Other languages
English (en)
French (fr)
Inventor
林渊灿
Original Assignee
腾讯科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 腾讯科技（深圳）有限公司 filed Critical 腾讯科技（深圳）有限公司
Publication of WO2019105239A1 publication Critical patent/WO2019105239A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/435—Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
    • H04N21/4788—Supplemental services, e.g. displaying phone caller identification, shopping application communicating with other users, e.g. chatting
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/85—Assembly of content; Generation of multimedia applications
    • H04N21/854—Content authoring
    • H04N21/8547—Content authoring involving timestamps for synchronizing content

Definitions

  • the embodiments of the present invention relate to the field of Internet technologies, and in particular, to a video stream sending method, a playing method, an apparatus, a device, and a storage medium.
  • the game video live broadcast business came into being.
  • the user In the process of providing the game video live broadcast service to the user, in addition to displaying the game screen to the user, the user is also presented with other game data, such as the nickname, location, rank, blood volume and the like of the game character, so that the user is more comprehensive. Learn about the game's game situation.
  • the game video live broadcast scheme provided by the related art is as follows: during the running of the game, the collecting end device collects the game screen according to the preset sampling frequency, encodes the collected game frames of each frame to generate a video stream, and sends the video stream to the server; At the same time, the collection device acquires game data in real time and sends the game data to the server. Correspondingly, the playback device acquires the video stream from the server, and decodes and plays the video stream. During the process of playing the game screen, the player device also obtains the game data from the server in real time, and displays the acquired game data.
  • the player device displays the acquired game data in the following manner: in the process of playing the game screen, the player device determines the started duration of the game according to the current time, the start time of the game game, and the delay time. And display the corresponding game data according to the game's started duration. Among them, the current time and the start time of the game match are determined based on the world time.
  • the delay time is related to various factors such as the encoding delay time, the network delay time, and the delay time of the video stream distribution, and the above delay times are unstable, it is difficult to accurately estimate the total delay time, which leads to the determination.
  • the game's opening time is not accurate enough, and eventually the displayed game data is not synchronized with the game screen. The error between the two will often reach about 5 seconds.
  • the embodiment of the present invention provides a video stream sending method, a playing method, a device, a device, and a storage medium, which can be used to solve the problem that the video live video and its associated data (such as game screen and game data) are not synchronized in the related art.
  • the technical solution is as follows:
  • the embodiment of the present application provides a video stream sending method, where the method includes:
  • an embodiment of the present application provides a video stream playing method, where the method includes:
  • the second video stream includes a first data frame and a second data frame that are arranged in intervals;
  • the live video of the video is played frame by frame, and the associated data corresponding to the live video of the currently played video is displayed.
  • an embodiment of the present application provides a video stream sending apparatus, where the apparatus includes:
  • a picture capturing module is configured to collect a live video screen
  • the data collection module is configured to acquire associated data corresponding to each live video frame of the video frame;
  • a video stream sending module configured to insert the data frame formed by the associated data into a video stream generated by the live video of the video, and send the data frame.
  • an embodiment of the present application provides a video stream playing device, where the device includes:
  • a video stream obtaining module configured to acquire a second video stream, where the second video stream includes a first data frame and a second data frame that are arranged in intervals;
  • a decoding module configured to decode the first data frame to obtain a live video screen
  • a data acquiring module configured to decode the second data frame to obtain associated data corresponding to the live video of the video
  • the display module is configured to play the live video of the video frame by frame, and display associated data corresponding to the live video of the currently played video.
  • the embodiment of the present application provides a video playing system, where the system includes: a collecting end device, a server, and a playing end device;
  • the collection end device includes the video stream sending device according to the above aspect;
  • the playback device includes the video streaming device as described in the above aspect.
  • an embodiment of the present application provides a computer device, where the computer device includes a processor and a memory, where the memory stores at least one instruction, at least one program, a code set, or a set of instructions, the at least one instruction, The at least one program, the code set, or the instruction set implements the video stream transmission method or the video stream playback method when executed by the processor.
  • the embodiment of the present application provides a computer readable storage medium, where the computer readable storage medium stores at least one instruction, at least one program, a code set, or a set of instructions, the at least one instruction, the The at least one program, the set of codes, or the set of instructions, when executed, implements the video stream transmitting method or the video stream playing method described above.
  • an embodiment of the present application provides a computer program product for implementing the above video stream sending method or video stream playing method when the computer program product is executed.
  • the data frame formed by the associated data corresponding to the live video of the video is inserted into the video stream generated by the video live broadcast, so that the playback device can generate the live video of the video according to the information carried in the data frame. Synchronize the associated data corresponding to the live video of the video to ensure that the live video of the video and the associated data are displayed in real time.
  • FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application.
  • FIG. 2 is a flowchart of a video stream sending method according to an embodiment of the present application.
  • FIG. 3 is a schematic diagram showing the results of the first video stream and the second video stream
  • FIG. 4 is a flowchart of a video stream playing method according to an embodiment of the present application.
  • FIG. 5 is a schematic diagram showing display contents of a playback device
  • FIG. 6 is a schematic diagram of an implementation environment provided by another embodiment of the present application.
  • FIG. 7 is a flowchart of a video stream playing method according to another embodiment of the present application.
  • Figure 8 is an interaction diagram corresponding to the method flow shown in Figure 7;
  • FIG. 9 is a block diagram of a video stream sending apparatus according to an embodiment of the present application.
  • FIG. 10 is a block diagram of a video stream sending apparatus according to another embodiment of the present application.
  • FIG. 11 is a block diagram of a video stream playing apparatus according to an embodiment of the present application.
  • FIG. 12 is a structural block diagram of a terminal according to an embodiment of the present application.
  • FIG. 1 shows a schematic diagram of an implementation environment provided by an embodiment of the present application.
  • the implementation environment may be a live video system, and the implementation environment may include: a collection device 11, a server 12, and a playback device 13.
  • the collection device 11 is configured to collect the live video and associated data, and send the collected content to the server 12, so that the playback device 13 obtains the content from the server 12 and displays the content.
  • the collection device 11 may be an electronic device such as a mobile phone, a tablet, a personal computer, or the like.
  • the server 12 is configured to receive the live video of the video and the associated data sent by the collection device 11 and distribute the content to the playback device 13.
  • the server 12 can be a server, a server cluster composed of multiple servers, or a cloud computing service center.
  • the server 13 establishes a communication connection with the collection device 11 and the playback device 13 through a wired or wireless network, respectively.
  • the player device 13 is configured to acquire a live video screen and associated data from the server 12 and present the content to the user.
  • the playback device 13 may be an electronic device such as a mobile phone, a tablet, a personal computer, or the like.
  • the technical solution provided by the embodiment of the present application is applicable to the synchronization problem between the live video of the video and the associated data in the live video scene.
  • the video live scene is a game video live scene
  • the live video screen is a game screen
  • the associated data corresponding to the live video screen is game data
  • the game data refers to real-time data related to the game screen, such as real-time status data of the game character.
  • At least one piece of data including nickname, location, grade, blood volume, equipment, kill status, and other real-time status data of each team in the game, such as the total economy of the team, the killing status of the team, and the like.
  • the live video of the video is a live online video of the network
  • the associated data of the live video may be a subtitle, a barrage, a chat content, an interactive content, audio data, and the like.
  • the technical solution of the game video is taken as an example to describe the technical solution provided by the embodiment of the present application.
  • the technical solution provided by the embodiment of the present application is also applicable to solving similar problems in other live scenarios.
  • FIG. 2 is a flowchart of a method for sending a video stream according to an embodiment of the present application.
  • the method is applicable to the collection device 11 of the implementation environment shown in FIG. 1, and the method may include the following steps. :
  • Step 201 Collect a live video screen.
  • the live video screen refers to a video screen used for live broadcast to the playback device.
  • the collecting end device collects the live video of the video according to the preset sampling frequency during the live broadcast of the video.
  • the preset sampling frequency may be preset according to actual requirements, for example, 25 frames per second, 30 frames per second, or 60 frames per second, etc., which is not limited in the embodiment of the present application.
  • the live video screen is the game screen.
  • the collecting device collects the game screen according to the preset sampling frequency.
  • the game device is installed and run in the collection device.
  • the collection device collects the game screen according to the preset sampling frequency.
  • the collection device runs another application (referred to as “recording application”) for implementing the method flow of the embodiment, and the recording application follows the preset sampling frequency. Capture game screens.
  • the recording application can capture game screens in several possible implementations.
  • the recording application provides an interface to the game application, and the game application sends the game screen to the recording application by calling the above interface.
  • the recording application intercepts the display content of the graphics card of the collection device to obtain the game screen.
  • the recording application obtains the game screen directly from the game application by means of cracking the game application.
  • the game application may also not run on the collection device, which is a device dedicated to collecting game screens.
  • the game application runs in a terminal, and the collection device can be a camera or a smart device with a camera, and the collection device collects a game screen of the game application running in the terminal through the camera.
  • Step 202 Acquire association data corresponding to each live video frame of the video frame.
  • the collection device needs to obtain the associated data of the live video of each frame in addition to the live video of the video.
  • the associated data refers to real-time data related to the live broadcast of the video when the live video screen is displayed. Taking the game video live scene as an example, in the process of running the game, the collection device needs to acquire the game data corresponding to each frame of the game screen in addition to the game screen.
  • the game data to be acquired may be determined according to actual display requirements. For example, when the real-time position of the game character needs to be displayed, the acquired game data includes the position of the game character in each frame of the game screen, and for example, the game needs to be displayed. In the real-time appearance and killing status of the character, the acquired game data includes the equipment and killing status of the game character in each frame of the game screen.
  • step 201 and step 202 are generally performed synchronously, that is, in the live broadcast process, the acquisition device collects the live video of the video frame by frame according to the progress of the live broadcast of the video, and acquires each video in real time. Associated data corresponding to the live video of the frame video.
  • Step 203 Insert the data frame formed by the associated data into the video stream generated by the live video of the video for transmission.
  • the collecting end device generates a data frame according to the associated data, and then inserts the data frame into the video stream generated by the live video of the video, and then sends the video stream to the server.
  • the video stream generated by the video capture live video is the first video stream
  • the video stream generated by inserting the data frame into the first video stream is the second video stream.
  • this step includes the following sub-steps:
  • Step 203 The video live picture is encoded to generate a first video stream, where the first video stream includes a first data frame.
  • Each of the first data frames carries data obtained by encoding a live video frame of one frame.
  • the technology used for encoding the live video picture is not limited, and any mainstream video coding technology may be adopted.
  • the collecting end device encodes the live video picture by using the H.264 encoding technology to generate a first video stream, and the first video stream may be referred to as a H.264 bare stream, and is a network abstraction layer unit (NALU).
  • the abstract layer unit consists of a byte stream.
  • Each frame of the live video is encoded by H.264 encoding technology to obtain one or more NALUs. That is, each first data frame includes one or more NALUs.
  • the above H.264 encoding technology is only exemplary. In practical applications, the video live video can be encoded according to actual needs, such as the Moving Picture Experts Group (MPEG) standard.
  • MPEG Moving Picture Experts Group
  • the coding algorithm, the related algorithm of the video codec (VC), the H.265 coding algorithm, and the like are not limited in this embodiment of the present application.
  • Step 203b Insert a second data frame in the first video stream to generate a second video stream, where the second data frame carries information for the playing end device to obtain the associated data.
  • the second data frame in which the information of the associated data corresponding to the live broadcast picture of the ith frame is obtained by the playback device is inserted before or after the ith first data frame, and the ith first data frame carries the ith frame
  • the data obtained by encoding the live video of the video, i is a positive integer.
  • the second data frame carries information for the playback device to acquire the associated data, that is, the playback device can obtain the associated data according to the foregoing information carried in the second data frame.
  • the second data frame and the ith first data frame where the information of the associated data corresponding to the live broadcast picture of the i-th frame is obtained by the playback device is obtained, and the corresponding associated data is acquired and sent in real time. There are no other first data frames spaced apart, or a smaller number of other first data frames.
  • the collecting end device inserts a second data frame after each first data frame of the first video stream.
  • the second data frame inserted after the ith first data frame is the ith second data frame, and the ith second data frame carries the associated data corresponding to the live broadcast picture of the ith frame of the ith frame.
  • Information As shown in FIG. 3, the video stream live video is encoded by the H.264 encoding technology to generate a first video stream.
  • the first video stream includes a NALU corresponding to the live video of the first frame and a video live broadcast of the second frame. NALU, the NALU corresponding to the live video of the third frame, and so on.
  • the acquiring device adds a second data frame after the NALU corresponding to the video live picture of the first frame, where the second data frame includes one or more NALUs, and the one or more NALUs include the association corresponding to the video live picture of the first frame.
  • the acquiring device adds a second data frame after the NALU corresponding to the live video frame of the second frame, where the second data frame includes one or more NALUs, and the one or more NALUs include the second frame video.
  • the sink device may also insert a second data frame before each first data frame of the first video stream.
  • the second data frame inserted before the ith first data frame is the ith second data frame, and the ith second data frame carries the associated data corresponding to the live broadcast picture of the ith frame of the ith frame. Information.
  • first data frame and the second data frame are spaced one by one, which can maximize the synchronization effect between the live video picture and the associated data.
  • one or more second data frames may be inserted every plurality of first data frames, which is not limited in this embodiment of the present application.
  • one second data frame is inserted every five first data frames, and each second data frame carries an association corresponding to a video live picture for acquiring five first data frames located before the second data frame.
  • Information about the data For example, two second data frames are inserted every eight first data frames, and the two second data frames carry live video for acquiring eight first data frames located before the two second data frames. The information of the associated data corresponding to the screen.
  • the information carried in the second data frame for the playing end device to obtain the associated data includes an identifier corresponding to the live video of the video.
  • the ith second data frame carries an identifier corresponding to the video live picture of the ith frame.
  • the identifier corresponding to the live video screen is used to uniquely indicate the live video of the video, and different live video screens have different identifiers.
  • the identifier corresponding to the live video of the video is a timestamp corresponding to the live video of the video.
  • the timestamp corresponding to the live broadcast of the video of the i-th frame refers to the started duration of the live broadcast of the live video of the ith frame.
  • the time stamp corresponding to the i-th frame game screen may refer to the started duration of the game when the ith frame game screen is displayed.
  • the time frame of the live video of the first frame is 40 ms
  • the time stamp corresponding to the live video of the second frame is 80 ms.
  • the time corresponding to the live video of the third frame is 80 ms.
  • the stamp is 120ms, and so on.
  • the recording application when applied to a game video live scene, obtains a timestamp corresponding to each frame of the game screen by:
  • the recording application receives a timestamp corresponding to each frame of the game screen provided by the client of the game through an interface calling manner;
  • the recording application provides an interface to the game application, and the game application sends a timestamp corresponding to each frame of the game screen to the recording application by calling the above interface.
  • the recording application uses image recognition technology to identify a timestamp corresponding to each frame of the game screen from each frame of the game screen.
  • the recording application can recognize the time stamp corresponding to each frame of the game screen from each frame of the game screen using image recognition technology.
  • the first possible implementation manner is used to obtain a time stamp faster, and the processing overhead of the collection device is smaller.
  • the collecting device When the information carried in the second data frame for the associated device to obtain the associated data includes the identifier corresponding to the video live video, the collecting device also sends the identifier and associated data corresponding to the live video of each frame to the server, and the server uses each The identifier corresponding to the live video frame and the associated data are stored in association, so that the subsequent playback device requests to obtain the associated data from the server.
  • the information carried in the second data frame for the playing end device to acquire the associated data may directly include the associated data.
  • the i-th second data frame carries the associated data corresponding to the video live picture of the i-th frame.
  • the playback device can directly extract the associated data from the second data frame for display, and does not need to request the data corresponding to the live video of the video from the server according to the identifier corresponding to the live video of the video.
  • the playing end device obtains the associated data more efficiently, but directly carrying the associated data in the video stream may result in an increase in the video bit rate and a higher requirement on the bandwidth.
  • the associated data carried in the second data frame refers to all associated data corresponding to the live video of the video, for example, the i-th second data frame carries the corresponding live video of the i-th frame. All associated data.
  • the associated data includes key associated data and non-critical associated data.
  • all the associated data corresponding to the live broadcast picture of the i-th frame includes the key associated data corresponding to the live broadcast picture of the i-th frame and the non-key related data corresponding to the live broadcast picture of the i-th frame.
  • the manner of dividing the key associated data and the non-key related data is not limited.
  • the key associated data and the non-key related data may be distinguished according to the type of the associated data. Taking the game data as an example, the killing status is the key. Game data, nickname, location, rank, blood volume, equipment, and other types of game data are non-key game data.
  • the key associated data refers to the associated data that requires high real-time performance
  • the non-critical associated data refers to the associated data that has low real-time requirements.
  • the information carried by the second data frame for the playing end device to obtain the associated data includes the identifier corresponding to the live video of the video and the key associated data.
  • the i-th second data frame carries an identifier corresponding to the live broadcast picture of the i-th frame, and key associated data corresponding to the live broadcast picture of the i-th frame.
  • the collecting end device also sends the identifier corresponding to the video live broadcast screen of each frame to the server, and the server is configured to associate the identifier corresponding to each live video of the video frame with the non-key related data, so that the subsequent playback device requests the server from the server.
  • Non-critical associated data is configured to associate the identifier corresponding to each live video of the video frame with the non-key related data.
  • the collecting end device may also send the identifier corresponding to each video live broadcast screen and all associated data to the server, and the server is configured to store the identifier corresponding to each video live broadcast screen and all associated data, and the present application The embodiment does not limit this.
  • the third possible implementation manner described above is a compromise of the first and second possible implementation manners described above, which can ensure timely display of key associated data without excessively increasing the code rate and bandwidth of the second video stream.
  • the identifier corresponding to the video live broadcast screen may be carried in the second data frame according to the actual situation, or all the associated data may be carried in the second data frame, or the identifier corresponding to the live video screen may be carried in the second data frame. Associated with key data.
  • the first data frame and the second data frame have different type identifiers, so that the playback end device distinguishes the first data frame and the second data frame according to the type identifier when parsing the second video stream.
  • the type identifier of the second data frame is 12, which is used to represent the extended data frame; and the type identifier of the first data frame may be any value between 0 and 11, or other A value other than 12 that is used to represent a frame of video data.
  • step 203c the second video stream is sent to the server.
  • the collecting end device After the collecting end device generates the second video stream, the second video stream is sent to the server, and the second video stream is distributed by the server to the playing end device for playing.
  • the collecting end device encapsulates the second video stream according to the preset streaming media format, generates a streaming media file in a corresponding format, and then sends the streaming media file to the server.
  • One or more data frames may be included in one streaming media file.
  • the foregoing streaming media format may be an FLV (Flash Video) format, which has the advantages of small file size, fast loading speed, and the like, and is suitable for a scene in which a game video is broadcast live.
  • the data frame formed by the associated data corresponding to the live video of the video is inserted into the video stream generated by the live video display, so that the playback device can generate the live video of the video in the parsed video stream.
  • the associated data corresponding to the live video of the video is synchronously acquired according to the information carried in the data frame, so as to ensure that the live video of the video and the associated data are displayed in real time.
  • the technical solution provided by the embodiment of the present application has a good application value in a live video game scene.
  • the synchronization error between the live video picture and the associated data is only a time interval of one frame of data, and the synchronization error is shortened from 5 seconds to a millisecond level.
  • the second video stream generated may have an increased code rate and bandwidth compared to the first video stream. Ignore it, it does not affect the normal transmission of the video stream.
  • FIG. 4 is a flowchart of a video stream playing method according to an embodiment of the present application.
  • the method is applicable to the playing end device 13 in the implementation environment shown in FIG. 1, and the method may include the following steps. :
  • Step 401 Acquire a second video stream, where the second video stream includes a first data frame and a second data frame that are arranged in intervals.
  • the player device acquires the second video stream from the server.
  • the second video stream on the server side is sent by the collecting end device to the server.
  • the specific process of recording and generating the second video stream by the collecting end device and sending the second video stream to the server is described in the embodiment of FIG. 2 above. I won't go into details here.
  • the second video stream includes a first data frame and a second data frame.
  • the first data frame carries data obtained by encoding a live video picture.
  • each of the first data frames carries data obtained by encoding a live video frame of one frame.
  • the i-th first data frame carries data obtained by encoding the live video picture of the i-th frame, and i is a positive integer.
  • the second data frame carries information for the playback device to acquire the associated data.
  • the second data frame inserted before and/or after the ith first data frame carries information for the playback device to acquire the associated data corresponding to the live video frame of the ith frame, and the associated data corresponding to the live video of the ith frame is Refers to real-time data related to live video broadcast when the live video of the i-th frame is displayed.
  • a second data frame is inserted after each first data frame.
  • the second data frame inserted after the ith first data frame is the ith second data frame, where the ith second data frame carries the association corresponding to the live broadcast picture of the ith frame of the ith frame.
  • Information about the data is inserted before each first data frame.
  • the second data frame inserted before the ith first data frame is the ith second data frame, and the ith second data frame carries the association corresponding to the live broadcast picture of the ith frame of the ith frame.
  • the playback device decapsulates the streaming media file and parses the second video stream therefrom.
  • Step 402 Decode the first data frame to obtain a live video picture.
  • the playing end device extracts the first data frame from the second video stream, and decodes the first data frame by using a decoding manner corresponding to the foregoing encoding manner to obtain a live video frame of each frame.
  • the playback device uses the H.264 decoding algorithm to decode each first data frame to obtain a live video of each frame.
  • Step 403 Decode the second data frame to obtain associated data corresponding to the live video of the video.
  • the playback device further extracts the second data frame from the second video stream, and obtains the associated data corresponding to each of the live video frames of the frame according to the information carried in the second data frame.
  • the playing end device may distinguish the first data frame and the second data frame from the second video stream according to different types of identifiers of the first data frame and the second data frame.
  • the information carried in the second data frame for the playing end device to obtain the associated data includes an identifier corresponding to the live video of the video.
  • the identifier corresponding to the live video screen is used to uniquely indicate the live video of the video, and different live video screens have different identifiers.
  • the above step 403 can include the following sub-steps:
  • the playback device sends a data acquisition request to the server, where the data acquisition request carries an identifier corresponding to the live video screen;
  • the server is configured to store the identifier corresponding to each live video of the video frame and the associated data. After receiving the data acquisition request sent by the device, the server obtains the associated data corresponding to the identifier carried in the request from the stored associated data, and sends the obtained associated data to the playback device.
  • the playback device receives the associated data corresponding to the live video of the video sent by the server.
  • the identifier corresponding to the live video of the video is a timestamp corresponding to the live video of the video.
  • the timestamp corresponding to the live broadcast of the video of the i-th frame refers to the started duration of the live broadcast of the live video of the ith frame.
  • the 40-frame video live broadcast screen is collected by the collecting device.
  • the timestamp of the live video of the first frame is 25 ms
  • the timestamp of the live video of the second frame is 50 ms.
  • the stamp is 75ms, and so on.
  • the playback device when the information carried by the playback device for acquiring the associated data in the second data frame includes the associated data corresponding to the live video of the video, the playback device directly extracts each frame from the second data frame. The associated data of the live video screen.
  • the information carried by the second data frame for the playback device to obtain the association data includes the identifier corresponding to the live video of the video and the key associated data.
  • the playback device obtains the key association data corresponding to the live video of the live video from the second data frame; and the playback device sends a data acquisition request to the server, where the data acquisition request carries the identifier corresponding to the live video of the video, and the server uses The identifier corresponding to the frame video live broadcast is stored in association with the non-key related data.
  • the server After receiving the data acquisition request sent by the device, the server obtains the non-key related data corresponding to the identifier carried in the request from the stored associated data.
  • the playing device receives the non-key related data corresponding to the live video of the video sent by the server.
  • the key associated data and the non-critical associated data refer to the embodiment of FIG. 2 above, which is not described in this embodiment.
  • each time the playback device requests to obtain the associated data from the server it may request to obtain the associated data of the first preset duration, where the first preset duration is greater than the duration of the live video of the video frame, for example, the first The preset duration is 3 seconds, 5 seconds, 10 seconds, and so on.
  • the playing end device sends a data acquisition request to the server to request to acquire the associated data of the subsequent display, before the second preset duration before the related data display that has been requested to be obtained is completed.
  • the second preset duration is less than the first preset duration. For example, when the first preset duration is 3 seconds, the second preset duration is 1 second.
  • steps 402 and 403 are generally performed synchronously, that is, the playback device synchronously parses the first data frame and the second data frame to obtain the live video and associated data respectively.
  • Step 404 The video live broadcast screen is played frame by frame, and the associated data corresponding to the currently played live video screen is displayed.
  • the playback device plays the live video broadcast frame by frame, so that the live broadcast video is played on the playback device. Moreover, in the process of playing the live video, the player device displays the associated data corresponding to the live video of the currently played video, so as to achieve the effect of synchronous display of the live video of the video and its corresponding associated data.
  • the associated data may be displayed on a different display screen or in a different area of the same display screen.
  • the associated data may also be displayed on the upper layer of the game screen or displayed in other manners, which is not limited in the embodiment of the present application.
  • the display screen of the playback device includes a first display area and a second display area, the playback device plays a live video screen in the first display area, and displays the current display in the first display area in the second display area.
  • Associated data corresponding to the live video of the video In another example, the playback device includes a first display screen and a second display screen, the playback device plays a live video screen in the first display screen, and displays the current display in the first display screen in the second display screen.
  • the associated data corresponding to the live video of the video is a game screen, and the associated data is game data.
  • the player device can display the game data in a combination of text, table, picture, graphic and the like.
  • the play end device displays a virtual scene
  • the virtual scene may be a preset game scene, such as the game map, the game background, the game theme, and the like.
  • the player device renders the game data corresponding to the currently played game screen through the game engine to obtain an image element; and displays the image element in the virtual scene.
  • the virtual scene is a game map
  • the game data includes a nickname and a position of each game character
  • the player device adds a character image of each game character at a corresponding position of the game map according to the nickname and position of each game character by the game engine.
  • the display screen of the playback device includes a first display area 51 and a second display area 52.
  • a game screen is displayed in the first display area 51, and a game map is displayed in the second display area 52.
  • the position of each game character synchronized with the currently playing game screen in real time is displayed.
  • the data frame formed by the associated data corresponding to the live video of the video is inserted into the video stream generated by the live video display, so that the playback device can generate the live video of the video in the parsed video stream.
  • the associated data corresponding to the live video of the video is synchronously acquired according to the information carried in the data frame, so as to ensure that the live video of the video and the associated data are displayed in real time.
  • the synchronization error between the live video picture and the associated data is only a time interval of one frame of data, and the synchronization error is shortened from 5 seconds to a millisecond level.
  • FIG. 6 shows a schematic diagram of an implementation environment provided by another embodiment of the present application.
  • the implementation environment may be a video playing system, and the implementation environment may include: a collecting end device 11, a video server 121, a data server 122, and a playing end device 13.
  • the collection device 11 is configured to collect the live video and the associated data, and send the collected live video to the video server 121, and send the collected associated data to the data server 122.
  • the video server 121 is configured to receive the live video of the video sent by the collection device 11 and distribute the live video of the video to the playback device 13 for playback.
  • the data server 122 is configured to receive the associated data sent by the collection device 11 and provide the associated data to the playback device 13 in response to the request of the playback device 13.
  • the player device 13 is configured to acquire a live video screen from the video server 121, and acquire associated data from the data server 122, and present the content to the user.
  • FIG. 7 is a flowchart of a video stream playing method according to another embodiment of the present application.
  • the method is applicable to the implementation environment shown in FIG. 6 , and the method may include the following steps:
  • Step 701 During the live broadcast of the video, the collecting end device 11 collects a live video screen according to a preset sampling frequency.
  • Step 702 The collection end device 11 acquires a timestamp corresponding to each video live broadcast screen, and acquires associated data corresponding to each video live broadcast screen.
  • Step 703 The collecting end device 11 encodes the live video picture to generate a first video stream, where the first video stream includes a first data frame.
  • Each of the first data frames carries data obtained by encoding a live video frame of one frame.
  • Step 704 The collection end device 11 inserts a second data frame in the first video stream to generate a second video stream, where the second data frame carries a timestamp corresponding to the live video of the video.
  • step 705 the collection device 11 sends a second video stream to the video server 121.
  • the video server 121 is a CDN (Content Delivery Network) server. After receiving the second video stream, the video server 121 sends the second video stream to the playback device 13 that requests the second video stream.
  • CDN Content Delivery Network
  • step 706 the collection end device 11 sends the timestamp and associated data corresponding to each live video frame of the video to the data server 122.
  • the data server 122 stores timestamps and associated data corresponding to each live video of the video frame.
  • step 707 the playback device 13 requests or receives the second video stream from the video server 121.
  • step 708 the playback device 13 decodes each of the first data frames by using a decoder to obtain a live video of each frame.
  • Step 709 The playback device 13 obtains timestamps corresponding to the live video frames of each frame according to the information carried in the second data frame.
  • step 710 the playback device 13 sends a data acquisition request to the data server 122, where the data acquisition request carries a timestamp corresponding to the live video of the video.
  • step 711 the data server 122 sends the associated data corresponding to the live video of the video to the playback device 13.
  • the data server 122 After receiving the data acquisition request, the data server 122 acquires the associated data corresponding to the timestamp carried in the request from the stored associated data, and sends the acquired associated data to the playback device 13.
  • step 712 the playback device 13 plays the live video of the video on a frame-by-frame basis, and displays the associated data corresponding to the live video of the currently played video.
  • FIG. 8 shows an interaction diagram corresponding to the method flow shown in FIG. 7 in the game video live scene.
  • the collection device 11 collects a game screen, and encodes the game screen to generate a first video stream. At the same time, the collection device 11 acquires a time stamp and game data corresponding to each frame of the game screen, and displays a time stamp corresponding to each frame of the game screen.
  • the game data is sent to the data server 122.
  • the collecting end device 11 inserts a second data frame carrying a time stamp corresponding to the game screen in the first video stream to generate a second video stream, and the collecting end device 11 encapsulates the second video stream to generate a streaming media file, and then streams the stream.
  • the media file is sent to the video server 121.
  • the playback device 13 obtains a streaming media file from the video server 121, and decapsulates the streaming media file to obtain a second video stream.
  • the playing end device 13 extracts the first data frame and the second data frame respectively from the second video stream, and the playing end device 13 decodes each first data frame by using a decoder to obtain a game screen of each frame and plays, and at the same time, plays
  • the end device 13 pulls the game data from the data server 122 according to the time stamp corresponding to each frame game screen carried in the second data frame, and displays the game data corresponding to the currently played game screen.
  • the module for transmitting the video stream of the collecting end device 11 and the video server 121 need not be modified, and the existing A module for sending video streams.
  • FIG. 9 is a block diagram of a video stream sending apparatus provided by an embodiment of the present application.
  • the apparatus has a function of implementing an example of the above-described video stream transmission method, and the function may be implemented by hardware or may be implemented by hardware by executing corresponding software.
  • the device may be the collection device described above or may be disposed in the collection device.
  • the device may include: a picture collection module 910, a data collection module 920, and a video stream sending module 930.
  • the screen collection module 910 is configured to collect a live video screen.
  • the data collection module 920 is configured to acquire associated data corresponding to each live video of the video frame.
  • the video stream sending module 930 is configured to insert a data frame formed by the associated data into a video stream generated by the live video of the video for transmission.
  • the data frame formed by the associated data corresponding to the live video of the video is inserted into the video stream generated by the live video display, so that the playback device can generate the live video of the video in the parsed video stream.
  • the associated data corresponding to the live video of the video is synchronously acquired according to the information carried in the data frame, so as to ensure that the live video of the video and the associated data are displayed in real time.
  • the associated data refers to data related to the live video of the video that needs to be displayed synchronously when the live video of the video is displayed.
  • the video stream sending module 930 includes an encoding streamizing unit 930a, a data frame inserting unit 930b, and a video stream sending unit 930c.
  • the encoding and streaming unit 930a is configured to encode the live video of the video to generate a first video stream, where the first video stream includes a first data frame, and each of the first data frames carries a video of a live video frame. The data obtained afterwards.
  • the data frame insertion unit 930b is configured to insert a second data frame into the first video stream to generate a second video stream, where the second data frame carries information for the playing end device to acquire the associated data.
  • the video stream sending unit 930c is configured to send the second video stream to the server.
  • the data frame insertion unit 930b is configured to insert a second data frame after each first data frame of the first video stream.
  • the second data frame inserted after the ith first data frame is the ith second data frame, and the ith first data frame carries the video stream of the ith frame.
  • the i-th second data frame carries information for the playback device to acquire the associated data corresponding to the live video frame of the ith frame, where i is a positive integer.
  • the information carried in the ith second data frame includes: an identifier corresponding to the video live picture of the ith frame.
  • the information carried in the ith second data frame further includes: key associated data corresponding to the video live picture of the ith frame;
  • the associated data corresponding to the live broadcast screen of the ith frame includes the key associated data and non-key related data, and the real-time requirement of the key associated data for displaying is higher than the real-time performance of the non-key associated data.
  • the identifier corresponding to the video live picture of the ith frame is a timestamp corresponding to the video live picture of the ith frame, where the timestamp corresponding to the video live picture of the ith frame refers to displaying the The duration of the video live broadcast when the i frame video live video is displayed.
  • the apparatus further includes an identification acquisition module 940 and a data transmission module 950.
  • the identifier obtaining module 940 is configured to obtain an identifier corresponding to each live video frame of the video frame.
  • the data sending module 950 is configured to send, to the server, an identifier and associated data corresponding to a live video frame of each frame, where the server is configured to associate the identifier corresponding to each live video of the video frame with the associated data.
  • the apparatus further includes an identifier acquisition module 940 and a data transmission module 950.
  • the identifier obtaining module 940 is configured to obtain an identifier corresponding to each live video frame of the video frame.
  • the data sending module 950 is configured to send, to the server, an identifier corresponding to each live video frame of the video frame and non-key related data, where the server is configured to store the identifier corresponding to each live video video frame and the non-key related data.
  • FIG. 11 is a block diagram of a video stream playing apparatus provided by an embodiment of the present application.
  • the device has a function of realizing an example of the above video stream playing method, and the function may be implemented by hardware, or may be implemented by hardware corresponding software.
  • the device may be the playback device described above or may be disposed in the playback device.
  • the apparatus may include a video stream acquisition module 1010, a decoding module 1020, a data acquisition module 1030, and a presentation module 1040.
  • the video stream obtaining module 1010 is configured to acquire a second video stream, where the second video stream includes a first data frame and a second data frame that are arranged in intervals.
  • the decoding module 1020 is configured to decode the first data frame to obtain a live video picture.
  • the data obtaining module 1030 is configured to decode the second data frame to obtain associated data corresponding to the live video of the video.
  • the display module 1040 is configured to play the live video of the video frame by frame, and display associated data corresponding to the live video of the currently played video.
  • the data frame formed by the associated data corresponding to the live video of the video is inserted into the video stream generated by the live video display, so that the playback device can generate the live video of the video in the parsed video stream.
  • the associated data corresponding to the live video of the video is synchronously acquired according to the information carried in the data frame, so as to ensure that the live video of the video and the associated data are displayed in real time.
  • the associated data refers to data related to the live video of the video that needs to be displayed synchronously when the live video of the video is displayed.
  • a second data frame is inserted after each first data frame; wherein the second data frame inserted after the ith first data frame is the ith a second data frame, wherein the ith first data frame carries data obtained by encoding a video live picture of the i-th frame, where the ith second data frame is carried by the playback device The information of the associated data corresponding to the live broadcast screen of the i-th frame, wherein the i is a positive integer.
  • the information carried by the second data frame for the playback device to acquire the associated data includes an identifier corresponding to a live video screen.
  • the data obtaining module 1030 is configured to: send a data acquisition request to the server, where the data acquisition request carries an identifier corresponding to the live video of the video; wherein the server is configured to correspond to each live video of the video frame.
  • the identifier and the associated data are stored in association; and the associated data corresponding to the live video of the video sent by the server is received.
  • the associated data includes key associated data and non-critical associated data
  • the associated data carried by the second data frame is acquired by the playing end device.
  • the information includes the identification and key associated data corresponding to the live video of the video.
  • the data obtaining module 1030 is configured to:
  • the server Sending a data acquisition request to the server, where the data acquisition request carries an identifier corresponding to the live video of the video, where the server is configured to associate the identifier corresponding to each live video of the video frame with the non-key related data; Correlation data corresponding to the live video of the video sent by the server.
  • the identifier corresponding to the video live broadcast screen is a timestamp corresponding to the video live broadcast screen, where the timestamp corresponding to the live broadcast screen of the ith frame is when the ith frame video live broadcast screen is displayed.
  • the live broadcast of the video has been started.
  • the display module 1040 is configured to: render the associated data corresponding to the currently played video live video to obtain an image element; display the virtual scene The image element.
  • An exemplary embodiment of the present application further provides a video playing system, where the system includes: a collecting end device, a server, and a playing end device.
  • the collecting end device includes the video stream sending device provided in the embodiment of FIG. 9 or an optional embodiment thereof.
  • the playback device includes a video stream playback device as provided in the embodiment of FIG. 10 or an alternative embodiment thereof.
  • FIG. 12 is a structural block diagram of a terminal 1100 according to an embodiment of the present invention.
  • the terminal 1100 can be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), and an MP4 (Moving Picture Experts Group Audio Layer IV). Level 4) Player, laptop or desktop computer.
  • Terminal 1100 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, and the like.
  • the terminal 1100 can be implemented as the collection device 11 or the playback device 13 in the above embodiment.
  • the terminal 1100 includes a processor 1101 and a memory 1102.
  • the processor 1101 can include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like.
  • the processor 1101 may be configured by at least one of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). achieve.
  • the processor 1101 may also include a main processor and a coprocessor.
  • the main processor is a processor for processing data in an awake state, which is also called a CPU (Central Processing Unit); the coprocessor is A low-power processor for processing data in standby.
  • the processor 1101 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and rendering of content that needs to be displayed on the display screen.
  • the processor 1101 may further include an AI (Artificial Intelligence) processor for processing computational operations related to machine learning.
  • AI Artificial Intelligence
  • Memory 1102 can include one or more computer readable storage media, which can be non-transitory. Memory 1102 can also include high speed random access memory, as well as non-volatile memory, such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer readable storage medium in the memory 1102 is configured to store at least one instruction for execution by the processor 1101 to implement the video stream provided by the method embodiment of the present application. Send method or video stream playback method.
  • the terminal 1100 further optionally includes: a peripheral device interface 1103 and at least one peripheral device.
  • the processor 1101, the memory 1102, and the peripheral device interface 1103 may be connected by a bus or a signal line.
  • Each peripheral device can be connected to the peripheral device interface 1103 via a bus, signal line or circuit board.
  • the peripheral device includes at least one of a radio frequency circuit 1104, a touch display screen 1105, a camera head 1106, an audio circuit 1107, a positioning component 1108, and a power source 1109.
  • the peripheral device interface 1103 can be used to connect at least one peripheral device associated with an I/O (Input/Output) to the processor 1101 and the memory 1102.
  • processor 1101, memory 1102, and peripheral interface 1103 are integrated on the same chip or circuit board; in some other embodiments, any of processor 1101, memory 1102, and peripheral interface 1103 or The two can be implemented on a separate chip or circuit board, which is not limited in this embodiment.
  • the RF circuit 1104 is configured to receive and transmit an RF (Radio Frequency) signal, also called an electromagnetic signal.
  • the RF circuit 1104 communicates with the communication network and other communication devices via electromagnetic signals.
  • the radio frequency circuit 1104 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal.
  • the radio frequency circuit 1104 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like.
  • the radio frequency circuit 1104 can communicate with other terminals via at least one wireless communication protocol.
  • the wireless communication protocols include, but are not limited to, the World Wide Web, a metropolitan area network, an intranet, generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and/or WiFi (Wireless Fidelity) networks.
  • the radio frequency circuit 1104 may further include an NFC (Near Field Communication) related circuit, which is not limited in this application.
  • the display 1105 is used to display a UI (User Interface).
  • the UI can include graphics, text, icons, video, and any combination thereof.
  • the display 1105 also has the ability to capture touch signals over the surface or surface of the display 1105.
  • the touch signal can be input to the processor 1101 as a control signal for processing.
  • the display 1105 can also be used to provide virtual buttons and/or virtual keyboards, also referred to as soft buttons and/or soft keyboards.
  • the display screen 1105 may be one, and the front panel of the terminal 1100 is disposed; in other embodiments, the display screen 1105 may be at least two, respectively disposed on different surfaces of the terminal 1100 or in a folded design; In still other embodiments, the display screen 1105 can be a flexible display screen disposed on a curved surface or a folded surface of the terminal 1100. Even the display screen 1105 can be set to a non-rectangular irregular pattern, that is, a profiled screen.
  • the display 1105 can be made of a material such as an LCD (Liquid Crystal Display) or an OLED (Organic Light-Emitting Diode).
  • Camera component 1106 is used to capture images or video.
  • camera assembly 1106 includes a front camera and a rear camera.
  • the front camera is placed on the front panel of the terminal, and the rear camera is placed on the back of the terminal.
  • the rear camera is at least two, which are respectively a main camera, a depth camera, a wide-angle camera, and a telephoto camera, so as to realize the background blur function of the main camera and the depth camera, and the main camera Combine with a wide-angle camera for panoramic shooting and VR (Virtual Reality) shooting or other integrated shooting functions.
  • the camera assembly 1106 can also include a flash.
  • the flash can be a monochrome temperature flash or a two-color temperature flash.
  • the two-color temperature flash is a combination of a warm flash and a cool flash that can be used for light compensation at different color temperatures.
  • the audio circuit 1107 can include a microphone and a speaker.
  • the microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals for input to the processor 1101 for processing, or to the RF circuit 1104 for voice communication.
  • the microphones may be multiple, and are respectively disposed at different parts of the terminal 1100.
  • the microphone can also be an array microphone or an omnidirectional acquisition microphone.
  • the speaker is then used to convert electrical signals from the processor 1101 or the RF circuit 1104 into sound waves.
  • the speaker can be a conventional film speaker or a piezoelectric ceramic speaker.
  • the audio circuit 1107 can also include a headphone jack.
  • the positioning component 1108 is configured to locate the current geographic location of the terminal 1100 to implement navigation or LBS (Location Based Service).
  • the positioning component 1108 can be a positioning component based on a US-based GPS (Global Positioning System), a Chinese Beidou system, or a Russian Galileo system.
  • a power supply 1109 is used to power various components in the terminal 1100.
  • the power source 1109 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery.
  • the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery.
  • a wired rechargeable battery is a battery that is charged by a wired line
  • a wireless rechargeable battery is a battery that is charged by a wireless coil.
  • the rechargeable battery can also be used to support fast charging technology.
  • terminal 1100 also includes one or more sensors 1110.
  • the one or more sensors 1110 include, but are not limited to, an acceleration sensor 1111, a gyro sensor 1112, a pressure sensor 1113, a fingerprint sensor 1114, an optical sensor 1115, and a proximity sensor 1116.
  • the acceleration sensor 1111 can detect the magnitude of the acceleration on the three coordinate axes of the coordinate system established by the terminal 1100.
  • the acceleration sensor 1111 can be used to detect components of gravity acceleration on three coordinate axes.
  • the processor 1101 can control the touch display screen 1105 to display the user interface in a landscape view or a portrait view according to the gravity acceleration signal collected by the acceleration sensor 1111.
  • the acceleration sensor 1111 can also be used for the acquisition of game or user motion data.
  • the gyro sensor 1112 can detect the body direction and the rotation angle of the terminal 1100, and the gyro sensor 1112 can cooperate with the acceleration sensor 1111 to collect the 3D motion of the user to the terminal 1100. Based on the data collected by the gyro sensor 1112, the processor 1101 can implement functions such as motion sensing (such as changing the UI according to the user's tilting operation), image stabilization at the time of shooting, game control, and inertial navigation.
  • functions such as motion sensing (such as changing the UI according to the user's tilting operation), image stabilization at the time of shooting, game control, and inertial navigation.
  • the pressure sensor 1113 may be disposed at a side border of the terminal 1100 and/or a lower layer of the touch display screen 1105.
  • the pressure sensor 1113 When the pressure sensor 1113 is disposed on the side frame of the terminal 1100, the user's holding signal to the terminal 1100 can be detected, and the processor 1101 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 1113.
  • the operability control on the UI interface is controlled by the processor 1101 according to the user's pressure operation on the touch display screen 1105.
  • the operability control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.
  • the fingerprint sensor 1114 is configured to collect the fingerprint of the user, and the processor 1101 identifies the identity of the user according to the fingerprint collected by the fingerprint sensor 1114, or the fingerprint sensor 1114 identifies the identity of the user according to the collected fingerprint. Upon identifying that the user's identity is a trusted identity, the processor 1101 authorizes the user to perform related sensitive operations including unlocking the screen, viewing encrypted information, downloading software, paying and changing settings, and the like.
  • the fingerprint sensor 1114 can be provided with the front, back or side of the terminal 1100. When the physical button or vendor logo is set on the terminal 1100, the fingerprint sensor 1114 can be integrated with the physical button or the manufacturer logo.
  • Optical sensor 1115 is used to collect ambient light intensity.
  • the processor 1101 can control the display brightness of the touch display 1105 based on the ambient light intensity acquired by the optical sensor 1115. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 1105 is raised; when the ambient light intensity is low, the display brightness of the touch display screen 1105 is lowered.
  • the processor 1101 can also dynamically adjust the shooting parameters of the camera assembly 1106 according to the ambient light intensity acquired by the optical sensor 1115.
  • Proximity sensor 1116 also referred to as a distance sensor, is typically disposed on the front panel of terminal 1100. Proximity sensor 1116 is used to collect the distance between the user and the front side of terminal 1100. In one embodiment, when the proximity sensor 1116 detects that the distance between the user and the front side of the terminal 1100 is gradually decreasing, the processor 1101 controls the touch display screen 1105 to switch from the bright screen state to the interest screen state; when the proximity sensor 1116 detects When the distance between the user and the front side of the terminal 1100 gradually becomes larger, the processor 1101 controls the touch display screen 1105 to switch from the state of the screen to the bright state.
  • terminal 1100 does not constitute a limitation to terminal 1100, may include more or fewer components than illustrated, or may combine certain components, or employ different component arrangements.
  • a computer device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or a set of instructions.
  • the computer device is the collection device 11 in the above embodiment, the at least one instruction, the at least one program, the code set or the instruction set is configured to be executed by one or more processors to implement the video stream transmission.
  • the computer device is the player device 13 in the above embodiment, the at least one instruction, the at least one program, the code set or the instruction set is configured to be executed by one or more processors to implement the video stream playback described above. method.
  • a computer readable storage medium having stored therein at least one instruction, at least one program, a code set or a set of instructions, the at least one instruction, the at least one program
  • the code set or the instruction set implements the above video stream transmission method or video stream playback method when executed by a processor of a computer device.
  • the computer readable storage medium described above may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device.
  • a computer program product for implementing the above-described video stream transmitting method or video stream playing method when the computer program product is executed.
  • a plurality as referred to herein means two or more.
  • "and/or” describing the association relationship of the associated objects, indicating that there may be three relationships, for example, A and/or B, which may indicate that there are three cases where A exists separately, A and B exist at the same time, and B exists separately.
  • the character "/" generally indicates that the contextual object is an "or" relationship.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • General Engineering & Computer Science (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

本申请实施例公开了一种视频流发送方法、播放方法、装置、设备及存储介质。所述方法包括：采集视频直播画面；获取每帧视频直播画面对应的关联数据；将所述关联数据形成的数据帧插入到所述视频直播画面生成的视频流中发送。本申请实施例通过将视频直播画面对应的关联数据形成的数据帧插入到视频直播画面生成的视频流中发送，以使得播放端设备能够在解析视频流生成视频直播画面的过程中，根据数据帧中携带的信息同步获取视频直播画面对应的关联数据，确保将视频直播画面与关联数据进行实时同步地展示。

Description

视频流发送方法、播放方法、装置、设备及存储介质
本申请要求于2017年12月01日提交的，申请号为201711250078.9、发明名称为“视频流发送方法、播放方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及互联网技术领域，特别涉及一种视频流发送方法、播放方法、装置、设备及存储介质。
背景技术
随着流媒体技术的发展，游戏视频直播业务应运而生。在向用户提供游戏视频直播业务的过程中，除了向用户展示游戏画面之外，还向用户展示其它一些游戏数据，例如游戏角色的昵称、位置、等级、血量等数据，以便用户更为全面地了解游戏对局情况。
相关技术提供的游戏视频直播方案如下：在游戏运行的过程中，采集端设备按照预设采样频率采集游戏画面，对采集到的各帧游戏画面进行编码生成视频流，向服务器发送该视频流；与此同时，采集端设备实时获取游戏数据，并将游戏数据发送给服务器。相应地，播放端设备从服务器获取视频流，对视频流进行解码播放，在播放游戏画面的过程中，播放端设备还会从服务器实时获取游戏数据，并将获取到的游戏数据进行展示。在相关技术中，播放端设备通过如下方式将获取到的游戏数据进行展示：播放端设备在播放游戏画面的过程中，根据当前时间、游戏对局的开始时间以及延迟时间确定游戏的已开局时长，并根据游戏的已开局时长显示相应的游戏数据。其中，当前时间、游戏对局的开始时间均是基于世界时间确定的。
由于延迟时间与编码延迟时间、网络延迟时间、视频流分发的延迟时间等多方面因素有关，而上述各项延迟时间均不稳定，因此很难准确估算出总的延迟时间，这就导致所确定的游戏的已开局时长不够准确，最终导致显示的游戏数据与游戏画面不同步，两者之间的误差往往会达到5秒左右。
发明内容
本申请实施例提供了一种视频流发送方法、播放方法、装置、设备及存储介质，可用于解决相关技术中视频直播画面与其关联数据(如游戏画面与游戏数据)不同步的问题。所述技术方案如下：
一方面，本申请实施例提供一种视频流发送方法，所述方法包括：
采集视频直播画面；
获取每帧视频直播画面对应的关联数据；
将所述关联数据形成的数据帧插入到所述视频直播画面生成的视频流中发送。
另一方面，本申请实施例提供一种视频流播放方法，所述方法包括：
获取第二视频流，所述第二视频流中包含有间隔排列的第一数据帧和第二数据帧；
解码所述第一数据帧得到视频直播画面；
解码所述第二数据帧得到所述视频直播画面对应的关联数据；
逐帧播放所述视频直播画面，并展示当前播放的视频直播画面对应的关联数据。
另一方面，本申请实施例提供一种视频流发送装置，所述装置包括：
画面采集模块，用于采集视频直播画面；
数据采集模块，用于获取每帧视频直播画面对应的关联数据；
视频流发送模块，用于将所述关联数据形成的数据帧插入到所述视频直播画面生成的视频流中发送。
另一方面，本申请实施例提供一种视频流播放装置，所述装置包括：
视频流获取模块，用于获取第二视频流，所述第二视频流中包含有间隔排列的第一数据帧和第二数据帧；
解码模块，用于解码所述第一数据帧得到视频直播画面；
数据获取模块，用于解码所述第二数据帧得到所述视频直播画面对应的关联数据；
展示模块，用于逐帧播放所述视频直播画面，并展示当前播放的视频直播画面对应的关联数据。
另一方面，本申请实施例提供一种视频播放系统，所述系统包括：采集端 设备、服务器和播放端设备；
所述采集端设备包括如上述方面所述的视频流发送装置；
所述播放端设备包括如上述方面所述的视频流播放装置。
再一方面，本申请实施例提供一种计算机设备，所述计算机设备包括处理器和存储器，所述存储器中存储有至少一条指令、至少一段程序、代码集或指令集，所述至少一条指令、所述至少一段程序、所述代码集或所述指令集在被所述处理器执行时实现上述视频流发送方法或者视频流播放方法。
又一方面，本申请实施例提供了一种计算机可读存储介质，所述计算机可读存储介质中存储有至少一条指令、至少一段程序、代码集或指令集，所述至少一条指令、所述至少一段程序、所述代码集或所述指令集在被执行时实现上述视频流发送方法或者视频流播放方法。
又一方面，本申请实施例提供了一种计算机程序产品，当该计算机程序产品被执行时，其用于实现上述视频流发送方法或者视频流播放方法。
本申请实施例提供的技术方案可以带来如下有益效果：
通过将视频直播画面对应的关联数据形成的数据帧插入到视频直播画面生成的视频流中发送，以使得播放端设备能够在解析视频流生成视频直播画面的过程中，根据数据帧中携带的信息同步获取视频直播画面对应的关联数据，确保将视频直播画面与关联数据进行实时同步地展示。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本申请一个实施例提供的实施环境的示意图；
图2是本申请一个实施例提供的视频流发送方法的流程图；
图3示出了第一视频流和第二视频流的结果示意图；
图4是本申请一个实施例提供的视频流播放方法的流程图；
图5示出了播放端设备的显示内容的示意图；
图6是本申请另一个实施例提供的实施环境的示意图；
图7是本申请另一个实施例提供的视频流播放方法的流程图；
图8是与图7所示的方法流程相对应的交互图；
图9是本申请一个实施例提供的视频流发送装置的框图；
图10是本申请另一个实施例提供的视频流发送装置的框图；
图11是本申请一个实施例提供的视频流播放装置的框图；
图12是本申请一个实施例提供的终端的结构框图。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请实施方式作进一步地详细描述。
请参考图1，其示出了本申请一个实施例提供的实施环境的示意图。该实施环境可以是一视频直播系统，该实施环境可以包括：采集端设备11、服务器12和播放端设备13。
采集端设备11用于采集视频直播画面和关联数据，并将采集到的上述内容发送给服务器12，以便播放端设备13从服务器12中获取上述内容并进行展示。采集端设备11可以是诸如手机、平板电脑、个人计算机等电子设备。
服务器12用于接收采集端设备11发送的视频直播画面和关联数据，并将上述内容分发给播放端设备13。服务器12可以是一台服务器，也可以是由多台服务器组成的服务器集群，或者是一个云计算服务中心。服务器13通过有线或无线网络，分别与采集端设备11和播放端设备13建立通信连接。
播放端设备13用于从服务器12获取视频直播画面和关联数据，并将上述内容展示给用户。播放端设备13可以是诸如手机、平板电脑、个人计算机等电子设备。
本申请实施例提供的技术方案，适用于视频直播场景下的视频直播画面与关联数据的同步问题。典型的，上述视频直播场景为游戏视频直播场景，视频直播画面为游戏画面，视频直播画面对应的关联数据为游戏数据，游戏数据是指与游戏画面相关的实时数据，如游戏角色的实时状态数据，包括昵称、位置、等级、血量、装备、击杀状况等至少一项数据，又例如游戏中各个团队的实时状态数据，如团队的总经济、团队的击杀状况等至少一项数据。
在其它可能的应用场景中，视频直播画面为网络在线视频直播画面，视频 直播画面对应的关联数据可以是字幕、弹幕、聊天内容、互动内容、音频数据等。在本申请实施例中，主要以游戏视频直播场景为例，对本申请实施例提供的技术方案进行介绍说明，但本申请实施例提供的技术方案对于解决其他直播场景中的类似问题同样适用。
请参考图2，其示出了本申请一个实施例提供的视频流发送方法的流程图，该方法可应用于图1所示实施环境的采集端设备11中，该方法可以包括如下几个步骤：
步骤201，采集视频直播画面。
视频直播画面是指用于向播放端设备进行直播展示的视频画面。可选地，采集端设备在视频直播过程中，按照预设采样频率采集视频直播画面。预设采样频率可根据实际需求预先设定，例如25帧每秒、30帧每秒或者60帧每秒等，本申请实施例对此不作限定。
以游戏视频直播场景为例，视频直播画面即为游戏画面。在游戏运行的过程中，采集端设备按照预设采样频率采集游戏画面。采集端设备中安装运行游戏应用程序，在游戏运行的过程中，若有将游戏画面进行直播展示的需求，则采集端设备按照预设采样频率采集游戏画面。可选地，采集端设备中除了运行有游戏应用程序之外，还运行有用于实现本实施例方法流程的另一应用程序(记为“录制应用程序”)，录制应用程序按照预设采样频率采集游戏画面。
录制应用程序可通过如下几种可能的实施方式采集游戏画面。在一种可能的实施方式中，录制应用程序向游戏应用程序提供接口，游戏应用程序通过调用上述接口向录制应用程序发送游戏画面。在另一种可能的实施方式中，录制应用程序截获采集端设备的显卡的显示内容，从而获取游戏画面。在另一种可能的实施方式中，录制应用程序采用破解游戏应用程序的方式，直接从游戏应用程序中获取游戏画面。
此外，在其它可能的应用场景中，游戏应用程序也可以不在采集端设备运行，采集端设备是一个专用于采集游戏画面的设备。例如，游戏应用程序在一终端中运行，采集端设备可以是一摄像头或者是一具有摄像头的智能设备，采集端设备通过摄像头采集终端中运行的游戏应用程序的游戏画面。
步骤202，获取每帧视频直播画面对应的关联数据。
在视频直播过程中，采集端设备除了需要获取视频直播画面之外，还需要 获取每帧视频直播画面对应的关联数据。关联数据是指在显示视频直播画面时与视频直播相关的实时数据。以游戏视频直播场景为例，在游戏运行的过程中，采集端设备除了需要获取游戏画面之外，还需要获取每帧游戏画面对应的游戏数据。
在实际应用中，可根据实际的展示需求确定所要获取的游戏数据，例如需要展示游戏角色的实时位置时，所获取的游戏数据中包括每帧游戏画面中游戏角色的位置，又例如需要展示游戏角色的实时出装和击杀状况时，所获取的游戏数据中包括每帧游戏画面中游戏角色的装备和击杀状况。
需要说明的是，在实际应用中，上述步骤201和步骤202通常同步执行，也即在视频直播过程中，随着视频直播的进度，采集端设备逐帧采集视频直播画面，并实时获取每一帧视频直播画面对应的关联数据。
步骤203，将关联数据形成的数据帧插入到视频直播画面生成的视频流中发送。
采集端设备根据关联数据生成数据帧，而后将该数据帧插入到视频直播画面生成的视频流，然后将该视频流发送给服务器。上述采集视频直播画面生成的视频流为第一视频流，将数据帧插入到第一视频流中生成的视频流为第二视频流。
可选地，本步骤包括如下几个子步骤：
步骤203a，对视频直播画面进行编码生成第一视频流，第一视频流包括第一数据帧。
每个第一数据帧中携带对一帧视频直播画面进行编码后得到的数据。在本申请实施例中，对视频直播画面进行编码所采用的技术不作限定，其可以采用任意主流的视频编码技术。示例性地，采集端设备采用H.264编码技术对视频直播画面进行编码生成第一视频流，该第一视频流可以称为H.264裸流，是一个由NALU(Network Abstraction Layer Unit，网络抽象层单元)组成的字节流。每一帧视频直播画面采用H.264编码技术进行编码后，得到一个或多个NALU。也即，每个第一数据帧包括一个或多个NALU。
上述H.264编码技术仅是示例性的，在实际应用中，可根据实际需求选用合适的视频编码技术对视频直播画面进行编码，如动态图像专家组(Moving Picture Experts Group，MPEG)标准提供的编码算法、视频编码器(Video Codec，VC)涉及的相关算法、H.265编码算法等，本申请实施例对此不作限定。
步骤203b，在第一视频流中插入第二数据帧，生成第二视频流，第二数据帧中携带供播放端设备获取关联数据的信息。
供播放端设备获取第i帧视频直播画面对应的关联数据的信息所在的第二数据帧插入在第i个第一数据帧之前或者之后，该第i个第一数据帧中携带对第i帧视频直播画面进行编码后得到的数据，i为正整数。第二数据帧中携带有供播放端设备获取关联数据的信息，也即播放端设备能够根据第二数据帧中携带的上述信息获取到关联数据。由于每帧视频直播画面及对应的关联数据是实时获取并发送的，因此供播放端设备获取第i帧视频直播画面对应的关联数据的信息所在的第二数据帧与第i个第一数据帧之间无间隔其它的第一数据帧，或者间隔较少数量其它的第一数据帧。
可选地，采集端设备在第一视频流的每个第一数据帧之后插入一个第二数据帧。其中，在第i个第一数据帧之后插入的第二数据帧为第i个第二数据帧，第i个第二数据帧中携带供播放端设备获取第i帧视频直播画面对应的关联数据的信息。如图3所示，以采用H.264编码技术对视频直播画面进行编码生成第一视频流为例，第一视频流包括第1帧视频直播画面对应的NALU、第2帧视频直播画面对应的NALU、第3帧视频直播画面对应的NALU，以此类推。采集端设备在第1帧视频直播画面对应的NALU后面添加一第二数据帧，该第二数据帧包括一个或多个NALU，该一个或多个NALU中包括第1帧视频直播画面对应的关联数据；类似地，采集端设备在第2帧视频直播画面对应的NALU后面添加一第二数据帧，该第二数据帧包括一个或多个NALU，该一个或多个NALU中包括第2帧视频直播画面对应的关联数据；采集端设备在第3帧视频直播画面对应的NALU后面添加一第二数据帧，该第二数据帧包括一个或多个NALU，该一个或多个NALU中包括第3帧视频直播画面对应的关联数据，以此类推。
在其它可能的实施方式中，采集端设备也可在第一视频流的每个第一数据帧之前插入一个第二数据帧。其中，在第i个第一数据帧之前插入的第二数据帧为第i个第二数据帧，第i个第二数据帧中携带供播放端设备获取第i帧视频直播画面对应的关联数据的信息。
需要说明的一点是，第一数据帧和第二数据帧一一间隔，能够使得视频直播画面与关联数据之间的同步效果达到最大化。但在其它可能的实施方式中，也可以每隔多个第一数据帧插入一个或多个第二数据帧，本申请实施例对此不 作限定。例如，每隔5个第一数据帧插入1个第二数据帧，每一个第二数据帧中携带用于获取位于该第二数据帧之前的5个第一数据帧的视频直播画面对应的关联数据的信息。又例如，每隔8个第一数据帧插入2个第二数据帧，这2个第二数据帧中携带用于获取位于该2个第二数据帧之前的8个第一数据帧的视频直播画面对应的关联数据的信息。
可选地，第二数据帧中携带的供播放端设备获取关联数据的信息包括视频直播画面对应的标识。例如，第i个第二数据帧中携带第i帧视频直播画面对应的标识。视频直播画面对应的标识用于唯一指示该视频直播画面，不同的视频直播画面具有不同的标识。可选地，视频直播画面对应的标识为视频直播画面对应的时间戳；其中，第i帧视频直播画面对应的时间戳是指在显示第i帧视频直播画面时视频直播的已开始时长。以游戏视频直播场景为例，第i帧游戏画面对应的时间戳可以是指在显示第i帧游戏画面时游戏的已开局时长。以采集端设备每秒采集25帧视频直播画面为例，第1帧视频直播画面对应的时间戳为40ms，第2帧视频直播画面对应的时间戳为80ms，第3帧视频直播画面对应的时间戳为120ms，以此类推。
可选地，应用于游戏视频直播场景时，录制应用程序通过如下方式获取每帧游戏画面对应的时间戳：
在一种可能的实施方式中，录制应用程序接收游戏的客户端通过接口调用方式提供的每帧游戏画面对应的时间戳；
录制应用程序向游戏应用程序提供接口，游戏应用程序通过调用上述接口向录制应用程序发送每帧游戏画面对应的时间戳。
在另一种可能的实施方式中，录制应用程序采用图像识别技术从每帧游戏画面中识别得到每帧游戏画面对应的时间戳。
如果游戏画面中显示有已开局时长，则录制应用程序可采用图像识别技术从每帧游戏画面中识别得到每帧游戏画面对应的时间戳。
相较于上述第二种可能的实施方式，采用上述第一种可能的实施方式获取时间戳的速度更快，且采集端设备的处理开销更小。
当第二数据帧中携带的供播放端设备获取关联数据的信息包括视频直播画面对应的标识时，采集端设备还向服务器发送每帧视频直播画面对应的标识和关联数据，服务器用于将每帧视频直播画面对应的标识和关联数据关联存储，以便后续播放端设备从服务器请求获取关联数据。
此外，在另一种可能的实施方式中，第二数据帧中携带的供播放端设备获取关联数据的信息可直接包括关联数据。例如，第i个第二数据帧中携带第i帧视频直播画面对应的关联数据。这样，播放端设备可直接从第二数据帧中提取关联数据进行展示，无需根据视频直播画面对应的标识从服务器中请求获取视频直播画面对应的数据。这种方式播放端设备获取关联数据的效率更高，但在视频流中直接携带关联数据会导致视频码率的增加，对带宽提出了较高要求。需要说明的一点是，在本实施方式中，第二数据帧中携带的关联数据是指视频直播画面对应的全部关联数据，例如第i个第二数据帧中携带第i帧视频直播画面对应的全部关联数据。
在又一种可能的实施方式中，关联数据包括关键关联数据和非关键关联数据。例如，第i帧视频直播画面对应的全部关联数据包括第i帧视频直播画面对应的关键关联数据和该第i帧视频直播画面对应的非关键关联数据。在本申请实施例中，对关键关联数据与非关键关联数据的划分方式不作限定，例如可以按照关联数据的类型区分关键关联数据和非关键关联数据，以游戏数据为例，击杀状况为关键游戏数据，昵称、位置、等级、血量、装备等其它类型的游戏数据为非关键游戏数据。可选地，关键关联数据是指对展示实时性要求较高的关联数据，非关键关联数据是指对展示实时性要求较低的关联数据。
当关联数据包括关键关联数据和非关键关联数据时，第二数据帧中携带的供播放端设备获取关联数据的信息包括视频直播画面对应的标识和关键关联数据。例如，第i个第二数据帧中携带第i帧视频直播画面对应的标识，以及该第i帧视频直播画面对应的关键关联数据。采集端设备还向服务器发送每帧视频直播画面对应的标识和非关键关联数据，服务器用于将每帧视频直播画面对应的标识和非关键关联数据关联存储，以便后续播放端设备从服务器请求获取非关键关联数据。当然，在本实施方式中，采集端设备也可以向服务器发送每帧视频直播画面对应的标识和全部关联数据，服务器用于将每帧视频直播画面对应的标识和全部关联数据关联存储，本申请实施例对此不作限定。
上述第3种可能的实施方式是上述第1种和第2种可能的实施方式的折中方案，可以确保关键关联数据及时显示，且不会过分地增加第二视频流的码率和带宽。在实际应用中，可以根据实际情况选择在第二数据帧中携带视频直播画面对应的标识，或者在第二数据帧中携带全部关联数据，或者在第二数据帧中携带视频直播画面对应的标识和关键关联数据。
需要说明的一点是，第一数据帧和第二数据帧具有不同的类型标识，以供播放端设备在解析第二视频流时，根据上述类型标识对第一数据帧和第二数据帧进行区分。示例性地，当采用H.264编码方式时，第二数据帧的类型标识为12，用于表示扩展数据帧；而第一数据帧的类型标识可以是0至11间的任意数值，或者其它非12的数值，用于表示视频数据帧。
步骤203c，向服务器发送第二视频流。
采集端设备生成第二视频流之后，将第二视频流发送给服务器，由服务器将第二视频流分发给播放端设备进行播放。
可选地，采集端设备按照预设的流媒体格式对第二视频流进行封装，生成相应格式的流媒体文件，而后将上述流媒体文件发送给服务器。一个流媒体文件中可以包括一个或多个数据帧(包括第一数据帧和第二数据帧)。示例性地，上述流媒体格式可以是FLV(Flash Video)格式，其具有文件小、加载速度快等优势，适用于游戏视频直播的场景。
另外，有关播放端设备对视频流进行播放的具体过程，在下述图4实施例中进行介绍说明。
在本申请实施例提供的方案中，通过将视频直播画面对应的关联数据形成的数据帧插入到视频直播画面生成的视频流中发送，以使得播放端设备能够在解析视频流生成视频直播画面的过程中，根据数据帧中携带的信息同步获取视频直播画面对应的关联数据，确保将视频直播画面与关联数据进行实时同步地展示。本申请实施例提供的技术方案在游戏视频直播场景中具有较好的应用价值。
另外，当每个第一数据帧之后插入有一个第二数据帧时，使得视频直播画面与关联数据之间的同步误差仅有一帧数据的时间间隔，将同步误差由5秒缩短至毫秒级别。
另外，当第二数据帧中携带的供播放端设备获取关联数据的信息为视频直播画面对应的标识时，最终生成的第二视频流与第一视频流相比，增加的码率和带宽可忽略不计，并不会影响到视频流的正常传输。
请参考图4，其示出了本申请一个实施例提供的视频流播放方法的流程图，该方法可应用于图1所示实施环境的播放端设备13中，该方法可以包括如下几个步骤：
步骤401，获取第二视频流，第二视频流中包含有间隔排列的第一数据帧和第二数据帧。
可选地，播放端设备从服务器获取第二视频流。服务器端的第二视频流由采集端设备发送给服务器，有关采集端设备录制生成第二视频流以及将上述第二视频流发送给服务器的具体过程已在上文图2实施例中介绍说明，此处不再赘述。
第二视频流中包括第一数据帧和第二数据帧。第一数据帧中携带对视频直播画面进行编码后得到的数据。在本申请实施例中，每个第一数据帧中携带对一帧视频直播画面进行编码后得到的数据。例如，第i个第一数据帧中携带对第i帧视频直播画面进行编码后得到的数据，i为正整数。第二数据帧中携带供播放端设备获取关联数据的信息。插入在第i个第一数据帧之前和/或之后的第二数据帧中携带供播放端设备获取第i帧视频直播画面对应的关联数据的信息，第i帧视频直播画面对应的关联数据是指在显示第i帧视频直播画面时与视频直播相关的实时数据。
可选地，每个第一数据帧之后插入有一个第二数据帧。其中，在第i个第一数据帧之后插入的第二数据帧为第i个第二数据帧，该第i个第二数据帧中携带供播放端设备获取第i帧视频直播画面对应的关联数据的信息。或者，每个第一数据帧之前插入有一个第二数据帧。其中，在第i个第一数据帧之前插入的第二数据帧为第i个第二数据帧，该第i个第二数据帧中携带供播放端设备获取第i帧视频直播画面对应的关联数据的信息。
另外，若播放端设备从服务器获取到流媒体文件，则播放端设备对流媒体文件进行解封装，从中解析出第二视频流。
步骤402，解码第一数据帧得到视频直播画面。
播放端设备从第二视频流中提取第一数据帧，并采用与上述编码方式相对应的解码方式对第一数据帧进行解码，得到各帧视频直播画面。例如，在采集端设备的编码过程采用H.264编码算法的情况下，播放端设备采用H.264解码算法对各个第一数据帧进行解码得到各帧视频直播画面。
步骤403，解码第二数据帧得到视频直播画面对应的关联数据。
播放端设备还从第二视频流中提取第二数据帧，并根据第二数据帧中携带的信息获取各帧视频直播画面分别对应的关联数据。播放端设备可以根据第一数据帧和第二数据帧所具有的不同类型标识，从第二视频流中区分提取出第一 数据帧和第二数据帧。
可选地，第二数据帧中携带的供播放端设备获取关联数据的信息包括视频直播画面对应的标识。视频直播画面对应的标识用于唯一指示该视频直播画面，不同的视频直播画面具有不同的标识。上述步骤403可以包括如下几个子步骤：
1、播放端设备向服务器发送数据获取请求，该数据获取请求中携带视频直播画面对应的标识；
服务器用于将每帧视频直播画面对应的标识和关联数据关联存储。服务器接收到播放端设备发送的数据获取请求之后，从存储的关联数据中获取与上述请求中携带的标识相对应的关联数据，并将获取到的关联数据发送给播放端设备。
2、播放端设备接收服务器发送的上述视频直播画面对应的关联数据。
可选地，视频直播画面对应的标识为视频直播画面对应的时间戳；其中，第i帧视频直播画面对应的时间戳是指在显示第i帧视频直播画面时视频直播的已开始时长。以采集端设备每秒采集40帧视频直播画面为例，第1帧视频直播画面对应的时间戳为25ms，第2帧视频直播画面对应的时间戳为50ms，第3帧视频直播画面对应的时间戳为75ms，以此类推。
在另一种可能的实施方式中，当第二数据帧中携带的供播放端设备获取关联数据的信息包括视频直播画面对应的关联数据时，播放端设备直接从第二数据帧中提取各帧视频直播画面分别对应的关联数据。
在又一种可能的实施方式中，当关联数据包括关键关联数据和非关键关联数据，第二数据帧中携带的供播放端设备获取关联数据的信息包括视频直播画面对应的标识和关键关联数据时，播放端设备从第二数据帧中获取视频直播画面对应的关键关联数据；以及，播放端设备向服务器发送数据获取请求，数据获取请求中携带视频直播画面对应的标识，服务器用于将每帧视频直播画面对应的标识和非关键关联数据关联存储，服务器接收到播放端设备发送的数据获取请求之后，从存储的关联数据中获取与上述请求中携带的标识相对应的非关键关联数据，并将获取到的非关键关联数据发送给播放端设备，相应地，播放端设备接收服务器发送的视频直播画面对应的非关键关联数据。对于关键关联数据和非关键关联数据的说明参见上文图2实施例，本实施例对此不再赘述。
需要说明的一点是，播放端设备每次从服务器请求获取关联数据时，其可 以请求获取第一预设时长的关联数据，该第一预设时长大于一帧视频直播画面的时长，例如第一预设时长为3秒、5秒、10秒等。此外，播放端设备在已请求获取的关联数据展示完成之前的第二预设时长，向服务器发送数据获取请求，以请求获取后续展示的关联数据。其中，第二预设时长小于第一预设时长，例如当第一预设时长为3秒时，第二预设时长为1秒。通过上述方式，能够减少播放端设备与服务器之间交互的请求次数，且确保播放端设备在已请求获取的关联数据展示完成之前就已经获取到后续需要展示的关联数据，避免中断。
还需要说明的一点是，在实际应用中，上述步骤402和步骤403通常同步执行，也即播放端设备同步解析第一数据帧和第二数据帧，分别获取视频直播画面和关联数据。
步骤404，逐帧播放视频直播画面，并展示当前播放的视频直播画面对应的关联数据。
播放端设备在解码得到视频直播画面之后，逐帧播放视频直播画面，实现在播放端设备播放直播视频。并且，在播放直播视频的过程中，播放端设备展示当前播放的视频直播画面对应的关联数据，以达到视频直播画面与其对应的关联数据同步展示的效果。
关联数据可以与游戏画面在不同的显示屏或者同一显示屏的不同区域中显示，关联数据也可以在游戏画面的上层显示，或者以其他方式进行显示，本申请实施例对此不作限定。
在一个示例中，播放端设备的显示屏包括第一显示区域和第二显示区域，播放端设备在第一显示区域播放视频直播画面，在第二显示区域中展示当前在第一显示区域中播放的视频直播画面对应的关联数据。在另一个示例中，播放端设备包括第一显示屏和第二显示屏，播放端设备在第一显示屏中播放视频直播画面，在第二显示屏中展示当前在第一显示屏中播放的视频直播画面对应的关联数据。示例性地，上述视频直播画面为游戏画面，关联数据为游戏数据。
另外，对于游戏数据，播放端设备可以将游戏数据以文字、表格、图片、图文结合等方式进行展示。在一种可能的实施方式中，播放端设备显示有虚拟场景，该虚拟场景可以是预先设定的一游戏场景，如该虚拟场景可以是游戏地图、游戏背景、游戏主题等。播放端设备通过游戏引擎对当前播放的游戏画面对应的游戏数据进行渲染，得到图像元素；在虚拟场景中显示图像元素。例如，虚拟场景为游戏地图，游戏数据包括各个游戏角色的昵称和位置，则播放端设 备通过游戏引擎根据各个游戏角色的昵称和位置，在游戏地图的相应位置处添加各个游戏角色的人物形象。
如图5所示，播放端设备的显示屏包括第一显示区域51和第二显示区域52，第一显示区域51中显示有游戏画面，第二显示区域52中显示有游戏地图，该游戏地图中显示有与当前播放的游戏画面实时同步的各个游戏角色的位置。
在本申请实施例提供的方案中，通过将视频直播画面对应的关联数据形成的数据帧插入到视频直播画面生成的视频流中发送，以使得播放端设备能够在解析视频流生成视频直播画面的过程中，根据数据帧中携带的信息同步获取视频直播画面对应的关联数据，确保将视频直播画面与关联数据进行实时同步地展示。
另外，当每个第一数据帧之后插入有一个第二数据帧时，使得视频直播画面与关联数据之间的同步误差仅有一帧数据的时间间隔，将同步误差由5秒缩短至毫秒级别。
请参考图6，其示出了本申请另一个实施例提供的实施环境的示意图。该实施环境可以是一视频播放系统，该实施环境可以包括：采集端设备11、视频服务器121、数据服务器122和播放端设备13。
采集端设备11用于采集视频直播画面和关联数据，并将采集到的视频直播画面发送给视频服务器121，以及将采集到的关联数据发送给数据服务器122。
视频服务器121用于接收采集端设备11发送的视频直播画面，并将视频直播画面分发给播放端设备13进行播放。
数据服务器122用于接收采集端设备11发送的关联数据，并响应于播放端设备13的请求，向播放端设备13提供其所需要的关联数据。
播放端设备13用于从视频服务器121获取视频直播画面，以及从数据服务器122获取关联数据，并将上述内容展示给用户。
请参考图7，其示出了本申请另一个实施例提供的视频流播放方法的流程图，该方法可应用于图6所示的实施环境中，该方法可以包括如下几个步骤：
步骤701，在视频直播过程中，采集端设备11按照预设采样频率采集视频直播画面。
步骤702，采集端设备11获取每帧视频直播画面对应的时间戳，以及获取每帧视频直播画面对应的关联数据。
步骤703，采集端设备11对视频直播画面进行编码生成第一视频流，第一视频流包括第一数据帧。
每个第一数据帧中携带对一帧视频直播画面进行编码后得到的数据。
步骤704，采集端设备11在第一视频流中插入第二数据帧，生成第二视频流，第二数据帧中携带视频直播画面对应的时间戳。
步骤705，采集端设备11向视频服务器121发送第二视频流。
可选地，视频服务器121为CDN(Content Delivery Network，内容分发网络)服务器。视频服务器121接收到第二视频流之后，向各个请求第二视频流的播放端设备13发送该第二视频流。
步骤706，采集端设备11向数据服务器122发送每帧视频直播画面对应的时间戳和关联数据。
数据服务器122存储每帧视频直播画面对应的时间戳和关联数据。
步骤707，播放端设备13从视频服务器121请求或接收第二视频流。
步骤708，播放端设备13采用解码器对各个第一数据帧进行解码得到各帧视频直播画面。
步骤709，播放端设备13根据第二数据帧中携带的信息获取各帧视频直播画面分别对应的时间戳。
步骤710，播放端设备13向数据服务器122发送数据获取请求，数据获取请求中携带视频直播画面对应的时间戳。
步骤711，数据服务器122向播放端设备13发送视频直播画面对应的关联数据。
数据服务器122接收到数据获取请求之后，从存储的关联数据中获取与上述请求中携带的时间戳相对应的关联数据，并将获取到的关联数据发送给播放端设备13。
步骤712，播放端设备13逐帧播放视频直播画面，并展示当前播放的视频直播画面对应的关联数据。
请参考图8，其示出了在游戏视频直播场景中，与上述图7所示的方法流程相对应的交互图。采集端设备11采集游戏画面，对游戏画面进行编码生成第一视频流，与此同时，采集端设备11获取每帧游戏画面对应的时间戳和游 戏数据，将每帧游戏画面对应的时间戳和游戏数据发送给数据服务器122。采集端设备11在第一视频流中插入携带有游戏画面对应的时间戳的第二数据帧，生成第二视频流，采集端设备11对第二视频流进行封装生成流媒体文件，而后将流媒体文件发送给视频服务器121。播放端设备13从视频服务器121获取流媒体文件，对流媒体文件进行解封装得到第二视频流。播放端设备13从第二视频流中分别提取第一数据帧和第二数据帧，播放端设备13采用解码器对各个第一数据帧进行解码得到各帧游戏画面并播放，与此同时，播放端设备13根据第二数据帧中携带的各帧游戏画面分别对应的时间戳，从数据服务器122拉取游戏数据，并展示当前播放的游戏画面对应的游戏数据。
在本申请实施例中，通过将第二数据帧内嵌至第一数据帧中，从而无需对采集端设备11和视频服务器121的用于发送视频流的模块进行改造，可以复用现有的用于发送视频流的模块。
下述为本申请装置实施例，可以用于执行本申请方法实施例。对于本申请装置实施例中未披露的细节，请参照本申请方法实施例。
请参考图9，其示出了本申请一个实施例提供的视频流发送装置的框图。该装置具有实现上述视频流发送方法示例的功能，所述功能可以由硬件实现，也可以由硬件执行相应的软件实现。该装置可以是上文介绍的采集端设备，也可以设置在采集端设备中。该装置可以包括：画面采集模块910、数据采集模块920和视频流发送模块930。
画面采集模块910，用于采集视频直播画面。
数据采集模块920，用于获取每帧视频直播画面对应的关联数据。
视频流发送模块930，用于将所述关联数据形成的数据帧插入到所述视频直播画面生成的视频流中发送。
在本申请实施例提供的方案中，通过将视频直播画面对应的关联数据形成的数据帧插入到视频直播画面生成的视频流中发送，以使得播放端设备能够在解析视频流生成视频直播画面的过程中，根据数据帧中携带的信息同步获取视频直播画面对应的关联数据，确保将视频直播画面与关联数据进行实时同步地展示。
在基于图9实施例提供的一个可选实施例中，所述关联数据是指显示所述视频直播画面时需要同步显示的与所述视频直播画面相关的数据。
在基于图9实施例提供的另一个可选实施例中，如图10所示，所述视频流发送模块930，包括：编码流化单元930a、数据帧插入单元930b和视频流发送单元930c。
编码流化单元930a，用于对所述视频直播画面进行编码生成第一视频流，所述第一视频流包括第一数据帧，每个第一数据帧中携带对一帧视频直播画面进行编码后得到的数据。
数据帧插入单元930b，用于在所述第一视频流中插入第二数据帧，生成第二视频流，其中，所述第二数据帧中携带供播放端设备获取所述关联数据的信息。
视频流发送单元930c，用于向服务器发送所述第二视频流。
可选地，所述数据帧插入单元930b，用于在第一视频流的每个第一数据帧之后插入一个第二数据帧。其中，在所述第i个第一数据帧之后插入的第二数据帧为第i个第二数据帧，所述第i个第一数据帧中携带对第i帧视频直播画面进行编码后得到的数据，所述第i个第二数据帧中携带供所述播放端设备获取所述第i帧视频直播画面对应的关联数据的信息，所述i为正整数。
可选地，所述第i个第二数据帧中携带的信息包括：所述第i帧视频直播画面对应的标识。
可选地，所述第i个第二数据帧中携带的信息还包括：所述第i帧视频直播画面对应的关键关联数据；
其中，所述第i帧视频直播画面对应的关联数据包括所述关键关联数据和非关键关联数据，所述关键关联数据对展示的实时性要求高于所述非关键关联数据对展示的实时性要求。
可选地，所述第i帧视频直播画面对应的标识为所述第i帧视频直播画面对应的时间戳；其中，所述第i帧视频直播画面对应的时间戳是指在显示所述第i帧视频直播画面时视频直播的已开始时长。
在一个示例中，如图10所示，所述装置还包括：标识获取模块940和数据发送模块950。
标识获取模块940，用于获取每帧视频直播画面对应的标识。
数据发送模块950，用于向所述服务器发送每帧视频直播画面对应的标识和关联数据，所述服务器用于将每帧视频直播画面对应的标识和关联数据关联存储。
在另一个示例中，如图10所示，所述装置还包括：标识获取模块940和数据发送模块950。
标识获取模块940，用于获取每帧视频直播画面对应的标识。
数据发送模块950，用于向所述服务器发送每帧视频直播画面对应的标识和非关键关联数据，所述服务器用于将每帧视频直播画面对应的标识和非关键关联数据关联存储。
请参考图11，其示出了本申请一个实施例提供的视频流播放装置的框图。该装置具有实现上述视频流播放方法示例的功能，所述功能可以由硬件实现，也可以由硬件执行相应的软件实现。该装置可以是上文介绍的播放端设备，也可以设置在播放端设备中。该装置可以包括：视频流获取模块1010、解码模块1020、数据获取模块1030和展示模块1040。
视频流获取模块1010，用于获取第二视频流，所述第二视频流中包含有间隔排列的第一数据帧和第二数据帧。
解码模块1020，用于解码所述第一数据帧得到视频直播画面。
数据获取模块1030，用于解码所述第二数据帧得到所述视频直播画面对应的关联数据。
展示模块1040，用于逐帧播放所述视频直播画面，并展示当前播放的视频直播画面对应的关联数据。
在本申请实施例提供的方案中，通过将视频直播画面对应的关联数据形成的数据帧插入到视频直播画面生成的视频流中发送，以使得播放端设备能够在解析视频流生成视频直播画面的过程中，根据数据帧中携带的信息同步获取视频直播画面对应的关联数据，确保将视频直播画面与关联数据进行实时同步地展示。
在基于图11实施例提供的一个可选实施例中，所述关联数据是指显示所述视频直播画面时需要同步显示的与所述视频直播画面相关的数据。
在基于图11实施例提供的一个可选实施例中，每个第一数据帧之后插入有一个第二数据帧；其中，在第i个第一数据帧之后插入的第二数据帧为第i个第二数据帧，所述第i个第一数据帧中携带对第i帧视频直播画面进行编码后得到的数据，所述第i个第二数据帧中携带供所述播放端设备获取所述第i帧视频直播画面对应的关联数据的信息，所述i为正整数。
在基于图11实施例提供的另一个可选实施例中，所述第二数据帧中携带的供所述播放端设备获取所述关联数据的信息包括视频直播画面对应的标识。
所述数据获取模块1030，用于：向所述服务器发送数据获取请求，所述数据获取请求中携带所述视频直播画面对应的标识；其中，所述服务器用于将每帧视频直播画面对应的标识和关联数据关联存储；接收所述服务器发送的所述视频直播画面对应的关联数据。
在基于图11实施例提供的另一个可选实施例中，所述关联数据包括关键关联数据和非关键关联数据，所述第二数据帧中携带的供所述播放端设备获取所述关联数据的信息包括视频直播画面对应的标识和关键关联数据。
所述数据获取模块1030，用于：
从所述第二数据帧中获取所述视频直播画面对应的关键关联数据；以及，
向所述服务器发送数据获取请求，所述数据获取请求中携带所述视频直播画面对应的标识，其中，所述服务器用于将每帧视频直播画面对应的标识和非关键关联数据关联存储；接收所述服务器发送的所述视频直播画面对应的关联数据。
可选地，所述视频直播画面对应的标识为所述视频直播画面对应的时间戳；其中，所述第i帧视频直播画面对应的时间戳是指在显示所述第i帧视频直播画面时所述视频直播的已开始时长。
在基于图11实施例提供的另一个可选实施例中，所述展示模块1040，用于：对所述当前播放的视频直播画面对应的关联数据进行渲染，得到图像元素；在虚拟场景中显示所述图像元素。
本申请一示例性实施例还提供了一种视频播放系统，所述系统包括：采集端设备、服务器和播放端设备。所述采集端设备包括如图9实施例或者其可选实施例提供的视频流发送装置。所述播放端设备包括如图10实施例或者其可选实施例提供的视频流播放装置。
需要说明的是，上述实施例提供的装置在实现其功能时，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将设备的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。另外，上述实施例提供的装置与方法实施例属于同 一构思，其具体实现过程详见方法实施例，这里不再赘述。
请参考图12，其示出了本发明一个实施例提供的终端1100的结构框图。该终端1100可以是：智能手机、平板电脑、MP3播放器(Moving Picture Experts Group Audio Layer III，动态影像专家压缩标准音频层面3)、MP4(Moving Picture Experts Group Audio Layer IV，动态影像专家压缩标准音频层面4)播放器、笔记本电脑或台式电脑。终端1100还可能被称为用户设备、便携式终端、膝上型终端、台式终端等其他名称。该终端1100可以实现成为上述实施例中的采集端设备11或者播放端设备13。
通常，终端1100包括有：处理器1101和存储器1102。
处理器1101可以包括一个或多个处理核心，比如4核心处理器、8核心处理器等。处理器1101可以采用DSP(Digital Signal Processing，数字信号处理)、FPGA(Field－Programmable Gate Array，现场可编程门阵列)、PLA(Programmable Logic Array，可编程逻辑阵列)中的至少一种硬件形式来实现。处理器1101也可以包括主处理器和协处理器，主处理器是用于对在唤醒状态下的数据进行处理的处理器，也称CPU(Central Processing Unit，中央处理器)；协处理器是用于对在待机状态下的数据进行处理的低功耗处理器。在一些实施例中，处理器1101可以在集成有GPU(Graphics Processing Unit，图像处理器)，GPU用于负责显示屏所需要显示的内容的渲染和绘制。一些实施例中，处理器1101还可以包括AI(Artificial Intelligence，人工智能)处理器，该AI处理器用于处理有关机器学习的计算操作。
存储器1102可以包括一个或多个计算机可读存储介质，该计算机可读存储介质可以是非暂态的。存储器1102还可包括高速随机存取存储器，以及非易失性存储器，比如一个或多个磁盘存储设备、闪存存储设备。在一些实施例中，存储器1102中的非暂态的计算机可读存储介质用于存储至少一个指令，该至少一个指令用于被处理器1101所执行以实现本申请中方法实施例提供的视频流发送方法或者视频流播放方法。
在一些实施例中，终端1100还可选包括有：外围设备接口1103和至少一个外围设备。处理器1101、存储器1102和外围设备接口1103之间可以通过总线或信号线相连。各个外围设备可以通过总线、信号线或电路板与外围设备接口1103相连。具体地，外围设备包括：射频电路1104、触摸显示屏1105、摄 像头1106、音频电路1107、定位组件1108和电源1109中的至少一种。
外围设备接口1103可被用于将I/O(Input/Output，输入/输出)相关的至少一个外围设备连接到处理器1101和存储器1102。在一些实施例中，处理器1101、存储器1102和外围设备接口1103被集成在同一芯片或电路板上；在一些其他实施例中，处理器1101、存储器1102和外围设备接口1103中的任意一个或两个可以在单独的芯片或电路板上实现，本实施例对此不加以限定。
射频电路1104用于接收和发射RF(Radio Frequency，射频)信号，也称电磁信号。射频电路1104通过电磁信号与通信网络以及其他通信设备进行通信。射频电路1104将电信号转换为电磁信号进行发送，或者，将接收到的电磁信号转换为电信号。可选地，射频电路1104包括：天线系统、RF收发器、一个或多个放大器、调谐器、振荡器、数字信号处理器、编解码芯片组、用户身份模块卡等等。射频电路1104可以通过至少一种无线通信协议来与其它终端进行通信。该无线通信协议包括但不限于：万维网、城域网、内联网、各代移动通信网络(2G、3G、4G及5G)、无线局域网和/或WiFi(Wireless Fidelity，无线保真)网络。在一些实施例中，射频电路1104还可以包括NFC(Near Field Communication，近距离无线通信)有关的电路，本申请对此不加以限定。
显示屏1105用于显示UI(User Interface，用户界面)。该UI可以包括图形、文本、图标、视频及其它们的任意组合。当显示屏1105是触摸显示屏时，显示屏1105还具有采集在显示屏1105的表面或表面上方的触摸信号的能力。该触摸信号可以作为控制信号输入至处理器1101进行处理。此时，显示屏1105还可以用于提供虚拟按钮和/或虚拟键盘，也称软按钮和/或软键盘。在一些实施例中，显示屏1105可以为一个，设置终端1100的前面板；在另一些实施例中，显示屏1105可以为至少两个，分别设置在终端1100的不同表面或呈折叠设计；在再一些实施例中，显示屏1105可以是柔性显示屏，设置在终端1100的弯曲表面上或折叠面上。甚至，显示屏1105还可以设置成非矩形的不规则图形，也即异形屏。显示屏1105可以采用LCD(Liquid Crystal Display，液晶显示屏)、OLED(Organic Light-Emitting Diode,有机发光二极管)等材质制备。
摄像头组件1106用于采集图像或视频。可选地，摄像头组件1106包括前置摄像头和后置摄像头。通常，前置摄像头设置在终端的前面板，后置摄像头设置在终端的背面。在一些实施例中，后置摄像头为至少两个，分别为主摄像头、景深摄像头、广角摄像头、长焦摄像头中的任意一种，以实现主摄像头和 景深摄像头融合实现背景虚化功能、主摄像头和广角摄像头融合实现全景拍摄以及VR(Virtual Reality，虚拟现实)拍摄功能或者其它融合拍摄功能。在一些实施例中，摄像头组件1106还可以包括闪光灯。闪光灯可以是单色温闪光灯，也可以是双色温闪光灯。双色温闪光灯是指暖光闪光灯和冷光闪光灯的组合，可以用于不同色温下的光线补偿。
音频电路1107可以包括麦克风和扬声器。麦克风用于采集用户及环境的声波，并将声波转换为电信号输入至处理器1101进行处理，或者输入至射频电路1104以实现语音通信。出于立体声采集或降噪的目的，麦克风可以为多个，分别设置在终端1100的不同部位。麦克风还可以是阵列麦克风或全向采集型麦克风。扬声器则用于将来自处理器1101或射频电路1104的电信号转换为声波。扬声器可以是传统的薄膜扬声器，也可以是压电陶瓷扬声器。当扬声器是压电陶瓷扬声器时，不仅可以将电信号转换为人类可听见的声波，也可以将电信号转换为人类听不见的声波以进行测距等用途。在一些实施例中，音频电路1107还可以包括耳机插孔。
定位组件1108用于定位终端1100的当前地理位置，以实现导航或LBS(Location Based Service，基于位置的服务)。定位组件1108可以是基于美国的GPS(Global Positioning System，全球定位系统)、中国的北斗系统或俄罗斯的伽利略系统的定位组件。
电源1109用于为终端1100中的各个组件进行供电。电源1109可以是交流电、直流电、一次性电池或可充电电池。当电源1109包括可充电电池时，该可充电电池可以是有线充电电池或无线充电电池。有线充电电池是通过有线线路充电的电池，无线充电电池是通过无线线圈充电的电池。该可充电电池还可以用于支持快充技术。
在一些实施例中，终端1100还包括有一个或多个传感器1110。该一个或多个传感器1110包括但不限于：加速度传感器1111、陀螺仪传感器1112、压力传感器1113、指纹传感器1114、光学传感器1115以及接近传感器1116。
加速度传感器1111可以检测以终端1100建立的坐标系的三个坐标轴上的加速度大小。比如，加速度传感器1111可以用于检测重力加速度在三个坐标轴上的分量。处理器1101可以根据加速度传感器1111采集的重力加速度信号，控制触摸显示屏1105以横向视图或纵向视图进行用户界面的显示。加速度传感器1111还可以用于游戏或者用户的运动数据的采集。
陀螺仪传感器1112可以检测终端1100的机体方向及转动角度，陀螺仪传感器1112可以与加速度传感器1111协同采集用户对终端1100的3D动作。处理器1101根据陀螺仪传感器1112采集的数据，可以实现如下功能：动作感应(比如根据用户的倾斜操作来改变UI)、拍摄时的图像稳定、游戏控制以及惯性导航。
压力传感器1113可以设置在终端1100的侧边框和/或触摸显示屏1105的下层。当压力传感器1113设置在终端1100的侧边框时，可以检测用户对终端1100的握持信号，由处理器1101根据压力传感器1113采集的握持信号进行左右手识别或快捷操作。当压力传感器1113设置在触摸显示屏1105的下层时，由处理器1101根据用户对触摸显示屏1105的压力操作，实现对UI界面上的可操作性控件进行控制。可操作性控件包括按钮控件、滚动条控件、图标控件、菜单控件中的至少一种。
指纹传感器1114用于采集用户的指纹，由处理器1101根据指纹传感器1114采集到的指纹识别用户的身份，或者，由指纹传感器1114根据采集到的指纹识别用户的身份。在识别出用户的身份为可信身份时，由处理器1101授权该用户执行相关的敏感操作，该敏感操作包括解锁屏幕、查看加密信息、下载软件、支付及更改设置等。指纹传感器1114可以被设置终端1100的正面、背面或侧面。当终端1100上设置有物理按键或厂商Logo时，指纹传感器1114可以与物理按键或厂商Logo集成在一起。
光学传感器1115用于采集环境光强度。在一个实施例中，处理器1101可以根据光学传感器1115采集的环境光强度，控制触摸显示屏1105的显示亮度。具体地，当环境光强度较高时，调高触摸显示屏1105的显示亮度；当环境光强度较低时，调低触摸显示屏1105的显示亮度。在另一个实施例中，处理器1101还可以根据光学传感器1115采集的环境光强度，动态调整摄像头组件1106的拍摄参数。
接近传感器1116，也称距离传感器，通常设置在终端1100的前面板。接近传感器1116用于采集用户与终端1100的正面之间的距离。在一个实施例中，当接近传感器1116检测到用户与终端1100的正面之间的距离逐渐变小时，由处理器1101控制触摸显示屏1105从亮屏状态切换为息屏状态；当接近传感器1116检测到用户与终端1100的正面之间的距离逐渐变大时，由处理器1101控制触摸显示屏1105从息屏状态切换为亮屏状态。
本领域技术人员可以理解，图12中示出的结构并不构成对终端1100的限定，可以包括比图示更多或更少的组件，或者组合某些组件，或者采用不同的组件布置。
在示例中实施例中，还提供了一种计算机设备，所述计算机设备包括处理器和存储器，所述存储器中存储有至少一条指令、至少一段程序、代码集或指令集。当所述计算机设备为上述实施例中的采集端设备11时，所述至少一条指令、至少一段程序、代码集或指令集经配置以由一个或者一个以上处理器执行，以实现上述视频流发送方法。当所述计算机设备为上述实施例中的播放端设备13时，所述至少一条指令、至少一段程序、代码集或指令集经配置以由一个或者一个以上处理器执行，以实现上述视频流播放方法。
在示例性实施例中，还提供了一种计算机可读存储介质，所述存储介质中存储有至少一条指令、至少一段程序、代码集或指令集，所述至少一条指令、所述至少一段程序、所述代码集或所述指令集在被计算机设备的处理器执行时实现上述视频流发送方法或者视频流播放方法。
可选地，上述计算机可读存储介质可以是ROM、随机存取存储器(RAM)、CD-ROM、磁带、软盘和光数据存储设备等。
在示例性实施例中，还提供了一种计算机程序产品，当该计算机程序产品被执行时，其用于实现上述视频流发送方法或者视频流播放方法。
应当理解的是，在本文中提及的“多个”是指两个或两个以上。“和/或”，描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。字符“/”一般表示前后关联对象是一种“或”的关系。
上述本申请实施例序号仅仅为了描述，不代表实施例的优劣。
以上所述仅为本申请的示例性实施例，并不用以限制本申请，凡在本申请的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。

Claims (14)

  1. 一种视频流发送方法，所述方法包括：
    采集视频直播画面；
    获取每帧视频直播画面对应的关联数据；
    将所述关联数据形成的数据帧插入到所述视频直播画面生成的视频流中发送。
  2. 根据权利要求1所述的方法，其中，所述关联数据是指显示所述视频直播画面时需要同步显示的与所述视频直播画面相关的数据。
  3. 根据权利要求1或2所述的方法，其中，所述将所述关联数据形成的数据帧插入到所述视频直播画面生成的视频流中发送，包括：
    对所述视频直播画面进行编码生成第一视频流，所述第一视频流包括第一数据帧，每个第一数据帧中携带对一帧视频直播画面进行编码后得到的数据；
    在所述第一视频流中插入第二数据帧，生成第二视频流，其中，所述第二数据帧中携带供播放端设备获取所述关联数据的信息；
    向服务器发送所述第二视频流。
  4. 根据权利要求3所述的方法，其中，所述在所述第一视频流中插入第二数据帧，生成第二视频流，包括：
    在所述第一视频流的每个第一数据帧之后插入一个第二数据帧；
    其中，在第i个第一数据帧之后插入的第二数据帧为第i个第二数据帧，所述第i个第一数据帧中携带对第i帧视频直播画面进行编码后得到的数据，所述第i个第二数据帧中携带供所述播放端设备获取所述第i帧视频直播画面对应的关联数据的信息，所述i为正整数。
  5. 根据权利要求4所述的方法，其中，所述第i个第二数据帧中携带的信息包括：所述第i帧视频直播画面对应的标识。
  6. 根据权利要求5所述的方法，其中，所述第i个第二数据帧中携带的信息还包括：所述第i帧视频直播画面对应的关键关联数据；
    其中，所述第i帧视频直播画面对应的关联数据包括所述关键关联数据和非关键关联数据，所述关键关联数据对展示的实时性要求高于所述非关键关联数据对展示的实时性要求。
  7. 根据权利要求5所述的方法，其中，所述第i帧视频直播画面对应的标识为所述第i帧视频直播画面对应的时间戳；其中，所述第i帧视频直播画面对 应的时间戳是指在显示所述第i帧视频直播画面时视频直播的已开始时长。
  8. 一种视频流播放方法，所述方法包括：
    获取第二视频流，所述第二视频流中包含有间隔排列的第一数据帧和第二数据帧；
    解码所述第一数据帧得到视频直播画面；
    解码所述第二数据帧得到所述视频直播画面对应的关联数据；
    逐帧播放所述视频直播画面，并展示当前播放的视频直播画面对应的关联数据。
  9. 根据权利要求8所述的方法，其中，所述展示当前播放的视频直播画面对应的关联数据，包括：
    对所述当前播放的视频直播画面对应的关联数据进行渲染，得到图像元素；
    在直播场景中显示所述图像元素。
  10. 一种视频流发送装置，所述装置包括：
    画面采集模块，用于采集视频直播画面；
    数据采集模块，用于获取每帧视频直播画面对应的关联数据；
    视频流发送模块，用于将所述关联数据形成的数据帧插入到所述视频直播画面生成的视频流中发送。
  11. 一种视频流播放装置，所述装置包括：
    视频流获取模块，用于获取第二视频流，所述第二视频流中包含有间隔排列的第一数据帧和第二数据帧；
    解码模块，用于解码所述第一数据帧得到视频直播画面；
    数据获取模块，用于解码所述第二数据帧得到所述视频直播画面对应的关联数据；
    展示模块，用于逐帧播放所述视频直播画面，并展示当前播放的视频直播画面对应的关联数据。
  12. 一种视频播放系统，所述系统包括：采集端设备、服务器和播放端设备；
    所述采集端设备包括如权利要求10所述的视频流发送装置；
    所述播放端设备包括如权利要求11所述的视频流播放装置。
  13. 一种计算机设备，所述计算机设备包括处理器和存储器，所述存储器中存储有至少一条指令、至少一段程序、代码集或指令集，所述至少一条指令、 所述至少一段程序、所述代码集或所述指令集在被所述处理器执行时实现如权利要求1至7任一项所述的视频流发送方法，或者实现如权利要求8至9任一项所述的视频流播放方法。
  14. 一种计算机可读存储介质，所述计算机可读存储介质中存储有至少一条指令、至少一段程序、代码集或指令集，所述至少一条指令、所述至少一段程序、所述代码集或所述指令集在被执行时实现如权利要求1至7任一项所述的视频流发送方法，或者实现如权利要求8至9任一项所述的视频流播放方法。
PCT/CN2018/115745 2017-12-01 2018-11-15 视频流发送方法、播放方法、装置、设备及存储介质 WO2019105239A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201711250078.9 2017-12-01
CN201711250078.9A CN109874043B (zh) 2017-12-01 2017-12-01 视频流发送方法、播放方法及装置

Publications (1)

Publication Number Publication Date
WO2019105239A1 true WO2019105239A1 (zh) 2019-06-06

Family

ID=66663958

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/115745 WO2019105239A1 (zh) 2017-12-01 2018-11-15 视频流发送方法、播放方法、装置、设备及存储介质

Country Status (2)

Country Link
CN (1) CN109874043B (zh)
WO (1) WO2019105239A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111918099A (zh) * 2020-09-16 2020-11-10 Oppo广东移动通信有限公司 视频处理方法、装置、电子设备及存储介质
CN112702624A (zh) * 2020-12-22 2021-04-23 山东鲁能软件技术有限公司 一种短视频播放效率的优化方法、系统、介质及设备

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110850983B (zh) * 2019-11-13 2020-11-24 腾讯科技（深圳）有限公司 视频直播中的虚拟对象控制方法、装置和存储介质
CN111212320B (zh) * 2020-01-08 2023-07-14 腾讯科技（深圳）有限公司 一种资源合成方法、装置、设备及存储介质
CN111629223B (zh) * 2020-06-11 2022-09-13 网易（杭州）网络有限公司 视频同步方法及装置、计算机可读存储介质以及电子设备
CN112261465A (zh) * 2020-10-26 2021-01-22 南京维沃软件技术有限公司 视频处理方法及装置
CN114449200B (zh) * 2020-10-30 2023-06-06 华为技术有限公司 音视频通话方法、装置及终端设备
CN112714333A (zh) * 2020-12-29 2021-04-27 维沃移动通信有限公司 多媒体数据处理方法和电子设备

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1487214A1 (en) * 2003-06-11 2004-12-15 Digital Multimedia Technologies S.P.A. A method and a system for synchronizing MHP applications in a data packet stream
WO2005099385A2 (en) * 2004-04-07 2005-10-27 Nielsen Media Research, Inc. Data insertion apparatus and methods for use with compressed audio/video data
CN102223487A (zh) * 2011-07-21 2011-10-19 杭州海康威视数字技术股份有限公司 在视频码流中存储、播放附加信息的方法及其装置
US8843957B2 (en) * 2010-06-21 2014-09-23 Accenture Global Services Limited Frame accurate content insertion system
CN104918016A (zh) * 2015-06-09 2015-09-16 柳州桂通科技股份有限公司 一种多媒体多信息同步重现的系统
CN106101808A (zh) * 2016-07-13 2016-11-09 乐视控股（北京）有限公司 一种视频直播方法及装置
CN106686424A (zh) * 2017-01-06 2017-05-17 广州四三九九信息科技有限公司 基于移动端录屏直播方法
CN107360440A (zh) * 2017-06-16 2017-11-17 北京米可世界科技有限公司 基于直播流中引入游戏进程的深度交互系统及交互方法

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8559438B2 (en) * 2008-04-11 2013-10-15 Mobitv, Inc. Modification of live streams
US8079054B1 (en) * 2008-04-14 2011-12-13 Adobe Systems Incorporated Location for secondary content based on data differential
CN102802039B (zh) * 2012-08-14 2015-04-15 武汉微创光电股份有限公司 多路视频混合解码输出方法及装置
CN109743335A (zh) * 2014-08-01 2019-05-10 广州华多网络科技有限公司 交互系统、服务器、客户端和交互方法
EP2953344A1 (en) * 2014-12-22 2015-12-09 Axis AB Method for improving live video
US10554992B2 (en) * 2015-06-03 2020-02-04 Avago Technologies International Sales Pte. Limited Inline codec switching
CN105740029B (zh) * 2016-03-03 2019-07-05 腾讯科技（深圳）有限公司 一种内容呈现的方法、用户设备及系统
CN106777205A (zh) * 2016-12-23 2017-05-31 北京奇虎科技有限公司 游戏类搜索词的搜索方法及装置
CN107147949B (zh) * 2017-05-05 2020-05-05 中广热点云科技有限公司 一种直播时移的播放进度操控方法及系统
CN107277636B (zh) * 2017-06-15 2020-04-03 广州华多网络科技有限公司 一种直播过程中的互动方法、用户端、主播端及系统
CN107172477B (zh) * 2017-06-16 2019-12-31 广州市网星信息技术有限公司 投票方法及装置

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1487214A1 (en) * 2003-06-11 2004-12-15 Digital Multimedia Technologies S.P.A. A method and a system for synchronizing MHP applications in a data packet stream
WO2005099385A2 (en) * 2004-04-07 2005-10-27 Nielsen Media Research, Inc. Data insertion apparatus and methods for use with compressed audio/video data
US8843957B2 (en) * 2010-06-21 2014-09-23 Accenture Global Services Limited Frame accurate content insertion system
CN102223487A (zh) * 2011-07-21 2011-10-19 杭州海康威视数字技术股份有限公司 在视频码流中存储、播放附加信息的方法及其装置
CN104918016A (zh) * 2015-06-09 2015-09-16 柳州桂通科技股份有限公司 一种多媒体多信息同步重现的系统
CN106101808A (zh) * 2016-07-13 2016-11-09 乐视控股（北京）有限公司 一种视频直播方法及装置
CN106686424A (zh) * 2017-01-06 2017-05-17 广州四三九九信息科技有限公司 基于移动端录屏直播方法
CN107360440A (zh) * 2017-06-16 2017-11-17 北京米可世界科技有限公司 基于直播流中引入游戏进程的深度交互系统及交互方法

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111918099A (zh) * 2020-09-16 2020-11-10 Oppo广东移动通信有限公司 视频处理方法、装置、电子设备及存储介质
CN112702624A (zh) * 2020-12-22 2021-04-23 山东鲁能软件技术有限公司 一种短视频播放效率的优化方法、系统、介质及设备
CN112702624B (zh) * 2020-12-22 2023-04-07 山东鲁软数字科技有限公司 一种短视频播放效率的优化方法、系统、介质及设备

Also Published As

Publication number Publication date
CN109874043B (zh) 2021-07-27
CN109874043A (zh) 2019-06-11

Similar Documents

Publication Publication Date Title
WO2019105239A1 (zh) 视频流发送方法、播放方法、装置、设备及存储介质
CN109600678B (zh) 信息展示方法、装置及系统、服务器、终端、存储介质
CN108401124B (zh) 视频录制的方法和装置
WO2019114514A1 (zh) 在直播间显示音高信息的方法及装置、存储介质
WO2020186988A1 (zh) 资讯的展示方法、装置、终端及存储介质
CN110830811B (zh) 直播互动方法及装置、系统、终端、存储介质
WO2019128592A1 (zh) 进行直播的方法和装置
CN108419113B (zh) 字幕显示方法及装置
CN112118477B (zh) 虚拟礼物展示方法、装置、设备以及存储介质
CN108900859A (zh) 直播方法及系统
CN111093108B (zh) 音画同步判断方法、装置、终端及计算机可读存储介质
US11962897B2 (en) Camera movement control method and apparatus, device, and storage medium
CN109413453B (zh) 视频播放方法、装置、终端及存储介质
US20220191557A1 (en) Method for displaying interaction data and electronic device
WO2021043121A1 (zh) 一种图像换脸的方法、装置、系统、设备和存储介质
CN110418152B (zh) 进行直播提示的方法及装置
WO2021073293A1 (zh) 动画文件的生成方法、装置及存储介质
CN110958465A (zh) 视频流推送方法、装置及存储介质
CN108600778B (zh) 媒体流发送方法、装置、系统及服务器、终端、存储介质
CN112104648A (zh) 数据处理方法、装置、终端、服务器及存储介质
CN113411680A (zh) 多媒体资源播放方法、装置、终端及存储介质
CN110958464A (zh) 直播数据处理方法、装置、服务器、终端及存储介质
CN114116053A (zh) 资源展示方法、装置、计算机设备及介质
CN111083513B (zh) 直播画面处理方法、装置、终端及计算机可读存储介质
CN112511889B (zh) 视频播放方法、装置、终端及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18882509

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18882509

Country of ref document: EP

Kind code of ref document: A1