WO2017092433A1

WO2017092433A1 - Method and device for video real-time playback

Info

Publication number: WO2017092433A1
Application number: PCT/CN2016/096989
Authority: WO
Inventors: 郑吉剑
Original assignee: 乐视控股（北京）有限公司; 乐视致新电子科技（天津）有限公司
Priority date: 2015-12-03
Filing date: 2016-08-26
Publication date: 2017-06-08
Also published as: CN105916011A

Abstract

Provided are a video real-time playback method and a video real-time playback device. The video real-time playback method comprises: storing a first frame image of a video packet and a timestamp corresponding to the first frame image to a database; converting the first frame image into a thumbnail image, displaying the thumbnail image and the corresponding timestamp in a preview area; receiving an operation request, and play back a video on the basis of the operation request. Also provided is a video real-time playback device. Several thumbnail images can be displayed in the preview area to be previewed by a user, thus enhancing user experience.

Description

Video real-time playing method and device

The present application claims priority to Chinese Patent Application No. 2015108835314, the entire disclosure of which is incorporated herein in

Technical field

The present application relates to the field of audio and video technologies, and in particular, to a video real-time playing method and apparatus.

Background technique

The HLS (HTTP Live Streaming) protocol is an extended HTTP protocol developed by Apple Inc. for use in live webcast and on-demand scenarios. The protocol encapsulates data using a Transport Stream (TS). Each transport stream carries data information such as video (Video), audio (Audio), and the like, and carries flow index information such as a Program Association Table (PAT) and a Program Map Table (PMT).

When playing a program that follows the HLS protocol, sometimes the client needs to perform a preview operation, but the existing HLS protocol does not support the preview operation through the thumbnail image, so the user experience is poor.

Summary of the invention

The embodiment of the present invention provides a video real-time playing method and device, which are used to solve the technical problem that the preview operation cannot be performed by the thumbnail image in the prior art.

The embodiment of the present application provides a video real-time playing method, including:

And storing a first frame image of the video data packet and a time stamp corresponding to the first frame image into a database;

Converting the first frame image into a thumbnail image, and displaying the thumbnail image and a corresponding time stamp in a preview area;

Receiving an operation request and playing a video according to the operation request.

The embodiment of the present application further provides a video real-time playing device, including:

a preview information obtaining module, configured to store a first frame image of the video data packet and a time stamp corresponding to the first frame image into a database;

a preview information display module, configured to convert the first frame image into a thumbnail image, and display the thumbnail image and a corresponding time stamp in a preview area;

a playing module, configured to receive an operation request, and play a video according to the operation request.

Further, the device further includes:

a download module for downloading slices in the transport stream;

And a demultiplexing module, configured to demultiplex the slice in the transport stream to obtain a grouped video original stream and program specific information.

The embodiment of the present application provides a non-transitory computer readable storage medium, wherein the non-transitory computer readable storage medium can store a computer program, and the program can implement a part of the video real-time playing method described above when executed. Or all steps.

An embodiment of the present application further provides an electronic device, including: one or more processors; and a memory; wherein the memory stores instructions executable by the one or more processors, the instructions being The one or more processors are executed to enable the one or more processors to perform the video real time playback method of the present application.

An embodiment of the present application provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, The computer is caused to execute the above-mentioned video real-time playing method in the embodiment of the present application.

The video real-time playing method and device provided by the embodiment of the present application captures the first frame image of each slice in the program transmission stream and the corresponding time stamp into the database when playing the program, and converts the first frame image into a thumbnail. Slightly image, then display several thumbnail images in the preview area for the user to preview to enhance the user's experience.

DRAWINGS

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings to be used in the embodiments or the prior art description will be briefly described below. Obviously, the drawings in the following description are Some embodiments of the present application can also obtain other drawings based on these drawings without departing from the prior art by those skilled in the art.

1 is a schematic diagram of a format of a slice of a transport stream in the prior art;

2 is a schematic diagram showing the structure of a packet of a slice of a transport stream in the prior art;

3 is a flowchart of a video real-time playing method according to an embodiment of the present application;

4 is a schematic diagram of a video real-time playing device according to an embodiment of the present application;

FIG. 5 is a schematic structural diagram of hardware of a device for playing a video in real time according to an embodiment of the present disclosure.

detailed description

The technical solutions in the embodiments of the present application are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present application. It is a part of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present application without departing from the inventive scope are the scope of the present application.

According to the existing regulations of the HLS protocol, in order to implement video on demand, the server first divides the video data into different slices according to a certain time interval, different media encoding formats and code rate sizes, and then stores all the slices in the server. The information of each slice and its length of time is stored as an M3U8 format file; before the client plays, the server first transmits the M3U8 format file to the client, and the M3U8 file includes a program association table, a program mapping table, and a timestamp.

After the client receives the M3U8 format file, if the user wants to perform the on-demand operation, the user may issue an operation request to the client, and the client performs the SEEK operation to download the fragment file specified in the operation request from the server for parsing and playing.

The inventor of the present application discovered that the SEEK operation provided in the existing HLS protocol can only be clicked according to information such as time stamps, and cannot provide an image corresponding to the time stamp to the user for previewing, so the experience is experienced. Poor. To this end, the inventors have proposed the following technical solutions of the present application to improve the user's experience.

FIG. 1 is a schematic diagram of a format of a slice of a transport stream in the prior art.

The transport stream is a hierarchical encapsulation format, and the program association table needs to be first found to obtain the number of program map tables in the transport stream, thereby obtaining the number of programs in the transport stream, where each program map corresponds to one program, and then A video packet identifier (PID) and an audio packet identifier of the program are obtained according to the program map.

According to the HLS protocol, a long-term program is usually cut into a plurality of slices of 10 seconds. Therefore, in the prior art, the slice of the transport stream is used to facilitate the user to receive the transport stream at any time and ensure that the audio and video data in the slice is not pirated, and the transmission end periodically inserts the program association table and the program map into the transport stream. In the slice.

For example, in FIG. 1, a program is cut into n slices, user A can receive from the program association table 1, and user B can receive from the program association table n to find the program map in the vicinity, thereby facilitating live broadcast.

2 is a schematic diagram showing the structure of a packet of a slice of a transport stream in the prior art.

A slice consists of multiple packets, each consisting of a header and data, each of which is 188 bytes or 204 bytes, of which 204 bytes are added after the 188-byte packet. The CRC check data of the bytes is formed.

The header can also include an extended self-use area, the length of the header is 4 bytes, and the self-use area and the packet data share 184 bytes. The packet header consists of a sync byte, a transmission error indicator, a payload unit start indicator, a transmission priority, a packet identifier, a transmission scrambling control, an adaptive zone control, and a continuous counter.

Among them, the packet correlation in the data stream can be detected by the automatic correlation feature of the bit string of the synchronization byte, and the packet synchronization can be established. When the error code indicator is transmitted, the error correction decoder can be used. Indicates a 1-bit error, but cannot correct the error; the payload unit start indicator indicates whether the packet has a determined start message; the transmission priority is used to assign a priority to the packet; the packet identifier value is determined by the user Determining, the decoder distinguishes different original stream (ES) packets according to the packet identifier to reconstruct the original stream; the transmission scrambling control can be used to indicate whether the packet content is scrambled; the adaptive area control is used to indicate that there is No adaptive zone, usually represented by 2 bits, for example, "01" means that there is useful information, but there is no adaptive zone, "10" means no useful information, and there is an adaptive zone, "11" means that there is useful information and there is In the adaptive zone, “00” means no definition; the continuous counter can count the transmission order of the packet identifier. According to the reading of the continuous counter, the receiving end can judge whether there is packet loss and packet transmission. Error. In summary, the packet header has synchronization, identification, error detection and encryption functions for the transport stream.

In addition, the packet identifier is a unique identifier in the transport stream, and the data content in the packet is determined by the packet identifier. If the packet identifier in the header of a packet in a transport stream is 0, then the data content of the packet is the program association table. If it is not 0, it is video data, audio data, program map, or other type of data.

FIG. 3 is a flowchart of a video real-time playing method according to an embodiment of the present application.

S301: Download a slice in the transport stream.

In this embodiment, the slice is downloaded simultaneously by two threads. In other embodiments, the slice may be downloaded by one or more threads. The number of threads may be based on the processing capability of the client and the server, and the network. The bandwidth is set, which is not limited in this application.

Specifically, in this embodiment, the first thread starts downloading from the first slice of the transport stream, and simultaneously passes the first Two threads start downloading from the last slice of the transport stream to speed up the download.

S302: Demultiplexing the slice in the transport stream to obtain a grouped video original stream and program specific information.

The slice is formed by multiplexing the program specific information, the packetized audio original stream, and the grouped video original stream, and having a length of 188 bytes. The program specific information typically includes at least a program association table, a program map table, and a time stamp.

S303: Decode the video original stream of the packet to obtain a video data packet.

The video original stream includes a plurality of access units, each of which is encoded data of an image. The video coding method may adopt transform coding, entropy coding, motion estimation and motion compensation, or hybrid coding, and the video data packet includes images of several frames.

S304: Store the first frame image of the video data packet and the time stamp corresponding to the first frame image into the database. Specifically, the first frame image of the slice and the corresponding time stamp are stored in a database named vod.databse.db. Of course, the data may also be named as another name, which is not limited in this application.

S305: Convert the first frame image into a thumbnail image, and display the thumbnail image and the corresponding time stamp in the preview area.

In this embodiment, the preview area is located below the play area, and may also be disposed above the play area, even on the left or right side, and may be set as needed during actual implementation. The thumbnail image may be displayed by reducing the first frame image, and the entire preview area is arranged by a plurality of thumbnail images. If the preview area on the screen cannot display all the thumbnail images, you can also display them by clicking the arrow at one end of the preview area or other buttons.

When the user selects a certain thumbnail image, the time stamp corresponding to the thumbnail image will be popped up at the same time, so that the user can directly observe, the playing time and the picture, which improves the user experience.

S306: Receive an operation request, and play a video according to the operation request.

When the user selects an image on the preview area through the User Interface (UI), the SEEK operation is triggered. The flow of the entire SEEK operation is as follows:

First, the timestamp corresponding to the preview image is read. If the timestamp has no value, the seek flag is directly cleared and the data packet is returned. If the timestamp has a value, the value of the timestamp is converted to microseconds and compared with the incoming time of the seek. If the value of the timestamp is greater than the seek time, it indicates that the read packet reaches the seek requirement, otherwise it continues to read. Take the packet until the value of the timestamp is equal to the seek time. When the value of the timestamp is equal to the seek time, it is checked whether the flag of the data packet is a key frame, and if so, the data packet is returned, otherwise the data packet is continued.

FIG. 4 is a schematic diagram of a video real-time playing device according to an embodiment of the present application.

The video real-time playing device of the embodiment of the present application includes a downloading module 100, a demultiplexing module 200, a decoding module 300, a preview information acquiring module 400, a preview information display module 500, and a playing module 600.

The download module 100 is configured to download a slice in the transport stream. The download module 100 specifically includes a first download submodule 101 and a second download submodule 102. The first download sub-module 101 is configured to start downloading from the first slice of the transport stream by using the first thread, and the second download sub-module 102, for downloading from the last slice of the transport stream by the second thread.

Specifically, in this embodiment, the first thread starts downloading from the first slice of the transport stream, and the second thread starts downloading from the last slice of the transport stream to speed up the download speed.

The demultiplexing module 200 is configured to demultiplex the slice in the transport stream to obtain a grouped video original stream and program specific information.

The slice is formed by multiplexing the program specific information, the packetized audio original stream, and the grouped video original stream. It is a 188-byte slice. The program specific information typically includes at least a program association table, a program map table, and a time stamp.

The decoding module 300 is configured to decode the video original stream of the packet to obtain a video data packet.

The preview information obtaining module 400 is configured to store the first frame image of the video data packet and the time stamp corresponding to the first frame image into the database.

Specifically, the first frame image of the slice and the corresponding time stamp are stored in a database named vod.databse.db. Of course, the data may also be named as another name, which is not limited in this application.

The preview information display module 500 is configured to convert the first frame image into a thumbnail image, and display the thumbnail image and the corresponding time stamp in the preview area.

The playing module 600 is configured to receive an operation request, and play the video according to the operation request.

First read the timestamp corresponding to the preview image. If the timestamp has no value, the seek flag is directly cleared and returned. Back to the packet. If the timestamp has a value, the value of the timestamp is converted to microseconds and compared with the incoming time of the seek. If the value of the timestamp is greater than the seek time, it indicates that the read packet reaches the seek requirement, otherwise it continues to read. Take the packet until the value of the timestamp is equal to the seek time. When the value of the timestamp is equal to the seek time, it is checked whether the flag of the data packet is a key frame, and if so, the data packet is returned, otherwise the data packet is continued.

The above technical solution of the embodiment of the present application is to store a first frame image of each slice in the program transmission stream and a corresponding time stamp into the database when the program is being played, and convert the first frame image into a thumbnail image, and then convert the first frame image into a thumbnail image, and then Several thumbnail images are displayed in the preview area for the user to preview to enhance the user's experience.

The embodiment of the present application further provides a non-transitory computer readable storage medium, where the non-transitory computer readable storage medium can store a program, and when executed, the program can implement a video real-time provided by any one of the foregoing embodiments. Some or all of the various implementations of the playback method.

FIG. 5 is a schematic structural diagram of hardware of an electronic device for playing a video in real time according to an embodiment of the present disclosure. As shown in FIG. 5, the device includes:

One or more processors 510 and memory 520, one processor 510 is taken as an example in FIG.

The device that performs the video real-time playing method may further include: an input device 530 and an output device 540.

The processor 510, the memory 520, the input device 530, and the output device 540 may be connected by a bus or other means, as exemplified by a bus connection in FIG.

The memory 520 is used as a non-transitory computer readable storage medium, and can be used for storing a non-volatile software program, a non-volatile computer executable program, and a module, such as a program instruction corresponding to the video real-time playing method in the embodiment of the present application. / module. The processor 510 executes various functional applications and data processing of the electronic device by executing non-volatile software programs, instructions, and modules stored in the memory 520, that is, the video real-time playing method of the foregoing method embodiments.

The memory 520 may include a storage program area and an storage data area, wherein the storage program area may store an operating system, an application required for at least one function; the storage data area may store data created according to use of the video real-time playback device, and the like. In addition, the memory 520 may include a high speed random access memory, and may also include a non- Volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 520 can optionally include memory remotely located relative to processor 510, which can be connected to the video real-time playback device over a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

Input device 530 can receive input numeric or character information and generate key signal inputs related to user settings and function control of the video real-time playback device. The output device 540 can include a display device such as a display screen.

The one or more modules are stored in the memory 520, and when executed by the one or more processors 510, perform a video real-time playback method in any of the above method embodiments.

The above products can perform the methods provided by the embodiments of the present application, and have the corresponding functional modules and beneficial effects of the execution method. For technical details that are not described in detail in this embodiment, reference may be made to the method provided by the embodiments of the present application.

The electronic device of the embodiment of the invention exists in various forms, including but not limited to:

(1) Mobile communication devices: These devices are characterized by mobile communication functions and are mainly aimed at providing voice and data communication. Such terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.

(2) Ultra-mobile personal computer equipment: This type of equipment belongs to the category of personal computers, has computing and processing functions, and generally has mobile Internet access. Such terminals include: PDAs, MIDs, and UMPC devices, such as the iPad.

(3) Portable entertainment devices: These devices can display and play multimedia content. Such devices include: audio, preview players (such as iPod), handheld game consoles, e-books, and smart toys and portable car navigation devices.

(4) Server: A device that provides computing services. The server consists of a processor, a hard disk, a memory, a system bus, etc. The server is similar to a general-purpose computer architecture, but because of the need to provide highly reliable services, processing power and stability High reliability in terms of reliability, security, scalability, and manageability.

(5) Other electronic devices with data interaction functions.

The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical. Units can be located in one place or distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the embodiment. Those of ordinary skill in the art can understand and implement without deliberate labor.

Through the description of the above embodiments, those skilled in the art can clearly understand that the various embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on such understanding, the above-described technical solutions may be embodied in the form of software products in essence or in the form of software products, which may be stored in a computer readable storage medium such as ROM/RAM, magnetic Discs, optical discs, etc., include instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to perform the methods described in various embodiments or portions of the embodiments.

Finally, it should be noted that the above embodiments are only used to explain the technical solutions of the present application, and are not limited thereto; although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still The technical solutions described in the foregoing embodiments are modified, or the equivalents of the technical features are replaced by the equivalents. The modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

A video real-time playing method, comprising:

And storing a first frame image of the video data packet and a time stamp corresponding to the first frame image into a database;

Converting the first frame image into a thumbnail image, and displaying the thumbnail image and a corresponding time stamp in a preview area;

Receiving an operation request and playing a video according to the operation request.
The method according to claim 1, wherein the step of storing the first frame image of the video data packet and the time stamp corresponding to the first frame image into the database comprises:

Download the slice in the transport stream;

Demultiplexing the slices in the transport stream to obtain a packetized video original stream and program specific information.
The method according to claim 2, wherein the step of storing the first frame image of the video data packet and the time stamp corresponding to the first frame image into the database further comprises:

Decoding the video original stream of the packet to obtain the video data packet.
The method according to claim 2 or 3, wherein the program specific information comprises a program association table, a program map table, and a time stamp.
The method according to claim 2, wherein the step of downloading the slice in the transport stream specifically comprises:

The download is started from the first slice of the transport stream by the first thread while the download is started from the last slice of the transport stream by the second thread.
A video real-time playing device, comprising:

a preview information obtaining module, configured to: image a first frame of the video data packet, and the first frame image Store the corresponding timestamp into the database;

a preview information display module, configured to convert the first frame image into a thumbnail image, and display the thumbnail image and a corresponding time stamp in a preview area;

a playing module, configured to receive an operation request, and play a video according to the operation request.
The device according to claim 6, further comprising:

a download module for downloading slices in the transport stream;

And a demultiplexing module, configured to demultiplex the slice in the transport stream to obtain a grouped video original stream and program specific information.
The device according to claim 7, further comprising:

And a decoding module, configured to decode the video original stream of the packet, to obtain the video data packet.
The apparatus according to claim 7 or 8, wherein said program specific information comprises a program association table, a program map table, and a time stamp.
The device according to claim 7, wherein the downloading module specifically comprises:

a first download submodule for downloading from the first slice of the transport stream by the first thread;

a second download submodule for downloading from the last slice of the transport stream by the second thread.
A non-transitory computer readable storage medium, characterized in that the non-transitory computer readable storage medium stores a computer program for causing the computer to perform the method of any of claims 1-5 .
An electronic device comprising:

One or more processors; and,

Memory; characterized in that

The memory stores instructions executable by the one or more processors, the instructions being Executing by one or more processors to cause the one or more processors to:

And storing a first frame image of the video data packet and a time stamp corresponding to the first frame image into a database;

Converting the first frame image into a thumbnail image, and displaying the thumbnail image and a corresponding time stamp in a preview area;

Receiving an operation request and playing a video according to the operation request.
A computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to execute The method of claims 1-5.