WO2021109354A1

WO2021109354A1 - Media stream data playback method and device

Info

Publication number: WO2021109354A1
Application number: PCT/CN2020/079175
Authority: WO
Inventors: 王良; 武兵; 李双增
Original assignee: 青岛海信传媒网络技术有限公司
Priority date: 2019-12-03
Filing date: 2020-03-13
Publication date: 2021-06-10
Also published as: CN111107428A

Abstract

The present application provides a media stream data playback method, comprising: acquiring first media stream data comprising first video data and first audio data, as well as second media stream data comprising at least second video data; outputting the first video data to a first window on a display and outputting the first audio data to an audio output interface; rendering the second video data into continuous video frames and outputting same to a second window on the display according to a particular frame rate.

Description

Media stream data playback method and equipment

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office, the application number is 201911222288.6, and the invention title is "A method and display device for playing dual-channel media stream data" on December 3, 2019, and its entire content Incorporated in this application by reference.

Technical field

This application relates to the field of display technology, and in particular to a two-channel media stream data playback method and display device.

Background technique

With the popularization of network technology, display devices can be used not only to play TV signals, but also to play network signals. As user needs increase, display devices need to support the playback of dual-channel media stream data. How to achieve this has become a problem that needs to be solved.

Summary of the invention

In view of this, the present application provides a dual-channel media stream data playback method and display device, which are used to implement dual-channel media stream data playback and improve user experience.

Specifically, this application is implemented through the following technical solutions:

In the first aspect, this application provides a display device, including:

monitor;

Audio output interface;

Browser function module, controlled by the controller, used to obtain two channels of media stream data;

A demultiplexing module, controlled by the controller, for demultiplexing the media stream data acquired by the browser function module into video data and audio data;

The decoding module is used to decode the demultiplexed media stream data to obtain the first channel of media stream data including the first channel of video data and the first channel of audio data, and the second channel including at least the second channel of video data Media stream data;

The playback module, controlled by the controller, is used to output the first channel of video data in the decoded first channel of media stream data to the first window on the display, and output the first channel of audio data to all The audio output interface;

The browser function module, controlled by the controller, is also used to render the second channel of video data in the decoded second channel of media stream data into continuous video frames, and output to the display according to a certain frame rate The second window.

In the second aspect, the present application also provides a method for playing dual-channel media stream data. The method is applied to a display device, and the method includes:

The controller controls the browser function module to obtain two channels of media stream data;

Controlling the demultiplexing module to demultiplex the two media stream data acquired by the browser function module into video data and audio data;

The decoding module decodes the demultiplexed media stream data to obtain the first channel of media stream data including the first channel of video data and the first channel of audio data, and the second channel of media stream including at least the second channel of video data data;

The controller controls the playback module to output the first channel of video data in the decoded first channel of media stream data to the first window on the display, and output the first channel of audio data to the audio output interface;

The controller controls the browser function module to render the second channel of video data in the decoded second channel of media stream data into continuous video frames, and output to the second window on the display according to a certain frame rate.

Description of the drawings

FIG. 1A exemplarily shows a schematic diagram of an operation scene between a display device and a control device;

FIG. 1B exemplarily shows a configuration block diagram of the control device 100 in FIG. 1A;

FIG. 1C exemplarily shows a configuration block diagram of the display device 200 in FIG. 1A;

FIG. 1D exemplarily shows a block diagram of the architecture configuration of the operating system in the memory of the display device 200;

Fig. 1E exemplarily shows a schematic diagram of dual-channel media stream data playback in the display;

Figures 2-1 and 2-2 exemplarily show a flow chart of the second video data compression;

Fig. 3 exemplarily shows a flow chart of rendering;

Fig. 4 exemplarily shows an interactive flowchart of a two-channel video playback method;

Fig. 5 exemplarily shows a flowchart of a two-channel video playback method.

Detailed ways

The exemplary embodiments will be described in detail here, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the application as detailed in the appended claims.

The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit the application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and/or" as used herein refers to and includes any or all possible combinations of one or more associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, the information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determination".

FIG. 1A exemplarily shows a schematic diagram of an operation scene between the display device and the control device. As shown in FIG. 1A, the control device 100 and the display device 200 can communicate in a wired or wireless manner.

Among them, the control device 100 is configured to control the display device 200, which can receive operation instructions input by the user, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, and act as an intermediary for the interaction between the user and the display device 200 effect. For example, the user operates the channel addition and subtraction keys on the control device 100, and the display device 200 responds to the channel addition and subtraction operations.

The control device 100 may be a remote controller 100A, including infrared protocol communication or Bluetooth protocol communication, and other short-distance communication methods, etc., to control the display device 200 in a wireless or other wired manner. The user can control the display device 200 by inputting user instructions through keys on the remote control, voice input, control panel input, etc. For example, the user can control the display device 200 by inputting corresponding control commands through the volume plus and minus keys, channel control keys, up/down/left/right movement keys, voice input keys, menu keys, and power on/off keys on the remote control. Function.

The control device 100 may also be a smart device, such as a mobile terminal 100B, a tablet computer, a computer, a notebook computer, and the like. For example, an application program running on a smart device is used to control the display device 200. The application can be configured to provide users with various controls through an intuitive user interface (UI) on the screen associated with the smart device.

Exemplarily, the mobile terminal 100B may install a software application with the display device 200, realize connection communication through a network communication protocol, and realize the purpose of one-to-one control operation and data communication. For example, the mobile terminal 100B can establish a control instruction protocol with the display device 200, and realize the functions of the physical keys arranged in the remote control 100A by operating various function keys or virtual buttons of the user interface provided on the mobile terminal 100B. The audio and video content displayed on the mobile terminal 100B can also be transmitted to the display device 200 to realize the synchronous display function.

The display device 200 may provide a broadcast receiving function and a network TV function of a computer support function. The display device can be implemented as digital TV, Internet TV, Internet Protocol TV (IPTV), and so on.

The display device 200 may be a liquid crystal display, an organic light emitting display, or a projection device. The specific display device type, size and resolution are not limited.

The display device 200 also performs data communication with the server 300 through a variety of communication methods. Here, the display device 200 may be allowed to communicate through a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 300 may provide various contents and interactions to the display device 200. For example, the display device 200 can send and receive information, such as receiving electronic program guide (EPG) data, receiving software program updates, or accessing a remotely stored digital media library. The server 300 can be one group or multiple groups, and can be one type or multiple types of servers. The server 300 provides other network service content such as video-on-demand and advertising services.

FIG. 1B exemplarily shows a configuration block diagram of the control device 100. As shown in FIG. 1B, the control device 100 includes a controller 110, a memory 120, a communicator 130, a user input interface 140, an output interface 150, and a power supply 160.

The controller 110 includes a random access memory (RAM) 111, a read only memory (ROM) 112, a processor 113, a communication interface, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication and cooperation between internal components, and external and internal data processing functions.

Exemplarily, when an interaction of a user pressing a button arranged on the remote control 100A or an interaction of touching a touch panel arranged on the remote control 100A is detected, the controller 110 may control to generate a signal corresponding to the detected interaction, and This signal is sent to the display device 200.

The memory 120 is used to store various operating programs, data, and applications for driving and controlling the control device 100 under the control of the controller 110. The memory 120 can store various control signal instructions input by the user.

The communicator 130 realizes the communication of control signals and data signals with the display device 200 under the control of the controller 110. For example, the control device 100 sends a control signal (such as a touch signal or a button signal) to the display device 200 via the communicator 130, and the control device 100 can receive the signal sent by the display device 200 via the communicator 130. The communicator 130 may include an infrared signal interface 131 and a radio frequency signal interface 132. For example, in the case of an infrared signal interface, the user input instruction needs to be converted into an infrared control signal according to the infrared control protocol, and sent to the display device 200 via the infrared sending module. For another example, in the case of a radio frequency signal interface, a user input instruction needs to be converted into a digital signal, which is then modulated according to the radio frequency control signal modulation protocol, and then sent to the display device 200 by the radio frequency sending terminal.

The user input interface 140 may include at least one of a microphone 141, a touch panel 142, a sensor 143, a button 144, etc., so that the user can input user instructions for controlling the display device 200 to the control device through voice, touch, gesture, pressing, etc. 100.

The output interface 150 outputs a user instruction received by the user input interface 140 to the display device 200, or outputs an image or voice signal received by the display device 200. Here, the output interface 150 may include an LED interface 151, a vibration interface 152 that generates vibration, a sound output interface 153 that outputs a sound, a display 154 that outputs an image, and the like. For example, the remote controller 100A can receive output signals such as audio, video, or data from the output interface 150, and display the output signals as images on the display 154, as audio on the sound output interface 153, or as vibration on the vibration interface 152. form.

The power supply 160 is used to provide operating power support for each element of the control device 100 under the control of the controller 110. The form can be battery and related control circuit.

FIG. 1C exemplarily shows a block diagram of the hardware configuration of the display device 200. As shown in FIG. 1C, the display device 200 may include a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a memory 260, a user interface 265, a video processor 270, a display 275, Audio processor 280, audio output interface 285, and power supply 290.

The tuner and demodulator 210, which receives broadcast television signals through wired or wireless means, can perform modulation and demodulation processing such as amplification, mixing and resonance, and is used to demodulate the television selected by the user from multiple wireless or cable broadcast television signals The audio and video signals carried in the frequency of the channel, as well as additional information (such as EPG data).

The tuner and demodulator 210 can be selected by the user and controlled by the controller 250 to respond to the frequency of the television channel selected by the user and the television signal carried by the frequency.

The tuner and demodulator 210 can receive signals in many ways according to different broadcasting formats of TV signals, such as terrestrial broadcasting, cable broadcasting, satellite broadcasting or Internet broadcasting; and according to different modulation types, it can be digital modulation or analog Modulation method; and according to different types of received TV signals, analog signals and digital signals can be demodulated.

In some other exemplary embodiments, the tuner demodulator 210 may also be in an external device, such as an external set-top box. In this way, the set-top box outputs a TV signal after modulation and demodulation, and is input to the display device 200 through the external device interface 240.

The communicator 220 is a component used to communicate with external devices or external servers according to various types of communication protocols. For example, the display device 200 may transmit content data to an external device connected via the communicator 220, or browse and download content data from an external device connected via the communicator 220. The communicator 220 may include a network communication protocol module such as a WIFI module 221, a Bluetooth communication protocol module 222, and a wired Ethernet communication protocol module 223 or a near field communication protocol module, so that the communicator 220 can receive the control device 100 according to the control of the controller 250 Control signals, and implement the control signals as WIFI signals, Bluetooth signals, radio frequency signals, etc.

The detector 230 is a component of the display device 200 for collecting signals from the external environment or interacting with the outside. The detector 230 may include a sound collector 231, such as a microphone, which may be used to receive a user's voice, such as a voice signal of a control instruction for the user to control the display device 200; or, it may collect environmental sounds used to identify the type of environmental scene to realize display The device 200 can adapt to environmental noise.

In some other exemplary embodiments, the detector 230 may also include an image collector 232, such as a camera, a camera, etc., which may be used to collect external environment scenes to adaptively change the display parameters of the display device 200; and to collect The attributes of the user or interactive gestures with the user to achieve the function of interaction between the display device and the user.

In some other exemplary embodiments, the detector 230 may further include a light receiver, which is used to collect the ambient light intensity to adapt to changes in display parameters of the display device 200 and so on.

In some other exemplary embodiments, the detector 230 may also include a temperature sensor. For example, by sensing the ambient temperature, the display device 200 may adaptively adjust the display color temperature of the image. Exemplarily, when the temperature is relatively high, the display device 200 can be adjusted to display an image with a cooler color temperature; when the temperature is relatively low, the display device 200 can be adjusted to display an image with a warmer color temperature.

The external device interface 240 is a component that provides the controller 250 to control data transmission between the display device 200 and external devices. The external device interface 240 can be connected to external devices such as set-top boxes, game devices, notebook computers, etc. in a wired/wireless manner, and can receive external devices such as video signals (such as moving images), audio signals (such as music), and additional information (such as EPG). ) And other data.

Among them, the external device interface 240 may include: a high-definition multimedia interface (HDMI) terminal 241, a composite video blanking synchronization (CVBS) terminal 242, an analog or digital component terminal 243, a universal serial bus (USB) terminal 244, and a component (Component) Any one or more of terminals (not shown in the figure), red, green and blue (RGB) terminals (not shown in the figure), etc.

The controller 250 controls the work of the display device 200 and responds to user operations by running various software control programs (such as an operating system and various application programs) stored on the memory 260.

As shown in FIG. 1C, the controller 250 includes a random access memory (RAM) 251, a read only memory (ROM) 252, a graphics processor 253, a CPU processor 254, a communication interface 255, and a communication bus 256. Among them, the RAM 251, the ROM 252, the graphics processor 253, and the CPU processor 254 communication interface 255 are connected via a communication bus 256.

ROM252, used to store various system startup instructions. For example, when the power-on signal is received, the power of the display device 200 starts to start, and the CPU processor 254 runs the system start-up instruction in the ROM 252, and copies the operating system stored in the memory 260 to the RAM 251 to start running the start-up operating system. After the operating system is started up, the CPU processor 254 copies various application programs in the memory 260 to the RAM 251, and then starts to run and start various application programs.

The graphics processor 253 is used to generate various graphics objects, such as icons, operation menus, and user input instructions to display graphics. The graphics processor 253 may include an arithmetic unit, which is used to perform operations by receiving various interactive instructions input by the user, and then display various objects according to the display attributes; and includes a renderer, which is used to generate various objects obtained based on the arithmetic unit, and perform operations. The rendered result is displayed on the display 275.

The CPU processor 254 is configured to execute operating system and application program instructions stored in the memory 260. And according to the received user input instructions, to execute various applications, data and content processing, so as to finally display and play various audio and video content.

In some exemplary embodiments, the CPU processor 254 may include multiple processors. The multiple processors may include a main processor and multiple or one sub-processors. The main processor is configured to perform some initialization operations of the display device 200 in the display device preloading mode, and/or, to display screen operations in the normal mode. Multiple or one sub-processor, used to perform an operation in the standby mode of the display device.

The communication interface 255 may include the first interface to the nth interface. These interfaces may be network interfaces connected to external devices via a network.

The controller 250 may control the overall operation of the display device 200. For example, in response to receiving a user input command for selecting a GUI object displayed on the display 275, the controller 250 may perform an operation related to the object selected by the user input command.

Among them, the object can be any one of the selectable objects, such as a hyperlink or an icon. The operation related to the selected object, for example, the operation of displaying the page, document, image, etc. connected to the hyperlink, or the operation of executing the program corresponding to the object. The user input command for selecting the GUI object may be a command input through various input devices (for example, a mouse, a keyboard, a touch pad, etc.) connected to the display device 200 or a voice command corresponding to a voice spoken by the user.

The memory 260 is used to store various types of data, software programs or application programs for driving and controlling the operation of the display device 200. The memory 260 may include volatile and/or non-volatile memory. The term “memory” includes the memory 260, the RAM 251 and ROM 252 of the controller 250, or the memory card in the display device 200.

In some embodiments, the memory 260 is specifically used to store the operating program that drives the controller 250 in the display device 200; to store various application programs built in the display device 200 and downloaded from an external device by the user; and to store the configuration provided by the display 275 Data such as various GUIs, various objects related to the GUI, and visual effect images of the selector used to select GUI objects.

In some embodiments, the memory 260 is specifically used to store drivers and related data of the tuner and demodulator 210, the communicator 220, the detector 230, the external device interface 240, the video processor 270, the display 275, the audio processor 280, etc. For example, external data (such as audio and video data) received from an external device interface or user data (such as key information, voice information, touch information, etc.) received from a user interface.

In some embodiments, the memory 260 specifically stores software and/or programs for representing an operating system (OS). These software and/or programs may include, for example, a kernel, middleware, application programming interface (API), and/or application. Exemplarily, the kernel can control or manage system resources and functions implemented by other programs (such as the middleware, API, or application program); at the same time, the kernel can provide interfaces to allow middleware, API, or application program access control To control or manage system resources.

FIG. 1D exemplarily shows a block diagram of the architecture configuration of the operating system in the memory of the display device 200. The operating system architecture consists of the application layer, the middleware layer, and the kernel layer from top to bottom.

Application layer, system built-in applications and non-system-level applications belong to the application layer. Responsible for direct interaction with users. The application layer can include multiple applications, such as settings applications, e-post applications, media center applications, and so on. These applications can be implemented as web applications, which are executed based on the WebKit engine, and can be developed and executed based on HTML5, Cascading Style Sheets (CSS) and JavaScript, usually the browser function module in the display device to control the operation or execution .

Here, HTML, the full name of HyperText Markup Language (HyperText Markup Language), is a standard markup language used to create web pages. Web pages are described through markup tags. HTML tags are used to describe text, graphics, animations, sounds, tables, For links, the browser will read the HTML document, explain the content of the tags in the document, and display it in the form of a web page.

CSS, the full name of Cascading Style Sheets (Cascading Style Sheets), is a computer language used to express the style of HTML files, and can be used to define style structures, such as fonts, colors, and positions. CSS styles can be directly stored in HTML web pages or in separate style files to achieve control over styles in web pages.

JavaScript is a language used in web page programming, which can be inserted into HTML pages and interpreted and executed by the browser. The interaction logic of the web application is implemented through JavaScript. JavaScript can encapsulate the JavaScript extension interface through the browser to realize the communication with the kernel layer,

The middleware layer can provide some standardized interfaces to support the operation of various environments and systems. For example, the middleware layer can be implemented as the Multimedia and Hypermedia Information Coding Expert Group (MHEG) of the middleware related to data broadcasting, and can also be implemented as the DLNA middleware of the middleware related to external device communication, and can also be implemented as providing Display the middleware of the browser environment in which each application in the device runs.

The kernel layer provides core system services, such as file management, memory management, process management, network management, system security authority management and other services. The kernel layer can be implemented as a kernel based on various operating systems, for example, a kernel based on the Linux operating system.

The kernel layer also provides communication between system software and hardware, and provides device driver services for various hardware, such as: providing display drivers for displays, camera drivers for cameras, button drivers for remote controls, and WIFI modules Provide WiFi driver, audio driver for audio output interface, power management driver for power management (PM) module, etc.

The user interface 265 receives various user interactions. Specifically, it is used to send the input signal of the user to the controller 250, or to transmit the output signal from the controller 250 to the user. Exemplarily, the remote control 100A may send input signals input by the user, such as a power switch signal, a channel selection signal, and a volume adjustment signal, to the user interface 265, and then the user interface 265 transfers to the controller 250; or the remote control 100A may Receive output signals such as audio, video, or data output from the user interface 265 after the controller 250 processes, and display the received output signal or output the received output signal as audio or vibration.

In some embodiments, the user may input a user command on a graphical user interface (GUI) displayed on the display 275, and the user interface 265 receives the user input command through the GUI. Specifically, the user interface 265 may receive user input commands for controlling the position of the selector in the GUI to select different objects or items.

Alternatively, the user may input a user command by inputting a specific sound or gesture, and the user interface 265 recognizes the sound or gesture through the sensor to receive the user input command. The video processor 270 is used to receive external video signals, and perform video data processing such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, and image synthesis according to the standard codec protocol of the input signal. The video signal displayed or played directly on the display 275.

For example, the video processor 270 includes a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, a display formatting module, and the like.

Among them, the demultiplexing module is used to demultiplex the input audio and video data stream, such as the input MPEG-2 stream (based on the compression standard of digital storage media moving images and voice), then the demultiplexing module will demultiplex it Multiplexed into video signals and audio signals, etc.

The video decoding module is used to process the demultiplexed video signal, including decoding and scaling.

An image synthesis module, such as an image synthesizer, is used to superimpose and mix the GUI signal generated by the graphics generator with the zoomed video image according to the user input or itself, so as to generate an image signal for display.

The frame rate conversion module is used to convert the frame rate of the input video, such as converting the frame rate of the input 60Hz video to a frame rate of 120Hz or 240Hz, and the usual format is realized by such as frame interpolation.

The display formatting module is used to change the signal output by the frame rate conversion module to a signal conforming to the display format such as a display, for example, format the signal output by the frame rate conversion module to output RGB data signals.

The display 275 is used to receive the image signal input from the video processor 270 to display video content, images, and a menu control interface. The displayed video content can be from the video content in the broadcast signal received by the tuner and demodulator 210, or from the video content input by the communicator 220 or the external device interface 240. The display 275 simultaneously displays a user manipulation interface UI generated in the display device 200 and used to control the display device 200.

And, the display 275 may include a display screen component for presenting a picture and a driving component for driving image display. Alternatively, if the display 275 is a projection display, it may also include a projection device and a projection screen.

The audio processor 280 is used to receive external audio signals, and perform decompression and decoding according to the standard codec protocol of the input signal, as well as audio data processing such as noise reduction, digital-to-analog conversion, and amplification processing, so that the audio data can be processed in the speaker 286 The audio signal to be played.

Exemplarily, the audio processor 280 may support various audio formats. Such as MPEG-2, MPEG-4, Advanced Audio Coding (AAC), High Efficiency AAC (HE-AAC) and other formats.

The audio output interface 285 is used to receive the audio signal output by the audio processor 280 under the control of the controller 250. The audio output interface 285 may include a speaker 286, or output to an external audio output terminal 287 of a generator of an external device, such as a headset Output terminal.

In some other exemplary embodiments, the video processor 270 may include one or more chips. The audio processor 280 may also include one or more chips.

And, in some other exemplary embodiments, the video processor 270 and the audio processor 280 may be separate chips, or may be integrated with the controller 250 in one or more chips.

The power supply 290 is used to provide power supply support for the display device 200 with power input from an external power supply under the control of the controller 250. The power supply 290 may be a built-in power supply circuit installed inside the display device 200, or may be a power supply installed outside the display device 200.

With reference to the display device in FIG. 1B, when the display device of the present application realizes dual-channel video playback, some components in the display device specifically perform the following operations:

The monitor is used to display the playback screen.

Audio output interface, used to output audio data;

Browser function module, controlled by the controller, used to obtain two channels of media stream data from the server;

Exemplarily, as shown in FIG. 1E, one channel of media stream data, such as a variety show, is played in full screen on the display; another channel of media stream data, such as an advertisement video, is played in the window at the upper left corner of the display. In this way, while not affecting users to watch variety shows, it can also provide users with advertising information to improve user experience.

In an embodiment, the controller may specifically be used to control the decoding module to compress the second video data in RGB format into YUV data and then send it to the browser function module. Specifically, the media stream data is processed by the decoding module to obtain the second video data. The second video data is RGB data. Generally speaking, the video image adopts the RGB mode. RGB is actually a combination of the three primary colors of red, green and blue. Under normal circumstances, each pixel can be composed of a 3bit RGB data set. For example, for 4×2 pictures, the RGB data storage method is as follows:

R1R2R3R4R5R6R7R8G1G2G3G4G5G6G7G8B1B2B3B4B5B6B7B8;

In order to improve the processing performance and further compress the data storage space, the RGB data format can be converted into the YUV data format, where Y is used to represent the brightness, that is, the gray value, while U and V represent the chromaticity, and the role is to describe The color and saturation of the image are used to specify the color of the pixel. Take the data format of YUV420 as an example, where 4 Ys share a set of UV components. For 4×2 pictures, the storage method of YUV420 data format includes the following two forms:

NV12 data format:

Y1Y2Y3Y4Y5Y6Y7Y8U1V1U2V2;

YV12 data format:

Y1Y2Y3Y4Y5Y6Y7Y8U1U2V1V2;

Through the comparison of the above RGB data format and YUV data format, it is found that since YUV data can save half of the memory storage space by sharing UV, in this embodiment, RGB data can be converted into YUV data to achieve the purpose of data compression.

Among them, there are two ways to compress RGB data into YUV data:

The first is that the controller can control the decoding module to convert the second video data in the RGB format into YUV data, and then sort the three groups of Y, U, and V data in the YUV data into a third video in a preset format. Data, sending the third video data to the browser function module. The preset format is NV12 format or YV12 format.

For example, as shown in Figure 2-1, the decoding module can decode H264 format media stream data through hardware to obtain RGB data. Assuming that the RGB data is 4×2 pixels, the RGB data can be expressed as:

R1R2R3R4R5R6R7R8G1G2G3G4G5G6G7G8B1B2B3B4B5B6B7B8;

The RGB data is converted into YUV data through a conversion algorithm, specifically:

Y: Y1Y2Y3Y4Y5Y6Y7Y8;

U: U1U2;

V: V1V2;

After encoding (here, NV12 is taken as an example), NV12 data is obtained, namely:

Y1Y2Y3Y4Y5Y6Y7Y8U1V1U2V2.

The NV12 data is the second video data that is finally input to the browser function module.

The second is that the controller can also control the decoding module to convert the second video data in the RGB format into YUV data, and then encapsulate the three groups of Y, U, and V data in the YUV data to obtain virtual YV12 data. , Sending the virtual YV12 data to the browser function module.

For example, as shown in Figure 2-2, the decoding module can decode H264 format media stream data through hardware to obtain RGB data. Assuming that the RGB data is 4×2 pixels, the RGB data can be expressed as:

R1R2R3R4R5R6R7R8G1G2G3G4G5G6G7G8B1B2B3B4B5B6B7B8;

Y: Y1Y2Y3Y4Y5Y6Y7Y8;

U: U1U2;

V: V1V2;

The YUV data is directly encapsulated to obtain the virtual YV12 data, namely:

Y: Y1Y2Y3Y4Y5Y6Y7Y8;

U: U1U2;

V: V1V2.

The virtual YV12 data is the second video data finally input to the browser function module.

Because in the first method above, the three groups of Y, U, and V data in the YUV data need to be sorted into video data in a preset format (such as NV12); and in the second method above, you can use the U in the YV12 data The U and V data are directly stored separately. Therefore, after the RGB data is decoded to obtain the YUV data, it is not necessary to perform the YUV sort processing according to the format of NV12 or YV12, but according to the Y, V data in the YUV data. The U and V three sets of data are respectively encapsulated to obtain virtual YV12 data, so that the virtual YV12 data is sent to the browser function module. Therefore, the second method reduces the data sorting process compared with the first method, and further optimizes the encoding and decoding process.

In one embodiment, the browser function module is specifically configured to render the second video data in the YUV format through OPENGL to obtain continuous video frame images. The browser function module can encapsulate the second video data into a designated data packet for each frame of the second video data in the YUV format, and obtain the position coordinates of the second window on the display screen of the display, according to the The position coordinates of the second window determine the position coordinates of each pixel in the second video data, the data in the specified data packet is filled into the second window based on the position coordinates of the pixel, and the display is displayed on the screen through the display. The video frame image corresponding to the second video data is displayed in the second window above.

The specific implementation mode can be shown in Figure 3. First, in the process A stage, the browser function module will construct a RenderTree by parsing HTML and CSS. The data nodes in the rendering tree determine the video playback window in the display (ie, the second window). ) Position and size, the position and size are expressed by coordinates. Then in the B stage of the process, the browser function module can encapsulate the second video data into a specified data package, such as DecodeTarget data, confirm the position coordinates of each pixel by packaging Box and layout Layout, and then convert each pixel in the DecodeTarget data The point is filled into the display area where the second window is located according to its position coordinates, so that the video frame image is displayed in the second window.

Figure 4 exemplarily shows the interactive process of the dual-channel video playback method, where the controller can control the browser function module to obtain two-channel media stream data from the server, as shown in step ① in Figure 4, and then control the de-multiplexing A module is used to demultiplex the two-channel media stream data obtained by the browser function module into video data and audio data. In this embodiment, the main focus is on the processing of video data, and the processing of audio data can refer to operations of related technologies, such as forwarding audio data to an audio output interface for audio playback.

The decoding module decodes the demultiplexed media stream data, as shown in step ② in Figure 4, to obtain the first channel of video data and the second channel of video data. The controller controls the video playing module to output the first channel of video data in the decoded first channel of media stream data to the first window on the display.

The controller controls the decoding module to compress the second video data in the RGB format into the second video data in the YUV format, as shown in step ③ in FIG. 4.

The controller controls the browser function module to first encapsulate the YUV data into a specified data packet for each frame of YUV data, as shown by arrow ④ in Figure 4, where the specified data packet can be a DecodeTarget data packet.

Then the browser function module can obtain the position coordinates of the second window on the display screen, determine the position coordinates of each pixel in the video image according to the position coordinates of the second window, and combine the data in the specified data packet The position coordinates based on the pixel are filled into the second window. This process is the rendering process of the video image, as shown by the arrow ⑤ in FIG. 4.

The rendered video frame image can be sent from the memory of the GPU to the memory of the display, so that the video frame image corresponding to the second video data can be displayed through the second window on the display screen of the display. As shown by the arrow ⑥ in Figure 4. Since the browser performs the above operation for each frame of second video data, the continuous video frame images can be displayed in the second window.

In some embodiments, if the controller (such as a chip) has a powerful dual-channel playback capability, for example, it has a dual-channel decoding module to support dual-channel video decoding, and a dual-channel playback module (such as a player provided by the controller) to support Dual-channel video playback, then the dual-channel video playback function can be realized. However, in some embodiments, if the controller (such as a chip) does not have a dual-channel video playback function, for example, it has only one decoding module or only one playback module, on the one hand, video playback can be realized based on the one-channel playback module, and on the other hand, Based on the rendering function of the browser itself, the video frame can be rendered frame by frame, so as to achieve the effect of video playback, so as to realize the function of dual-channel video playback on the display device.

In some embodiments, the video processor of the present application compresses the second video data in RGB format into YUV data and sends it to the browser, thereby improving data transmission efficiency and data processing, thereby making video playback smoother.

Fig. 5 exemplarily shows a flowchart of a two-channel video playback method.

In combination with the above-mentioned display device structure, the method for dual-channel video playback on the display device includes the following steps 501 to 505:

Step 501: The controller controls the browser function module to obtain two channels of media stream data from the server.

Step 502: Control the demultiplexing module to demultiplex the two channels of media stream data acquired by the browser function module into video data and audio data.

Step 503: The decoding module decodes the demultiplexed media stream data to obtain the first channel of media stream data including the first channel of video data and the first channel of audio data, and the second channel of media stream data including at least the second channel of video data. Road media stream data.

Step 504: The controller controls the playback module to output the first channel of video data in the decoded first channel of media stream data to the first window on the display, and output the first channel of audio data to the audio output interface .

Step 505: The controller controls the browser function module to render the second-channel video data in the decoded second-channel media stream data into continuous video frames, and output them to the second window on the display according to a certain frame rate.

In one embodiment, the decoding module compresses the second video data in RGB format into YUV data and then sends it to the browser function module.

In an embodiment, the decoding module compresses the second video data in RGB format into YUV data and sends it to the browser function module, which specifically includes: after the decoding module converts the second video data in RGB format into YUV data , And then encapsulate the three groups of Y, U, and V data in the YUV data to obtain virtual YV12 data, and send the virtual YV12 data to the browser.

For the specific embodiment of the method for dual-channel video playback on the display device, reference may be made to the embodiment of the processing of each module in the display device, which will not be repeated here.

In the above embodiment, the display device can obtain two channels of media stream data from the server through the browser function module, and the demultiplexing module demultiplexes the two channels of media stream data into video data and audio data; the decoding module demultiplexes the data after demultiplexing. The media stream data is decoded to obtain the first channel of media stream data including the first channel of video data and the first channel of audio data, and the second channel of media stream data including at least the second channel of video data; the playback module will decode the The first channel of video data in the first channel of media stream data is output to the first window on the display, and the first channel of audio data is output to the audio output interface; the browser function module will decode the second channel of video data Rendered as a continuous video frame, and output to the second window on the display according to a certain frame rate.

On the one hand, the first channel of media stream data can be played through the playback module in the display device; on the other hand, the video frame can be rendered frame by frame based on the browser's own rendering function, so as to realize the playback of the second channel of media stream data; In this way, the function of dual-channel media stream data playback on the display device is realized.

In some embodiments, in this application, the second video data in RGB format can be compressed into YUV data and then sent to the browser function module, so that the efficiency of data transmission and data processing can be improved, and the video playback can be made smoother.

Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they learn the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

Obviously, those skilled in the art can make various changes and modifications to the application without departing from the spirit and scope of the application. In this way, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these modifications and variations.

Claims

A display device, characterized in that it comprises:

monitor;

Audio output interface;

The controller is used to control the browser function module, the demultiplexing module, the decoding module and the playback module; wherein the browser function module is used to obtain two channels of media stream data;

The demultiplexing module is used to demultiplex the two channels of media stream data obtained by the browser function module into video data and audio data respectively;

The decoding module is used to decode the two channels of demultiplexed media stream data respectively to obtain the first channel of media stream data including the first channel of video data and the first channel of audio data, and at least the second channel of video data. The second media stream data;

A playback module, configured to output the first channel of video data in the decoded first channel of media stream data to the first window on the display, and output the first channel of audio data to the audio output interface;

The browser function module is also used to render the second channel of video data in the decoded second channel of media stream data into continuous video frames, and output to the second window on the display according to a certain frame rate.
The display device according to claim 1, wherein the browser function module is specifically used for:

The second video data in the YUV format is rendered through OPENGL to obtain continuous video frame images.
The display device according to claim 2, wherein the browser function module is specifically used for:

For each frame of second video data in YUV format, encapsulate the second video data into a specified data packet;

Acquiring the position coordinates of the second window on the display screen of the display;

Determine the position coordinates of each pixel in the second video data according to the position coordinates of the second window;

The data in the designated data packet is filled into the second window based on the position coordinates of the pixel, and the video frame image corresponding to the second video data is displayed in the second window through a display.
The display device of claim 1, wherein the controller is configured to:

Control the second video data in RGB format to be compressed into YUV data and then sent to the browser function module.
The display device according to claim 4, wherein the controller is further configured to:

After controlling the conversion of the second video data in the RGB format into YUV data, the three groups of Y, U, and V data in the YUV data are sorted into third video data in a preset format, and the third video data is sent To the browser function module.
The display device according to claim 5, wherein:

The preset format is NV12 format or YV12 format.
The display device according to claim 4, wherein the controller is further configured to:

After controlling the conversion of the second video data in the RGB format into YUV data, the three groups of Y, U, and V data in the YUV data are respectively encapsulated to obtain virtual YV12 data, and the virtual YV12 data is sent to the browser functional module.
A method for playing dual-channel media stream data in a display device, characterized in that the method includes:

Obtain two channels of media stream data;

Demultiplex the acquired two-channel media stream data into video data and audio data;

Decode the two channels of demultiplexed media stream data to obtain the first channel of media stream data including the first channel of video data and the first channel of audio data, and the second channel of media stream data including at least the second channel of video data ；

Output the first channel of video data in the decoded first channel of media stream data to the first window on the display, and output the first channel of audio data to the audio output interface;

Render the second channel of video data in the decoded second channel of media stream data into continuous video frames, and output them to the second window on the display according to a certain frame rate.
The method according to claim 8, wherein the method further comprises:

Compress the second video data in RGB format into YUV data.
The method according to claim 9, wherein the method further comprises:

After the second video data in the RGB format is converted into YUV data, the three groups of Y, U, and V data in the YUV data are sorted into the sorted second video data in a preset format, or the virtual data is encapsulated separately. YV12 data for subsequent rendering.
The method according to claim 9, wherein the method further comprises:

For each frame of second video data in YUV format, encapsulate the second video data into a specified data packet;

Acquiring the position coordinates of the second window on the display screen of the display;

Determine the position coordinates of each pixel in the second video data according to the position coordinates of the second window;

The data in the designated data packet is filled into the second window based on the position coordinates of the pixel, and the video frame image corresponding to the second video data is displayed in the second window through a display.