CN114095778A

CN114095778A - Audio hard decoding method of application-level player and display equipment

Info

Publication number: CN114095778A
Application number: CN202010862300.6A
Authority: CN
Inventors: 吕鹏; 李斌
Original assignee: Hisense Visual Technology Co Ltd
Current assignee: Hisense Visual Technology Co Ltd
Priority date: 2020-08-25
Filing date: 2020-08-25
Publication date: 2022-02-25

Abstract

The application discloses an audio hard decoding method and display equipment of an application-level player, wherein the configured application-level player is used for creating an audio hard decoder based on audio hard decoding parameters; and calling an audio hard decoder to perform audio hard decoding processing on the audio data before decoding to obtain decoded audio data in a pbuffer structure storage form, converting the decoded audio data in the pbuffer structure storage form into decoded audio data in an avframe structure storage form, and writing the decoded audio data into an audio decoded data queue to play the specified audio and video file. Therefore, the method and the display equipment provided by the invention can enable the application-level player to decode the audio data by adopting a hard decoding mode, can preprocess the sound effect, and decode the audio data into multiple channels through decoding to achieve the effect of enhancing the sound effect.

Description

Audio hard decoding method of application-level player and display equipment

Technical Field

The present application relates to the field of audio decoding technologies, and in particular, to an audio hard decoding method and a display device for an application-level player.

Background

Along with the rapid development of display equipment, the function of display equipment will be more and more abundant, and the performance is also more and more powerful, and at present, display equipment includes intelligent TV set, double-screen laser TV, intelligent STB, intelligent box to and have intelligent display screen etc..

In the prior art, when playing an audio/video file (a video source), an application-level player is usually adopted, and the application-level player decodes audio data and video data of the video source to realize playing. Application level players, including ijkplayer, exoplyer, adoplyer, pplvetvplayer, etc., typically use ffmpeg (third party open source decapsulation decoder), audio extrator (android native decapsulator), or self-encapsulated decoders when decoding audio.

However, most of the film sources played by the application-level player are network film sources, the audio coding format is single, the audio channel output after decoding by adopting the decoding mode is generally 2 channels, and the sampling rate and the bit rate are low, so that a good audio-visual effect cannot be obtained when the audio-video file is played.

Disclosure of Invention

The application provides an audio hard decoding method and display equipment of an application-level player, and aims to solve the problem that a good audio-visual effect cannot be obtained in the existing decoding mode.

In a first aspect, the present application provides a display device comprising:

a controller having an application-level player configured therein for playing a specified audio-video file, the application-level player configured to:

acquiring audio hard decoding parameters and audio data before decoding, wherein the audio data before decoding refers to the audio data obtained by de-encapsulating the specified audio and video file, and the audio hard decoding parameters refer to parameters required by audio hard decoding processing on the audio data before decoding;

creating an audio hard decoder based on the audio hard decoding parameters;

calling the audio hard decoder to perform audio hard decoding processing on the audio data before decoding to obtain decoded audio data;

converting the decoded audio data into decoded audio data in an avframe structure storage form, wherein the avframe structure storage form is a storage form adopted by an application-level player;

and writing the decoded audio data in the storage form of the avframe structure body into an audio decoded data queue to play the specified audio and video file.

In some embodiments of the present application, the application-level player, in executing the creating an audio hard decoder based on the audio hard decoding parameters, is further configured to:

creating an audio hard decoder;

configuring the audio hard decoder based on the audio hard decoding parameters;

creating a hard decoding input thread and a hard decoding output thread in an audio hard decoder configured with the audio hard decoding parameters.

In some embodiments of the present application, after executing the audio hard decoder to perform audio hard decoding processing on the pre-decoding audio data by the application-level player, obtaining decoded audio data, the application-level player is further configured to:

calling a hard decoding input thread in the audio hard decoder to perform audio hard decoding processing on the audio data before decoding to obtain decoded audio data;

and writing the decoded audio data into a hard decoding output thread in the audio hard decoder to obtain the decoded audio data in a pbuffer structure storage form, wherein the pbuffer structure storage form is the storage form adopted by the audio hard decoder.

In some embodiments of the present application, the application level player, in executing the creating the audio hard decoder, is further configured to:

acquiring running environment parameters of an audio hard decoder, wherein the running environment parameters refer to parameters required when the audio hard decoder is called;

compiling the operating environment parameters to generate a cross-language calling file comprising function names;

acquiring a function name corresponding to the audio hard decoding parameter, and matching the function name corresponding to the audio hard decoding parameter with a function name in the cross-language calling file;

when the function names match, an audio hard decoder is created.

In some embodiments of the present application, the application-level player, in executing the invoking of the hard decode input thread in the audio hard decoder to perform audio hard decode processing on the pre-decode audio data, is further configured to:

calling a hard decoding input thread in the audio hard decoder to acquire the audio data before decoding and an input buffer index;

and writing the audio data before decoding into the input buffer index for audio hard decoding processing to obtain the decoded audio data.

In some embodiments of the present application, the application-level player, prior to performing the obtaining the input buffer index, is further configured to:

judging whether the audio data needs to be emptied based on user operation when the specified audio and video file is played;

if the audio data needs to be emptied, the audio data stored in the audio hard decoder is emptied;

if the audio data does not need to be emptied, the step of obtaining the input buffer index is performed.

In some embodiments of the present application, after executing the hard decoding output thread that writes the decoded audio data into the audio hard decoder, the application-level player obtains the decoded audio data in the form of pbuffer structure storage, and is further configured to:

calling a hard decoding output thread in the audio hard decoder, and acquiring an output buffer index from the audio hard decoder;

and writing the decoded audio data into the output buffer index to obtain the decoded audio data in a pbuffer structure storage form.

In some embodiments of the present application, the application-level player, in performing the converting the decoded audio data into decoded audio data in the form of an avframe structure store, is further configured to:

acquiring audio data output format information and decoded audio data in a pbuffer structure storage form, wherein the audio data output format information refers to information required by output in an avframe structure storage form;

acquiring an audio data offset from the decoded audio data in the storage form of the pbuffer structure;

obtaining real decoded audio data in a pbuffer structure storage form based on the audio data offset and the decoded audio data;

creating an avframe structure based on the real decoded audio data in the pbuffer structure storage form;

and writing the audio data output format information into the avframe structure to obtain decoded audio data in an avframe structure storage form.

In some embodiments of the present application, the application level player is further configured to:

calling a standard decoding interface to obtain a comparison table comprising decoding formats and bottom layer decoding names which correspond to one another, wherein each bottom layer decoding name corresponds to an application chip;

determining a first bottom layer decoding name based on the comparison table and a decoding format corresponding to the audio hard decoding parameter;

obtaining a second bottom layer decoding name from the configured static file, and scoring the first bottom layer decoding name and the second bottom layer decoding name;

and determining the bottom layer decoding name with the highest score as a target bottom layer decoding name, and establishing connection with the application chip corresponding to the target bottom layer decoding name.

In a second aspect, the present application further provides an audio hard decoding method for an application-level player, the method including:

creating an audio hard decoder based on the audio hard decoding parameters;

In a third aspect, the present application further provides a storage medium, where the computer storage medium may store a program, and the program may implement, when executed, some or all of the steps in the embodiments of the audio hard decoding method including the application-level player provided in the present application.

As can be seen from the foregoing technical solutions, in the audio hard decoding method and the display device of the application-level player provided in the embodiments of the present invention, the configured application-level player is configured to obtain an audio hard decoding parameter and audio data before decoding, and create an audio hard decoder based on the audio hard decoding parameter; and calling an audio hard decoder to perform audio hard decoding processing on the audio data before decoding to obtain decoded audio data in a pbuffer structure storage form, converting the decoded audio data in the pbuffer structure storage form into decoded audio data in an avframe structure storage form, and writing the decoded audio data into an audio decoded data queue to play the specified audio and video file. Therefore, the method and the display device provided by the embodiment of the invention can enable the application-level player to decode the audio data by adopting a hard decoding mode, can preprocess the sound effect, and can decode the audio data into multiple channels through decoding to achieve the effect of enhancing the sound effect.

Drawings

In order to more clearly explain the technical solution of the present application, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious to those skilled in the art that other drawings can be obtained according to the drawings without any creative effort.

Fig. 1 is a schematic diagram illustrating an operational scenario between a display device and a control apparatus according to some embodiments;

a block diagram of a hardware configuration of a display device 200 according to some embodiments is illustrated in fig. 2;

a block diagram of the hardware configuration of the control device 100 according to some embodiments is illustrated in fig. 3;

a schematic diagram of a software configuration in a display device 200 according to some embodiments is illustrated in fig. 4;

FIG. 5 illustrates an icon control interface display diagram of an application in the display device 200, according to some embodiments;

a flow diagram of a method of audio hard decoding for an application level player according to some embodiments is illustrated in fig. 6;

FIG. 7 illustrates a block diagram of an application level player, according to some embodiments;

a method flow diagram for creating an audio hard decoder according to some embodiments is illustrated in fig. 8;

a data flow diagram of a hard decode input thread according to some embodiments is illustrated in fig. 9;

FIG. 10 is a flow diagram illustrating a method of execution of a hard decoded output thread in accordance with some embodiments;

a data flow diagram of a hard decode output thread according to some embodiments is illustrated in fig. 11;

a flowchart of a method of converting an avframe structure according to some embodiments is illustrated in fig. 12;

a data flow diagram of the transform avframe structure according to some embodiments is illustrated in fig. 13;

a method flow diagram for multi-chip OMX layer decoding format compatibility according to some embodiments is illustrated in fig. 14.

Detailed Description

To make the objects, embodiments and advantages of the present application clearer, the following description of exemplary embodiments of the present application will clearly and completely describe the exemplary embodiments of the present application with reference to the accompanying drawings in the exemplary embodiments of the present application, and it is to be understood that the described exemplary embodiments are only a part of the embodiments of the present application, and not all of the embodiments.

All other embodiments, which can be derived by a person skilled in the art from the exemplary embodiments described herein without inventive step, are intended to be within the scope of the claims appended hereto. In addition, while the disclosure herein has been presented in terms of one or more exemplary examples, it should be appreciated that aspects of the disclosure may be implemented solely as a complete embodiment.

It should be noted that the brief descriptions of the terms in the present application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of the present application. These terms should be understood in their ordinary and customary meaning unless otherwise indicated.

The terms "first," "second," "third," and the like in the description and claims of this application and in the above-described drawings are used for distinguishing between similar or analogous objects or entities and are not necessarily intended to limit the order or sequence of any particular one, Unless otherwise indicated. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments described herein are, for example, capable of operation in sequences other than those illustrated or otherwise described herein.

Furthermore, the terms "comprises" and "comprising," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a product or device that comprises a list of elements is not necessarily limited to those elements explicitly listed, but may include other elements not expressly listed or inherent to such product or device.

The term "module," as used herein, refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and/or software code that is capable of performing the functionality associated with that element.

The term "remote control" as used in this application refers to a component of an electronic device (such as the display device disclosed in this application) that is typically wirelessly controllable over a relatively short range of distances. Typically using infrared and/or Radio Frequency (RF) signals and/or bluetooth to connect with the electronic device, and may also include WiFi, wireless USB, bluetooth, motion sensor, etc. For example: the hand-held touch remote controller replaces most of the physical built-in hard keys in the common remote control device with the user interface in the touch screen.

The term "gesture" as used in this application refers to a user's behavior through a change in hand shape or an action such as hand motion to convey a desired idea, action, purpose, or result.

A schematic diagram of an operational scenario between a display device and a control apparatus according to some embodiments is illustrated in fig. 1. As shown in fig. 1, a user may operate the display device 200 through the mobile terminal 300 and the control apparatus 100.

In some embodiments, the control apparatus 100 may be a remote controller, and the communication between the remote controller and the display device includes an infrared protocol communication or a bluetooth protocol communication, and other short-distance communication methods, etc., and the display device 200 is controlled by wireless or other wired methods. The user may input a user command through a key on a remote controller, voice input, control panel input, etc. to control the display apparatus 200. Such as: the user can input a corresponding control command through a volume up/down key, a channel control key, up/down/left/right moving keys, a voice input key, a menu key, a power on/off key, etc. on the remote controller, to implement the function of controlling the display device 200.

In some embodiments, mobile terminals, tablets, computers, laptops, and other smart devices may also be used to control the display device 200. For example, the display device 200 is controlled using an application program running on the smart device. The application, through configuration, may provide the user with various controls in an intuitive User Interface (UI) on a screen associated with the smart device.

In some embodiments, the mobile terminal 300 may install a software application with the display device 200 to implement connection communication through a network communication protocol for the purpose of one-to-one control operation and data communication. Such as: the mobile terminal 300 and the display device 200 can establish a control instruction protocol, synchronize a remote control keyboard to the mobile terminal 300, and control the display device 200 by controlling a user interface on the mobile terminal 300. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200, so as to realize the synchronous display function.

As also shown in fig. 1, the display apparatus 200 also performs data communication with the server 400 through various communication means. The display device 200 may be allowed to be communicatively connected through a Local Area Network (LAN), a Wireless Local Area Network (WLAN), and other networks. The server 400 may provide various contents and interactions to the display apparatus 200. Illustratively, the display device 200 receives software program updates, or accesses a remotely stored digital media library, by sending and receiving information, as well as Electronic Program Guide (EPG) interactions. The server 400 may be a cluster or a plurality of clusters, and may include one or more types of servers. Other web service contents such as video on demand and advertisement services are provided through the server 400.

The display device 200 may be a liquid crystal display, an OLED display, a projection display device. The particular display device type, size, resolution, etc. are not limiting, and those skilled in the art will appreciate that the display device 200 may be modified in performance and configuration as desired.

The display apparatus 200 may additionally provide an intelligent network tv function of a computer support function including, but not limited to, a network tv, an intelligent tv, an Internet Protocol Tv (IPTV), and the like, in addition to the broadcast receiving tv function.

A block diagram of a hardware configuration of a display device 200 according to some embodiments is illustrated in fig. 2.

In some embodiments, at least one of the controller 250, the tuner demodulator 210, the communicator 220, the detector 230, the input/output interface 255, the display 275, the audio output interface 285, the memory 260, the power supply 290, the user interface 265, and the external device interface 240 is included in the display apparatus 200.

In some embodiments, a display 275 receives image signals originating from the first processor output and displays video content and images and components of the menu manipulation interface.

In some embodiments, the display 275, includes a display screen assembly for presenting a picture, and a driving assembly that drives the display of an image.

In some embodiments, the video content is displayed from broadcast television content, or alternatively, from various broadcast signals that may be received via wired or wireless communication protocols. Alternatively, various image contents received from the network communication protocol and sent from the network server side can be displayed.

In some embodiments, the display 275 is used to present a user-manipulated UI interface generated in the display apparatus 200 and used to control the display apparatus 200.

In some embodiments, a driver assembly for driving the display is also included, depending on the type of display 275.

In some embodiments, display 275 is a projection display and may also include a projection device and a projection screen.

In some embodiments, communicator 220 is a component for communicating with external devices or external servers according to various communication protocol types. For example: the communicator 220 may include at least one of a Wifi module 221, a bluetooth module 222, a wired ethernet module 223, and other network communication protocol modules or near field communication protocol modules, and an infrared receiver.

In some embodiments, the display apparatus 200 may establish control signal and data signal transmission and reception with the external control apparatus 100 or the content providing apparatus through the communicator 220.

In some embodiments, the user interface 265 may be configured to receive infrared control signals from a control device 100 (e.g., an infrared remote control, etc.).

In some embodiments, the detector 230 is a signal used by the display device 200 to collect an external environment or interact with the outside.

In some embodiments, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light, and parameters changes can be adaptively displayed by collecting the ambient light, and the like.

In some embodiments, the detector 230 may further include an image collector 232, such as a camera, a video camera, etc., which may be configured to collect external environment scenes, collect attributes of the user or gestures interacted with the user, adaptively change display parameters, and recognize user gestures, so as to implement a function of interaction with the user.

In some embodiments, the detector 230 may also include a temperature sensor or the like, such as by sensing ambient temperature.

In some embodiments, the display apparatus 200 may adaptively adjust a display color temperature of an image. For example, the display apparatus 200 may be adjusted to display a cool tone when the temperature is in a high environment, or the display apparatus 200 may be adjusted to display a warm tone when the temperature is in a low environment.

In some embodiments, the detector 230 further comprises a sound collector 231 or the like, such as a microphone, which may be used to receive the user's voice. Illustratively, a voice signal including a control instruction of the user to control the display device 200, or to collect an ambient sound for recognizing an ambient scene type, so that the display device 200 can adaptively adapt to an ambient noise.

In some embodiments, as shown in fig. 2, the input/output interface 255 is configured to allow data transfer between the controller 250 and external other devices or other controllers 250. Such as receiving video signal data and audio signal data of an external device, or command instruction data, etc.

In some embodiments, the external device interface 240 may include, but is not limited to, the following: the interface can be any one or more of a high-definition multimedia interface (HDMI), an analog or data high-definition component input interface, a composite video input interface, a USB input interface, an RGB port and the like. The plurality of interfaces may form a composite input/output interface.

In some embodiments, as shown in fig. 2, the tuning demodulator 210 is configured to receive a broadcast television signal through a wired or wireless receiving manner, perform modulation and demodulation processing such as amplification, mixing, resonance, and the like, and demodulate an audio and video signal from a plurality of wireless or wired broadcast television signals, where the audio and video signal may include a television audio and video signal carried in a television channel frequency selected by a user and an EPG data signal.

In some embodiments, the frequency points demodulated by the tuner demodulator 210 are controlled by the controller 250, and the controller 250 can send out control signals according to user selection, so that the modem responds to the television signal frequency selected by the user and modulates and demodulates the television signal carried by the frequency.

In some embodiments, the broadcast television signal may be classified into a terrestrial broadcast signal, a cable broadcast signal, a satellite broadcast signal, an internet broadcast signal, or the like according to the broadcasting system of the television signal. Or may be classified into a digital modulation signal, an analog modulation signal, and the like according to a modulation type. Or the signals are classified into digital signals, analog signals and the like according to the types of the signals.

In some embodiments, the controller 250 and the modem 210 may be located in different separate devices, that is, the modem 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box. Therefore, the set top box outputs the television audio and video signals modulated and demodulated by the received broadcast television signals to the main body equipment, and the main body equipment receives the audio and video signals through the first input/output interface.

In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 may control the overall operation of the display apparatus 200. For example: in response to receiving a user command for selecting a UI object to be displayed on the display 275, the controller 250 may perform an operation related to the object selected by the user command.

In some embodiments, the object may be any one of selectable objects, such as a hyperlink or an icon. Operations related to the selected object, such as: displaying an operation connected to a hyperlink page, document, image, or the like, or performing an operation of a program corresponding to the icon. The user command for selecting the UI object may be a command input through various input means (e.g., a mouse, a keyboard, a touch pad, etc.) connected to the display apparatus 200 or a voice command corresponding to a voice spoken by the user.

As shown in fig. 2, the controller 250 includes at least one of a Random Access Memory 251 (RAM), a Read-Only Memory 252 (ROM), a video processor 270, an audio processor 280, other processors 253 (e.g., a Graphics Processing Unit (GPU), a Central Processing Unit 254 (CPU), a Communication Interface (Communication Interface), and a Communication Bus 256(Bus), which connects the respective components.

In some embodiments, RAM 251 is used to store temporary data for the operating system or other programs that are running, and in some embodiments, ROM 252 is used to store instructions for various system boots.

In some embodiments, the ROM 252 is used to store a Basic Input Output System (BIOS). The system is used for completing power-on self-test of the system, initialization of each functional module in the system, a driver of basic input/output of the system and booting an operating system.

In some embodiments, upon receiving the power-on signal, the display device 200 starts to power up, and the processor 254 executes system boot instructions in the ROM 252 to copy temporary data of the operating system stored in the memory into the RAM 251 in order to boot or run the operating system. After the start of the operating system is completed, the processor 254 copies the temporary data of the various application programs in the memory to the RAM 251, and then, the various application programs are started or run.

In some embodiments, processor 254 is used to execute operating system and application program instructions stored in memory. And executing various application programs, data and contents according to various interactive instructions received from the outside so as to finally display and play various audio and video contents.

In some demonstrative embodiments, processor 254 may include a plurality of processors. The plurality of processors may include a main processor and one or more sub-processors. A main processor for performing some operations of the display apparatus 200 in a pre-power-up mode and/or operations of displaying a screen in a normal mode. One or more sub-processors for one operation in a standby mode or the like.

In some embodiments, the graphics processor 253 is used to generate various graphics objects, such as: icons, operation menus, user input instruction display graphics, and the like. The display device comprises an arithmetic unit which carries out operation by receiving various interactive instructions input by a user and displays various objects according to display attributes. And the system comprises a renderer for rendering various objects obtained based on the arithmetic unit, wherein the rendered objects are used for being displayed on a display.

In some embodiments, the video processor 270 is configured to receive an external video signal, and perform video processing such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, image synthesis, and the like according to a standard codec protocol of the input signal, so as to obtain a signal that can be displayed or played on the direct display device 200.

In some embodiments, video processor 270 includes a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, a display formatting module, and the like.

The demultiplexing module is used for demultiplexing the input audio and video data stream, and if the input MPEG-2 is input, the demultiplexing module demultiplexes the input audio and video data stream into a video signal and an audio signal.

And the video decoding module is used for processing the video signal after demultiplexing, including decoding, scaling and the like.

And the image synthesis module is used for carrying out superposition mixing processing on the GUI signal input by the user or generated by the user and the video image after the zooming processing by the graphic generator so as to generate an image signal for display.

The frame rate conversion module is configured to convert an input video frame rate, such as a 60Hz frame rate into a 120Hz frame rate or a 240Hz frame rate, and the normal format is implemented in, for example, an interpolation frame mode.

The display format module is used for converting the received video output signal after the frame rate conversion, and changing the signal to conform to the signal of the display format, such as outputting an RGB data signal.

In some embodiments, the graphics processor 253 and the video processor may be integrated or separately configured, and when the graphics processor and the video processor are integrated, the graphics processor and the video processor may perform processing of graphics signals output to the display, and when the graphics processor and the video processor are separately configured, the graphics processor and the video processor may perform different functions, respectively, for example, a GPU + frc (frame Rate conversion) architecture.

In some embodiments, the audio processor 280 is configured to receive an external audio signal, decompress and decode the received audio signal according to a standard codec protocol of the input signal, and perform noise reduction, digital-to-analog conversion, and amplification processes to obtain an audio signal that can be played in a speaker.

In some embodiments, video processor 270 may comprise one or more chips. The audio processor may also comprise one or more chips.

In some embodiments, the video processor 270 and the audio processor 280 may be separate chips or may be integrated together with the controller in one or more chips.

In some embodiments, the audio output, under the control of controller 250, receives sound signals output by audio processor 280, such as: the speaker 286, and an external sound output terminal of a generating device that can output to an external device, in addition to the speaker carried by the display device 200 itself, such as: external sound interface or earphone interface, etc., and may also include a near field communication module in the communication interface, for example: and the Bluetooth module is used for outputting sound of the Bluetooth loudspeaker.

The power supply 290 supplies power to the display device 200 from the power input from the external power source under the control of the controller 250. The power supply 290 may include a built-in power supply circuit installed inside the display apparatus 200, or may be a power supply interface installed outside the display apparatus 200 to provide an external power supply in the display apparatus 200.

A user interface 265 for receiving an input signal of a user and then transmitting the received user input signal to the controller 250. The user input signal may be a remote controller signal received through an infrared receiver, and various user control signals may be received through the network communication module.

In some embodiments, the user inputs a user command through the control apparatus 100 or the mobile terminal 300, the user input interface responds to the user input through the controller 250 according to the user input, and the display device 200 responds to the user input through the controller 250.

In some embodiments, a user may enter user commands on a Graphical User Interface (GUI) displayed on the display 275, and the user input interface receives the user input commands through the Graphical User Interface (GUI). Alternatively, the user may input the user command by inputting a specific sound or gesture, and the user input interface receives the user input command by recognizing the sound or gesture through the sensor.

In some embodiments, a "user interface" is a media interface for interaction and information exchange between an application or operating system and a user that enables conversion between an internal form of information and a form that is acceptable to the user. A commonly used presentation form of the User Interface is a Graphical User Interface (GUI), which refers to a User Interface related to computer operations and displayed in a graphical manner. It may be an interface element such as an icon, a window, a control, etc. displayed in the display screen of the electronic device, where the control may include a visual interface element such as an icon, a button, a menu, a tab, a text box, a dialog box, a status bar, a navigation bar, a Widget, etc.

The memory 260 includes a memory storing various software modules for driving the display device 200. Such as: various software modules stored in the first memory, including: at least one of a basic module, a detection module, a communication module, a display control module, a browser module, and various service modules.

The base module is a bottom layer software module for signal communication between various hardware in the display device 200 and for sending processing and control signals to the upper layer module. The detection module is used for collecting various information from various sensors or user input interfaces, and the management module is used for performing digital-to-analog conversion and analysis management.

For example, the voice recognition module comprises a voice analysis module and a voice instruction database module. The display control module is used for controlling the display to display the image content, and can be used for playing the multimedia image content, UI interface and other information. And the communication module is used for carrying out control and data communication with external equipment. And the browser module is used for executing a module for data communication between browsing servers. And the service module is used for providing various services and modules including various application programs. Meanwhile, the memory 260 may store a visual effect map for receiving external data and user data, images of various items in various user interfaces, and a focus object, etc.

Fig. 3 illustrates a block diagram of a configuration of the control apparatus 100 according to some embodiments. As shown in fig. 3, the control apparatus 100 includes a controller 110, a communication interface 130, a user input/output interface, a memory, and a power supply source.

The control device 100 is configured to control the display device 200 and may receive an input operation instruction of a user and convert the operation instruction into an instruction recognizable and responsive by the display device 200, serving as an interaction intermediary between the user and the display device 200. Such as: the user responds to the channel up and down operation by operating the channel up and down keys on the control device 100.

In some embodiments, the control device 100 may be a smart device. Such as: the control apparatus 100 may install various applications that control the display apparatus 200 according to user demands.

In some embodiments, as shown in fig. 1, a mobile terminal 300 or other intelligent electronic device may function similar to the control device 100 after installing an application that manipulates the display device 200. Such as: the user may implement the functions of controlling the physical keys of the device 100 by installing applications, various function keys or virtual buttons of a graphical user interface available on the mobile terminal 300 or other intelligent electronic device.

The controller 110 includes a processor 112 and RAM 113 and ROM 114, a communication interface 130, and a communication bus. The controller is used to control the operation of the control device 100, as well as the communication cooperation between the internal components and the external and internal data processing functions.

The communication interface 130 enables communication of control signals and data signals with the display apparatus 200 under the control of the controller 110. Such as: the received user input signal is transmitted to the display apparatus 200. The communication interface 130 may include at least one of a WiFi chip 131, a bluetooth module 132, an NFC module 133, and other near field communication modules.

A user input/output interface 140, wherein the input interface includes at least one of a microphone 141, a touch pad 142, a sensor 143, keys 144, and other input interfaces. Such as: the user can realize a user instruction input function through actions such as voice, touch, gesture, pressing, and the like, and the input interface converts the received analog signal into a digital signal and converts the digital signal into a corresponding instruction signal, and sends the instruction signal to the display device 200.

The output interface includes an interface that transmits the received user instruction to the display apparatus 200. In some embodiments, the interface may be an infrared interface or a radio frequency interface. Such as: when the infrared signal interface is used, the user input instruction needs to be converted into an infrared control signal according to an infrared control protocol, and the infrared control signal is sent to the display device 200 through the infrared sending module. The following steps are repeated: when the rf signal interface is used, a user input command needs to be converted into a digital signal, and then the digital signal is modulated according to the rf control signal modulation protocol and then transmitted to the display device 200 through the rf transmitting terminal.

In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input-output interface 140. The control device 100 is provided with a communication interface 130, such as: the WiFi, bluetooth, NFC, etc. modules may transmit the user input command to the display device 200 through the WiFi protocol, or the bluetooth protocol, or the NFC protocol code.

A memory 190 for storing various operation programs, data and applications for driving and controlling the control apparatus 200 under the control of the controller. The memory 190 may store various control signal commands input by a user.

And a power supply 180 for providing operational power support to the various elements of the control device 100 under the control of the controller. A battery and associated control circuitry.

In some embodiments, the system may include a Kernel (Kernel), a command parser (shell), a file system, and an application program. The kernel, shell, and file system together make up the basic operating system structure that allows users to manage files, run programs, and use the system. After power-on, the kernel is started, kernel space is activated, hardware is abstracted, hardware parameters are initialized, and virtual memory, a scheduler, signals and interprocess communication (IPC) are operated and maintained. And after the kernel is started, loading the Shell and the user application program. The application program is compiled into machine code after being started, and a process is formed.

A schematic diagram of the software configuration in the display device 200 according to some embodiments is illustrated in fig. 4. Referring to fig. 4, in some embodiments, the system is divided into four layers, which are an Application (Applications) layer (abbreviated as "Application layer"), an Application Framework (Application Framework) layer (abbreviated as "Framework layer"), an Android runtime (Android runtime) and system library layer (abbreviated as "system runtime library layer"), and a kernel layer from top to bottom.

In some embodiments, at least one application program runs in the application program layer, and the application programs can be Window (Window) programs carried by an operating system, system setting programs, clock programs, camera applications and the like; or may be an application developed by a third party developer such as a hi program, a karaoke program, a magic mirror program, or the like. In specific implementation, the application packages in the application layer are not limited to the above examples, and may actually include other application packages, which is not limited in this embodiment of the present application.

The framework layer provides an Application Programming Interface (API) and a programming framework for the application program of the application layer. The application framework layer includes a number of predefined functions. The application framework layer acts as a processing center that decides to let the applications in the application layer act. The application program can access the resource in the system and obtain the service of the system in execution through the API interface

As shown in fig. 4, in the embodiment of the present application, the application framework layer includes a manager (Managers), a Content Provider (Content Provider), and the like, where the manager includes at least one of the following modules: an Activity Manager (Activity Manager) is used for interacting with all activities running in the system; the Location Manager (Location Manager) is used for providing the system service or application with the access of the system Location service; a Package Manager (Package Manager) for retrieving various information related to an application Package currently installed on the device; a Notification Manager (Notification Manager) for controlling display and clearing of Notification messages; a Window Manager (Window Manager) is used to manage the icons, windows, toolbars, wallpapers, and desktop components on a user interface.

In some embodiments, the activity manager is to: managing the life cycle of each application program and the general navigation backspacing function, such as controlling the exit of the application program (including switching the user interface currently displayed in the display window to the system desktop), opening, backing (including switching the user interface currently displayed in the display window to the previous user interface of the user interface currently displayed), and the like.

In some embodiments, the window manager is configured to manage all window processes, such as obtaining a display size, determining whether a status bar is available, locking a screen, intercepting a screen, controlling a display change (e.g., zooming out, dithering, distorting, etc.) and the like.

In some embodiments, the system runtime layer provides support for the upper layer, i.e., the framework layer, and when the framework layer is used, the android operating system runs the C/C + + library included in the system runtime layer to implement the functions to be implemented by the framework layer.

In some embodiments, the kernel layer is a layer between hardware and software. As shown in fig. 4, the core layer includes at least one of the following drivers: audio drive, display drive, bluetooth drive, camera drive, WIFI drive, USB drive, HDMI drive, sensor drive (such as fingerprint sensor, temperature sensor, touch sensor, pressure sensor, etc.), and so on.

In some embodiments, the kernel layer further comprises a power driver module for power management.

In some embodiments, software programs and/or modules corresponding to the software architecture of fig. 4 are stored in the first memory or the second memory shown in fig. 2 or 3.

In some embodiments, taking the magic mirror application (photographing application) as an example, when the remote control receiving device receives a remote control input operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the input operation into an original input event (including information such as a value of the input operation, a timestamp of the input operation, etc.). The raw input events are stored at the kernel layer. The application program framework layer obtains an original input event from the kernel layer, identifies a control corresponding to the input event according to the current position of the focus and uses the input operation as a confirmation operation, the control corresponding to the confirmation operation is a control of a magic mirror application icon, the magic mirror application calls an interface of the application framework layer to start the magic mirror application, and then the kernel layer is called to start a camera driver, so that a static image or a video is captured through the camera.

In some embodiments, for a display device with a touch function, taking a split screen operation as an example, the display device receives an input operation (such as a split screen operation) that a user acts on a display screen, and the kernel layer may generate a corresponding input event according to the input operation and report the event to the application framework layer. The window mode (such as multi-window mode) corresponding to the input operation, the position and size of the window and the like are set by an activity manager of the application framework layer. And the window management of the application program framework layer draws a window according to the setting of the activity manager, then sends the drawn window data to the display driver of the kernel layer, and the display driver displays the corresponding application interface in different display areas of the display screen.

An icon control interface display diagram of an application in display device 200 according to some embodiments is illustrated in fig. 5. In some embodiments, as shown in fig. 5, the application layer containing at least one application may display a corresponding icon control in the display, such as: the system comprises a live television application icon control, a video on demand application icon control, a media center application icon control, an application center icon control, a game application icon control and the like.

In some embodiments, the live television application may provide live television via different signal sources. For example, a live television application may provide television signals using input from cable television, radio broadcasts, satellite services, or other types of live television services. And, the live television application may display video of the live television signal on the display device 200.

In some embodiments, a video-on-demand application may provide video from different storage sources. Unlike live television applications, video on demand provides a video display from some storage source. For example, the video on demand may come from a server side of the cloud storage, from a local hard disk storage containing stored video programs.

In some embodiments, the media center application may provide various applications for multimedia content playback. For example, a media center, which may be other than live television or video on demand, may provide services that a user may access to various images or audio through a media center application.

In some embodiments, an application center may provide storage for various applications. The application may be a game, an application, or some other application associated with a computer system or other device that may be run on the smart television. The application center may obtain these applications from different sources, store them in local storage, and then be operable on the display device 200.

In some embodiments, the display device is configured with an application-level player, such as ijkplayer, exolayer, adoplayer, pplvetvplayer, and so on, when playing the audio/video file. When an application-level player is used to play audio/video files, the audio/video files are usually unpacked, the audio data, the video data, and the subtitle data are decoded, and the audio/video files are rendered for playing.

The application level player generally uses soft decoding mode in audio decoding, for example, ffmpeg (third party open source decapsulation decoder), audio extrator (android native decapsulation program), or self-encapsulated decoder. Since most of the application level players play the network film sources, and the audio coding format is single, the use of soft decoding has little influence on the system expense.

However, with the diversification of network playing and the development of the bottom chip technology of the display device, a better audio-visual effect needs to be provided, and more cpu and memory are occupied when audio decoding is performed through soft decoding, the audio channel is generally 2 channels, and the sampling rate and the bit rate are lower, so that the decoded audio-visual file cannot achieve a higher audio-visual effect. Therefore, the application player is required to use hard decoding (mediaodec) to achieve better audio-visual effects.

An application-level player in the display equipment decodes audio data in a hard decoding mode, can preprocess dts dolby sound effect at a decoding end, and decodes the audio data (2 channels before decoding) into 8 channels or 6 channels through decoding to achieve the effect of enhancing the sound effect. And with the development of audio and video technology, the audio coding technology is more complex, and the use of hard decoding can effectively reduce the system overhead and prepare for backward compatibility.

To this end, an embodiment of the present invention provides a display device, which includes a controller, and an application-level player configured in the controller for playing a specified audio and video file. The display equipment can realize that the audio decoding stream of the application-level player based on the ijkplayer calls a mediacodec (android native decoder) hard decoding function, increases multichannel output support at a mediacodec end, and simultaneously realizes compatible support for the diversity of audio hard decoding formats of the existing chip Nova, Mtk and Hisi schemes.

A flow diagram of a method of audio hard decoding for an application level player according to some embodiments is illustrated in fig. 6. In a display device provided in an embodiment of the present invention, when executing an audio hard decoding method of an application-level player, the configured application-level player is configured to execute the following steps:

and S1, obtaining audio hard decoding parameters and audio data before decoding, wherein the audio data before decoding refers to the audio data obtained by de-encapsulating the specified audio and video file, and the audio hard decoding parameters refer to parameters required by audio hard decoding processing of the audio data before decoding.

When the display device plays a specified audio and video file, an application-level player needs to be started first, and the application-level player sets parameters, namely audio hard decoding parameters, required for audio hard decoding processing of audio data in the audio and video file through a SetOption (player alone sets an upper functional interface) interface.

When setting the audio hard decoding parameters, the setting can be performed based on the selection of the user, for example, if the user wants to achieve the best sound effect when playing the specified audio/video file, the option of the dolby sound effect can be selected, and at this time, the audio hard decoding parameters for realizing the dolby sound effect can be set.

A block diagram of an application level player according to some embodiments is illustrated in fig. 7. Referring to fig. 7, the application level player includes a download and decapsulation module, a pre-decoding data queue, a decoding module, a post-decoding data queue, and a rendering module.

And the downloading and decapsulating module is used for downloading the film source of the specified audio and video file and decapsulating the specified audio and video file to obtain audio data, video data and subtitle data. The pre-decoding data queue is used for storing Audio data in a pre-decoding Audio data queue (Audio Pkg queue), storing Video data in a pre-decoding Video data queue (Video Pkg queue), and storing Subtitle data in a pre-decoding Subtitle data queue (Subtitle Pkg queue). The decoding module is used for hard decoding (MediaCodec) or soft decoding (avcodec) of audio data, video data, and subtitle data. The decoded data queue is used for placing the decoded audio data into a decoded audio data queue (Asmq), placing the decoded video data into a decoded video data queue (Vpicq), and placing the decoded subtitle data into a decoded subtitle data queue (subpacq). The rendering module is used for rendering the decoded video data, audio data and subtitle data, and playing the appointed audio and video file with higher sound effect.

In some embodiments, when performing hard decoding processing on audio data, the application-level player obtains audio data before decoding obtained through decapsulation, and places the audio data before decoding into an audio data queue before decoding, so that the decoding module performs audio hard decoding processing subsequently.

Since the decoding module has two decoding modes, the decoding mode needs to be selected after the application-level player acquires the pre-decoding audio data.

In some embodiments, the application level player, after performing the obtaining the pre-decoding audio data, is further configured to: selecting a corresponding audio decoding mode for decoding the specified audio and video file based on the audio hard decoding parameters; and if the selected audio decoding mode is the audio hard decoding mode, carrying out audio hard decoding processing on the specified audio and video file.

The audio hard decoding parameters can identify whether the user selects to perform sound effect enhancement processing, and if the user selects sound effect enhancement when playing a specified audio and video file, such as Dolby sound effect selection, the audio hard decoding mode can be selected according to the audio hard decoding parameters to perform audio hard decoding processing on the specified audio and video file.

And S2, creating an audio hard decoder based on the audio hard decoding parameters.

After the application-level player stores the audio data before decoding in the audio data queue before decoding, the audio data before decoding stored in the audio data queue before decoding can be subjected to audio hard decoding processing by the decoding module. When the decoding module performs audio hard decoding processing, an audio hard decoder (Android media codec) needs to be created according to the audio hard decoding parameters. The audio hard decoder adopts an asynchronous decoding mode, so a hard decoding input thread and a hard decoding output thread need to be created in the audio hard decoder.

In some embodiments, the application-level player, in performing creating the audio hard decoder based on the audio hard decoding parameters, is further configured to perform the steps of:

step 21, creating an audio hard decoder.

And step 22, configuring an audio hard decoder based on the audio hard decoding parameters.

And step 23, creating a hard decoding input thread and a hard decoding output thread in the audio hard decoder configured with the audio hard decoding parameters.

When the audio hard decoding processing is carried out, the application-level player calls the decoding module to create an audio hard decoder, and the audio hard decoder is used for carrying out the audio hard decoding processing on the audio data before decoding.

In order to enable the audio data after the audio hard decoding processing to have parameters required by high sound effects such as high sampling rate, high bit rate, increased number of channels and the like, an audio hard decoder needs to be configured according to the audio hard decoding parameters. The audio hard decoding parameters comprise decoding related media parameters such as sampling rate, bit rate, channel number, audio format and the like which can realize high sound effect.

Meanwhile, in order to realize an asynchronous decoding mode, the direct dependency relationship before and after decoding is avoided, namely if the decoded data cannot be obtained due to the abnormality before decoding, the phenomenon of jamming occurs, and the normal decoding processing process is influenced. Thus, in some embodiments, a hard decode input thread and a hard decode output thread are created in an audio hard decoder configured with audio hard decode parameters.

The hard decoding input thread and the hard decoding output thread are not directly connected but are two mutually independent threads, the hard decoding input thread stores the decoded audio data into the audio hard decoder, and when the audio data needs to be output, the hard decoding output thread acquires the decoded audio data from the audio hard decoder and then carries out subsequent operation. And an asynchronous decoding mode is adopted, so that the condition of jamming caused by exception of a certain thread can be avoided.

The decoding module needs to communicate with the audio hard decoder when the audio hard decoder is created and a hard decoding input thread and a hard decoding output thread in the audio hard decoder are called.

A flow diagram of a method of creating an audio hard decoder according to some embodiments is illustrated in fig. 8. To this end, in some embodiments, referring to fig. 8, the application level player, in performing creating the audio hard decoder, is further configured to:

s211, obtaining the operation environment parameters of the audio hard decoder, wherein the operation environment parameters refer to parameters required when the audio hard decoder is called.

S212, compiling the operating environment parameters to generate a cross-language calling file comprising the function name;

s213, acquiring a function name corresponding to the audio hard decoding parameter, and matching the function name corresponding to the audio hard decoding parameter with a function name in a cross-language calling file;

and S214, when the function names are matched and consistent, creating an audio hard decoder.

If the decoding module in the application-level player calls the audio hard decoder, the decoding module needs to have parameters related to the called audio hard decoder, that is, the decoding module needs to have the operating environment parameters of the audio hard decoder.

The application-level player compiles the operating environment parameters which can realize the calling of the audio hard decoder to generate a cross-language calling file (jni file). The cross-language calling file comprises a plurality of operating environment parameters, and the operating environment parameters of the same class can be used as a java class name, namely a function name. The function names in the cross-language call file include, but are not limited to: setting an audio media format method, starting a decoding method, acquiring an input buffer index method, acquiring an output data method, stopping a decoding method, and the like.

After setting parameters (audio hard decoding parameters) required for audio hard decoding processing on audio data in an audio and video file by an application-level player, a plurality of function names are stored in a decoding module in the application-level player, and the function names corresponding to the audio hard decoding parameters include but are not limited to: setting audio media format, starting decoding, obtaining input buffer index, obtaining output data, stopping decoding, etc.

Therefore, the function name corresponding to the audio hard decoding parameter is matched with the function name in the cross-language calling file, so that whether the decoding module in the application-level player can create the audio hard decoder can be determined.

If the function name corresponding to the audio hard decoding parameter is matched and consistent with the function name in the cross-language calling file, the decoding module in the application-level player can create an audio hard decoder through the jni layer, and the decoding module in the application-level player calls the audio hard decoder.

And S3, calling the audio hard decoder to perform audio hard decoding processing on the audio data before decoding to obtain decoded audio data.

After the creation of the audio hard decoder comprising the hard decoding input thread and the hard decoding output thread is completed, the audio hard decoding processing can be performed on the audio data before decoding of the specified audio and video file. Since audio data obtained after audio hard decoding (MediaCodec) processing is in the form of a pbuffer structure (a storage unit of decoded audio data), decoded audio data in the form of pbuffer structure storage is obtained after a hard decoding input thread and a hard decoding output thread.

In some embodiments, the application-level player, after executing the step of invoking an audio hard decoder to perform audio hard decoding processing on the pre-decoding audio data to obtain decoded audio data, is further configured to:

step 301, calling a hard decoding input thread in an audio hard decoder to perform audio hard decoding processing on audio data before decoding to obtain decoded audio data;

step 302, writing the decoded audio data into a hard decoding output thread in the audio hard decoder to obtain the decoded audio data in a pbuffer structure storage form, where the pbuffer structure storage form is a storage form adopted by the audio hard decoder.

Because the audio decoder comprises the hard decoding input thread and the hard decoding output thread, the hard decoding input thread is called to carry out audio hard decoding on the audio data of the decoder to obtain the decoded audio data. And storing the decoded audio data to a hard decoding output thread to obtain the decoded audio data in a pbuffer structure storage form.

A data flow diagram for a hard decode input thread according to some embodiments is illustrated in fig. 9. In some embodiments, referring to fig. 9, in the step 301 of executing the application-level player, that is, calling a hard decoding input thread in the audio hard decoder to perform audio hard decoding processing on the pre-decoding audio data, the application-level player is further configured to execute the following steps:

step 311, call a hard decoding input thread in the audio hard decoder, and obtain the audio data before decoding and the input buffer index.

Step 312, writing the audio data before decoding into the input buffer index for audio hard decoding processing, so as to obtain the decoded audio data.

During the audio hard decoding process, the application-level player starts the audio hard decoder, i.e. calls the hard decoding input thread to perform the hard decoding process. And the hard decoding input thread is provided with an exit flag bit, and if the exit flag bit is identified, the hard decoding input thread exits. Therefore, when the hard decoding input thread is executed, whether the exit flag bit exists in the hard decoding input thread is identified in real time, and if the exit flag bit is not identified, the hard decoding process can be executed, namely, the pre-decoding audio data and the input buffer index required by the hard decoding processing are obtained.

Since the audio/video file is played while being decoded, the user may perform operations on the audio/video file, such as fast forward, fast backward, etc., during playing. When a user operates, redundant data can be generated, and in order to ensure the effect of hard decoding processing, the redundant data needs to be processed before the input buffer index is acquired.

To this end, in some embodiments, the application level player, prior to performing the obtaining the input buffer index, is further configured to perform the steps of:

step 3111, determining whether the audio data needs to be cleared based on a user operation when the specified audio/video file is played.

Step 3112, if the audio data needs to be cleared, clearing the audio data stored in the audio hard decoder.

Step 3113, if the audio data does not need to be cleared, executing the step of obtaining the input buffer index.

In some embodiments, the manner in which the redundant data is processed may be by flushing audio data. Therefore, if the application-level player recognizes that the user generates a corresponding operation, such as fast-backward, fast-forward, etc., it can determine that the audio data needs to be cleared, and at this time, the audio data stored in the audio hard decoder needs to be cleared. The audio data stored in the audio decoder is the audio data obtained by performing the hard decoding processing at the previous time.

The hard decoding input thread is a process of performing hard decoding processing on the audio data before decoding in a circulating manner, the hard decoding processing is performed once by the input buffer index every time one frame of audio data before decoding is acquired, and the decoded audio data is stored in the hard decoding output thread.

If the user does not operate the playing of the specified audio and video file, the audio data does not need to be cleared, and then the input buffer index can be obtained for hard decoding. Namely, the audio data before decoding is written into the input buffer index for audio hard decoding processing, and the decoded audio data is obtained.

When the decoding module (SDL _ amedia codec) in the application level player writes the pre-decoding audio data into the input buffer index, it needs to establish communication between the decoding module and the hard decoding input thread.

The decoding module in the application level player needs to be realized through a jni (Java Native Interface: Java cross-language intermediate module) layer when calling the audio hard decoder, namely, the decoding module in the application level player needs to transmit through the jni layer when writing the audio data before decoding into the input buffer index.

To this end, in some embodiments, the application level player performs an audio hard decoding process in performing writing of pre-decoded audio data into the input buffer index, further configured to perform the steps of:

3121, obtaining a function name corresponding to the audio data before decoding and a cross-language call file including the function name, where the cross-language call file is a file generated based on the operating environment parameters of the audio hard decoder.

And 3122, matching the function name corresponding to the audio data before decoding with the function name in the cross-language calling file.

And 3123, when the function names are matched, writing the audio data before decoding into the input buffer index, and performing audio hard decoding processing.

If a decoding module in an application-level player calls a hard decoding input thread in an audio hard decoder, a cross-language calling file needs to be acquired first. The cross-language call file is a file generated based on parameters required for calling the audio hard decoder when the application-level player creates the audio hard decoder, and the specific generation process may refer to the implementation processes of steps S211 to S212, which are not described herein again. After the file is generated, the file can be directly acquired for use when an audio hard decoder is subsequently called.

The cross-language calling file includes function names corresponding to a plurality of operating environment parameters, a decoding module in the application-level player stores a plurality of function names corresponding to the audio data before decoding, and the function names corresponding to the audio data before decoding include but are not limited to: setting audio media format, starting decoding, obtaining input buffer index, obtaining output data, stopping decoding, etc.

Therefore, matching the function name corresponding to the audio data before decoding with the function name in the cross-language call file can determine whether the decoding module in the application-level player can call the hard decoding input thread in the audio hard decoder.

If the function name corresponding to the audio data before decoding is matched with the function name in the cross-language calling file consistently, the communication between the decoding module in the application-level player and the audio hard decoder can be established through the jni layer, and the decoding module in the application-level player calls the hard decoding input thread. Therefore, after the decoding module communicates with the audio hard decoder, the decoding module in the application-level player can write the audio data before decoding into the hard decoding input thread, specifically, write the input buffer index, and perform hard decoding processing to obtain the decoded audio data.

And storing the decoded audio data into a hard decoding output thread for subsequent audio output. The audio hard decoder stores the decoded audio data in the form of the pbuffer structure, so that the decoded audio data in the form of the pbuffer structure can be obtained after the decoded audio data is stored in the hard decoding output thread.

FIG. 10 is a flow diagram illustrating a method of execution of a hard decoded output thread in accordance with some embodiments; a data flow diagram of a hard decode output thread according to some embodiments is illustrated in fig. 11. In some embodiments, referring to fig. 10 and 11, in step 302, the application-level player writes the decoded audio data into a hard decoding output thread in an audio hard decoder, resulting in decoded audio data in the form of pbuffer structure storage, and is further configured to:

s321, calling a hard decoding output thread in the audio hard decoder, and acquiring an output buffer index from the audio hard decoder.

And S322, writing the decoded audio data into an output buffer index to obtain the decoded audio data in a pbuffer structure storage form.

In the process of audio hard decoding, after a hard decoding input thread in an audio hard decoder finishes hard decoding of audio data before decoding of one frame, the obtained decoded audio data is stored in the audio hard decoder. When the audio data needs to be output for rendering processing to realize the playing of the specified audio and video file, the application-level player calls a hard decoding output thread to acquire the decoded audio data.

And the hard decoding output thread is provided with an exit flag bit, and if the exit flag bit is identified, the hard decoding output thread exits. Therefore, when the hard decoding output thread is executed, whether the exit flag bit exists in the hard decoding output thread is identified in real time, and if the exit flag bit is not identified, the process of acquiring the decoded audio data can be executed.

When the audio hard decoder stores the decoded audio data written in the input buffer index, the decoded audio data is stored in the output buffer index. Therefore, a hard decoding output thread is started, an output buffer index is obtained from the audio hard decoder, and the decoded audio data is stored in the output buffer index. Because the audio hard decoder adopts the pbuffer structure to store the audio data, the decoded audio data can be obtained after being stored to the output buffer index.

And S4, converting the decoded audio data into decoded audio data in an avframe structure storage form, wherein the avframe structure storage form is a storage form adopted by an application-level player.

Because the application-level player usually uses ffmpeg as a decapsulation tool, and the decapsulated audio storage data structure of the application-level player is highly matched with the avframe (decoded audio storage unit) used by the ffmpeg soft decoding module, the application-level player needs to unify the decoded audio data into an avframe format, that is, the decoded audio data stored in the pbuffer structure is converted into the decoded audio data stored in the avframe structure, so as to ensure the output of the decoded audio data.

A flowchart of a method of converting an avframe structure according to some embodiments is illustrated in fig. 12; a data flow diagram for the transform avframe structure is illustrated in fig. 13, in accordance with some embodiments. Referring to fig. 12 and 13, in some embodiments, the application-level player, in performing the conversion of the decoded audio data into decoded audio data in the form of an avframe structure store, is further configured to perform the following steps:

and S41, acquiring audio data output format information and decoded audio data in a pbuffer structure storage form, wherein the audio data output format information refers to information required by output in an avframe structure storage form.

And S42, acquiring the audio data offset from the decoded audio data in the storage form of the pbuffer structure.

And S43, obtaining the real decoded audio data in a pbuffer structure storage form based on the audio data offset and the decoded audio data.

S44, creating an avframe structure based on the real decoded audio data in the form of pbuffer structure storage.

And S45, writing the audio data output format information into the avframe structure to obtain the decoded audio data in the storage form of the avframe structure.

After being processed by the audio hard decoder, the obtained decoded audio data is usually in a storage form using a pbuffer structure. When the decoded audio data in the pbuffer structure storage form is converted into the decoded audio data in the avframe structure storage form, the audio data output format information required by the conversion structure needs to be acquired first, and the audio data output format information includes information required by the avframe structure storage form. And meanwhile, acquiring decoded audio data in a pbuffer structure storage form from the output buffer index.

When a decoding module (SDL _ amediatecodec) in an application level player acquires decoded audio data obtained by hard decoding processing from a hard decoding output thread, communication between the decoding module and the hard decoding output thread needs to be established.

When calling an audio hard decoder, a decoding module in the application level player needs to be realized through a jni (Java Native Interface: Java cross-language intermediate module) layer, that is, when the decoding module in the application level player acquires decoded audio data obtained by hard decoding from a hard decoding output thread (output buffer index), the decoding module needs to transmit through the jni layer.

In some embodiments, the application-level player, in performing the obtaining of the decoded audio data in the form of pbuffer structure storage, is further configured to perform the steps of:

step 411, obtaining a function name corresponding to the decoded audio data and a cross-language call file including the function name, where the cross-language call file is a file generated based on the operating environment parameters of the audio hard decoder.

And step 412, matching the function name corresponding to the decoded audio data with the function name in the cross-language calling file.

And 413, when the function names are matched and consistent, acquiring the decoded audio data in the storage form of the pbuffer structure from the output buffer index of the hard decoding output thread.

If the decoding module in the application-level player calls the output buffer index of the hard decoding output thread in the audio hard decoder, a cross-language calling file needs to be acquired first. The cross-language call file is a file generated based on parameters required for calling the audio hard decoder when the application-level player creates the audio hard decoder, and the specific generation process may refer to the implementation processes of steps S211 to S212, which are not described herein again. After the file is generated, the file can be directly acquired for use when an audio hard decoder is subsequently called.

After the decoded audio data is obtained, a plurality of function names are stored in a decoding module in the application-level player, and the function names corresponding to the decoded audio data include, but are not limited to: setting audio media format, starting decoding, obtaining input buffer index, obtaining output data, stopping decoding, etc. The cross-language calling file comprises function names corresponding to a plurality of operating environment parameters, so that the function names corresponding to the decoded audio data are matched with the function names in the cross-language calling file, and whether a decoding module in the application-level player can call an audio hard decoder can be determined, so that the decoded audio data can be obtained from an output buffer index in a hard decoding output thread.

If the function name corresponding to the decoded audio data is matched with the function name in the cross-language calling file consistently, the communication between the decoding module in the application-level player and the hard decoding output thread can be established through the jni layer, and the decoding module in the application-level player calls the hard decoding output thread. Therefore, after the decoding module communicates with the hard decoding output thread, the decoding module in the application-level player can acquire the decoded audio data in the storage form of the pbuffer structure from the output buffer index of the hard decoding output thread.

Because all decoded audio data obtained by the audio hard decoder through the hard decoding may not be valid data, it is necessary to acquire an audio data offset from the decoded audio data, and determine, based on the audio data offset, real decoded audio data that can be output and that adopts a pbuffer structure storage form.

For example, if the audio data offset of the decoded audio data stored in the pbuffer structure is 10 bytes, the audio data is shifted backward by 10 bytes from the first byte of the decoded audio data, i.e., the data of the first 10 bytes is deleted, and the data of the bytes from the 11 th byte and beyond is the real decoded audio data in the form stored in the pbuffer structure.

And creating an avframe structure used for ffmpeg decapsulation according to the real decoded audio data in the storage form of the pbuffer structure, so as to realize the conversion from the pbuffer structure to the avframe structure.

And then, writing the audio data output format information into an avframe structure to obtain the decoded audio data in the storage form of the avframe structure. The audio data output format information comprises a time stamp, the number of channels, the number of samples, the sampling rate and the like, and in order to decode the audio data into multiple channels, the audio data output format information corresponding to each channel is sequentially written into an avframe structure body based on the number of channels required by audio output.

After the writing of the audio data output format information with the specified number of channels is finished, when the number of channels in which data can be written is 0, the decoded audio data which has multiple channels, high sampling number and high sampling rate and adopts an avframe structure storage form can be obtained and further written into a decoded audio data queue.

And S5, writing the decoded audio data in the storage form of the avframe structure into an audio decoded data queue to play the specified audio and video file.

After the decoded audio data in the storage form of the avframe structure required by the output of the application-level player is obtained, in order to play a specified audio and video file, the decoded audio data in the storage form of the avframe structure is written into a decoded data queue, and specifically into an audio decoded data queue.

When the appointed audio and video file is played, the rendering module renders the decoded audio data in the decoded audio data queue in the decoded data queue, the decoded video data in the decoded video data queue and the decoded caption data in the decoded caption data queue, so that the appointed audio and video file is played.

Therefore, the display device provided by the embodiment of the invention can realize that the application-level player calls a hard decoding function, the audio data is decoded in a hard decoding mode, the dts dolby (dolby) sound effect can be preprocessed at the decoding end, and the audio data (2 channels before decoding) is decoded into 8 channels or 6 channels through decoding, so that the effect of enhancing the sound effect is achieved.

Since different application chips, such as Nova, Mtk, Hisi, etc., may be configured in the display device. Audio formats of OMX (openmax, android bottom layer docking decoding module) layers of different application chips are not uniform, and in order to ensure that a method for decoding audio data by an application-level player in a hard decoding mode can be applied to the OMX layers of different application chips, the application-level player needs to have a compatible characteristic in the hard decoding mode.

A method flow diagram for multi-chip OMX layer decoding format compatibility according to some embodiments is illustrated in fig. 14. Referring to fig. 14, in the display device provided by the embodiment of the present invention, on the basis of executing the foregoing audio hard decoding method of the application level player, the application level player is further configured to execute the following steps:

s61, calling a standard decoding interface, and obtaining a comparison table comprising decoding formats and bottom layer decoding names which are in one-to-one correspondence, wherein each bottom layer decoding name corresponds to an application chip.

And S62, determining the first bottom layer decoding name based on the decoding format corresponding to the comparison table and the audio hard decoding parameters.

And S63, obtaining a second bottom layer decoding name from the configured static file, and scoring the first bottom layer decoding name and the second bottom layer decoding name.

And S64, determining the bottom layer decoding name with the highest score as the target bottom layer decoding name, and establishing connection with the application chip corresponding to the target bottom layer decoding name.

In order to realize the compatibility of the decoding formats of the multi-chip OMX layer, the application-level player calls a standard decoding interface (MediaCodecList. getCodecinfo) to obtain all the supported decoding formats and bottom layer decoding names (omxtype), and each decoding format corresponds to one bottom layer decoding name to form a comparison table. Each underlying decoding name corresponds to an application chip.

And acquiring a decoding format corresponding to the audio hard decoding parameter, and searching the omxtype corresponding to the input format in the comparison table by a fuzzy matching method, namely searching the first bottom layer decoding name corresponding to the decoding format corresponding to the audio hard decoding parameter in the comparison table.

And simultaneously, reading a configured static file, wherein the static file comprises the decoding formats of the existing application chips and the corresponding omxtype, so that a second bottom layer decoding name corresponding to the decoding format corresponding to the audio hard decoding parameter can be determined.

And respectively scoring the first bottom layer decoding name and the second bottom layer decoding name, and finally taking the bottom layer decoding name omxtype with the highest score as the target bottom layer decoding name, wherein at the moment, the application-level player can be connected with an application chip corresponding to the target bottom layer decoding name to realize the multi-chip bottom layer difference compatibility.

Therefore, the display device provided by the embodiment of the invention can realize that the audiodec (android native decoder) hard decoding function is called by the audio decoding stream of the ijkplayer-based application-level player, so that the multichannel output support of the audiodec end is increased, and meanwhile, the audio hard decoding format diversity compatible support is provided for the existing chips Nova, Mtk and Hisi.

As can be seen from the foregoing technical solutions, in the audio hard decoding method and the display device of the application-level player provided in the embodiments of the present invention, the configured application-level player is configured to obtain an audio hard decoding parameter and audio data before decoding, and based on the audio hard decoding parameter, create an audio hard decoder including a hard decoding input thread and a hard decoding output thread; and calling a hard decoding input thread to perform audio hard decoding processing on the audio data before decoding, writing the obtained decoded audio data into the hard decoding output thread to obtain decoded audio data in a pbuffer structure storage form, converting the decoded audio data in the pbuffer structure storage form into decoded audio data in an avframe structure storage form, and writing the decoded audio data into an audio decoded data queue to play the specified audio and video file. Therefore, the method and the display device provided by the embodiment of the invention can enable the application-level player to decode the audio data by adopting a hard decoding mode, can preprocess the sound effect, and can decode the audio data into multiple channels through decoding to achieve the effect of enhancing the sound effect.

A flow diagram of a method of audio hard decoding for an application level player according to some embodiments is illustrated in fig. 6. Referring to fig. 6, the present application further provides an audio hard decoding method of an application-level player, which is executed by the application-level player in the display device provided in the foregoing embodiment, and the method includes:

s1, obtaining audio hard decoding parameters and audio data before decoding, wherein the audio data before decoding is audio data obtained by de-encapsulating a specified audio and video file, and the audio hard decoding parameters are parameters required by the audio hard decoding of the audio data before decoding;

s2, creating an audio hard decoder based on the audio hard decoding parameters;

s3, calling the audio hard decoder to perform audio hard decoding processing on the audio data before decoding to obtain decoded audio data;

s4, converting the decoded audio data into decoded audio data in an avframe structure storage form, wherein the avframe structure storage form is a storage form adopted by an application-level player;

s5, writing the decoded audio data in the storage form of the avframe structure into an audio decoded data queue to play the specified audio and video file.

In specific implementation, the present invention further provides a computer storage medium, where the computer storage medium may store a program, and the program may include some or all of the steps in the embodiments of the audio hard decoding method for an application-level player provided by the present invention when executed. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a Random Access Memory (RAM).

Those skilled in the art will readily appreciate that the techniques of the embodiments of the present invention may be implemented as software plus a required general purpose hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be essentially or partially implemented in the form of a software product, which may be stored in a storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments.

The same and similar parts in the various embodiments in this specification may be referred to each other. In particular, for the embodiment of the audio hard decoding method of the application level player, since it is basically similar to the embodiment of the display device, the description is relatively simple, and for the relevant points, refer to the description in the embodiment of the display device.

Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.

The foregoing description, for purposes of explanation, has been presented in conjunction with specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the embodiments to the precise forms disclosed above. Many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles and the practical application, to thereby enable others skilled in the art to best utilize the embodiments and various embodiments with various modifications as are suited to the particular use contemplated.

Claims

1. A display device, comprising:

creating an audio hard decoder based on the audio hard decoding parameters;

2. The display device of claim 1, wherein the application level player, in performing the creating an audio hard decoder based on the audio hard decoding parameters, is further configured to:

creating an audio hard decoder;

configuring the audio hard decoder based on the audio hard decoding parameters;

3. The display device of claim 2, wherein the application-level player, after performing the audio hard decoder to perform audio hard decoding on the pre-decoding audio data to obtain decoded audio data, is further configured to:

4. The display device of claim 2, wherein the application level player, in executing the create audio hard decoder, is further configured to:

when the function names match, an audio hard decoder is created.

5. The display device of claim 3, wherein the application-level player, in performing the invoking of the hard decode input thread in the audio hard decoder, is to perform audio hard decode processing on the pre-decode audio data, and is further configured to:

6. The display device of claim 5, wherein the application-level player, prior to performing the obtaining the input buffer index, is further configured to:

7. The display device of claim 3, wherein the application-level player, in executing the hard decode output thread that writes decoded audio data into the audio hard decoder, results in decoded audio data in the form of pbuffer structure storage, further configured to:

8. The display device of claim 3, wherein the application-level player, in performing the converting the decoded audio data into decoded audio data in the form of an avframe structure store, is further configured to:

9. The display device of claim 1, wherein the application-level player is further configured to:

10. A method for audio hard decoding for an application level player, the method comprising:

creating an audio hard decoder based on the audio hard decoding parameters;