WO2023142720A1 - 基于Linux桌面显卡的安卓视频解码硬件加速方法 - Google Patents

基于Linux桌面显卡的安卓视频解码硬件加速方法 Download PDF

Info

Publication number
WO2023142720A1
WO2023142720A1 PCT/CN2022/138089 CN2022138089W WO2023142720A1 WO 2023142720 A1 WO2023142720 A1 WO 2023142720A1 CN 2022138089 W CN2022138089 W CN 2022138089W WO 2023142720 A1 WO2023142720 A1 WO 2023142720A1
Authority
WO
WIPO (PCT)
Prior art keywords
video
android
linux
graphics card
video decoding
Prior art date
Application number
PCT/CN2022/138089
Other languages
English (en)
French (fr)
Inventor
马超
黄彩龙
李翔
黄晟
杨硕
彭韬
游远
赵坤
谢辉军
杨陈博寰
Original Assignee
麒麟软件有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 麒麟软件有限公司 filed Critical 麒麟软件有限公司
Publication of WO2023142720A1 publication Critical patent/WO2023142720A1/zh

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H04N21/440218—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/40—Transformation of program code
    • G06F8/41—Compilation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/60—Software deployment
    • G06F8/61—Installation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/8166—Monomedia components thereof involving executable data, e.g. software
    • H04N21/8193—Monomedia components thereof involving executable data, e.g. software dedicated tools, e.g. video decoder software or IPMP tool

Definitions

  • the invention relates to the technical field of Android video decoding, in particular to an Android video decoding hardware acceleration method based on a Linux desktop graphics card.
  • Android has a wide user group in the field of mobile terminals, and has a mature development platform and application software group, which has certain reference value for solving the current dilemma of the Linux operating system.
  • the Android operating system Since the Android operating system is developed based on the Linux kernel, it has made great improvements and enhancements in the user space for the characteristics of mobile terminal devices, so it provides great possibility and convenience for the Linux operating system to be compatible with Android applications, making mobile terminal devices Rich applications are likely to run successfully on the PC side.
  • the compatible support of the Android application ecosystem on the Linux operating system has become a research hotspot. Many researchers at home and abroad have tried to migrate the Android operating environment to a Linux-like computer platform to achieve the effect of compatible support for the Android application ecosystem on the Linux operating system. , derived from various compatible technologies, including Google's ARC (App Runtime for Chrome) technology, virtual machine and emulator technology, Anbox (Android in a Box) project, etc.
  • the purpose of the present invention is to provide a Linux desktop graphics card-based Android video decoding hardware acceleration method to solve the problem of low video decoding efficiency in existing Linux-compatible Android systems.
  • a kind of Android video decoding hardware acceleration method based on Linux desktop graphics card of the present invention is applied in the environment of Linux system compatible Android container, based on the graphics card device supporting VAAPI of Linux kernel and Linux desktop, the method specifically includes the following steps:
  • the video decoder initializes a video decoding hardware acceleration interface
  • the video decoder sends the video data packet to the video decoding hardware acceleration interface
  • the video decoding hardware acceleration interface transmits the data packet to the Linux kernel and decodes the data packet into a video frame by the graphics card device;
  • the video decoder retrieves the decoded video frame through the video decoding hardware acceleration interface
  • the video decoding service sends the video frame to a video program for display.
  • step S1 the following specific steps are also included before step S1:
  • the Android system and the Linux system share the same Linux kernel, and the Android system is run in the container of the Linux system to form an Android container;
  • step S2 specifically includes the following steps:
  • the FFmpeg video decoder initializes its own context configuration, and initializes the context of the video decoding hardware acceleration interface.
  • step S3 specifically includes the following steps:
  • the video application in the Android container generates a video data packet
  • the video decoding service in the Android container obtains the video data packet
  • the video decoder acquires video data packets from the input data stream of the video decoding service
  • the video decoder uses the FFmpeg interface to send the data to the libva library and the VAAPI implementation program related to the graphics card.
  • step S5 specifically includes the following steps:
  • the graphics card device completes video data decoding to generate a video frame
  • the video decoder retrieves data of the video frame from the graphics card device
  • the video decoder scales the data of the video frame.
  • step S6 specifically includes the following steps:
  • the video decoding service sends the data of the video frame to the video program of the client that calls the Android video decoding interface;
  • the client performs pixel format conversion and display range cropping on the video frame data
  • the client displays the processed video frame data.
  • a kind of Android video decoding hardware acceleration method based on Linux desktop graphics card connects the VAAPI of the Linux operating system to the Android container environment, and the Android container environment can realize video decoding hardware acceleration by using the Linux kernel, that is, Android can Directly operate the graphics card device through the Linux kernel, and on this basis, connect libva library, FFmpeg library, and VAAPI of Intel graphics card and AMD graphics card in Mesa to the Android container environment, provide Android environment, enable VAAPI function for FFmpeg library, and provide Android
  • the support for VAAPI in the environment reduces the performance loss caused by using the CPU for video decoding, and improves the efficiency of Android video decoding.
  • Fig. 1 is a schematic flow sheet of the present invention
  • FIG. 2 is a structural diagram of implementing video decoding hardware acceleration in the Android container environment of the present invention.
  • a kind of Android video decoding hardware acceleration method based on Linux desktop graphics card is applied in the environment where Linux system is compatible with Android container, based on the graphics card device supporting VAAPI of Linux kernel and Linux desktop, the method Specifically include the following steps:
  • the video decoder initializes the video decoding hardware acceleration interface, in this embodiment, initializes FFmpeg and uses the interface of FFmpeg to initialize VAAPI;
  • the video decoder sends the video data packet to the video decoding hardware acceleration interface
  • the video decoding hardware acceleration interface passes data packets into the Linux kernel and the graphics card device decodes the data packets into video frames.
  • the video decoding hardware acceleration interface passes data packets into the Linux kernel Afterwards, the data packet is decoded into a video frame by the GPU of the graphics card device;
  • the video decoder retrieves the decoded video frame through the video decoding hardware acceleration interface
  • the video decoding service sends the video frame to a video program for display.
  • Fig. 2 is a structural diagram of realizing hardware acceleration of video decoding in an Android container environment, as can be seen from the figure, Linux and Android share the same kernel (Linux kernel), and the Android environment runs in a container, that is, an Android container,
  • the Linux platform maps the graphics card device to the Android container.
  • the Android video decoding service uses FFmpeg to decode.
  • FFmpeg uses the libva library to load the graphics card-related va library.
  • the graphics card-related VA library includes the intel graphics card va library and the amd graphics card va library.
  • Video decoding is a structural diagram of realizing hardware acceleration of video decoding in an Android container environment, as can be seen from the figure, Linux and Android share the same kernel (Linux kernel), and the Android environment runs in a container, that is, an Android container,
  • the Linux platform maps the graphics card device to the Android container.
  • the Android video decoding service uses FFmpeg to decode.
  • FFmpeg uses the libva library to
  • step S1 the following specific steps are also included before step S1:
  • the Android system and the Linux system share the same Linux kernel, and the Android system is run in the container of the Linux system to form an Android container;
  • the video decoding of the native Android system does not provide hardware acceleration, and each mobile device manufacturer needs to implement a video decoding program supporting hardware acceleration for the mobile device according to the hardware.
  • the Android container running under the Linux desktop environment needs to use the related technologies that the Linux desktop can provide to realize the video decoding hardware acceleration in the Android container system.
  • video decoding hardware acceleration can be achieved through VAAPI.
  • VAAPI support can be provided in the native Android system.
  • the graphics card device needs to support VAAPI.
  • the graphics cards that support VAAPI include Intel and AMD graphics cards. In addition to installing the above graphics cards on the Linux desktop, it is also necessary to ensure that the Android container can use the graphics card device normally.
  • Linux needs to map the device nodes of the above graphics devices to Android.
  • the container and set reasonable permissions for the device node in the container to ensure that the Android environment can normally access the device node of the graphics card.
  • the Android system supports the OpenMax multimedia framework, and can load and use codec plug-ins that comply with the OpenMax standard. For this reason, the decoding program that supports VAAPI needs to be implemented as an OMX library. Considering that the types of graphics cards installed in the above-mentioned Linux desktop environment may be different, the video encoding formats supported by the video decoding hardware acceleration of different types of graphics cards are different.
  • FFmpeg is a commonly used codec library that supports CPU decoding of multiple video encodings and VAAPI hardware acceleration of video decoding. Therefore, FFmpeg is selected as the specific implementation of the video decoding program. For this purpose, FFmpeg needs to be connected to the Android container environment, including compiling FFmpeg
  • the library implements the OMX plug-in library based on FFmpeg, that is, the FFmpeg video decoder plug-in library, and transforms the Android multimedia framework to support loading the OMX plug-in library of FFmpeg.
  • step S2 specifically includes the following steps:
  • the FFmpeg video decoder initializes its own context configuration, and initializes the context of the video decoding hardware acceleration interface.
  • the libva library provides a general interface of VAAPI and selects and loads the VAAPI library provided by the graphics card according to the graphics card. Because different graphics cards have different specific implementations of video decoding hardware acceleration, each graphics card has a dedicated VAAPI implementation library. For example, the specific implementation for Intel graphics cards is i965_drv_video.so, and Mesa provides r600_drv_video.so and radeonsi_drv_video.so for AMD graphics cards. In addition, the FFmpeg library uses CPU decoding by default.
  • FFmpeg In order to use VAAPI hardware acceleration, you need to configure the FFmpeg compilation option, enable the decoding option of the relevant video stream format, and enable the VAAPI hardware acceleration option of the decoding option.
  • the FFmpeg decoding program When the FFmpeg decoding program is actually running, in addition to initializing its own decoder context, it also needs to initialize the video decoding hardware acceleration support (ie VAAPI) context.
  • VAAPI video decoding hardware acceleration support
  • step S3 specifically includes the following steps:
  • the video application in the Android container generates a video data packet
  • the video decoding service in the Android container obtains the video data packet
  • the video decoder acquires video data packets from the input data stream of the video decoding service
  • the video decoder uses the FFmpeg interface to send the data to the libva library and the VAAPI implementation program related to the graphics card.
  • the video data is generated by the Android application, and exists in the form of local files, network transmission byte streams, and the like.
  • the Android application For a data stream that contains both audio and video data, it is necessary to separate the audio stream from the video stream through the application or Android media.extractor, and then send the video stream data to the Android video decoding service in the form of packets.
  • the video decoding service creates different decoders for video stream formats according to the requirements of Android applications, such as H.264 decoders, H.265 decoders, etc.
  • the video decoding service gets the data packet, it passes the data packet to the decoder as the input of the decoder.
  • the decoder After the decoder gets the data packet, it uses the FFmpeg interface to hand over the data packet to the decoding program for decoding.
  • FFmpeg starts and initializes VAAPI
  • the data will be processed by the Linux kernel and graphics card device through libva and graphics card VAAPI implementation.
  • step S5 specifically includes the following steps:
  • the graphics card device completes video data decoding to generate a video frame
  • the video decoder retrieves data of the video frame from the graphics card device
  • the video decoder scales the data of the video frame.
  • the Linux desktop graphics card has better image processing capabilities and can be used for video decoding.
  • the UVD function of the graphics card is used for the AMD graphics card Mesa VAAPI.
  • the video decoder needs to retrieve the video frame data from the graphics card for further processing of the picture.
  • retrieving video frame data is also performed using the FFmpeg interface, and FFmpeg actually retrieves video frame data through the VAAPI interface.
  • the video frame data is in the YUV pixel format, and the YUV parameters of the video frame data need to be consistent with the YUV parameters of the display rendering program in the final display, so the video frame data needs to be scaled.
  • step S6 specifically includes the following steps:
  • the video decoding service sends the data of the video frame to the video program of the client that calls the Android video decoding interface;
  • the client performs pixel format conversion and display range cutting on the data of the video frame, wherein the process of this step is implemented inside the Android video decoding interface;
  • the client displays the processed video frame data.
  • the client's video program provides a carrier that can be used for display, a surface, and the Android video decoding interface internally fills data into this surface to realize display, but the process of filling data occurs in the client's video program process.
  • the decoding server sends the data to the decoding interface client program. Since the Surface used for display in the application may not support the YUV pixel format, it is necessary to convert the YUV pixel format to a pixel format that Surface can support, such as RGBA pixel format. For applications, the data can be further cropped to select the video area to be displayed. Finally, after pixel format conversion and range clipping, the data is sent to the display buffer queue of the application video Surface for display.
  • the Android video decoding hardware acceleration method based on the Linux desktop graphics card connects the VAAPI of the Linux operating system to the Android environment, and the Android container environment can realize video decoding hardware acceleration by using the Linux kernel, that is, Android can directly
  • the graphics card is operated through the kernel, and on this basis, the libva library, FFmpeg library, and the VAAPI of the Intel graphics card and the AMD graphics card in Mesa are connected to the Android environment, and the VAAPI function is enabled for the FFmpeg library in the Android environment.
  • the support of VAAPI reduces the performance loss caused by using CPU for video decoding, and improves the efficiency of Android video decoding.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Telephone Function (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

本发明公开了一种基于Linux桌面显卡的安卓视频解码硬件加速方法，应用在Linux系统兼容安卓容器的环境中，包括：S1、加载安卓容器的视频解码服务并初始化视频解码器；S2、视频解码器初始化视频解码硬件加速接口；S3、视频解码器将视频数据包发送至视频解码硬件加速接口；S4、视频解码硬件加速接口将数据包传入Linux内核并由显卡设备将数据包解码为视频帧；S5、视频解码器通过视频解码硬件加速接口取回已解码的视频帧；S6、视频解码服务将视频帧发送到视频程序进行显示，本发明减少使用CPU进行视频解码带来的性能损失，提高了Android视频解码的效率。

Description

基于Linux桌面显卡的安卓视频解码硬件加速方法 技术领域
本发明涉及Android视频解码技术领域，尤指一种基于Linux桌面显卡的安卓视频解码硬件加速方法。
 背景技术
近年来，随着Android（安卓）用户的增长以及Android软硬件产品的成熟和发展，Android应用生态呈现出爆发式发展，在很多领域都已经超过Windows应用生态和iOS应用生态，成为目前使用最广泛的应用生态。Android基于Linux内核发展而来，是目前移动终端市场中市场份额最高的Linux类操作系统，在移动终端领域具有极好的生态环境；Windows系统占据着PC市场的最大份额，其作为美国闭源操作系统，在信息网络安全方面存在巨大安全隐患；而Linux操作系统，其作为Android的前身，由于开发人员数目和市场份额的限制，发展较为缓慢，生态环境长期处于不良状态，一直不见起色。随着信息技术的飞速发展，国家对信息安全的重视程度日益提升，国内基于开源Linux操作系统的自主操作系统日益发展壮大，但Linux操作系统生态应用的严重缺乏仍旧是困扰Linux操作系统真正推向市场的一大痛点。
 技术问题
要解决Linux操作系统的生态问题，必须解决用户数不足和平台应用软件数不足这两个关键点。Android在移动终端领域具有广泛的用户群，并且具有成熟的开发平台和应用软件群体，对于解决Linux操作系统目前的窘境具有一定的参考价值。
由于Android操作系统基于Linux内核发展而来，其针对移动终端设备的特点在用户空间做了大幅的改进和增强，所以给Linux操作系统兼容Android应用提供了极大的可能性和便利，使得移动端的丰富应用很有可能成功运行在PC端。近年来，Android应用生态在Linux操作系统上的兼容支持成为了研究热点，国内外有许多研究人员尝试将Android运行环境迁移到Linux类计算机平台以实现Android应用生态在Linux操作系统上兼容支持的效果，各种兼容技术衍生而出，其中就包括Google的 ARC(App Runtime for Chrome)技术、虚拟机和模拟器技术、Anbox（Android in a Box）项目等，这些技术将PC端和移动端进行融合，成功将Android应用生态迁移到Linux上来，极大地推动了Linux应用生态的发展，很大程度上弥补了Linux应用缺乏的诟病，方便了Linux操作系统用户的日常办公、娱乐等需求，也加快了Linux操作系统走向消费市场的步伐，但它们都还存在着无法忽视的用户体验问题，包括稳定性、易用性、功能完备性等，导致这块技术领域没有显著的提升。
对于Linux桌面而言，通用硬件设备并不像Android移动设备一样配置了专门的硬件实现视频解码硬件加速，所以只能使用显卡的图像处理能力来实现视频解码硬件加速。Android模拟器和虚拟机由于无法直接访问到宿主机的硬件设备，目前采取的方式主要是实现纯软件的解码插件库，完全使用CPU解码，对CPU的使用率较高，加上模拟器和虚拟机方案本身具有一定的性能损耗，视频解码效率进一步降低。再比如Anbox，虽然Anbox不是虚拟机或模拟器，但由于采用的仍然是Android原本的纯软件解码器，解码效率完全依赖于CPU的性能，CPU的使用率也比较高。这种完全使用软件解码的方案，总体来说解码效率比较低，并且对Android视频应用的体验有影响。
 技术解决方案
本发明的目的，在于提供一种基于Linux桌面显卡的安卓视频解码硬件加速方法，以解决现有的Linux兼容安卓系统中存在的视频解码效率低的问题。
为解决上述技术问题，本发明是这样实现的：
本发明的一种基于Linux桌面显卡的安卓视频解码硬件加速方法，应用在Linux系统兼容安卓容器的环境中，基于Linux内核及Linux桌面的支持VAAPI的显卡设备，所述方法具体包括以下步骤：
S1、加载安卓容器的视频解码服务并初始化视频解码器；
S2、所述视频解码器初始化视频解码硬件加速接口；
S3、所述视频解码器将视频数据包发送至所述视频解码硬件加速接口；
S4、所述视频解码硬件加速接口将数据包传入Linux内核并由显卡设备将所述数据包解码为视频帧；
S5、所述视频解码器通过所述视频解码硬件加速接口取回已解码的视频帧；
S6、所述视频解码服务将所述视频帧发送到视频程序进行显示。
其中，在步骤S1之前还包括以下具体步骤：
S01、将安卓系统和Linux系统安装于同一Linux平台上；
S02、将所述安卓系统和所述Linux系统共享同一Linux内核，Linux系统的容器中运行安卓系统以构成安卓容器；
S03、在Linux平台上安装支持VAAPI的显卡设备，将支持VAAPI的显卡设备的设备节点映射到安卓容器中；
S04、将Linux系统的FFmpeg对接到安卓中，并实现为安卓的视频解码器插件库。
其中，步骤S2具体包括以下步骤：
S21、将Linux的libva库对接到安卓容器中；
S22、将符合VAAPI的显卡插件库对接到安卓容器中；
S23、所述FFmpeg视频解码器插件库开启VAAPI功能；
S24、所述FFmpeg视频解码器初始化自身的上下文配置，并初始化视频解码硬件加速接口的上下文。
其中，步骤S3具体包括以下步骤：
S31、所述安卓容器中的视频应用产生视频数据包；
S32、所述安卓容器中的视频解码服务获取视频数据包；
S33、所述视频解码器从视频解码服务的输入数据流中获取视频数据包；
S34、所述视频解码器使用FFmpeg接口将数据发送给libva库及显卡相关VAAPI实现程序。
其中，步骤S5具体包括以下步骤：
S51、所述显卡设备完成视频数据解码生成视频帧；
S52、所述视频解码器从所述显卡设备中取回所述视频帧的数据；
S53、所述视频解码器对视频帧的数据进行缩放。
其中，步骤S6具体包括以下步骤：
S61、所述视频解码服务将视频帧的数据发送给调用安卓视频解码接口的客户端的视频程序；
S62、所述客户端对视频帧的数据进行像素格式转换和显示范围裁剪；
S63、所述客户端将处理过的视频帧的数据进行显示。
 有益效果
本发明的有益效果为：
本发明提供的一种基于Linux桌面显卡的安卓视频解码硬件加速方法，将Linux操作系统的VAAPI对接到安卓容器环境中，安卓容器环境可以通过使用Linux内核来实现视频解码硬件加速，即让安卓可以直接通过Linux内核操作显卡设备，并在此基础上将libva库、FFmpeg库以及intel显卡和Mesa中AMD显卡的VAAPI对接到安卓容器环境中，提供安卓环境中，对FFmpeg库开启VAAPI功能，提供Android环境中对VAAPI的支持，减少使用CPU进行视频解码带来的性能损失，提高了Android视频解码的效率。
 附图说明
下面结合附图和具体实施方式对本发明作进一步详细的说明。
图1为本发明的流程示意图；
图2为本发明的安卓容器环境实现视频解码硬件加速的结构图。
 本发明的最佳实施方式
下面结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述。在下面的描述中阐述了很多具体细节以便于充分理解本发明，但是本发明还可以采用其他不同于在此描述的其它方式来实施，本领域技术人员可以在不违背本发明内涵的情况下做类似推广，因此本发明不受下面公开的具体实施例的限制。
如图1和图2所示，一种基于Linux桌面显卡的安卓视频解码硬件加速方法，应用在Linux系统兼容安卓容器的环境中，基于Linux内核及Linux桌面的支持VAAPI的显卡设备，所述方法具体包括以下步骤：
S1、加载安卓容器的视频解码服务并初始化视频解码器；
S2、所述视频解码器初始化视频解码硬件加速接口，在本实施例中，是初始化FFmpeg并使用FFmpeg的接口初始化VAAPI；
S3、所述视频解码器将视频数据包发送至所述视频解码硬件加速接口；
S4、所述视频解码硬件加速接口将数据包传入Linux内核并由显卡设备将所述数据包解码为视频帧，在本实施例中，所述视频解码硬件加速接口将数据包传入Linux内核后，由显卡设备的GPU将所述数据包解码为视频帧；
S5、所述视频解码器通过所述视频解码硬件加速接口取回已解码的视频帧；
S6、所述视频解码服务将所述视频帧发送到视频程序进行显示。
在实施例中，图2是安卓容器环境实现视频解码硬件加速的结构图，从图中可以看出，Linux和安卓共享同一个内核（Linux内核），安卓环境运行在容器中，即安卓容器，Linux平台将显卡设备映射到安卓容器中，安卓视频解码服务使用FFmpeg解码，FFmpeg使用libva库加载显卡相关va库，显卡相关va库包括intel显卡va库和amd显卡va库，va库使用显卡设备进行视频解码。
其中，在步骤S1之前还包括以下具体步骤：
S01、将安卓系统和Linux系统安装于同一Linux平台上；
S02、将所述安卓系统和所述Linux系统共享同一Linux内核，Linux系统的容器中运行安卓系统以构成安卓容器；
S03、在Linux平台上安装支持VAAPI的显卡设备，将支持VAAPI的显卡设备映射到安卓容器中；
S04、将Linux系统的FFmpeg对接到安卓中，并实现为安卓的视频解码器插件库。其中，该步骤的“对接”是将Linux上的FFmpeg源码在安卓里配置并编译成二进制库。
在本实施例中，原生安卓系统的视频解码不提供硬件加速，各个移动设备厂商需要根据硬件实现针对该移动设备的支持硬件加速的视频解码程序。Linux桌面环境下运行的安卓容器，需要使用Linux桌面所能提供的相关技术，实现安卓容器系统里的视频解码硬件加速。在Linux桌面上，可以通过VAAPI的方式实现视频解码硬件加速，在此条件下，可以在原生安卓系统里提供VAAPI支持，为此需要解决一些问题。首先是需要显卡设备支持VAAPI，支持VAAPI的显卡有Intel和AMD的显卡，Linux桌面除了安装上述显卡外，还需要保证安卓容器能够正常使用显卡设备，Linux需要将上述显卡设备的设备节点映射到安卓容器里，并且容器里为设备节点设置合理的权限保证安卓环境能正常访问显卡的设备节点。另外，安卓系统支持OpenMax多媒体框架，可以加载并使用符合OpenMax标准的编解码插件程序，为此需要将支持VAAPI的解码程序实现为OMX库。考虑到上述Linux桌面环境安装的显卡型号可能不同，不同型号显卡视频解码硬件加速支持的视频编码格式有所不同，当显卡设备解码硬件加速不支持某种视频编码格式时，还需要能灵活地切换回CPU解码。FFmpeg是一个常用的编解码库，支持多种视频编码的CPU解码和视频解码VAAPI硬件加速，因此选择FFmpeg作为视频解码程序的具体实现，为此需要将FFmpeg对接到安卓容器环境里，包括编译FFmpeg库，实现基于FFmpeg的OMX插件库，即FFmpeg视频解码器插件库，改造Android多媒体框架以支持加载FFmpeg的OMX插件库。
其中，步骤S2具体包括以下步骤：
S21、将Linux的libva库对接到安卓容器中，其中，该处的对接为：将Linux上的libva源码在安卓里编译成二进制库；
S22、将符合VAAPI的显卡插件库对接到安卓容器中；
S23、所述FFmpeg视频解码器插件库开启VAAPI功能；
S24、所述FFmpeg视频解码器初始化自身的上下文配置，并初始化视频解码硬件加速接口的上下文。
在本实施例中，为了使用VAAPI进行视频解码硬件加速，需要将Linux的libva库对接到安卓环境里，libva库提供VAAPI的通用接口并根据显卡选择加载显卡提供的VAAPI库。由于不同的显卡视频解码硬件加速的具体实现不同，各个显卡都有专门的VAAPI实现库，例如专门针对Intel显卡的具体实现有i965_drv_video.so，Mesa提供了AMD显卡的r600_drv_video.so和radeonsi_drv_video.so。另外，FFmpeg库默认是使用CPU解码的，为了使用VAAPI硬件加速，需要配置FFmpeg的编译选项，打开相关视频流格式的解码选项，并且打开该解码选项的VAAPI硬件加速选项。在FFmpeg解码程序实际运行时，除了初始化自身的解码器上下文，还需要初始化视频解码硬件加速支持（即VAAPI）的上下文，在初始化上下文过程中，FFmpeg使用libva库加载所述的显卡设备的VAAPI实现库，完成硬件加速接口及环境的初始化。
其中，步骤S3具体包括以下步骤：
S31、所述安卓容器中的视频应用产生视频数据包；
S32、所述安卓容器中的视频解码服务获取视频数据包；
S33、所述视频解码器从视频解码服务的输入数据流中获取视频数据包；
S34、所述视频解码器使用FFmpeg接口将数据发送给libva库及显卡相关VAAPI实现程序。
在本实施例中，视频数据是由安卓应用产生的，以本地文件、网络传输字节流等形式存在。对于同时包含音频与视频数据的数据流而言，需要首先通过应用或者Android media.extractor将音频流与视频流分开，再将视频流数据以包的形式发送到安卓视频解码服务。视频解码服务根据安卓应用的要求针对视频流格式创建不同的解码器，例如H.264解码器，H.265解码器等。视频解码服务拿到数据包之后，将数据包作为解码器的输入传递给解码器。解码器拿到数据包之后，使用FFmpeg接口将数据包交给解码程序进行解码。FFmpeg在开启并初始化VAAPI的情况下，会将数据通过libva和显卡VAAPI具体实现交由Linux内核和显卡设备处理。
其中，步骤S5具体包括以下步骤：
S51、所述显卡设备完成视频数据解码生成视频帧；
S52、所述视频解码器从所述显卡设备中取回所述视频帧的数据；
S53、所述视频解码器对视频帧的数据进行缩放。
在本实施例中，Linux桌面显卡具有较好的图像处理能力，可以用于视频解码，例如，对于AMD显卡Mesa VAAPI使用显卡的UVD功能。在显卡完成视频解码之后生成视频帧数据，视频解码器对画面进行进一步处理就需要从显卡中取回视频帧数据。同样的，取回视频帧数据也是使用FFmpeg的接口进行，FFmpeg实际通过VAAPI接口取回视频帧数据。一般情况下，视频帧数据是YUV像素格式的，在最终显示时需要视频帧数据YUV参数与显示渲染程序YUV参数保持一致，因此需要对视频帧数据进行缩放处理。
其中，步骤S6具体包括以下步骤：
S61、所述视频解码服务将视频帧的数据发送给调用安卓视频解码接口的客户端的视频程序；
S62、所述客户端对视频帧的数据进行像素格式转换和显示范围裁剪，其中，该步骤的过程是在安卓视频解码接口内部实现的；
S63、所述客户端将处理过的视频帧的数据进行显示。其中，客户端的视频程序提供了一个可以用来显示的载体，一个surface，安卓视频解码接口内部将数据填充到这个surface实现显示，只是这个填充数据的过程发生在客户端的视频程序进程中。
在本实施例中，视频解码器在处理视频帧数据后，解码服务器将数据发送到解码接口客户端程序。由于应用中用来显示的Surface可能并不支持YUV像素格式，所以此时还需要将YUV像素格式转化为Surface可以支持显示的像素格式，例如RGBA像素格式。对于应用，还可以对数据做进一步的裁剪以选择想要显示的视频区域。最终，进过像素格式转换和范围裁剪后，将数据送到应用视频Surface的显示缓冲队列中，进行显示。
本发明提供的一种基于Linux桌面显卡的安卓视频解码硬件加速方法将Linux操作系统的VAAPI实现对接到Android环境中，安卓容器环境可以通过使用Linux内核来实现视频解码硬件加速，即让安卓可以直接通过内核操作显卡，并在此基础上将libva库，FFmpeg库以及intel显卡和Mesa中AMD显卡的VAAPI实现对接到Android环境中，提供Android环境中，对FFmpeg库开启VAAPI功能，提供Android环境中对VAAPI的支持，减少使用CPU进行视频解码带来的性能损失，提高了安卓视频解码的效率。
应当理解的是，本发明的上述具体实施方式仅仅用于示例性说明或解释本发明的原理，而不构成对本发明的限制。因此，在不偏离本发明的精神和范围的情况下所做的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。此外，本发明所附权利要求旨在涵盖落入所附权利要求范围和边界、或者这种范围和边界的等同形式内的全部变化和修改例。

Claims (4)

  1. 一种基于Linux桌面显卡的安卓视频解码硬件加速方法，其特征在于，应用在Linux系统兼容安卓容器的环境中，基于Linux内核及Linux桌面的支持VAAPI的显卡设备，所述方法具体包括以下步骤：
    S01、将安卓系统和Linux系统安装于同一Linux平台上；
    S02、将所述安卓系统和所述Linux系统共享同一Linux内核，Linux系统的容器中运行安卓系统以构成安卓容器；
    S03、在Linux平台上安装支持VAAPI的显卡设备，将支持VAAPI的显卡设备的设备节点映射到安卓容器中；
    S04、将Linux系统的FFmpeg对接到安卓中，并实现为安卓的视频解码器插件库；
    S1、加载安卓容器的视频解码服务并初始化视频解码器；
    S2、所述视频解码器初始化视频解码硬件加速接口，步骤S2具体包括以下步骤：
    S21、将Linux的libva库对接到安卓容器中；
    S22、将符合VAAPI的显卡插件库对接到安卓容器中；
    S23、所述FFmpeg视频解码器插件库开启VAAPI功能；
    S24、所述FFmpeg视频解码器初始化自身的上下文配置，并初始化视频解码硬件加速接口的上下文；
    S3、所述视频解码器将视频数据包发送至所述视频解码硬件加速接口；
    S4、所述视频解码硬件加速接口将数据包传入Linux内核并由显卡设备将所述数据包解码为视频帧；
    S5、所述视频解码器通过所述视频解码硬件加速接口取回已解码的视频帧；
    S6、所述视频解码服务将所述视频帧发送到视频程序进行显示。
  2. 根据权利要求1所述的基于Linux桌面显卡的安卓视频解码硬件加速方法，其特征在于，步骤S3具体包括以下步骤：
    S31、所述安卓容器中的视频应用产生视频数据包；
    S32、所述安卓容器中的视频解码服务获取视频数据包；
    S33、所述视频解码器从视频解码服务的输入数据流中获取视频数据包；
    S34、所述视频解码器使用FFmpeg接口将数据发送给libva库及显卡相关VAAPI实现程序。
  3. 根据权利要求1所述的基于Linux桌面显卡的安卓视频解码硬件加速方法，其特征在于，步骤S5具体包括以下步骤：
    S51、所述显卡设备完成视频数据解码生成视频帧；
    S52、所述视频解码器从所述显卡设备中取回所述视频帧的数据；
    S53、所述视频解码器对视频帧的数据进行缩放。
  4. 根据权利要求1所述的基于Linux桌面显卡的安卓视频解码硬件加速方法，其特征在于，步骤S6具体包括以下步骤：
    S61、所述视频解码服务将视频帧的数据发送给调用安卓视频解码接口的客户端的视频程序；
    S62、所述客户端对视频帧的数据进行像素格式转换和显示范围裁剪；
    S63、所述客户端将处理过的视频帧的数据进行显示。
PCT/CN2022/138089 2022-01-26 2022-12-09 基于Linux桌面显卡的安卓视频解码硬件加速方法 WO2023142720A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210088457.7A CN114125558B (zh) 2022-01-26 2022-01-26 基于Linux桌面显卡的安卓视频解码硬件加速方法
CN202210088457.7 2022-01-26

Publications (1)

Publication Number Publication Date
WO2023142720A1 true WO2023142720A1 (zh) 2023-08-03

Family

ID=80361350

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/138089 WO2023142720A1 (zh) 2022-01-26 2022-12-09 基于Linux桌面显卡的安卓视频解码硬件加速方法

Country Status (2)

Country Link
CN (1) CN114125558B (zh)
WO (1) WO2023142720A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117075984A (zh) * 2023-10-17 2023-11-17 海马云（天津）信息技术有限公司 模块处理方法、装置、通信设备及计算机可读存储介质

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114125558B (zh) * 2022-01-26 2022-04-22 麒麟软件有限公司 基于Linux桌面显卡的安卓视频解码硬件加速方法
CN115580735B (zh) * 2022-12-08 2023-05-02 安徽海马云科技股份有限公司 视频编解码处理与系统、编解码服务器与插件模块
CN115695857B (zh) * 2022-12-29 2023-03-21 北京海誉动想科技股份有限公司 云应用的视频编码方法及装置
CN116193201A (zh) * 2023-02-27 2023-05-30 深圳数拓科技有限公司 一种视频数据处理方法
CN116450149B (zh) * 2023-06-16 2023-08-11 成都阿加犀智能科技有限公司 一种硬件解码方法、设备及存储介质
CN117234655B (zh) * 2023-11-13 2024-01-26 成都阿加犀智能科技有限公司 基于安卓系统运行Linux桌面程序的方法、装置、设备及介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112099923A (zh) * 2020-11-17 2020-12-18 江苏北弓智能科技有限公司 一种运行于Linux操作系统上的Android虚拟化系统
CN112269603A (zh) * 2020-10-16 2021-01-26 北京技德系统技术有限公司 一种在Linux上兼容运行Android应用的图形显示方法与装置
CN112511840A (zh) * 2020-12-24 2021-03-16 北京睿芯高通量科技有限公司 一种基于ffmpeg与硬件加速设备的解码系统及方法
CN113891087A (zh) * 2021-10-12 2022-01-04 深圳市算筹信息技术有限公司 一种基于Linux系统实现虚拟化硬件编解码加速的方法及系统
CN114125558A (zh) * 2022-01-26 2022-03-01 麒麟软件有限公司 基于Linux桌面显卡的安卓视频解码硬件加速方法

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011135352A1 (en) * 2010-04-26 2011-11-03 Hu-Do Limited A computing device operable to work in conjunction with a companion electronic device
CN106572361A (zh) * 2016-11-15 2017-04-19 中兴通讯股份有限公司 一种视频文件的处理方法、装置和系统
CN108881916A (zh) * 2018-06-21 2018-11-23 深圳市斯迈龙科技有限公司 远程桌面的视频优化处理方法及装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112269603A (zh) * 2020-10-16 2021-01-26 北京技德系统技术有限公司 一种在Linux上兼容运行Android应用的图形显示方法与装置
CN112099923A (zh) * 2020-11-17 2020-12-18 江苏北弓智能科技有限公司 一种运行于Linux操作系统上的Android虚拟化系统
CN112511840A (zh) * 2020-12-24 2021-03-16 北京睿芯高通量科技有限公司 一种基于ffmpeg与硬件加速设备的解码系统及方法
CN113891087A (zh) * 2021-10-12 2022-01-04 深圳市算筹信息技术有限公司 一种基于Linux系统实现虚拟化硬件编解码加速的方法及系统
CN114125558A (zh) * 2022-01-26 2022-03-01 麒麟软件有限公司 基于Linux桌面显卡的安卓视频解码硬件加速方法

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117075984A (zh) * 2023-10-17 2023-11-17 海马云（天津）信息技术有限公司 模块处理方法、装置、通信设备及计算机可读存储介质
CN117075984B (zh) * 2023-10-17 2023-12-26 海马云（天津）信息技术有限公司 模块处理方法、装置、通信设备及计算机可读存储介质

Also Published As

Publication number Publication date
CN114125558B (zh) 2022-04-22
CN114125558A (zh) 2022-03-01

Similar Documents

Publication Publication Date Title
WO2023142720A1 (zh) 基于Linux桌面显卡的安卓视频解码硬件加速方法
US10542301B2 (en) Multimedia redirection method, device, and system
CN104780378A (zh) 一种用于解码视频的方法、装置和播放器
US9183025B2 (en) Systems and algorithm for interfacing with a virtualized computing service over a network using a lightweight client
CN108881916A (zh) 远程桌面的视频优化处理方法及装置
US9128893B2 (en) Remote direct memory access socket aggregation
US10791160B2 (en) Method and apparatus for cloud streaming service
CN103888485A (zh) 云计算资源的分配方法、装置及系统
CN104159150A (zh) 云终端、云服务器、媒体数据流播放系统及方法
CN110968395B (zh) 一种在模拟器中处理渲染指令的方法及移动终端
JP2013541295A (ja) 画像圧縮のためのエントロピーコーダー
US20230214243A1 (en) One-machine multi-control method, apparatus, system and electronic device
CN112200712A (zh) Gles图像渲染方法及装置、存储介质、计算机设备
CN110807111A (zh) 三维图形的处理方法及装置、存储介质、电子设备
US20070234317A1 (en) Mechanism for reducing detectable pauses in dynamic output caused by dynamic compilation
CN112882826B (zh) 一种资源协同调度方法以及装置
CN114339415B (zh) 客户端视频播放方法、装置、电子设备和可读介质
CN114222185B (zh) 视频播放方法、终端设备及存储介质
WO2022135105A1 (zh) 功能机视频配乐方法、装置、终端设备及存储介质
Fu et al. Research on audio/video codec based on Android
CN108235822B (zh) 虚拟sim卡的实现方法及装置、存储介质、电子设备
KR20090122703A (ko) 변환 프로그램 제공 서버 및 방법
KR20090121949A (ko) 모바일 콘텐츠 변환 방법 및 시스템
CN113840173B (zh) 网页视频播放方法、装置、设备、存储介质及程序产品
CN116149777B (zh) 一种基于资源透明映射的跨环境纹理渲染优化方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22923509

Country of ref document: EP

Kind code of ref document: A1