WO2018119611A1 - 多操作系统多媒体数据编解码方法、装置、电子设备和计算机程序产品 - Google Patents

多操作系统多媒体数据编解码方法、装置、电子设备和计算机程序产品 Download PDF

Info

Publication number
WO2018119611A1
WO2018119611A1 PCT/CN2016/112207 CN2016112207W WO2018119611A1 WO 2018119611 A1 WO2018119611 A1 WO 2018119611A1 CN 2016112207 W CN2016112207 W CN 2016112207W WO 2018119611 A1 WO2018119611 A1 WO 2018119611A1
Authority
WO
WIPO (PCT)
Prior art keywords
hardware codec
operating system
multimedia data
hardware
request
Prior art date
Application number
PCT/CN2016/112207
Other languages
English (en)
French (fr)
Inventor
李向远
Original Assignee
深圳前海达闼云端智能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳前海达闼云端智能科技有限公司 filed Critical 深圳前海达闼云端智能科技有限公司
Priority to CN201680002805.7A priority Critical patent/CN106797470A/zh
Priority to PCT/CN2016/112207 priority patent/WO2018119611A1/zh
Publication of WO2018119611A1 publication Critical patent/WO2018119611A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/70—Media network packetisation

Definitions

  • the present application relates to the field of virtualization technologies, and in particular, to a multi-operating system multimedia data encoding and decoding method, apparatus, electronic device, and computer program product.
  • Mobile terminals have become an important tool for users' daily work and life. While mobile terminals are driving the rapid development of mobile Internet, mobile terminals contain rich and diverse information (such as enterprise data access, business communication, social networks, financial management, Games, etc.), mobile terminals have become the new focus of cybercrime. For information security and privacy protection, mobile virtualization has become the future development direction of mobile terminals.
  • FIG. 1 shows a schematic diagram of the virtualization system architecture.
  • the hardware layer of the system which mainly includes the processor, memory, and input and output devices.
  • the virtualization layer which runs a virtual machine monitor (abbreviated as VMM or Hypervisor).
  • VMM virtual machine monitor
  • the main function of the virtual machine monitor is to manage the real physical hardware platform and provide a corresponding virtual hardware platform for each guest virtual machine (ie, client).
  • the virtualized system architecture can run multiple operating systems, such as the Linux operating system or the Android operating system, which usually includes a host operating system (Host OS) and one or more guest operating systems (Guest OS), host operations.
  • Host OS host operating system
  • Guest OS guest operating systems
  • the system runs on the host (Host), running one or more clients on the host operating system, each client running as an application of the host operating system, which can simulate the hardware device for the guest operating system running on the client For use by the guest operating system, various applications can be run in the guest operating system running on each client.
  • Hos HyperText
  • clients running one or more clients on the host operating system
  • each client running as an application of the host operating system, which can simulate the hardware device for the guest operating system running on the client
  • various applications can be run in the guest operating system running on each client.
  • CPUs, memory, and input and output devices provided by the hardware layer can be shared among multiple operating systems.
  • the isolation provided by the virtualization technology can isolate multiple operating systems on a single terminal, and prevent processes on each operating system from adversely affecting other operating systems.
  • a typical application scenario is a mobile device such as a mobile phone through virtualization technology.
  • the separation of the enterprise system and the personal system is realized, and the potential harm caused by the process on the personal system is prevented, and the demand for the BYOD (Bring Your Own Device) is met by some enterprises with security requirements.
  • each virtual machine of the terminal needs a multimedia codec.
  • the Guest OS usually implements software codec through the CPU.
  • the codec mode has the following for the Guest OS.
  • the guest's Guest OS can only implement software codec through the CPU, which results in video files of many formats cannot be supported, playback is not smooth, and power consumption is large.
  • the embodiment of the present application proposes a multi-operating system multimedia data encoding and decoding method, device, electronic device and computer program product, which are mainly used to improve data encoding and decoding speed.
  • an embodiment of the present application provides a virtual multi-operating system multimedia data encoding and decoding method, where the method includes:
  • the driver of the hardware codec is invoked according to the hardware codec request to perform hardware codec processing on the multimedia data.
  • the embodiment of the present application provides a virtual multi-operating system multimedia data encoding and decoding device, where the device includes:
  • a hardware codec front-end module configured to acquire, in a guest operating system, a hardware codec request that the multimedia framework calls a hardware codec to perform a multimedia data codec process, where the hardware codec request includes a hardware codec process Multimedia data; transmitting the hardware codec request to a host operating system;
  • a hardware codec backend module configured to: in a host operating system, invoke a driver of the hardware codec to perform hardware codec processing on the multimedia data according to the hardware codec request.
  • an embodiment of the present application provides an electronic device, including: a memory, one or more processors; and one or more modules, the one or more modules being Stored in the memory and configured to be executed by the one or more processors, the one or more modules including instructions for performing the various steps of any of the above methods.
  • embodiments of the present application provide a computer program product for use in conjunction with an electronic device, the computer program product comprising a computer program embedded in a computer readable storage medium, the computer program comprising The electronic device is caused to execute instructions of each of the above methods.
  • the hardware codec request is executed by the hardware codec driver front end running on the guest operating system. Send to hardware code running on the host operating system The decoder driver backend, and then drives the real hardware in the host operating system to complete the encoding and decoding processing of the multimedia data.
  • This application enables the Guest OS to support the processing of multimedia files of more formats, accelerate the data processing process, and make the playback more Smooth; and reduce system power consumption.
  • FIG. 1 is a schematic diagram showing the architecture of a virtualization system in the prior art
  • FIG. 2 is a schematic diagram showing a virtualized system architecture of a multi-operating system multimedia data encoding and decoding method according to Embodiment 1 of the present application;
  • FIG. 3 is a schematic flowchart of a multi-operating system multimedia data encoding and decoding method in Embodiment 1 of the present application;
  • FIG. 4 is a schematic structural diagram of another virtualization system applied to a multi-operating system multimedia data encoding and decoding method according to Embodiment 1 of the present application;
  • FIG. 5 is a schematic structural diagram of a multi-operating system multimedia data encoding and decoding apparatus in Embodiment 2 of the present application;
  • FIG. 6 is a schematic structural diagram of an electronic device in Embodiment 3 of the present application.
  • the present application provides a multi-operating system multimedia data encoding and decoding method.
  • the client runs through the client.
  • Operating system hardware codec driver The front end sends the hardware codec request to the hardware codec driver backend running on the host operating system, thereby driving the real hardware in the host operating system to complete the encoding and decoding of the multimedia data.
  • This application enables the Guest OS to support more formats. Processing of multimedia files, speeding up the data processing process to make playback smoother; and reducing system power consumption.
  • Embodiment 1 is a diagrammatic representation of Embodiment 1:
  • FIG. 2 is a schematic diagram showing a virtualized system architecture of a multi-operating system multimedia data encoding and decoding method according to the first embodiment of the present application, where the virtualization system architecture includes at least one guest operating system as an example, and the client operates A multimedia application, such as a video player, is run in the system.
  • the multimedia application needs to encode and decode certain multimedia data
  • the multimedia data encoding and decoding request is sent to the multimedia framework, for example, playing in a video player.
  • a video data decoding request is sent to the multimedia framework.
  • FIG. 3 is a schematic flowchart of a multi-operating system multimedia data encoding and decoding method according to Embodiment 1 of the present application. As shown in FIG. 3, the multi-operating system multimedia data encoding and decoding method includes:
  • Step 101 In the guest operating system, acquire a hardware codec request that the multimedia framework calls the hardware codec component to perform multimedia data encoding and decoding processing, where the hardware codec request includes multimedia data that needs to be subjected to hardware codec processing;
  • the hardware codec request is sent to the host operating system;
  • Step 102 In the host operating system, the driver of the hardware codec is invoked according to the hardware codec request to perform hardware codec processing on the multimedia data.
  • the multimedia application running in the guest operating system sends a multimedia data encoding and decoding request to the multimedia framework when the multimedia data needs to be encoded and decoded.
  • the multimedia frame is based on the type and format of the multimedia data. Or the resolution or the like is selected for the appropriate codec for corresponding processing.
  • the hardware codec in the guest operating system is invoked to initiate a request for hardware codecing of the multimedia data.
  • the multimedia framework in the guest operating system in this embodiment It can be OpenMAX or other existing multimedia framework. When the multimedia framework is OpenMAX, it has one or more hardware codec components, and the multimedia framework initiates hardware coding of multimedia data by calling each hardware codec component. Decode the processed request.
  • the real hardware codec does not exist in the guest operating system, but the hardware codec driver front end runs in the kernel layer of the guest operating system. When the hardware codec driver front end acquires the multimedia frame, the hardware codec is called. And after forwarding the request to the hardware codec driver running in the kernel layer of the host operating system, after performing the request for performing hardware codec processing on the multimedia data, forwarding the request together with the multimedia data that needs to be subjected to hardware codec processing end.
  • the hardware codec request is sent to the host operating system based on cross-system communication.
  • the interaction of the data between the front end and the back end of the aforementioned hardware codec driver can be realized by the existing QEMU PIPE, Socket and other cross-system communication methods.
  • the hardware codec driver backend running in the kernel layer of the host operating system receives the multimedia data sent by the front end of the hardware codec driver in the guest operating system and performs hardware codec on the multimedia data. After processing the request, the driver of the hardware codec is run in the host operating system to complete the hardware codec processing of the multimedia data by the real hardware codec.
  • the hardware codec driver may be a V4L2Driver or the like.
  • the method before the step 101, further includes: determining, according to the multimedia data, a hardware codec that performs hardware codec processing on the multimedia data.
  • all codecs supported by the guest operating system can be stored in a pre-configured configuration file.
  • the multimedia framework can select a suitable codec for corresponding processing according to the type, format or resolution of the multimedia data that needs to be coded and processed, and needs to determine the multimedia data when it is judged that the hardware codec is required to perform corresponding processing.
  • Corresponding hardware codec. 4 is a schematic structural diagram of another virtualization system applied to a multi-operating system multimedia data encoding and decoding method according to Embodiment 1 of the present application.
  • the multimedia framework can learn the existing software/hardware codec that can be coded and processed, when the multimedia framework in the guest operating system is based on the multimedia to be processed.
  • the OpenMAX Core can be loaded through the vendorOMXPlugin plugin, and the components of the corresponding hardware codec are loaded to forward the codec processing request to the hardware through the unified hardware codec driver front end.
  • the decoder driver backend runs the appropriate hardware codec driver in the host operating system to drive the appropriate hardware codec to encode and decode the multimedia data.
  • the multimedia framework when an application in a guest operating system plays an AVC-encoded video file and needs to decode the video, the multimedia framework further determines when the hardware codec is used for decoding processing.
  • the AVC hardware decoder performs the decoding process to invoke a component of the AVC hardware decoder in the guest operating system, and forwards the decoding request to the hardware codec driver of the host operating system through the hardware codec driver front end end.
  • each device may have a front end of the hardware codec driver and a corresponding back end, that is, after running multiple layers of the host operating system kernel layer. At the end, multiple front ends are running at the guest operating system kernel level, not shown in FIG.
  • the method further comprises: feeding back the processed multimedia data to the guest operating system; and in the guest operating system, feeding the processed multimedia data to the multimedia framework.
  • the hardware codec driver backend in the host operating system may perform the encoding and decoding process.
  • the data is sent to the hardware codec driver front end in the guest operating system through cross-system communication.
  • the hardware codec driver front end in the guest operating system can feed back the codec processed data to the multimedia framework, so that the multimedia frame feeds back the codec processed data to initiate the original decoding request multimedia application. program.
  • the hardware codec in the guest operating system, when the upper multimedia frame requires the hardware codec to perform hardware codec processing on the multimedia data, the hardware codec is executed by the hardware codec driver front end running on the guest operating system. The request is sent to the hardware codec driver backend running on the host operating system, and then the hardware codec in the host operating system is called to complete the hardware codec processing of the multimedia data.
  • This application enables the Guest OS to support more formats of multimedia. The processing of the file speeds up the data processing process to make the playback smoother; and reduces system power consumption.
  • the embodiment further discloses that the multimedia framework of the guest operating system can select an appropriate hardware codec, and based on the driver front end and the back end of the hardware codec, loading an appropriate hardware codec in the main operating system is completed.
  • Encoding and decoding of multimedia data multimedia data that is encoded and processed in the main operating system can be fed back to the multimedia framework in the guest operating system through the driver back end and front end of the aforementioned hardware codec, and then fed back to the client operation
  • Upper-layer multimedia applications in the system data exchange between the front-end and back-end of the hardware codec driver in the host operating system and the guest operating system through cross-system communication.
  • Embodiment 2 is a diagrammatic representation of Embodiment 1:
  • the multi-operating system multimedia data encoding and decoding apparatus 200 includes:
  • the hardware codec front-end module 201 is configured to acquire, in the guest operating system, a hardware codec request that the multimedia frame calls a hardware codec to perform multimedia data encoding and decoding processing, where the hardware codec request includes hardware codec processing Multimedia data; sending the hardware codec request to a host operating system;
  • the hardware codec backend module 202 is configured to, in the host operating system, invoke a driver of the hardware codec to perform hardware codec processing on the multimedia data according to the hardware codec request.
  • the apparatus 200 further includes a multimedia framework module 203, configured to acquire, at the hardware codec front end module 201, a multimedia framework call hardware codec for multimedia data codec processing in a guest operating system. Before the hardware codec request, a hardware codec for performing hardware codec processing on the multimedia data is determined according to the multimedia data.
  • the hardware codec backend module 202 is further configured to: after performing a hardware codec process on the multimedia data by calling a driver of the hardware codec according to the hardware codec request, The processed multimedia data is fed back to the guest operating system;
  • the hardware codec front-end module 201 is further configured to feed the processed multimedia data to the multimedia framework in a guest operating system.
  • the transmitting the hardware codec request to a host operating system comprises transmitting the hardware codec request to a host operating system based on cross-system communication.
  • Embodiment 3 is a diagrammatic representation of Embodiment 3
  • the electronic device 300 includes: a memory 301, one or more processors 302; and one or more modules, the one or more modules being stored in the memory and configured to Executed by the one or more processors, the one or more modules include instructions for performing the various steps of any of the above methods.
  • the electronic device may be a mobile phone, a tablet computer, a robot, or other smart device.
  • Embodiment 4 is a diagrammatic representation of Embodiment 4:
  • the embodiment of the present application further provides a computer program product for use in combination with an electronic device, the computer program product being embedded in a computer readable storage medium.
  • a computer program comprising instructions for causing the electronic device to perform various steps of any of the above methods.
  • embodiments of the present application can be provided as a method, system, or computer program product.
  • the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment in combination of software and hardware.
  • the application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.
  • the computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device.
  • the apparatus implements the functions specified in one or more blocks of a flow or a flow and/or block diagram of the flowchart.
  • These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device.
  • the instructions provide steps for implementing the functions specified in a block or blocks of a flow or a flow and/or a block diagram of a flowchart Step.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Software Systems (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

本申请提供了多操作系统多媒体数据编解码方法、装置、电子设备和计算机程序产品，所述方法包括，在客户机操作系统中，获取多媒体框架调用硬件编解码器进行多媒体数据编解码处理的硬件编解码请求，所述硬件编解码请求包括需要进行硬件编解码处理的多媒体数据；将所述硬件编解码请求发送至主机操作系统；在主机操作系统中，根据所述硬件编解码请求调用所述硬件编解码器的驱动程序对所述多媒体数据进行硬件编解码处理。本申请使Guest OS能够支持更多格式的多媒体文件的处理，加速所述数据处理过程，使播放更流畅；并且降低系统功耗。

Description

多操作系统多媒体数据编解码方法、装置、电子设备和计算机程序产品 技术领域
本申请涉及虚拟化技术领域，特别涉及多操作系统多媒体数据编解码方法、装置、电子设备和计算机程序产品。
背景技术
移动终端已成为用户日常工作生活的重要工具，在移动终端推动移动互联网快速发展的同时，由于移动终端包含了丰富多样化的信息(例如：企业数据的访问、商务沟通、社交网络、财务管理、游戏等等)，移动终端成了网络犯罪的新焦点。为了信息安全、保护隐私，移动虚拟化成为了移动终端未来的发展方向。
图1示出了虚拟化系统架构示意图，处于底层的是系统的硬件层，主要包括处理器、内存和输入输出设备等。在硬件层之上为虚拟化层，运行的是虚拟机监控器(缩写为VMM或Hypervisor)。虚拟机监控器的主要职能是：管理真实的物理硬件平台，并为每个客户虚拟机(即客户机)提供对应的虚拟硬件平台。虚拟化系统架构中可以运行多个操作系统，例如Linux操作系统或者Android操作系统等，其中通常包括一个主机操作系统(Host OS)，以及一个或者多个客户机操作系统(Guest OS)，主机操作系统运行于宿主机(Host)，在主操作系统上运行一个或者多个客户机，各客户机作为主机操作系统的应用程序运行，其可以为运行于该客户机上的客户机操作系统模拟硬件设备以供客户机操作系统使用，在各客户机运行的客户机操作系统中可运行各种应用程序。在虚拟化系统架构中，多个操作系统之间可共享硬件层提供的CPU，内存和输入输出设备。
通过虚拟化技术带来的隔离性可实现单一终端上多个操作系统的隔离，避免各操作系统上的进程对其他操作系统产生不利影响，典型的应用场景为通过虚拟化技术在手机等移动设备上实现企业系统和个人系统的分离，防范个人系统上的进程给企业带来的潜在危害，满足一些具有安全要求的企业对“自带设备”(BYOD，Bring Your Own Device)的需求。
随着多媒体应用在生活中运用的深入发展，终端的各个虚拟机都需要有多媒体编解码器，现有技术中Guest OS通常通过CPU来实现软件编解码，此种编解码方式对于Guest OS有以下特点：
1)格式支持受限：
对于普通分辨率的AVI、RMVB等文件，绝大多数的CPU都可以胜任，随着高清视频(1080i/p)的普及，CPU变的越来越难以胜任。所以很多格式的视频文件在只有软件编解码的终端上是无法支持的。
2)CPU占用率高、播放不流畅、功耗大：
由于高清视频的分辨率远远高于一般格式视频，使得高清视频的码率非常高，再加上VC-1和H.264编码的压缩率很高，解码运算的运算量很大。因此常规地直接用CPU解码(即常说的“软解”)会极大地消耗CPU的运算能力，一些较老的CPU软解H.264时CPU占用率往往高达90％以上，更老的CPU可能就根本无法播放了。
可见现有技术不足在于：
终端的Guest OS只能通过CPU实现软件编解码，导致很多格式的视频文件无法支持、播放不流畅，而且功耗大。
发明内容
本申请实施例提出了多操作系统多媒体数据编解码方法、装置、电子设备和计算机程序产品，主要用以提高数据编解码速度。
在一个方面，本申请实施例提供了一种虚拟多操作系统多媒体数据编解码方法，其特征在于，所述方法包括：
在客户机操作系统中，获取多媒体框架调用硬件编解码器进行多媒体数据编解码处理的硬件编解码请求，所述硬件编解码请求包括需要进行硬件编解码处理的多媒体数据；将所述硬件编解码请求发送至主机操作系统；
在主机操作系统中，根据所述硬件编解码请求调用所述硬件编解码器的驱动程序对所述多媒体数据进行硬件编解码处理。
在另一个方面，本申请实施例提供了一种虚拟多操作系统多媒体数据编解码装置，其特征在于，所述装置包括：
硬件编解码器前端模块，用于在客户机操作系统中，获取多媒体框架调用硬件编解码器进行多媒体数据编解码处理的硬件编解码请求，所述硬件编解码请求包括需要进行硬件编解码处理的多媒体数据；将所述硬件编解码请求发送至主机操作系统；
硬件编解码器后端模块，用于在主机操作系统中，根据所述硬件编解码请求调用所述硬件编解码器的驱动程序对所述多媒体数据进行硬件编解码处理。
在另一个方面，本申请实施例提供了一种电子设备，其特征在于，所述电子设备包括：存储器，一个或多个处理器；以及一个或多个模块，所述一个或多个模块被存储在所述存储器中，并被配置成由所述一个或多个处理器执行，所述一个或多个模块包括用于执行任一上述方法中各个步骤的指令。
在另一个方面，本申请实施例提供了一种与电子设备结合使用的计算机程序产品，所述计算机程序产品包括内嵌于计算机可读的存储介质中的计算机程序，所述计算机程序包括用于使所述电子设备执行任一上述方法中的各个步骤的指令。
本申请实施例的有益效果如下：
本申请中，在客户机操作系统中，当上层多媒体框架需要硬件编解码器对多媒体数据进行硬件编解码处理时，通过运行于客户机操作系统的硬件编解码器驱动程序前端将硬件编解码请求发送至运行于主机操作系统的硬件编 解码器驱动程序后端，进而驱动主机操作系统中的真实硬件完成多媒体数据的编解码处理，本申请使Guest OS能够支持更多格式的多媒体文件的处理，加速所述数据处理过程，使播放更流畅；并且降低系统功耗。
附图说明
下面将参照附图描述本申请的具体实施例，其中：
图1示出了现有技术中虚拟化系统架构示意图；
图2示出了本申请实施例一中多操作系统多媒体数据编解码方法应用的虚拟化系统架构示意图；
图3示出了本申请实施例一中多操作系统多媒体数据编解码方法的流程示意图；
图4示出了本申请实施例一中多操作系统多媒体数据编解码方法应用的另一种虚拟化系统架构示意图；
图5示出了本申请实施例二中多操作系统多媒体数据编解码装置的结构示意图；
图6示出了本申请实施例三中电子设备的结构示意图。
具体实施方式
为了使本申请的技术方案及优点更加清楚明白，以下结合附图对本申请的示例性实施例进行进一步详细的说明，显然，所描述的实施例仅是本申请的一部分实施例，而不是所有实施例的穷举。并且在不冲突的情况下，本说明中的实施例及实施例中的特征可以互相结合。
发明人在发明过程中注意到：终端的Guest OS只能通过CPU实现软件编解码，导致很多格式的视频文件无法支持、播放不流畅，而且功耗大。
针对上述不足，本申请提供了一种多操作系统多媒体数据编解码方法，在客户机操作系统中，当上层多媒体框架需要硬件编解码器对多媒体数据进行硬件编解码处理时，通过运行于客户机操作系统的硬件编解码器驱动程序 前端将硬件编解码请求发送至运行于主机操作系统的硬件编解码器驱动程序后端，进而驱动主机操作系统中的真实硬件完成多媒体数据的编解码处理，本申请使Guest OS能够支持更多格式的多媒体文件的处理，加速所述数据处理过程，使播放更流畅；并且降低系统功耗。
为了便于本申请的实施，下面以实例进行说明。
实施例一：
图2示出了本申请实施例一中多操作系统多媒体数据编解码方法应用的虚拟化系统架构示意图，以所述虚拟化系统架构中至少包括一个客户机操作系统为例，在该客户机操作系统中运行有多媒体应用程序，例如视频播放器等，当所述多媒体应用程序需要对某些多媒体数据进行编解码处理时，会向多媒体框架下发多媒体数据编解码请求，例如在视频播放器播放高清视频时会向多媒体框架下发视频数据解码请求。
图3示出了本申请实施例一中多操作系统多媒体数据编解码方法流程示意图，如图3所示，所述多操作系统多媒体数据编解码方法包括：
步骤101，在客户机操作系统中，获取多媒体框架调用硬件编解码器组件进行多媒体数据编解码处理的硬件编解码请求，所述硬件编解码请求包括需要进行硬件编解码处理的多媒体数据；将所述硬件编解码请求发送至主机操作系统；
步骤102，在主机操作系统中，根据所述硬件编解码请求调用所述硬件编解码器的驱动程序对所述多媒体数据进行硬件编解码处理。
在步骤101中，客户机操作系统中运行的多媒体应用程序在需要对某些多媒体数据进行编解码处理时，会向多媒体框架下发多媒体数据编解码请求，多媒体框架会根据多媒体数据的类型、格式或者分辨率等为其选择适当的编解码器进行相应处理。当多媒体框架判断所述多媒体数据需要由硬件编解码器进行相应处理时，调用客户机操作系统中的硬件编解码器发起对所述多媒体数据进行硬件编解码的请求。本实施例中客户机操作系统中的多媒体框架 可以为OpenMAX或者其他现有的多媒体框架，当所述多媒体框架为OpenMAX时，其具有一个或者多个硬件编解码器组件Component，多媒体框架通过调用各硬件编解码器组件发起对多媒体数据进行硬件编解码处理的请求。
在客户机操作系统中不存在真实的硬件编解码器，而是在客户机操作系统内核层中运行了硬件编解码器驱动程序前端，当硬件编解码器驱动程序前端获取多媒体框架调用硬件编解码器组件，以进行对所述多媒体数据进行硬件编解码处理的请求后，将所述请求连同需要进行硬件编解码处理的多媒体数据转发至主机操作系统内核层中运行的硬件编解码器驱动程序后端。
在一些实施方式中，将所述硬件编解码请求基于跨系统通信发送至主机操作系统。
前述硬件编解码器驱动程序的前端和后端间数据的交互可通过现有的QEMU PIPE、Socket等跨系统通信方式实现。
在步骤102中，主机操作系统内核层中运行的硬件编解码器驱动程序后端，当接收到客户机操作系统中硬件编解码器驱动程序前端发送的多媒体数据以及对该多媒体数据进行硬件编解码处理的请求后，在主机操作系统中运行硬件编解码器的驱动程序，以由真正的硬件编解码器完成对所述多媒体数据的硬件编解码处理。所述硬件编解码器驱动程序可以为V4L2Driver等。
在一些实施方式中，在上述步骤101之前还包括，根据所述多媒体数据确定对所述多媒体数据进行硬件编解码处理的硬件编解码器。
在多媒体框架中，在预先设置的配置文件中可以存储客户机操作系统支持的所有编解码器，包括各种硬件编解码器和软件编解码器。多媒体框架能够根据需要进行编解码处理的多媒体数据的类型、格式或者分辨率等为其选择适当的编解码器进行相应处理，当判断需要硬件编解码器进行相应处理时，需要确定所述多媒体数据对应的硬件编解码器。图4示出了本申请实施例一中多操作系统多媒体数据编解码方法应用的另一种虚拟化系统架构示意图， 以客户机操作系统为Andriod系统，其中多媒体框架为OpenMAX为例，在虚拟化系统架构中存在多种硬件编解码器，例如AVC(高级视频编码，Advanced Video Coding)硬件编解码器、MP4(Moving Picture Experts Group 4)硬件编解码器和其他的硬件编解码器等。通过修改客户机操作系统多媒体框架中的配置文件media_codec.xml，使多媒体框架能够获知可进行编解码处理的已有软件/硬件编解码器，当客户机操作系统中的多媒体框架根据待处理的多媒体数据确定了某一适当的硬件编解码器后，可通过vendorOMXPlugin插件加载OpenMAX Core，进而加载相应的硬件编解码器的组件将编解码处理请求经由统一的硬件编解码器驱动程序前端转发至硬件编解码器驱动程序后端，在主操作系统中运行适当的硬件编解码器驱动程序驱动适当的硬件编解码器对多媒体数据进行编解码处理。
以视频解码为例，当客户机操作系统系统中的应用程序播放AVC编码的视频文件，需要对该视频进行解码时，多媒体框架当判断需要采用硬件编解码器进行解码处理时，还进一步确定采用AVC硬件解码器进行所述解码处理，以便调用客户机操作系统中AVC硬件解码器的组件，将所述解码请求通过硬件编解码器驱动程序前端转发至主机操作系统的硬件编解码器驱动程序后端。
此外，当真实的硬件编解码器涉及多个硬件设备时，对应每个设备可能各有一个硬件编解码器驱动程序的前端和对应的后端，即在主机操作系统内核层运行了多个后端，在客户机操作系统内核层运行了多个前端，图4中未示出。
在一些实施方式中，在上述步骤102之后，还包括将处理后的多媒体数据反馈所述客户机操作系统；在客户机操作系统中，将所述处理后的多媒体数据反馈所述多媒体框架。
在主机操作系统的硬件编解码器对所述多媒体数据完成编解码处理之后，主机操作系统中的硬件编解码器驱动程序后端可以将进行编解码处理后 的数据通过跨系统通信发送至客户机操作系统中的硬件编解码器驱动程序前端。客户机操作系统中的硬件编解码器驱动程序前端在接收到编解码处理后的数据后可将其反馈至多媒体框架，以便多媒体框架将编解码处理后的数据反馈发起原编解码请求的多媒体应用程序。
本实施例中，在客户机操作系统中，当上层多媒体框架需要硬件编解码器对多媒体数据进行硬件编解码处理时，通过运行于客户机操作系统的硬件编解码器驱动程序前端将硬件编解码请求发送至运行于主机操作系统的硬件编解码器驱动程序后端，进而调用主机操作系统中的硬件编解码器完成多媒体数据的硬件编解码处理，本申请使Guest OS能够支持更多格式的多媒体文件的处理，加速所述数据处理过程，使播放更流畅；并且降低系统功耗。本实施例还公开了客户机操作系统的多媒体框架可以选择适当的硬件编解码器，并基于该硬件编解码器的驱动程序前端和后端实现在主操作系统中加载适当的硬件编解码器完成多媒体数据的编解码处理；在主操作系统中完成编解码处理的多媒体数据可通过前述硬件编解码器的驱动程序后端和前端反馈至客户机操作系统中的多媒体框架，进而反馈至客户机操作系统中上层多媒体应用程序；主机操作系统和客户机操作系统中的硬件编解码器驱动程序的前端和后端间可通过跨系统通信方式实现数据交互。
实施例二：
基于同一发明构思，本申请实施例中还提供了一种多操作系统多媒体数据编解码装置，由于这些装置解决问题的原理与多操作系统多媒体数据编解码方法相似，因此这些装置的实施可以参见方法的实施，重复之处不再赘述。如图5所示，所述多操作系统多媒体数据编解码装置200包括：
硬件编解码器前端模块201，用于在客户机操作系统中，获取多媒体框架调用硬件编解码器进行多媒体数据编解码处理的硬件编解码请求，所述硬件编解码请求包括需要进行硬件编解码处理的多媒体数据；将所述硬件编解码请求发送至主机操作系统；
硬件编解码器后端模块202，用于在主机操作系统中，根据所述硬件编解码请求调用所述硬件编解码器的驱动程序对所述多媒体数据进行硬件编解码处理。
在一些实施方式中，所述装置200还包括多媒体框架模块203，用于在客户机操作系统中，在所述硬件编解码器前端模块201获取多媒体框架调用硬件编解码器进行多媒体数据编解码处理的硬件编解码请求之前，根据所述多媒体数据确定对所述多媒体数据进行硬件编解码处理的硬件编解码器。
在一些实施方式中，所述硬件编解码器后端模块202还用于，在根据所述硬件编解码请求调用所述硬件编解码器的驱动程序对所述多媒体数据进行硬件编解码处理之后，将处理后的多媒体数据反馈所述客户机操作系统；
所述硬件编解码器前端模块201还用于，在客户机操作系统中，将所述处理后的多媒体数据反馈所述多媒体框架。
在一些实施方式中，所述将所述硬件编解码请求发送至主机操作系统，包括：将所述硬件编解码请求基于跨系统通信发送至主机操作系统。
实施例三：
基于同一发明构思，本申请实施例中还提供了一种电子设备，由于其原理与多操作系统多媒体数据编解码方法相似，因此其实施可以参见方法的实施，重复之处不再赘述。如图6所示，所述电子设备300包括：存储器301，一个或多个处理器302；以及一个或多个模块，所述一个或多个模块被存储在所述存储器中，并被配置成由所述一个或多个处理器执行，所述一个或多个模块包括用于执行任一上述方法中各个步骤的指令。
本实施例中，所述电子设备可以为手机、平板电脑、机器人或者其他智能设备。
实施例四：
基于同一发明构思，本申请实施例还提供了一种与电子设备结合使用的计算机程序产品，所述计算机程序产品包括内嵌于计算机可读的存储介质中 的计算机程序，所述计算机程序包括用于使所述电子设备执行上述任一方法中的各个步骤的指令。
为了描述的方便，以上所述装置的各部分以功能分为各种模块分别描述。当然，在实施本申请时可以把各模块或单元的功能在同一个或多个软件或硬件中实现。
本领域内的技术人员应明白，本申请的实施例可提供为方法、系统、或计算机程序产品。因此，本申请可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本申请可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本申请是参照根据本申请实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步 骤。
尽管已描述了本申请的优选实施例，但本领域内的技术人员一旦得知了基本创造性概念，则可对这些实施例作出另外的变更和修改。所以，所附权利要求意欲解释为包括优选实施例以及落入本申请范围的所有变更和修改。

Claims (10)

  1. 一种多操作系统多媒体数据编解码方法，其特征在于，所述方法包括：
    在客户机操作系统中，获取多媒体框架调用硬件编解码器进行多媒体数据编解码处理的硬件编解码请求，所述硬件编解码请求包括需要进行硬件编解码处理的多媒体数据；将所述硬件编解码请求发送至主机操作系统；
    在主机操作系统中，根据所述硬件编解码请求调用所述硬件编解码器的驱动程序对所述多媒体数据进行硬件编解码处理。
  2. 如权利要求1所述的方法，其特征在于，在所述在客户机操作系统中，获取多媒体框架调用硬件编解码器进行多媒体数据编解码处理的硬件编解码请求之前，还包括：
    根据所述多媒体数据确定对所述多媒体数据进行硬件编解码处理的硬件编解码器。
  3. 如权利要求1所述的方法，其特征在于，在所述在主机操作系统中，根据所述硬件编解码请求调用所述硬件编解码器的驱动程序对所述多媒体数据进行硬件编解码处理之后，还包括：
    将处理后的多媒体数据反馈所述客户机操作系统；
    在客户机操作系统中，将所述处理后的多媒体数据反馈所述多媒体框架。
  4. 如权利要求1所述的方法，其特征在于，所述将所述硬件编解码请求发送至主机操作系统，包括：
    将所述硬件编解码请求基于跨系统通信发送至主机操作系统。
  5. 一种多操作系统多媒体数据编解码装置，其特征在于，所述装置包括：
    硬件编解码器前端模块，用于在客户机操作系统中，获取多媒体框架调用硬件编解码器进行多媒体数据编解码处理的硬件编解码请求，所述硬件编解码请求包括需要进行硬件编解码处理的多媒体数据；将所述硬件编解码请求发送至主机操作系统；
    硬件编解码器后端模块，用于在主机操作系统中，根据所述硬件编解码请求调用所述硬件编解码器的驱动程序对所述多媒体数据进行硬件编解码处理。
  6. 如权利要求5所述的装置，其特征在于所述装置还包括：
    多媒体框架模块，用于在客户机操作系统中，在所述硬件编解码器前端模块获取多媒体框架调用硬件编解码器进行多媒体数据编解码处理的硬件编解码请求之前，根据所述多媒体数据确定对所述多媒体数据进行硬件编解码处理的硬件编解码器。
  7. 如权利要求5所述的装置，其特征在于，
    所述硬件编解码器后端模块还用于，在根据所述硬件编解码请求调用所述硬件编解码器的驱动程序对所述多媒体数据进行硬件编解码处理之后，将处理后的多媒体数据反馈所述客户机操作系统；
    所述硬件编解码器前端模块还用于，在客户机操作系统中，将所述处理后的多媒体数据反馈所述多媒体框架。
  8. 如权利要求5所述的装置，其特征在于，所述将所述硬件编解码请求发送至主机操作系统，包括：
    将所述硬件编解码请求基于跨系统通信发送至主机操作系统。
  9. 一种电子设备，其特征在于，所述电子设备包括：
    存储器，一个或多个处理器；以及一个或多个模块，所述一个或多个模块被存储在所述存储器中，并被配置成由所述一个或多个处理器执行，所述 一个或多个模块包括用于执行权利要求1-4中任一所述方法中各个步骤的指令。
  10. 一种与电子设备结合使用的计算机程序产品，所述计算机程序产品包括内嵌于计算机可读的存储介质中的计算机程序，所述计算机程序包括用于使所述电子设备执行权利要求1至4中任一所述方法中的各个步骤的指令。
PCT/CN2016/112207 2016-12-26 2016-12-26 多操作系统多媒体数据编解码方法、装置、电子设备和计算机程序产品 WO2018119611A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201680002805.7A CN106797470A (zh) 2016-12-26 2016-12-26 多操作系统多媒体数据编解码方法、装置、电子设备和计算机程序产品
PCT/CN2016/112207 WO2018119611A1 (zh) 2016-12-26 2016-12-26 多操作系统多媒体数据编解码方法、装置、电子设备和计算机程序产品

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2016/112207 WO2018119611A1 (zh) 2016-12-26 2016-12-26 多操作系统多媒体数据编解码方法、装置、电子设备和计算机程序产品

Publications (1)

Publication Number Publication Date
WO2018119611A1 true WO2018119611A1 (zh) 2018-07-05

Family

ID=58952232

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/112207 WO2018119611A1 (zh) 2016-12-26 2016-12-26 多操作系统多媒体数据编解码方法、装置、电子设备和计算机程序产品

Country Status (2)

Country Link
CN (1) CN106797470A (zh)
WO (1) WO2018119611A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112788343A (zh) * 2021-02-23 2021-05-11 北京睿芯高通量科技有限公司 一种用于视频处理服务端的软硬件编码协同系统、方法及电子装置
CN113050998A (zh) * 2021-03-30 2021-06-29 联想(北京)有限公司 电子设备及控制方法

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018119711A1 (zh) * 2016-12-27 2018-07-05 深圳前海达闼云端智能科技有限公司 多操作系统多媒体编解码方法、装置及电子设备
CN110198427B (zh) * 2018-07-05 2021-09-21 腾讯科技（深圳）有限公司 视频通话方法和装置
CN110830748A (zh) * 2019-12-19 2020-02-21 广东以诺通讯有限公司 一种视频通话方法及系统

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100299665A1 (en) * 2009-05-19 2010-11-25 Vmware, Inc. Interposition method suitable for hardware-assisted virtual machine
CN103139634A (zh) * 2011-11-30 2013-06-05 国云科技股份有限公司 云终端虚拟桌面视频播放方法
CN103810021A (zh) * 2014-02-19 2014-05-21 福建升腾资讯有限公司 一种vdi下基于宿主机的虚拟桌面显示方法及其系统
CN104023266A (zh) * 2014-05-27 2014-09-03 烽火通信科技股份有限公司 一种android系统的通信编解码组件的使用方法
CN104780378A (zh) * 2015-04-16 2015-07-15 腾讯科技（北京）有限公司 一种用于解码视频的方法、装置和播放器
US20150334126A1 (en) * 2014-05-15 2015-11-19 Lynx Software Technologies, Inc. Systems and Methods Involving Aspects of Hardware Virtualization Such as Hypervisor, Detection and Interception of Code or Instruction Execution Including API Calls, and/or Other Features
CN106210036A (zh) * 2016-07-08 2016-12-07 中霆云计算科技（上海）有限公司 虚拟桌面呈现设备中的视频显示加速方法

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101847105B (zh) * 2009-03-26 2013-09-04 联想(北京)有限公司 一种计算机及多操作系统共享内存的方法
CN102981887B (zh) * 2011-09-06 2016-07-06 联想(北京)有限公司 数据处理方法及电子设备

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100299665A1 (en) * 2009-05-19 2010-11-25 Vmware, Inc. Interposition method suitable for hardware-assisted virtual machine
CN103139634A (zh) * 2011-11-30 2013-06-05 国云科技股份有限公司 云终端虚拟桌面视频播放方法
CN103810021A (zh) * 2014-02-19 2014-05-21 福建升腾资讯有限公司 一种vdi下基于宿主机的虚拟桌面显示方法及其系统
US20150334126A1 (en) * 2014-05-15 2015-11-19 Lynx Software Technologies, Inc. Systems and Methods Involving Aspects of Hardware Virtualization Such as Hypervisor, Detection and Interception of Code or Instruction Execution Including API Calls, and/or Other Features
CN104023266A (zh) * 2014-05-27 2014-09-03 烽火通信科技股份有限公司 一种android系统的通信编解码组件的使用方法
CN104780378A (zh) * 2015-04-16 2015-07-15 腾讯科技（北京）有限公司 一种用于解码视频的方法、装置和播放器
CN106210036A (zh) * 2016-07-08 2016-12-07 中霆云计算科技（上海）有限公司 虚拟桌面呈现设备中的视频显示加速方法

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112788343A (zh) * 2021-02-23 2021-05-11 北京睿芯高通量科技有限公司 一种用于视频处理服务端的软硬件编码协同系统、方法及电子装置
CN113050998A (zh) * 2021-03-30 2021-06-29 联想(北京)有限公司 电子设备及控制方法

Also Published As

Publication number Publication date
CN106797470A (zh) 2017-05-31

Similar Documents

Publication Publication Date Title
CN106797388B (zh) 跨系统多媒体数据编解码方法、装置、电子设备和计算机程序产品
WO2018119611A1 (zh) 多操作系统多媒体数据编解码方法、装置、电子设备和计算机程序产品
WO2018119711A1 (zh) 多操作系统多媒体编解码方法、装置及电子设备
CN103810021B (zh) 一种vdi下基于宿主机的虚拟桌面显示方法及其系统
CN102981887B (zh) 数据处理方法及电子设备
CN107710720B (zh) 用于移动客户端设备的云计算
US11128879B2 (en) Hybrid decoding
US11012498B1 (en) Web browser multi-media redirection
US11588875B2 (en) Multi-media redirection for media applications
WO2014089793A1 (zh) 一种视频重定向的方法、装置、系统及计算机可读介质
WO2018119616A1 (zh) 多操作系统多媒体数据编解码方法、装置、电子设备和计算机程序产品
US20220417111A1 (en) Adaptable real-time communications plugin for virtual desktop infrastructure solutions
US10834164B2 (en) Virtualizing audio and video devices using synchronous A/V streaming
CN106210036A (zh) 虚拟桌面呈现设备中的视频显示加速方法
CN105657540B (zh) 一种适配安卓系统的视频解码方法及其装置
CN111866508A (zh) 视频处理方法、装置、介质及电子设备
US11704175B2 (en) Bridging virtual desktops under nested mode
US9336557B2 (en) Apparatus and methods for processing of media signals
WO2020243973A1 (zh) 基于模型的信号推理方法和装置
Wang et al. Implementation and optimization of multimedia framework on heterogeneous multi-core platform
CN105242972A (zh) 一种计算机资源解放系统及方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16925861

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 15/10/2019)

122 Ep: pct application non-entry in european phase

Ref document number: 16925861

Country of ref document: EP

Kind code of ref document: A1