WO2018119955A1

WO2018119955A1 - Cross-system multimedia data coding and decoding method, apparatus, electronic device and computer program product

Info

Publication number: WO2018119955A1
Application number: PCT/CN2016/113276
Authority: WO
Inventors: 李向远
Original assignee: 深圳前海达闼云端智能科技有限公司
Priority date: 2016-12-29
Filing date: 2016-12-29
Publication date: 2018-07-05
Also published as: CN106797388A; CN106797388B

Abstract

Provided by the present application are a cross-system multimedia data coding and decoding method, an apparatus, an electronic device, and a computer program product, the method comprising: in a first operating system, obtaining a coding and decoding request for multimedia data coding and decoding processing, the coding and decoding request comprising multimedia data requiring coding and decoding processing; sending the multimedia data to a shared memory; generating a coding and decoding instruction for coding and decoding processing of the multimedia data according to the coding and decoding request, the coding and decoding instruction comprising an offset address of the multimedia data in the shared memory; sending the coding and decoding instruction to a second operating system on the basis of cross-system communication; in the second operating system, obtaining the multimedia data from the shared memory according to the offset address in the coding and decoding instruction; and carrying out coding and decoding processing for the multimedia data according to the coding and decoding instruction, the shared memory being in a readable and writable state for both the first operating system and the second operating system. The present application enhances the performance of a virtual system and shortens the time for coding and decoding cross-system multimedia data.

Description

Cross-system multimedia data encoding and decoding method, device, electronic device and computer program product

Technical field

The present application relates to the field of virtualization technologies, and in particular, to a cross-system multimedia data encoding and decoding method, apparatus, electronic device, and computer program product.

Background technique

Mobile terminals have become an important tool for users' daily work and life. While mobile terminals are driving the rapid development of mobile Internet, mobile terminals contain rich and diverse information (such as enterprise data access, business communication, social networks, financial management, Games, etc.), mobile terminals have become the new focus of cybercrime. For information security and privacy protection, mobile virtualization has become the future development direction of mobile terminals.

Figure 1 shows a schematic diagram of a virtualized system architecture based on Qemu/KVM (Kernel-based Virtual Machine) technology. The bottom layer is the hardware layer of the system, which mainly includes the processor, memory, and input and output devices. Wait. Above the hardware layer is the virtualization layer, which runs a virtual machine monitor (abbreviated as VMM or Hypervisor). The main function of the virtual machine monitor is to manage the real physical hardware platform and provide a corresponding virtual hardware platform for each guest virtual machine (ie, client). The virtualized system architecture can run multiple operating systems, such as the Linux operating system or the Android operating system, which usually includes a host operating system (Host OS) and one or more guest operating systems (Guest OS), host operations. The system runs on a host (Host) running one or more clients on the host operating system, each client running as an application of the host operating system, which can emulate a hardware device for the guest operating system running on the client For use by the guest operating system, various applications can be run in the guest operating system running on each client. In a virtualized system architecture, hardware layers can be shared between multiple operating systems Provides CPU, memory, and input and output devices.

The isolation provided by the virtualization technology can isolate multiple operating systems on a single terminal, and prevent processes on each operating system from adversely affecting other operating systems. A typical application scenario is a mobile device such as a mobile phone through virtualization technology. The separation of the enterprise system and the personal system is realized, and the potential harm caused by the process on the personal system is prevented, and the demand for the BYOD (Bring Your Own Device) is met by some enterprises with security requirements.

With the in-depth development of multimedia applications in life, each virtual machine of the terminal needs a multimedia codec. In the prior art, the client operating system usually implements software codec through the CPU, and this codec mode will occupy the current The client CPU has a large amount of resources and consumes a large amount of power; and the client operating system can only implement software codec through the CPU, which results in video files of many formats cannot be supported and played smoothly. When a guest operating system requires other client operating systems or the host operating system to assist in completing the multimedia data encoding and decoding process, it is necessary to separately allocate corresponding buffers for the multimedia framework buffers in each guest operating system or host operating system. Physical storage area, and when a guest operating system needs other systems to assist in cross-system multimedia data encoding and decoding processing, it is necessary to copy the contents of the physical storage area corresponding to the buffer of the guest operating system multimedia framework to the assistance. In the physical storage area corresponding to the buffer of the operating system multimedia frame of the codec processing, after the codec processing is completed, the processing result needs to be copied back to the physical storage area corresponding to the buffer of the guest operating system multimedia frame that initiates the codec processing request. .

The shortcomings of the prior art are:

When encoding and decoding multimedia data across systems, it involves copying a large amount of data in different physical storage intervals. On the one hand, it occupies a large amount of computing resources and storage resources of the virtualized system architecture, resulting in reduced performance of the virtualized system, and on the other hand, extending the cross. System multimedia data encoding and decoding processing time.

Summary of the invention

The embodiment of the present application proposes a cross-system multimedia data encoding and decoding method, device, electronic device and computer program product, which are mainly used to improve the performance of the virtualization system and shorten the encoding and decoding processing time of the multimedia data.

In one aspect, an embodiment of the present application provides a cross-system multimedia data encoding and decoding method, where the method includes:

Acquiring, in the first operating system, a codec request for encoding and decoding a multimedia data, the codec request including multimedia data that needs to be subjected to codec processing; sending the multimedia data to a shared memory; according to the codec request Generating a codec instruction for performing codec processing on the multimedia data, the codec instruction including an offset address of the multimedia data in the shared memory; and transmitting the codec instruction to a second operation based on cross-system communication system;

In the second operating system, the multimedia data is obtained by the shared memory according to an offset address in the codec instruction; the multimedia data is encoded and decoded according to the codec instruction, and the sharing The memory is in a readable and writable state for both the first operating system and the second operating system.

In another aspect, an embodiment of the present application provides a cross-system multimedia data encoding and decoding apparatus, where the apparatus includes: a front end module, configured to acquire, in a first operating system, a codec processing process on a multimedia data. a codec request, the codec request includes multimedia data that needs to be subjected to codec processing; the multimedia data is sent to a shared memory; and a codec instruction for encoding and decoding the multimedia data is generated according to the codec request, The codec instruction includes an offset address of the multimedia data in the shared memory; and the codec instruction is sent to a second operating system based on cross-system communication;

a backend module, configured to obtain, by the shared memory, the multimedia data according to an offset address in the codec instruction in the second operating system; and compiling the multimedia data according to the codec instruction Decoding processing, the shared memory being in a readable and writable state for both the first operating system and the second operating system.

In another aspect, an embodiment of the present application provides an electronic device, including: a memory, one or more processors; and one or more modules, the one or more modules being Stored in the memory and configured to be executed by the one or more processors, the one or more modules including instructions for performing the various steps of any of the above methods.

In another aspect, embodiments of the present application provide a computer program product for use in conjunction with an electronic device, the computer program product comprising a computer program embedded in a computer readable storage medium, the computer program comprising The electronic device is caused to execute instructions of each of the above methods.

The beneficial effects of the embodiments of the present application are as follows:

In the application, in the first operating system, when the multimedia data needs to be encoded and decoded, the multimedia data may be sent to the second operating system through the shared memory to complete the editing of the multimedia data in the second operating system. Decoding processing, in the present application, the multimedia data is transmitted in the process of encoding and decoding the cross-system multimedia data through the shared memory, thereby reducing the number of data copies, thereby saving the computing resources and storage resources of the virtualized system architecture, and improving the performance of the virtualized system. On the other hand, the cross-system multimedia data encoding and decoding processing time is shortened.

DRAWINGS

Specific embodiments of the present application will be described below with reference to the accompanying drawings, in which:

FIG. 1 is a schematic diagram showing the architecture of a virtualization system in the prior art;

FIG. 2 is a schematic flowchart diagram of a method for encoding and decoding multimedia data in a cross-system according to Embodiment 1 of the present application;

FIG. 3 is a schematic diagram showing a virtualized system architecture of a cross-system multimedia data encoding and decoding method according to Embodiment 1 of the present application;

4 is a schematic structural diagram of another virtualization system applied to a cross-system multimedia data encoding and decoding method according to Embodiment 1 of the present application;

FIG. 5 is a schematic structural diagram of another virtualization system applied to a cross-system multimedia data encoding and decoding method according to Embodiment 1 of the present application;

6 is a schematic structural diagram of a cross-system multimedia data encoding and decoding apparatus according to Embodiment 2 of the present application;

FIG. 7 is a schematic structural diagram of an electronic device in Embodiment 3 of the present application.

detailed description

The exemplary embodiments of the present application are further described in detail below with reference to the accompanying drawings, in which the embodiments described are only a part of the embodiments of the present application, but not all embodiments. An exhaustive example. And in the case of no conflict, the features in the embodiments and the embodiments in the description can be combined with each other.

The inventor noticed in the process of invention that the encoding and decoding of multimedia data across the system involves the copying of a large amount of data in different physical storage intervals, and on the other hand occupies a large amount of computing resources and storage resources of the virtualized system architecture, resulting in a virtualized system. The performance is reduced, and on the other hand, the cross-system multimedia data encoding and decoding processing time is extended.

For the above-mentioned deficiencies, the present application provides a cross-system multimedia data encoding and decoding method. In the first operating system, when the multimedia data needs to be encoded and decoded, the multimedia data can be sent to the second operating system through the shared memory. In the second operating system, the codec processing of the multimedia data is completed. In the present application, the multimedia data is transmitted during the cross-system multimedia data encoding and decoding process by using the shared memory, thereby reducing the number of data copies, and saving virtualization on the one hand. The computing resources and storage resources of the system architecture improve the performance of the virtualized system, and on the other hand, shorten the processing time of codec processing of multimedia data across systems.

In order to facilitate the implementation of the present application, the following description will be made by way of example.

Embodiment 1:

When the virtualization system architecture is started, Qemu can divide a specific address space in the memory as shared memory that can be read and written by each operating system in the virtualization architecture, for example, the shared memory can be created by a system call. . Further, the shared memory space may be further divided to support access by multiple processes or threads. The size of the shared memory can be set by the developer and adapted to cross-system transfers that may be involved in the virtualized system architecture.

When the guest operating system is started, its multimedia framework can obtain the relevant configuration information from the guest operating system kernel layer for initialization, thereby obtaining the mapping of the shared memory to the address space of the guest operating system, and can also establish with Qemu. Synchronize control channels.

In the host operating system, Qemu may trigger a multimedia framework in the host operating system to perform initialization similar to the guest operating system to obtain the shared memory mapped to the address space of the host operating system, thereby establishing a connection with the shared memory across the system.

FIG. 2 is a schematic flowchart of a method for encoding and decoding multimedia data in a cross-system according to Embodiment 1 of the present application. As shown in FIG. 2, the method for encoding and decoding multimedia data in a cross-system includes:

Step 101: In the first operating system, acquiring a codec request for encoding and decoding a multimedia data, where the codec request includes multimedia data that needs to be subjected to codec processing; and sending the multimedia data to a shared memory; The codec request generates a codec instruction for performing codec processing on the multimedia data, the codec instruction including an offset address of the multimedia data in the shared memory; and sending the codec instruction based on cross-system communication to Second operating system;

Step 102: In the second operating system, obtaining, by the shared memory, the multimedia data according to an offset address in the codec instruction, and performing codec processing on the multimedia data according to the codec instruction, The shared memory is in a readable and writable state for both the first operating system and the second operating system;

In step 103, in the second operating system, the codec processing result of the multimedia data is fed back to the shared memory; in the first operating system, the codec processing result is obtained by the shared memory.

In step 101, after the application in the first operating system initiates the request for the multimedia data encoding and decoding process, the multimedia data that needs to be encoded and decoded included in the request is stored in the first operating system and the second operating system. Can be read and written in shared memory. The multimedia data may be multimedia data that is actually required to be subjected to codec processing, such as data that is actually required to be coded and processed in a video file or an audio file, or may be a location of a multimedia data that is actually required to be coded and processed in a memory. And other information, such as data format, time stamp, data size, and so on.

In the virtualized system architecture, a buffer (Buffer) for encoding and decoding multimedia data is allocated to the first operating system, and the physical memory corresponding to the buffer can be read by the first operating system and the second operating system. When writing shared memory, it can be directly based on address mapping in the first operating system The multimedia data is written to its assigned buffer. The kernel layer of the first operating system can map the shared memory to the user space of the first operating system; the first operating system can perform offset recording on the shared memory for which the data is written, that is, record the memory block currently writing the multimedia data. The offset address of the first address relative to the first address of the entire shared memory area.

And generating, according to the codec request, a codec instruction for performing codec processing on the multimedia data, where the codec instruction indicates, for the second operating system, how to perform codec processing on the delivered multimedia data, and may include codec Command functions and related parameters, etc. And transmitting, by the multimedia data, the offset address of the shared memory in the codec instruction to the second operating system. The codec instruction may be sent to the second operating system based on a cross-system communication manner such as QEMU PIPE, Socket, or shared memory.

In some embodiments, the codec instructions are sent to the second operating system based on a shared memory.

Because the shared memory-based cross-system communication method is faster than QEMU PIPE, Socket, etc., the code-based instruction is transmitted faster between the first operating system and the second operating system based on the shared memory mode, further shortening the cross-system multimedia data. The time of codec processing.

In some embodiments, the transmitting the multimedia data to the shared memory comprises: transmitting the multimedia data to the non-shared memory; and copying the multimedia data in the non-shared memory to the shared memory.

The physical memory corresponding to the buffer allocated by the first operating system can be directly read and written by the first operating system, but cannot be read or written by the second operating system, that is, the physical memory corresponding to the buffer is non-shared memory. And storing the multimedia data that needs to be coded and processed by the first operating system into the physical memory corresponding to the buffer, and then copying the multimedia data to the shared memory that can be read and written by the first operating system and the second operating system. In memory.

It should be understood that, in these cases, if the codec instruction is also transmitted across the system in a shared memory manner, the codec instruction needs to be first written into the non-shared memory of the first operating system, and then the Codec instructions in non-shared memory are copied to shared memory for second operation System acquisition.

In step 102, the kernel layer of the second operating system can map the shared memory to the user space of the second operating system. After acquiring the codec instruction, the second operating system may read the multimedia data according to the offset address in the codec instruction to a corresponding position of the shared memory, and then the multimedia data according to the codec instruction. The codec processing is performed to obtain the processing result. Specifically, in the second operating system, the codec driver may be invoked according to the codec instruction to drive the corresponding codec to complete the codec process; and the codec component may be loaded according to the codec instruction. Decoding processing; the multimedia data may be encoded and decoded according to the codec instruction by an OpenMAX integration layer (OpenMAX IL, OpenMAX Integration Layer) in the second operating system. The format and specific content of the codec instructions may also be different in various different situations as described above.

It should be understood that when the codec instruction is transferred from the first operating system to the second operating system based on the shared memory mode, the first operating system also needs to send the codec instruction in the shared memory to the second operating system. And shifting the address, so that the second operating system acquires the codec instruction from the shared memory according to the offset address of the codec instruction, and further acquires the multimedia data.

In step 103, in the second operating system, after the codec processing of the multimedia data is performed according to the codec instruction, a processing result is obtained, and the processing result is also transmitted to the first operating system through the shared memory. For example, the codec processing result is written into the shared memory, and the position of the processing result in the shared memory (first address offset address) is recorded, and the offset address is based on QEMU PIPE, Socket or shared memory. The system communication mode is sent to the first operating system, so that the first operating system can read data through the offset address of the processing result to the corresponding location of the shared memory. The processing result may be multimedia data after being encoded and processed, such as video or audio data in a decoded video file or a compressed audio file, or may be encoded and processed multimedia data in a memory. Location and other information such as data format, time stamp, data size, etc.

The first operating system is a guest operating system in a virtualized system architecture, and the second operation The system is the host operating system in the virtualized system architecture.

In some embodiments, the step 101 is: in the first operating system, acquiring a codec request of the multimedia frame calling codec to perform multimedia data encoding and decoding processing, where the codec request includes multimedia that needs to be coded and processed. Transmitting the codec request to the shared memory; generating a codec instruction for encoding and decoding the multimedia data according to the codec request, the codec instruction including the multimedia data in the shared memory Offset address; transmitting the codec instruction to the second operating system based on cross-system communication; the step 102 is, in the second operating system, by the shared memory according to the offset address in the codec instruction Obtaining the multimedia data; calling, according to the codec instruction, a driver of the codec to encode and decode the multimedia data, where the shared memory is used by the first operating system and the second operating system Readable and writable.

FIG. 3 is a schematic diagram of a virtualized system architecture for applying the cross-system multimedia data encoding and decoding method in the first embodiment of the present application, wherein the virtualized system architecture includes at least one guest operating system, and the first operation The system is a guest operating system, and the second operating system is a host operating system. A multimedia application, such as a video player, is run in the guest operating system. When the multimedia application needs to encode and decode certain multimedia data, the multimedia data encoding and decoding request is sent to the multimedia framework, for example, When the video player plays the HD video, the video data decoding request is sent to the multimedia framework.

In step 101, the multimedia application running in the guest operating system sends a multimedia data encoding and decoding request to the multimedia framework when the multimedia data needs to be encoded and decoded. The multimedia frame is based on the type and format of the multimedia data. Or the resolution or the like is selected for the appropriate codec for corresponding processing. When the multimedia framework determines that the multimedia data needs to be processed by the hardware codec, the hardware codec in the guest operating system is called to generate an instruction to perform hardware codec on the multimedia data. In this embodiment, the multimedia framework in the guest operating system may be OpenMAX or other existing multimedia framework. When the multimedia framework is OpenMAX, it has one or more hardware codec components, and a multimedia frame. The shelf generates instructions for hardware codec processing of the multimedia data by invoking each hardware codec component.

Instead of a real hardware codec in the guest operating system, the hardware codec driver front end is run in the guest operating system kernel layer, when the hardware codec driver front end receives the multimedia framework call hardware After performing the instruction of performing hardware codec processing on the multimedia data, the decoder component writes the multimedia data that needs to be subjected to hardware codec processing into the shared memory, and includes the offset address of the multimedia data in the shared memory. In the instructions, the instructions are passed to the hardware codec driver backend running in the host operating system kernel layer by cross-system communication.

In step 102, the hardware codec driver backend running in the kernel layer of the host operating system receives the instruction to perform hardware codec processing on the multimedia data sent by the front end of the hardware codec driver in the guest operating system. Obtaining multimedia data from the shared memory according to the offset address included in the instruction, and running a driver of the hardware codec in the host operating system to complete the multimedia data by the real hardware codec Hardware codec processing. The hardware codec driver may be a V4L2Driver or the like.

In this embodiment, after the foregoing step 102, the method may further include the step 103, feeding back the processed multimedia data to the guest operating system based on the shared memory; and in the guest operating system, the processed multimedia data. Feedback to the multimedia framework.

After the hardware codec of the host operating system completes the encoding and decoding process on the multimedia data, the hardware codec driver backend in the host operating system can transfer the coded and processed data to the client operation through the shared memory. The hardware codec driver front end in the system. The hardware codec driver front end in the guest operating system can feed back the codec processed data to the multimedia framework, so that the multimedia frame feeds back the codec processed data to initiate the original decoding request multimedia application. program.

In this embodiment, taking the codec as a hardware codec as an example, that is, the multimedia framework in the guest operating system determines that the codec request is a hardware codec request, and needs to invoke the host operation. The driver of the hardware codec in the system drives the hardware codec to perform corresponding hardware codec processing. It should be understood that when the multimedia framework in the guest operating system determines that the codec request is a software codec request, The corresponding software codec processing can be performed by the host operating system through the shared memory.

In some implementations, the step 101 is to acquire, in the first operating system OpenMAX integration layer, a codec request sent by the adaptation layer to perform codec processing on the multimedia data, where the codec request includes a codec that needs to be coded. Processing the multimedia data; transmitting the multimedia data to the shared memory; generating a codec instruction for encoding and decoding the multimedia data according to the codec request, the codec instruction including the multimedia data in the sharing An offset address of the memory; the codec instruction is sent to the second operating system based on the cross-system communication; the step 102 is, in the second operating system OpenMAX integration layer, according to the offset address in the codec instruction Obtaining the multimedia data by the shared memory; encoding and decoding the multimedia data according to the component of the codec loading codec, the shared memory pairing the first operating system and the second operation The system is both readable and writable.

FIG. 4 is a schematic diagram of another virtualization system architecture used in the cross-system multimedia data encoding and decoding method in the first embodiment of the present application. The technical solution provided in this embodiment is applicable to an operating system using the OpenMAX multimedia framework. The virtual operating system architecture includes at least one guest operating system, the first operating system is a guest operating system, and the second operating system is a host operating system. A multimedia application, such as a video player, is run in the guest operating system. When the multimedia application needs to encode and decode certain multimedia data, the multimedia data encoding and decoding request is sent to the multimedia framework, for example, When the video player plays the HD video, the video data decoding request is sent to the multimedia framework.

In step 101, after the multimedia application in the guest operating system issues a codec request for encoding and decoding the multimedia data, the adaptation layer of the guest operating system may be based on the type and format of the multimedia data or The resolution etc. selects the appropriate software or hardware codec for it. There are multiple codecs supporting the same format, which can include hardware codecs and The software codec can select the most suitable codec according to the preset rules when adapting the codec, for example, selecting a software codec for multimedia files with low resolution requirements, and selecting multimedia files with high resolution requirements. Hardware codec, etc. After receiving the codec request sent by the adaptation layer in the OpenMAX integration layer of the guest operating system, generating a codec instruction corresponding to the codec request, writing multimedia data into the shared memory, and storing the multimedia data in the shared memory The offset address is included in the codec instruction, and the codec instruction is delivered to the host operating system by cross-system communication. In this embodiment, only the multimedia data adapted to the hardware codec may be transferred to the host operating system based on the shared memory, and the multimedia data adapted to the software codec is still used in the guest operating system. The software codec encodes and decodes the multimedia file; and all multimedia data of the codec request may also be transmitted to the host operating system based on the shared memory.

In step 102, after the host operating system acquires the codec instruction sent by the guest operating system, the multimedia data is obtained by the shared memory according to the offset address included in the codec instruction, in the OpenMAX integration layer of the host operating system. And loading, by the component of the codec according to the codec instruction, encoding and decoding the multimedia data.

In some embodiments, the OpenMAX front end and the OpenMAX back end are respectively run in the OpenMAX integration layer of the guest operating system and the OpenMAX integration layer of the host operating system, and the codec request of the application is performed at the adaptation layer of the guest operating system. After matching the appropriate codec, the OpenMAX front end generates a codec instruction according to the codec request that determines the appropriate codec, writes the multimedia data into the shared memory, and stores the multimedia data in the shared memory. An offset address is included in the codec instruction to pass the codec instruction to the OpenMAX backend in the host operating system by cross-system communication. After receiving the codec instruction sent by the OpenMAX front end, the OpenMAX backend in the host operating system acquires multimedia data from the shared memory according to the offset address included in the instruction, and the codec instruction is integrated in the host operating system OpenMAX. The corresponding codec is loaded in the layer to encode and decode the multimedia data.

In this embodiment, after the foregoing step 102, the method may further include step 103, after the processing The multimedia data is fed back to the guest operating system based on the shared memory; in the guest operating system, the processed multimedia data is fed back to the adaptation layer.

After the host operating system OpenMAX integration layer loading software or hardware codec completes the encoding and decoding process of the multimedia data, the host operating system can pass the coded and processed data to the guest operating system based on the shared memory through the OpenMAX backend. The OpenMAX front end of the client operating system, after receiving the codec processed data, can feed it back to the adaptation layer and feed back to the multimedia application that initiated the original decoding request. In addition, after the host operating system OpenMAX integration layer loading software or hardware codec completes the encoding and decoding process on the multimedia data, the processing result may also be directly sent to the multimedia application in the host operating system for subsequent operations.

In some implementations, the step 101 is: acquiring, in the first operating system, a codec request for the multimedia data encoding and decoding process by the multimedia application, where the codec request includes multimedia data that needs to be subjected to codec processing; And the multimedia data is sent to the shared memory; and the codec instruction for encoding and decoding the multimedia data is generated according to the codec request, where the codec instruction includes an offset address of the multimedia data in the shared memory; Transmitting the codec instruction to the second operating system based on cross-system communication; in step 102, in the second operating system, obtaining the multimedia by the shared memory according to an offset address in the codec instruction Transmitting the codec instruction and the multimedia data to an OpenMAX integration layer in a second operating system, so that an OpenMAX integration layer in the second operating system performs the multimedia data according to the codec instruction Codec processing.

FIG. 5 is a schematic diagram of another virtualization system architecture applied to the cross-system multimedia data encoding and decoding method according to the first embodiment of the present application. The technical solution provided by the implementation manner is applicable to the multimedia multimedia framework standard. This can include Stagefright+OpenMAX or Opencore+OpenMAX. The virtual operating system architecture includes at least one guest operating system, the first operating system is a guest operating system, and the second operating system is a host operating system. Running a multimedia application, such as video playback, in the guest operating system And the like, when the multimedia application needs to encode and decode some multimedia data, the multimedia data encoding and decoding request is sent to the multimedia framework, for example, when the video player plays the high-definition video, the video data is sent to the multimedia frame. Decode the request.

In step 101, the multimedia application running in the guest operating system sends a multimedia data codec request to the multimedia framework when the multimedia data needs to be coded and processed, and the codec request includes codec decoding. Processing of multimedia data. In the guest operating system, the multimedia data is processed according to the codec request without running a complete multimedia framework, and only the codec instruction needs to be generated according to the codec request, and the multimedia data is written into the shared memory, and The offset address of the multimedia data in the shared memory is included in the codec instruction, and the codec instruction is delivered to the host operating system by means of cross-system communication.

In step 102, the complete OpenMAX integration layer is run in the host operating system, and after receiving the codec instruction of the guest operating system based on the shared memory transfer, the shared memory is obtained according to the offset address included in the codec instruction. The multimedia data, the codec instruction is sent to the OpenMAX integration layer in the host operating system along with the multimedia data to be encoded and decoded, and the multimedia data is encoded and decoded by the OpenMAX integration layer according to the codec instruction. It may include loading various software codec components for software codec processing; and may also include loading various hardware codec components and driving corresponding hardware codecs for hardware codec processing.

In some implementations, in the step 101, acquiring, in the OMX adaptation layer of the client operating system Stagefright framework, a codec request for the multimedia data encoding and decoding process by the multimedia application, where the codec request includes And encoding and decoding the multimedia data; generating, according to the codec request, a codec instruction for performing codec processing on the multimedia data, where the codec instruction includes an offset address of the multimedia data in the shared memory; The codec instruction is sent to the host operating system based on cross-system communication; in the step 102, in the host operating system OMX adaptation layer, the multimedia data is acquired by the shared memory according to the offset address in the codec instruction. Transmitting the codec instruction and the multimedia data to an OpenMAX integration layer in a host operating system to cause an OpenMAX integration layer in the host operating system to be The codec instruction performs codec processing on the multimedia data.

Taking the virtualization system architecture using the Android multimedia framework (Stagefright+OpenMAX) as an example, the Stagefright in the host operating system and the guest operating system respectively run the OMX adaptation layer, and the OMXClient is respectively referenced in the adaptation layer. In the OMX adaptation layer of the host operating system, AwesomePlayer can call the OMX Client::connect function, obtain the MediaPlayerService through the binder mechanism, and then create an OMX instance through the MediaPlayerService, so that the OMXClient in the host operating system is used as the entrance of the OMX. The binder mechanism obtains the services provided by OMX.

In the guest operating system, a new class OMXClient front end is created in the OMX adaptation layer of the Stagefright framework in the guest operating system, and the OMXClient front end can be to the class mediaplayer, class mediacodec, or other classes in the multimedia framework of the android system. Providing a standard class OMXClient interface, when the OMXClient front end in the guest operating system acquires a codec request for the multimedia data encoding and decoding process by the multimedia application, generating a codec instruction according to the codec request, and writing the multimedia data Sharing the memory, and including the offset address of the multimedia data in the shared memory in the codec instruction, and transmitting the codec instruction to the OMXClient in the host operating system OMX adaptation layer by means of cross-system communication End, the OMXClient backend can obtain the multimedia data from the shared memory according to the offset address included in the codec instruction, and send the codec instruction together with the multimedia data to be coded and processed to the MediaPlayerService through the Binder mechanism. , MediaCodecService or android system multimedia Other Service framework. In this implementation manner, the OMXClient backend is used as an entry of the OMX adaptation layer OMX of the host operating system, so that the codec instructions and multimedia data sent to the OMXClient front end in the guest operating system can be sent to the host operation through the OMXClient backend. The complete OpenMAX integration layer running in the system completes the codec processing of hardware or software.

In this embodiment, the information of the codec that performs the encoding and decoding process on the multimedia data may be further determined according to the codec request, and then the codec instruction corresponding to the codec is generated. The foregoing step 102 may be: obtaining, by the shared memory, the multimedia data according to an offset address in the codec instruction; and sending the codec instruction and the multimedia data to an OpenMAX integration layer in a host operating system, The multimedia data is coded and processed by an OpenMAX integration layer in the host operating system to load a component of the codec according to the codec instruction.

For the Stagefright framework, all codecs supported by the guest operating system, including various hardware codecs and software codecs, can be stored in a pre-configured configuration file. After obtaining the codec request sent by the multimedia application in the OMX adaptation layer OMXClient front end of the guest operating system, the Stagefright framework can determine the codec applicable to the multimedia data according to the multimedia data that needs to be codec processed, for example, according to multimedia. The type, format, or resolution of the data is selected as an appropriate software or hardware codec, and a codec instruction corresponding to the codec is generated.

After obtaining the codec instruction sent by the guest operating system by the Stagefright framework OMX adaptation layer OMXClient backend of the host operating system, obtaining the multimedia data by the shared memory according to the offset address in the codec instruction, Transmitting the codec instruction and the multimedia data to an OpenMAX integration layer in a host operating system, so that the OpenMAX integration layer loads an appropriate software or hardware codec component according to the codec request to perform the multimedia data. Codec processing.

In this embodiment, after the foregoing step 102, the method may further include the step 103, feeding back the processed multimedia data to the guest operating system based on the shared memory; and in the guest operating system, the processed multimedia data. Feedback the multimedia application.

After the software or hardware codec of the host operating system completes the encoding and decoding process on the multimedia data, the OMXClient backend in the host operating system OMX adaptation layer can transfer the coded and processed data to the client based on the shared memory. The OMXClient front-end in the operating system multimedia framework OMX adaptation layer, the OMXClient front-end in the OMX adaptation layer of the guest operating system, after receiving the codec-processed data, can feed it back to the multimedia requesting the original encoding and decoding request. Use the program.

In the foregoing implementation manner, when Qemu starts, a shared memory that is accessible to the host operating system and the guest operating system is created for the multimedia, and the file node of the shared memory is created in the host operating system, and the shared memory of the block is mapped to A guest operating system, such as a memory space mapped to a guest operating system PCI (Peripheral Component Interconnect) device. The multimedia front end (such as the codec driver front end, the OpenMAX front end, or the OMXClient front end) in the guest operating system is obtained by the kernel layer to obtain configuration information, and then obtains the shared memory space allocated by Qemu, that is, the shared memory is obtained in the client operation. The first address in the system. After the multimedia backend (such as the codec driver backend, the OpenMAX backend, or the OMXClient backend) in the host operating system is started, the connection request of the corresponding front end is monitored, and when the current end is connected with the back end, the multimedia back end is based on The file node mapping of the shared memory created by the foregoing Qemu obtains a shared memory space, that is, the first address of the shared memory in the host operating system is obtained. After that, the front end and the back end after establishing the connection can realize the cross-system transmission of the multimedia data or the codec instruction through the same block space in the shared memory through the first address of the shared memory in the operating system and the same offset address.

In this embodiment, a cross-system multimedia data encoding and decoding method is provided. In the first operating system, when the multimedia data needs to be encoded and decoded, the multimedia data may be sent to the second operating system through the shared memory. The codec processing of the multimedia data is completed in the second operating system. In the present application, the multimedia data is transmitted in the process of encoding and decoding the cross-system multimedia data through the shared memory, thereby reducing the number of data copies, and saving the virtualization system. The computing resources and storage resources of the architecture improve the performance of the virtualized system, and on the other hand, shorten the processing time of the cross-system multimedia data encoding and decoding. The application can also transfer codec instructions between the first operating system and the second operating system based on the shared memory mode, further shortening the time of the cross-system multimedia data encoding and decoding process; in the physical storage area of the buffer originally allocated by the first operating system When it is non-shared memory, the data written to the non-shared memory can be copied to the shared memory for the second operating system to read. In addition, the proposal also discloses the kernel layer of the first operating system and the second operating system, and OpenMAX integration. Layer or multimedia The adaptation layer of the framework can implement multimedia data encoding and decoding requests across systems based on shared memory, which can be adjusted according to user requirements and is more flexible.

Embodiment 2:

Based on the same inventive concept, a cross-system multimedia data encoding and decoding device is also provided in the embodiment of the present application. Since the principle of solving the problem is similar to the cross-system multimedia data encoding and decoding method, the implementation of these devices can refer to the implementation of the method. , the repetition will not be repeated. As shown in FIG. 6, the cross-system multimedia data encoding and decoding apparatus 200 includes:

The front end module 201 is configured to acquire, in the first operating system, a codec request for the codec processing of the multimedia data, where the codec request includes multimedia data that needs to be subjected to codec processing; and send the multimedia data to the shared memory; Generating, according to the codec request, a codec instruction for performing codec processing on the multimedia data, where the codec instruction includes an offset address of the multimedia data in the shared memory; and the codec instruction is based on a cross system Communication is sent to the second operating system;

The backend module 202 is configured to obtain, by the shared memory, the multimedia data according to an offset address in the codec instruction in the second operating system, and perform the multimedia data according to the codec instruction. Codec processing, the shared memory being in a readable and writable state for both the first operating system and the second operating system.

In some implementations, the transmitting the codec instruction to the second operating system based on cross-system communication comprises: transmitting the codec instruction to the second operating system based on a shared memory manner.

In some implementations, in the first operating system, acquiring a codec request for a multimedia data codec process, where the codec request includes multimedia data that needs to be subjected to codec processing, including:

In the first operating system, acquiring a multimedia framework call codec for multimedia data compilation a coded codec request, the codec request including multimedia data that requires codec processing;

The encoding and decoding processing the multimedia data according to the codec instruction includes:

And calling the driver of the codec according to the codec instruction to perform codec processing on the multimedia data.

In some implementations, the front-end module 201 is specifically configured to: acquire, in a first operating system OpenMAX integration layer, a codec request sent by an adaptation layer to perform codec processing on the multimedia data, where the codec request includes a requirement Performing codec processing of multimedia data; transmitting the multimedia data to a shared memory; generating, according to the codec request, a codec instruction for performing codec processing on the multimedia data, where the codec instruction includes the multimedia data The offset address of the shared memory; sending the codec instruction to the second operating system based on cross-system communication;

The backend module 202 is specifically configured to: obtain, in the second operating system OpenMAX integration layer, the multimedia data by using the shared memory according to an offset address in the codec instruction; according to the codec instruction The component loading the codec encodes and decodes the multimedia data, and the shared memory is in a readable and writable state for both the first operating system and the second operating system.

In the first operating system, acquiring a codec request for the multimedia data encoding and decoding process by the multimedia application, where the codec request includes multimedia data that needs to be subjected to codec processing;

Transmitting the codec instruction and the multimedia data to an OpenMAX integration layer in a second operating system, so that an OpenMAX integration layer in the second operating system encodes and decodes the multimedia data according to the codec instruction deal with.

In some embodiments, the backend module 202 is further configured to perform the codec according to the After the encoding and decoding process is performed on the multimedia data, the codec processing result of the multimedia data is fed back to the shared memory;

The front end module 201 is further configured to acquire the codec processing result by the shared memory in the first operating system.

Embodiment 3:

Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application. Since the principle is similar to the cross-system multimedia data encoding and decoding method, the implementation of the method may refer to the implementation of the method, and the repeated description is not repeated. As shown in FIG. 7, the electronic device 300 includes: a memory 301, one or more processors 302; and one or more modules, the one or more modules being stored in the memory and configured to Executed by the one or more processors, the one or more modules include instructions for performing the various steps of any of the above methods.

In this embodiment, the electronic device may be a mobile phone, a tablet computer, a robot, or other smart device.

Embodiment 4:

Based on the same inventive concept, an embodiment of the present application further provides a computer program product for use in combination with an electronic device, the computer program product comprising a computer program embedded in a computer readable storage medium, the computer program comprising An instruction to cause the electronic device to perform each of the steps of any of the above methods.

For the convenience of description, the various parts of the above-described apparatus are separately described by functions into various modules. Of course, the functions of each module or unit may be implemented in the same software or hardware in the implementation of the present application.

Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system, or computer program product. Thus, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment in combination of software and hardware. Moreover, the application may employ computer-usable storage media (including but not limited to disk) in one or more of the computer-usable program code embodied therein. The form of a computer program product implemented on a memory, CD-ROM, optical memory, or the like.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (system), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block of the flowchart illustrations and/or FIG. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine for the execution of instructions for execution by a processor of a computer or other programmable data processing device. Means for implementing the functions specified in one or more of the flow or in a block or blocks of the flow chart.

The computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device. The apparatus implements the functions specified in one or more blocks of a flow or a flow and/or block diagram of the flowchart.

These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device. The instructions provide steps for implementing the functions specified in one or more of the flow or in a block or blocks of a flow diagram.

While the preferred embodiment of the present application has been described, it will be apparent that those skilled in the art can make further changes and modifications to the embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and the modifications and

Claims

A cross-operating system multimedia data encoding and decoding method, characterized in that the method comprises:

Acquiring, in the first operating system, a codec request for encoding and decoding a multimedia data, the codec request including multimedia data that needs to be subjected to codec processing; sending the multimedia data to a shared memory; according to the codec request Generating a codec instruction for performing codec processing on the multimedia data, the codec instruction including an offset address of the multimedia data in the shared memory; and transmitting the codec instruction to a second operation based on cross-system communication system;

In the second operating system, the multimedia data is obtained by the shared memory according to an offset address in the codec instruction; the multimedia data is encoded and decoded according to the codec instruction, and the sharing The memory is in a readable and writable state for both the first operating system and the second operating system.
The method of claim 1, wherein the transmitting the codec instruction to the second operating system based on cross-system communication comprises:

Sending the codec instruction to the second operating system based on the shared memory mode.
The method of claim 1, wherein the transmitting the multimedia data to the shared memory comprises:

Transmitting the multimedia data to non-shared memory;

Copy the multimedia data in the non-shared memory to the shared memory.
The method according to claim 1, wherein in the first operating system, a codec request for encoding and decoding a multimedia data is acquired, the codec request including multimedia data that needs to be subjected to codec processing, including :

In the first operating system, acquiring a codec request of the multimedia frame calling codec to perform multimedia data encoding and decoding processing, where the codec request includes a multimedia number that needs to be codec processed according to;

The encoding and decoding processing the multimedia data according to the codec instruction includes:

And calling the driver of the codec according to the codec instruction to perform codec processing on the multimedia data.
The method of claim 1 wherein said method comprises:

Acquiring, in the OpenMAX integration layer of the first operating system, a codec request for encoding and decoding the multimedia data sent by the adaptation layer, where the codec request includes multimedia data that needs to be subjected to codec processing; and sending the multimedia data to a shared memory; generating, according to the codec request, a codec instruction for performing codec processing on the multimedia data, where the codec instruction includes an offset address of the multimedia data in the shared memory; and the codec instruction is Transmitting to the second operating system based on cross-system communication;

And in the second operating system OpenMAX integration layer, obtaining the multimedia data by the shared memory according to an offset address in the codec instruction; loading a component of the codec according to the codec instruction The multimedia data is subjected to codec processing, and the shared memory is in a readable and writable state for both the first operating system and the second operating system.
The method according to claim 1, wherein in the first operating system, a codec request for encoding and decoding a multimedia data is acquired, the codec request including multimedia data that needs to be subjected to codec processing, including :

In the first operating system, acquiring a codec request for the multimedia data encoding and decoding process by the multimedia application, where the codec request includes multimedia data that needs to be subjected to codec processing;

The encoding and decoding processing the multimedia data according to the codec instruction includes:

Transmitting the codec instruction and the multimedia data to an OpenMAX integration layer in a second operating system, so that an OpenMAX integration layer in the second operating system encodes and decodes the multimedia data according to the codec instruction deal with.
The method of claim 1 wherein said encoding and decoding fingers are After the encoding and decoding of the multimedia data, the method further includes:

And feeding back the codec processing result of the multimedia data to the shared memory;

In the first operating system, the codec processing result is obtained by the shared memory.
A cross-operating system multimedia data encoding and decoding device, characterized in that the device comprises:

a front-end module, configured to acquire, in the first operating system, a codec request for encoding and decoding a multimedia data, where the codec request includes multimedia data that needs to be subjected to codec processing; and the multimedia data is sent to the shared memory; The codec request generates a codec instruction for encoding and decoding the multimedia data, the codec instruction includes an offset address of the multimedia data in the shared memory; and the codec instruction is based on cross-system communication Send to the second operating system;

a backend module, configured to obtain, by the shared memory, the multimedia data according to an offset address in the codec instruction in the second operating system; and compiling the multimedia data according to the codec instruction Decoding processing, the shared memory being in a readable and writable state for both the first operating system and the second operating system.
The apparatus according to claim 8, wherein the sending the codec instruction to the second operating system based on cross-system communication comprises:

Sending the codec instruction to the second operating system based on the shared memory mode.
The device according to claim 8, wherein the transmitting the multimedia data to the shared memory comprises:

Transmitting the multimedia data to non-shared memory;

Copy the multimedia data in the non-shared memory to the shared memory.
The apparatus according to claim 8, wherein in the first operating system, a codec request for a multimedia data codec process is acquired, and the codec request includes a need to perform Codec processed multimedia data, including:

In the first operating system, acquiring a codec request of the multimedia frame calling codec to perform multimedia data encoding and decoding processing, where the codec request includes multimedia data that needs to be subjected to codec processing;

The encoding and decoding processing the multimedia data according to the codec instruction includes:

And calling the driver of the codec according to the codec instruction to perform codec processing on the multimedia data.
The device of claim 8 wherein:

The front-end module is specifically configured to acquire, in the OpenMAX integration layer of the first operating system, a codec request for encoding and decoding the multimedia data sent by the adaptation layer, where the codec request includes multimedia data that needs to be coded and processed. Transmitting the multimedia data to a shared memory; generating, according to the codec request, a codec instruction for performing codec processing on the multimedia data, where the codec instruction includes an offset of the multimedia data in the shared memory Addressing; transmitting the codec instruction to the second operating system based on cross-system communication;

The backend module is specifically configured to: obtain, in the second operating system OpenMAX integration layer, the multimedia data by using the shared memory according to an offset address in the codec instruction; loading according to the codec instruction The component of the codec performs codec processing on the multimedia data, and the shared memory is in a readable and writable state for both the first operating system and the second operating system.
The apparatus according to claim 8, wherein in the first operating system, a codec request for encoding and decoding a multimedia data is acquired, the codec request including multimedia data that needs to be subjected to codec processing, including :

In the first operating system, acquiring a codec request for the multimedia data encoding and decoding process by the multimedia application, where the codec request includes multimedia data that needs to be subjected to codec processing;

The encoding and decoding processing the multimedia data according to the codec instruction includes:

Transmitting the codec instruction and the multimedia data to an OpenMAX integration layer in a second operating system, so that an OpenMAX integration layer in the second operating system encodes and decodes the multimedia data according to the codec instruction deal with.
The device of claim 8 wherein:

The backend module is further configured to: after the encoding and decoding processing the multimedia data according to the codec instruction, feed back a codec processing result of the multimedia data to the shared memory;

The front end module is further configured to acquire the codec processing result by the shared memory in the first operating system.
An electronic device, comprising:

a memory, one or more processors; and one or more modules, the one or more modules being stored in the memory and configured to be executed by the one or more processors, the one or The plurality of modules includes instructions for performing the various steps of the method of any of claims 1-7.
A computer program product for use with an electronic device, the computer program product comprising a computer program embedded in a computer readable storage medium, the computer program comprising means for causing the electronic device to perform claims 1 to 7 The instructions of the various steps in any of the methods described.