CN117173309A - Image rendering methods, devices, equipment, media and program products - Google Patents

Abstract

Embodiments of the present application provide an image rendering method, device, equipment, media and program products. According to the opening instruction of the 2D application, the 2D application is started in the virtual screen. The 2D rendering synthesizer obtains multiple images of the 2D application from the virtual screen. layer, synthesize multiple layers of the 2D application to obtain the 2D image of the 2D application, and send the 2D image to the transmission component; the transmission component transmits the 2D image to the 3D rendering synthesizer; the 3D rendering synthesizer processes the 2D image and the 3D application. The rendering data is rendered and synthesized to obtain a 3D image of the 3D application, and the display screen displays the 3D image. The rendering of 2D applications and the rendering of 3D applications are executed independently. The rendering of 2D applications is implemented by the 2D rendering synthesizer, and the rendering of 3D applications is implemented by the 3D rendering synthesizer. The rendering of 3D applications no longer relies on the surface flinger of the Android system, making this implementation The example method is no longer limited to the Android system and can be flexibly run on various operating systems.

Description

Image rendering methods, devices, equipment, media and program products

Technical field

The embodiments of the present application relate to the field of virtual reality technology, and in particular, to an image rendering method, device, equipment, medium and program product.

Background technique

Extended Reality (XR) refers to the use of computers to combine reality and virtuality to create a virtual environment that allows human-computer interaction. XR is also virtual reality (VR), augmented reality (AR) and hybrid A collective name for various technologies such as Mixed Reality (MR). By integrating the three visual interaction technologies, the user can experience a "sense of immersion" that seamlessly transitions between the virtual world and the real world.

In XR devices, users have the need to use traditional 2D applications (applications, apps). In the existing technology, the system service module (surface flinger) of the Android system is mainly relied on for rendering. Surfaceflinger can complete the rendering of 2D applications. Image rendering can also complete the rendering of 3D images, thereby enabling the user interface (User Interface, UI) of 2D applications to be displayed in the extended reality scene provided by 3D applications.

However, the existing rendering method relies on the surface flinger of Android system and has limited use.

Contents of the invention

Embodiments of the present application provide an image rendering method, device, equipment, media and program products. The rendering of 2D applications and the rendering of 3D applications are executed independently. The rendering of 2D applications is implemented by a 2D rendering synthesizer, and the rendering of 3D applications is implemented by 3D rendering. With the synthesizer implementation, the rendering of 3D applications no longer relies on the surface flinger of the Android system, so that the method in this embodiment is no longer limited to the Android system and can be flexibly run on various operating systems.

In a first aspect, embodiments of the present application provide an image rendering method applied to an XR device. The XR device includes a 2D rendering synthesizer, a 3D rendering synthesizer and a display screen. The method includes:

According to the opening instruction of the 2D application, launch the 2D application in the virtual screen;

The 2D rendering synthesizer obtains multiple layers of the 2D application from the virtual screen, synthesizes the multiple layers of the 2D application to obtain a 2D image of the 2D application, and sends the 2D image to The transmission component of the virtual screen;

The transmission component transmits the 2D image to the 3D rendering synthesizer;

The 3D rendering synthesizer renders and synthesizes the 2D image and the data to be rendered of the 3D application to obtain a 3D image of the 3D application. The 3D image displays the virtual screen, and the virtual screen displays the The 2D image;

The display screen displays the 3D image.

In some embodiments, the transmission component includes a buffer queue, the buffer queue is used to store the 2D image, and the transmission component uses a producer-consumer model to transmit the 2D image.

In some embodiments, before the display screen displays the 3D image, it further includes:

The 3D rendering synthesizer performs anti-dispersion processing on the 3D image, and sends the anti-dispersion processed 3D image to the display screen.

In some embodiments, the XR device further includes a hardware synthesizer HWC, and before the display screen displays the 3D image, it also includes:

The 3D rendering compositor sends the 3D image to the HWC;

The HWC performs anti-dispersion processing on the 3D image, and sends the anti-dispersion processed 3D image to the display screen.

In some embodiments, multiple 2D applications are launched in the 3D application, and the multiple 2D applications are displayed through multiple virtual screens, and each virtual screen can only be associated with one 2D application.

In some embodiments, the rendering frequency of the 2D rendering synthesizer is less than or equal to the rendering frequency of the 3D rendering synthesizer.

In some embodiments, the 3D rendering synthesizer renders and synthesizes the 2D image and the to-be-rendered data of the 3D application to obtain the 3D image of the 3D application, including:

Divide the 2D image into a left eye 2D image and a right eye 2D image;

Obtain the left eye data and right eye data of the 3D application from the buffer of the 3D application;

The left eye image and the right eye image of the 3D application are rendered and synthesized according to the left eye data, right eye data, the left eye 2D image and the right eye 2D image of the 3D application.

In some embodiments, when the XR device adopts the Android system, the 2D rendering synthesizer is the system service module surface flinger, and the transmission component includes the surface and Surface texture modules of the virtual screen. The Surface texture module includes a first buffer queue, and the first buffer queue is used to store the 2D image.

In some embodiments, the transmission component transmits the 2D image to the 3D rendering compositor, including:

The surface of the virtual screen stores the 2D image into the first buffer queue of the Surface texture module;

The 3D rendering compositor reads the 2D image from the first buffer queue.

In some embodiments, the first buffer queue uses a multi-level buffer.

In a second aspect, embodiments of the present application provide an image rendering device, including:

A startup module, used to start the 2D application in the virtual screen according to the startup instruction of the 2D application;

A 2D rendering synthesizer, configured to obtain multiple layers of the 2D application from the virtual screen, synthesize the multiple layers of the 2D application to obtain a 2D image of the 2D application, and send the 2D image A transmission component for the virtual screen;

A transmission component for transmitting the 2D image to the 3D rendering synthesizer;

A 3D rendering synthesizer, configured to render and synthesize the 2D image and the data to be rendered of the 3D application to obtain a 3D image of the 3D application. The virtual screen is displayed in the 3D image, and the virtual screen is displayed in the virtual screen. the 2D image;

A display module is used to display the 3D image.

In a third aspect, embodiments of the present application provide an XR device. The XR includes: a processor and a memory. The memory is used to store a computer program. The processor is used to call and run the computer program stored in the memory. To perform the method described in any one of the above first aspects.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium, the computer-readable storage medium being used to store a computer program, the computer program causing the computer to execute the method described in any one of the above-mentioned first aspects.

In a fifth aspect, embodiments of the present application provide a computer program product, including a computer program that implements the method described in any one of the above first aspects when executed by a processor.

The image rendering method, device, equipment, media and program products provided by the embodiments of the present application start the 2D application in the virtual screen according to the opening instruction of the 2D application, and the 2D rendering synthesizer obtains multiple layers of the 2D application from the virtual screen , synthesize multiple layers of the 2D application to obtain the 2D image of the 2D application, and send the 2D image to the transmission component; the transmission component transmits the 2D image to the 3D rendering synthesizer; the 3D rendering synthesizer processes the 2D image and the 3D application's to-be-rendered The data is rendered and synthesized to obtain a 3D image of the 3D application, and the 3D image is displayed on the display screen. The rendering of 2D applications and the rendering of 3D applications are executed independently. The rendering of 2D applications is implemented by the 2D rendering synthesizer, and the rendering of 3D applications is implemented by the 3D rendering synthesizer. The rendering of 3D applications no longer relies on the surface flinger of the Android system, making this implementation The example method is no longer limited to the Android system and can be flexibly run on various operating systems.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 is a schematic flowchart of an image rendering method provided in Embodiment 1 of the present application;

Figure 2 is a schematic diagram of the principle of the image rendering method under the Android system;

Figure 3 is a flow chart of the image rendering method provided in Embodiment 2 of the present application;

Figure 4 is a schematic structural diagram of an image rendering device provided in Embodiment 3 of the present application;

Figure 5 is a schematic structural diagram of the XR equipment provided in Embodiment 4 of the present application.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without any creative work fall within the scope of protection of the present invention.

It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances so that the embodiments of the invention described herein are capable of being practiced in sequences other than those illustrated or described herein. Furthermore, the terms "include" and "having" and any variations thereof are intended to cover non-exclusive inclusions, e.g., a process, method, system, product or server that encompasses a series of steps or units and need not be limited to those explicitly listed. Those steps or elements may instead include other steps or elements not expressly listed or inherent to the process, method, product or apparatus.

In order to facilitate understanding of the embodiments of the present application, before describing each embodiment of the present application, some concepts involved in all the embodiments of the present application are first properly explained, as follows:

1) Virtual Reality (VR for short), a technology for creating and experiencing a virtual world, determines the generation of a virtual environment, and is a type of multi-source information (the virtual reality mentioned in this article at least includes visual perception, in addition to Including auditory perception, tactile perception, motion perception, and even taste perception, olfactory perception, etc.), realizing the integration of virtual environment, interactive three-dimensional dynamic vision and simulation of entity behavior, allowing users to immerse themselves in the simulated virtual reality environment It can be used in various virtual environments such as maps, games, videos, education, medical care, simulation, collaborative training, sales, assisted manufacturing, maintenance and repair, etc.

2) Virtual reality equipment (VR equipment), a terminal that achieves virtual reality effects, can usually be provided in the form of glasses, a Head Mount Display (HMD), or contact lenses to achieve visual perception and other forms. Of course, the form of virtual reality equipment is not limited to this, and can be further miniaturized or enlarged according to actual needs.

Optionally, the VR devices described in the embodiments of this application may include but are not limited to the following types:

2.1) External VR equipment. The VR equipment and the external equipment are connected through wired or wireless methods. The external equipment performs calculations related to virtual reality functions and outputs data to the VR equipment. The external equipment can be a smartphone or a personal computer (person computer). , PC) and other electronic devices, when the external device is a PC, the external VR device is also called a computer-side virtual reality (PCVR) device.

2.2) All-in-one VR equipment has a processor for performing related calculations of virtual functions, so it has independent virtual reality input and output functions. It does not need to be connected to a PC or mobile terminal, and has a high degree of freedom of use.

3) Mixed Reality (MR): refers to the combination of real and virtual worlds to create new environments and visualizations. Physical entities and digital objects coexist and can interact in real time to simulate real objects. Mixed reality, augmented reality, augmented virtual and virtual reality technologies. MR is a composite of virtual reality (VR) and augmented reality (AR). Mixed reality (MR) is an expansion and extension of virtual reality (VR) technology. It can increase user experience by presenting virtual scenes in real scenes. sense of reality. The field of MR involves computer vision. Computer vision is a science that studies how to make machines "see". Furthermore, it refers to the machine vision of cameras and computers replacing human eyes in identifying, tracking and measuring targets, and further image processing. , processed by computers into images more suitable for human eye observation or transmitted to instruments for detection.

That is, MR is a simulated set that integrates computer-created sensory input (e.g., virtual objects) with sensory input from the physical set or its representation. In some MR sets, the computer-created sensory input can be adapted to the input from the physical set. changes in sensory input. Additionally, some electronic systems used to render the MR scenery may monitor orientation and/or position relative to the physical scenery to enable virtual objects to interact with real objects (i.e., physical elements from the physical scenery or representations thereof). For example, the system can monitor motion so that virtual plants appear stationary relative to physical buildings.

Figure 1 is a schematic flowchart of an image rendering method provided in Embodiment 1 of the present application. The method is executed by an XR device. The XR device can be a VR device, an AR device, or an MR device. This embodiment takes a VR device as an example for illustration. VR The device can be an integrated VR device or an external VR device. The XR device includes a 2D rendering synthesizer, a 3D rendering synthesizer and a display screen. The 2D rendering synthesizer is used to synthesize images for 2D applications. The 3D rendering synthesizer is used to synthesize images for 3D applications. It also supports virtualization in 3D applications. Display the content of 2D applications in real space. As shown in Figure 1, the method provided in this embodiment includes the following steps.

S101. Start the 2D application in the virtual screen according to the opening instruction of the 2D application.

2D applications refer to traditional applications that run on mobile phones, computers, tablets and other electronic devices. The images displayed to users by 2D applications are 2D images. 3D applications refer to applications running on XR devices. The images displayed to users by 3D applications are 3D images. The 2D applications include but are not limited to: video playback applications, short video applications, music applications, instant messaging applications, shopping software, etc.

The usage scenario of the embodiment of the present application is to display the content of the 2D application in the virtual reality space (also called extended reality space or virtual scene) of the 3D application. The user can input the opening command during the running of the 3D application or in the 3D desktop environment. The XR device starts the 2D application in the corresponding virtual screen according to the opening command.

The virtual screen can be understood as a hosting container for 2D applications. In some operating systems, the virtual screen can also be a container.

In one implementation, each virtual screen can only be associated with one 2D application. Then, after receiving the opening instruction, the 3D application can create a virtual screen and the transmission component corresponding to the virtual screen, and transfer the 2D application to the 2D application. Associated with the corresponding virtual screen. The virtual screen has an identity (ID) and also has parameters such as size and resolution. After the virtual screen is created, the content of the 2D application is displayed through the virtual screen. The transmission component corresponding to the virtual screen is used to transmit the image of the 2D application corresponding to the virtual screen to the 3D rendering synthesizer. The transmission component corresponding to the virtual screen can also be understood as the transmission component corresponding to the 2D application.

In another implementation, each virtual screen can be associated with multiple 2D applications. The user can open a new application in the created virtual screen. At this time, there is no need to re-create the virtual screen. However, a virtual screen Only one 2D application can be displayed at the same time. When a new 2D application is opened, the new 2D application will be displayed on the virtual screen, and the previously displayed 2D application will be switched to the background. Correspondingly, the XR device switches the 2D application previously displayed in the virtual screen to the background according to the opening instruction of the 2D application, and starts the 2D application in the virtual screen.

S102. The 2D rendering synthesizer obtains multiple layers of the 2D application from the virtual screen, synthesizes the multiple layers of the 2D application to obtain a 2D image of the 2D application, and sends the 2D image to the transmission component of the virtual screen.

The method of this embodiment separates the rendering of 2D applications and the rendering of 3D applications, which are implemented by two independent modules. The rendering of 2D applications is implemented by a 2D rendering synthesizer, and the rendering of 3D applications is implemented by a 3D rendering synthesizer. 3D The rendering compositor, also known as the XR compositor, decouples the rendering of 3D applications from the rendering of 2D applications. The rendering of 3D applications no longer relies on the surface flinger of the Android system, that is, the 3D application rendering is decoupled from the surface flinger, making it more flexible and more portable. It is no longer limited to the Android system, so that the method of the embodiment of the present application can Applies to any existing operating system.

When rendering each frame image of the 2D application, the 2D rendering synthesizer obtains the contents of multiple layers of the current frame image of the 2D application from the virtual screen, and synthesizes the multiple layers of the current frame image to obtain the current frame image. , the current frame image is the 2D image of the 2D application. The 2D image is the 2D image corresponding to the current frame. The 2D image is usually composed of multiple layers. For example, taking the 2D application as a video application, the 2D image of the current frame of the video application may include the following windows: video window, pop-up window Screen window, control menu window, advertising window, each window can correspond to one layer, or multiple windows correspond to one layer, for example, the barrage window and the control menu window correspond to one layer.

In one implementation, the 2D rendering synthesizer reads each layer of the current frame from the virtual screen, and each layer is generated by the 2D application. In another implementation, the 2D rendering synthesizer reads the layers from the virtual screen. Each layer is generated according to the data of each layer of the current frame, where the data of each layer refers to the content displayed in each window. For example, the content of the video window is the video data.

In this embodiment, each window can correspond to a layer as an example. The 2D rendering synthesizer reads the layers corresponding to the video window, pop-up window, control menu window and advertising window from the virtual screen, and converts the video window, pop-up window The layers corresponding to the screen window, control menu window and advertising window are synthesized to obtain the current frame image of the video application.

For example, the 2D rendering compositor performs compositing based on multiple layers of the 2D application and other attributes of the image, for example, compositing based on the position of the window, transparency (such as semi-transparent, fully transparent), color and other attributes, and finally obtains the result. Image displayed by the user.

It is understandable that the 2D rendering compositor may have different names in different operating systems. For example, in the Android system, the 2D rendering compositor is the system service module surface flinger. Surface flinger is located at the system layer in the layered architecture of XR devices. , while 2D applications and 3D applications are located at the application layer.

S103. The transmission component transmits the 2D image to the 3D rendering synthesizer.

The transmission component can be understood as the transmission channel between the 2D rendering synthesizer and the 3D rendering synthesizer. Optionally, the transmission component includes a buffer queue. The buffer queue is used to store 2D images. The transmission component uses producer consumption. or model to transfer 2D images.

There are two roles in the producer-consumer model: producer and consumer. The producer and the consumer communicate through the buffer queue (memory buffer). The producer generates the data that the consumer needs, and the data produced by the producer is stored in In the buffer queue, consumers read data from the buffer queue for consumption without affecting each other and with low coupling.

If the producer-consumer model is not adopted, there is a direct calling relationship between the producer and the consumer. The generation speed and consumption speed may affect each other, and the next production must be carried out after consumption.

Optionally, the buffer queue uses a multi-level buffer (buffer), for example, level 3 buffer or level 4 buffer. When the multi-level buffer works in exchange mode, take the level 3 buffer as an example. For example, the first frame image uses The first buffer is used to draw, the second frame image is drawn using the second buffer, the third frame image is drawn using the third buffer, the fourth frame image is drawn using the first buffer, and so on. At the same time, the producer may be drawing data into the first buffer. At this time, the second buffer is used to store the synthesized data, and the consumer can consume the data in the second buffer. The third buffer may be a free buffer or a buffer that has been synthesized, that is, it is used to store the synthesized data.

In this embodiment, the 2D rendering synthesizer is the producer, and the 3D rendering synthesizer is the consumer. The 2D rendering synthesizer stores the produced 2D images into the buffer queue in the transmission component, and the 3D rendering synthesizer stores the generated 2D images in the buffer queue. The synthesized 2D image is consumed.

S104. The 3D rendering synthesizer renders and synthesizes the 2D image and the data to be rendered of the 3D application to obtain a 3D image of the 3D application. The 3D image displays a virtual screen, and the virtual screen displays a 2D image.

For example, the 3D rendering synthesizer divides the 2D image into a left-eye 2D image and a right-eye 2D image, because the data to be rendered in the 3D application is divided into left-eye data and right-eye data. Then, obtain the left eye data and right eye data of the 3D application from the buffer of the 3D application, and perform rendering and synthesis based on the left eye data, right eye data, left eye 2D image and right eye 2D image of the 3D application to obtain 3D The left eye image and the right eye image of the application.

The left eye data and right eye data of 3D applications can be stored in different buffers respectively. Usually two buffers are set up: left eye buffer and right eye buffer. The left eye buffer is used to store left eye data, and the right eye buffer is used to store right eye data. eye data. Optionally, the buffers of 3D applications also use multi-level buffers. For example, the left-eye buffer and the right-eye buffer both use 3-level buffers, and multi-level buffers work in exchange mode.

When rendering and synthesizing, the 3D rendering synthesizer renders and synthesizes the left eye data of the 3D application and the left eye 2D image to obtain the left eye image of the 3D application, and renders and synthesizes the right eye data and the right eye 2D image of the 3D application to obtain the 3D application. image of the right eye.

The 2D image is displayed through the virtual screen in the left eye image and the right eye image of the 3D application respectively, thereby enabling the 2D application to be displayed in the extended reality space of the 3D application.

Since the 2D rendering synthesizer and the 3D rendering synthesizer can render independently, the rendering frequency of the 2D rendering synthesizer and the 3D rendering synthesizer corresponds to the frame rate. The frame rate is the continuous image in frames. The frequency that appears on the display can be considered to be the rendering frequency equal to the frame rate.

In the prior art, surface flinger uses the same rendering frequency to render 2D applications and 3D applications. In this embodiment, the rendering frequency of the 2D rendering synthesizer can not only be equal to the rendering frequency of the 3D rendering synthesizer, but also the rendering frequency of the 2D rendering synthesizer. The rendering frequency can also be less than that of the 3D rendering compositor. When the rendering frequency of the 2D rendering synthesizer is less than the rendering frequency of the 3D rendering synthesizer, the update frequency of the extended reality space provided by the 3D application is greater than the update frequency of the 2D image displayed in the extended reality space.

In order to provide users with realistic scenes, 3D applications usually use higher frame rates, while users usually have lower display requirements for 2D applications and do not need to use the same frame rate as 3D applications. The methods of the existing technology, because 2D applications The rendering frequency of 2D applications must be the same as that of 3D applications, which increases rendering costs. However, according to the method of the embodiment, the rendering frequency of 2D applications can be lower than that of 3D applications, which reduces rendering costs.

In this embodiment, the 2D rendering synthesizer and the 3D rendering synthesizer are two independent modules. The 2D rendering synthesizer independently completes the rendering of 2D applications, and the 3D rendering synthesizer independently completes the rendering of 3D applications, and the 3D rendering synthesizer supports 2D applications. The 2D image is displayed in 3D space. The 3D rendering synthesizer is separated from the surface flinger of the Android system, so that the method in this embodiment is no longer limited to the Android system and can be flexibly run in various operating systems. The 2D rendering synthesizer and 3D rendering synthesizer can be developed, upgraded, and maintained independently, and low coupling makes management easier.

S105. Display the 3D image on the display screen.

After the 3D rendering synthesizer renders the 3D image, it submits the 3D image to the display screen for display.

In one implementation, before the 3D rendering synthesizer submits the 3D image to the display screen, the 3D rendering synthesizer performs anti-dispersion processing on the 3D image, and sends the anti-dispersion processed 3D image to the display screen.

Inverse dispersion is used to reversely compensate for the dispersion that occurs in 3D images, compensate for the color of the 3D image, and offset the impact of dispersion on the image.

In another implementation, the XR device also includes a hardware synthesizer (Hardware Composer, HWC). The 3D rendering synthesizer sends the 3D image to the HWC. The HWC performs anti-dispersion processing on the 3D image, and converts the anti-dispersion processed 3D image into the HWC. The image is sent to the display screen.

The method of this embodiment starts the 2D application in the virtual screen according to the opening instruction of the 2D application. The 2D rendering synthesizer obtains multiple layers of the 2D application from the virtual screen, and synthesizes the multiple layers of the 2D application to obtain the 2D application. The 2D image is sent to the transmission component; the transmission component transmits the 2D image to the 3D rendering synthesizer; the 3D rendering synthesizer renders and synthesizes the 2D image and the data to be rendered in the 3D application to obtain the 3D image of the 3D application, and displays the screen Display 3D images. The rendering of 2D applications and the rendering of 3D applications are independent. The rendering of 2D applications is implemented by the 2D rendering synthesizer, and the rendering of 3D applications is implemented by the 3D rendering synthesizer. The rendering of 3D applications no longer relies on the surface flinger of the Android system, making this implementation The example method is no longer limited to the Android system and can be flexibly run on various operating systems.

On the basis of the first embodiment, the second embodiment of the present application provides an image rendering method, which is applied in the Android system. Figure 2 is a schematic diagram of the principle of the image rendering method under the Android system. As shown in Figure 2, in the Android system The 2D rendering synthesizer is surface flinger. The transmission component includes the surface and Surface texture modules of the virtual screen. The Surface texture module includes a first buffer queue. The first buffer queue is used to store 2D images. The first buffer queue shown in the figure uses a multi-layer Level buffer, in the Android system, the buffer of the Surface texture module is also called the drawing buffer (ie Graphic buffer).

Figure 3 is a flow chart of an image rendering method provided in Embodiment 2 of the present application. The method of this embodiment is described with reference to Figures 2 and 3. As shown in Figure 3, the method provided in this embodiment includes the following steps.

S201. In the virtual reality space of the 3D application, according to the opening instruction of the 2D application, create a virtual screen and a surface of the virtual screen of the 2D application, and start the 2D application in the virtual screen.

As shown in Figure 2, multiple 2D applications can be opened in a 3D application: 2D app A, 2D app B, and 2D app C. Each 2D application corresponds to a virtual screen.

When the user triggers the opening command of any 2D application, the XR device creates a virtual screen for the 2D application according to the opening command, and creates the surface of the virtual screen. The 2D application is bound to the virtual screen and the surface of the virtual screen.

S202. The surface flinger obtains multiple layers of the 2D application from the virtual screen, synthesizes the multiple layers of the 2D application onto the surface of the virtual screen, and obtains a 2D image of the 2D application.

As shown in Figure 2, a 2D application can include multiple 2D activity components and can also include a 2Dwindow component.

The activity component is usually a separate screen in an Android application. It can display some controls and listen to and process user events to respond. The activity component usually provides a full-screen interface, while the window component provides a non-full-screen interface. , for example, pop-up windows, dialog boxes, etc.

Surface flinger obtains the layer of the 2D application from the activity component of the 2D application, or the activity component and window component of the 2D application. Normally, one activity component corresponds to one layer, and one window component corresponds to one layer. In some special cases, In some cases, an activity component may correspond to multiple layers. For example, when a video is displayed in the activity component, the video is a layer and the activity component itself is a layer.

The surface of the virtual screen can be understood as a virtual screen or a canvas of a 2D application, which is used to render the image of the 2D application onto the canvas so that the content of the 2D application can be displayed on the virtual screen.

S203. The surface of the virtual screen stores the 2D image in the first buffer queue of the Surface texture.

The surface of the virtual screen and the Surface texture are bound. The surface of the virtual screen can be understood as an interface between the surface flinger and the Surface texture. The 2D image synthesized by the surface flinger is finally stored in the first buffer queue of the Surface texture.

In this embodiment, the existing Surface texture in the Android system is used to transfer the 2D image to the 3D rendering synthesizer. The creation and principle of the Surface texture will not be described in detail here. The first buffer queue in the Surface texture can use a multi-level buffer. Data is transferred between multi-level buffers through exchange mode.

S204. The 3D rendering synthesizer reads the 2D image from the first buffer queue of the Surface texture.

When multiple 2D applications are opened in a 3D application, the 3D rendering synthesizer reads the 2D image from the Surface texture corresponding to each 2D application.

S205. The 3D rendering synthesizer reads left eye data from the left eye buffer of the 3D application and reads right eye data from the right eye buffer of the 3D application.

S206. The 3D rendering synthesizer performs rendering and synthesis based on the left eye data, right eye data and 2D image of the 3D application to obtain a 3D image, and sends the 3D image to the HWC.

When multiple 2D applications are opened in a 3D application, for each frame of image, the 3D rendering synthesizer starts rendering the current frame image after acquiring the 2D images of the multiple opened applications and the data to be rendered of the 3D application. The 3D rendering synthesizer divides the 2D image into a left-eye 2D image and a right-eye 2D image, renders the left-eye 2D image and the left-eye image of the 3D application together to obtain a 3D image of the left eye, and combines the right-eye 2D image and the 3D application The right eye image is rendered together to obtain the 3D image of the right eye.

The 3D image rendered and synthesized by the 3D rendering synthesizer is stored in the Graphic buffer of the 3D rendering synthesizer. In Figure 2, the data stored in the Graphic buffer of the rendering synthesizer and the Graphic buffer of the Surface texture are different. The data stored in the Graphic buffer of the 3D rendering synthesizer is different. The data is a 3D image, and the data stored in the Graphicbuffer of the Surface texture is a 2D image.

S207. The HWC performs anti-dispersion processing on the 3D image, and sends the anti-dispersion processed 3D image to the display screen.

S208. Display the 3D image on the display screen.

The 3D image includes the content of the 2D application, and the content of the 2D application is displayed on the virtual screen. In the example shown in Figure 2, the content of three 2D applications is displayed in the 3D image, that is, the virtual reality space provided by the 3D application. Among them, the initial position of the virtual screen of each 2D application may be at a fixed position. Users can adjust the position, size, posture, etc. of the virtual screen according to their own needs. They can also close the virtual screen. Closing the virtual screen means closing the virtual screen. Display 2D applications.

In this embodiment, the rendering of 3D applications is separated from the surface flinger of the Android system. The surface flinger performs image rendering of 2D applications. The rendering of 3D applications is completed by an independent 3D rendering synthesizer. The surface flinger renders images of 2D applications. After completion, the 2D image is passed to the 3D rendering synthesizer through the Surface texture. The 3D rendering synthesizer renders the 3D application's data to be rendered and the 2D image together to obtain a 3D image. The 3D rendering synthesizer is separated from surface flinger to facilitate the expansion and upgrade of the 3D rendering synthesizer.

In order to facilitate better implementation of the image rendering method in the embodiment of the present application, the embodiment of the present application also provides an image rendering device. Figure 4 is a schematic structural diagram of an image rendering device provided in Embodiment 3 of the present application. As shown in Figure 4, the image rendering device 100 may include:

The startup module 11 is used to start the 2D application in the virtual screen according to the startup instruction of the 2D application;

The 2D rendering synthesizer 12 is used to obtain multiple layers of the 2D application from the virtual screen, synthesize the multiple layers of the 2D application to obtain a 2D image of the 2D application, and convert the 2D image into Sent to the transmission component of the virtual screen;

Transmission component 13, used to transmit the 2D image to the 3D rendering synthesizer;

The 3D rendering synthesizer 14 is used to render and synthesize the 2D image and the to-be-rendered data of the 3D application to obtain the 3D image of the 3D application. The virtual screen is displayed in the 3D image, and the virtual screen displays There is said 2D image;

The display module 15 is used to display the 3D image.

In some embodiments, the transmission component includes a buffer queue, the buffer queue is used to store the 2D image, and the transmission component uses a producer-consumer model to transmit the 2D image.

In some embodiments, the 3D rendering synthesizer 14 is also configured to: perform anti-dispersion processing on the 3D image, and send the anti-dispersion processed 3D image to the display module 15 .

In some embodiments, the XR device further includes a hardware compositor HWC, and the 3D rendering compositor 14 is also used to: send the 3D image to the HWC;

The HWC is used to perform anti-dispersion processing on the 3D image, and send the anti-dispersion processed 3D image to the display module 15 .

In some embodiments, multiple 2D applications are launched in the 3D application, and the multiple 2D applications are displayed through multiple virtual screens, and each virtual screen can only be associated with one 2D application.

In some embodiments, the rendering frequency of the 2D rendering synthesizer is less than or equal to the rendering frequency of the 3D rendering synthesizer.

In some embodiments, the 3D rendering synthesizer 14 is specifically used to:

Divide the 2D image into a left eye 2D image and a right eye 2D image;

Obtain the left eye data and right eye data of the 3D application from the buffer of the 3D application;

The left eye image and the right eye image of the 3D application are rendered and synthesized according to the left eye data, right eye data, the left eye 2D image and the right eye 2D image of the 3D application.

In some embodiments, when the XR device adopts the Android system, the 2D rendering synthesizer is the system service module surface flinger, and the transmission component includes the surface and Surface texture modules of the virtual screen. The Surface texture module includes a first buffer queue, and the first buffer queue is used to store the 2D image.

In some embodiments, the surface of the virtual screen stores the 2D image into the first buffer queue of the Surfacetexture module;

The 3D rendering compositor reads the 2D image from the first buffer queue.

In some embodiments, the first buffer queue uses a multi-level buffer.

It should be understood that the device embodiments and the method embodiments may correspond to each other, and similar descriptions may refer to the method embodiments. To avoid repetition, they will not be repeated here.

The devices 100 and 200 in the embodiments of the present application are described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in the form of hardware, can also be implemented through instructions in the form of software, or can also be implemented through a combination of hardware and software modules. Specifically, each step of the method embodiments in the embodiments of the present application can be completed by integrated logic circuits of hardware in the processor and/or instructions in the form of software. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly embodied in hardware. The execution of the decoding processor is completed, or the execution is completed using a combination of hardware and software modules in the decoding processor. Optionally, the software module may be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in combination with its hardware.

An embodiment of the present application also provides an XR device. Figure 5 is a schematic structural diagram of an XR device provided in Embodiment 4 of the present application. As shown in Figure 5, the XR device 300 may include:

Memory 31 and processor 32. The memory 31 is used to store computer programs and transmit the program codes to the processor 32. In other words, the processor 32 can call and run the computer program from the memory 31 to implement the method in the embodiment of the present application.

For example, the processor 32 may be configured to execute method steps performed by the XR device or server in the above method embodiment according to instructions in the computer program. Correspondingly, the XR device 300 is an XR device or server.

In some embodiments of the present application, the processor 32 may include but is not limited to:

General processor, Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) or other programmable logic devices, discrete gates or Transistor logic devices, discrete hardware components, and more.

In some embodiments of the present application, the memory 31 includes but is not limited to:

Volatile memory and/or non-volatile memory. Among them, the non-volatile memory can be read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), electrically removable memory. Erase programmable read-only memory (Electrically EPROM, EEPROM) or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of illustration, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double DataRate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synch link DRAM, SLDRAM) ) and direct memory bus random access memory (Direct Rambus RAM, DR RAM).

In some embodiments of the present application, the computer program can be divided into one or more modules, and the one or more modules are stored in the memory 31 and executed by the processor 32 to complete the tasks provided by the present application. method. The one or more modules may be a series of computer program instruction segments capable of completing specific functions. The instruction segments are used to describe the execution process of the computer program in the XR device.

As shown in FIG. 5 , the XR device may further include a transceiver 33 , and the transceiver 33 may be connected to the processor 32 or the memory 31 .

The processor 32 can control the transceiver 33 to communicate with other devices. Specifically, it can send information or data to other devices, or receive information or data sent by other devices. Transceiver 33 may include a transmitter and a receiver. The transceiver 33 may further include an antenna, and the number of antennas may be one or more.

It can be understood that, although not shown in FIG. 5 , the XR device 300 may also include a camera module, a wireless fidelity WIFI module, a positioning module, a Bluetooth module, a display, a controller, etc., which will not be described again here.

It should be understood that various components in the XR device are connected through a bus system, where in addition to the data bus, the bus system also includes a power bus, a control bus and a status signal bus.

This application also provides a computer storage medium on which a computer program is stored. When executed by a computer, the computer program enables the computer to execute the method executed by the XR device or server in the above method embodiments, which will not be described again here.

The application also provides a computer program product. The computer program product includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of the XR device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the XR device executes the method executed by the XR device or the server in the above method embodiment, which will not be described again here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components may be combined or may be Integrated into another system, or some features can be ignored, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or modules, and may be in electrical, mechanical or other forms.

Modules described as separate components may or may not be physically separated, and components shown as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. For example, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically alone, or two or more modules can be integrated into one module.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, and they should be covered by within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims (14)

1. An image rendering method, applied to an XR device comprising a 2D rendering compositor, a 3D rendering compositor, and a display screen, the method comprising:

Starting the 2D application in the virtual screen according to an opening instruction of the 2D application;

the 2D rendering synthesizer acquires a plurality of layers of the 2D application from the virtual screen, and the 2D image of the 2D application is obtained by combining the plurality of layers of the 2D application, and the 2D image is sent to a transmission component of the virtual screen;

the transmission component transmits the 2D image to the 3D rendering compositor;

the 3D rendering synthesizer renders and synthesizes the 2D image and the data to be rendered of the 3D application to obtain a 3D image of the 3D application, the 3D image is displayed with the virtual screen, and the virtual screen is displayed with the 2D image;

the display screen displays the 3D image.

2. The method of claim 1, wherein the transmission component comprises a buffer queue therein for storing the 2D image, the transmission component transmitting the 2D image using a producer consumer model.

3. The method of claim 1, wherein prior to the displaying of the 3D image by the display screen, further comprising:

and the 3D rendering synthesizer carries out inverse dispersion processing on the 3D image and sends the 3D image subjected to the inverse dispersion processing to the display screen.

4. The method of claim 1, wherein the XR device further comprises a hardware synthesizer HWC, the display screen further comprising, prior to displaying the 3D image:

the 3D rendering compositor sends the 3D image to the HWC;

and the HWC carries out inverse dispersion processing on the 3D image, and sends the 3D image after the inverse dispersion processing to the display screen.

5. The method of claim 1, wherein a plurality of 2D applications are launched in the 3D application, the plurality of 2D applications being displayed through a plurality of virtual screens, each virtual screen being capable of associating only one 2D application.

6. The method of claim 1, wherein a rendering frequency of the 2D rendering compositor is less than or equal to a rendering frequency of the 3D rendering compositor.

7. The method according to claim 1, wherein the 3D rendering compositor renders and composites the 2D image and data to be rendered of a 3D application to obtain a 3D image of the 3D application, comprising:

dividing the 2D image into a left-eye 2D image and a right-eye 2D image;

acquiring left eye data and right eye data of the 3D application from a buffer zone of the 3D application;

And rendering and synthesizing according to the left-eye data, the right-eye data, the left-eye 2D image and the right-eye 2D image of the 3D application to obtain the left-eye image and the right-eye image of the 3D application.

8. The method of any of claims 1-7, wherein when the XR device employs an android system, the 2D rendering compositor is a system service flinger, the transport component comprises a Surface and a Surface texture module of the virtual screen, and the Surface texture module comprises a first buffer queue, the first buffer queue is configured to store the 2D images.

9. The method of claim 8, wherein the transmitting component transmits the 2D image to the 3D rendering compositor comprises:

the Surface of the virtual screen stores the 2D image into the first buffer queue of the Surface texture module;

the 3D rendering compositor reads the 2D image from the first buffer queue.

10. The method of claim 9, wherein the first buffer queue employs a multi-level buffer.

11. An image rendering apparatus, comprising:

the starting module is used for starting the 2D application in the virtual screen according to the starting instruction of the 2D application;

A 2D rendering synthesizer, configured to obtain multiple layers of the 2D application from the virtual screen, combine the multiple layers of the 2D application to obtain a 2D image of the 2D application, and send the 2D image to a transmission component of the virtual screen;

a transmission component for transmitting the 2D image to the 3D rendering compositor;

a 3D rendering synthesizer, configured to render and synthesize the 2D image and data to be rendered of a 3D application to obtain a 3D image of the 3D application, where the 3D image is displayed with the virtual screen, and the virtual screen is displayed with the 2D image;

and the display module is used for displaying the 3D image.

12. An XR device comprising:

a processor and a memory for storing a computer program, the processor being for invoking and running the computer program stored in the memory to perform the method of any of claims 1 to 10.

13. A computer readable storage medium storing a computer program for causing a computer to perform the method of any one of claims 1 to 10.

14. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1 to 10.