CN114764848A - Scene illumination distribution estimation method - Google Patents

Abstract

The application discloses an illumination distribution estimation method which is mainly applied to the fields of virtual reality and augmented reality. The method comprises the steps of constructing a cubic mapping, projecting an image shot by a camera onto the cubic mapping based on an equipment orientation matrix and a camera projection matrix, then constructing a mirror sphere inside a space formed by the cubic mapping, using the mirror sphere to map and project to obtain a scene environment map, and then determining the environment map with a high dynamic range based on a neural network. The environment graph obtained by the method can obtain the illumination effect with high reality and improve the quality of illumination distribution estimation.

Description

A method of scene illumination distribution estimation

technical field

The present application relates to the technical field of virtual reality and augmented reality, and in particular, to a method for estimating scene illumination distribution for mobile devices.

Background technique

In recent years, with the rapid development of hardware devices and algorithms, Virtual Reality (VR, Virtual Reality), Augmented Reality (AR, Augmented Reality) and Mixed Reality (MR, Mixed Reality) technologies are used in education and training, military, medical, entertainment It is widely used in manufacturing and other industries. Among them, the fusion of virtual and real with a high degree of realism is the core requirement of related applications. Therefore, only by keeping the light source and lighting model of the virtual object consistent with the real scene, a realistic rendering effect can be obtained. However, the traditional scene illumination estimation method has inconsistent camera parameters on different mobile phones and only uses a single image as input. Therefore, in practical applications, the rendering effect on different mobile phones is inconsistent, and for continuous scenes The input is jittery and renders poorly.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a scene illumination distribution estimation method, which can improve the accuracy of scene illumination estimation, so that when virtual objects are superimposed in a real scene, the virtual objects can have illumination information more consistent with the environment, and improve the accuracy of virtual objects. Display effect, improve user experience.

The above-mentioned objects and other objects are achieved by the features of the independent claims. Further implementations are embodied in the dependent claims, the description and the drawings.

A first aspect provides an illumination estimation method, comprising the following steps: acquiring a first image; determining a device orientation matrix and a camera projection matrix corresponding to the first image; according to the device orientation matrix and the camera projection matrix , project the first image onto the space cube map; according to the device orientation matrix and the space cube map, use the specular ball mapping projection to obtain a scene environment map; according to the first image and the scene environment map , to determine the high dynamic range environment map.

The technical solution described in the first aspect can combine cube maps and specular sphere mapping to obtain illumination information in real scenes; in the obtained high dynamic range environment map, through illumination estimation, unobserved pixels are supplemented The lighting information of the point improves the effect of lighting estimation.

According to the first aspect, in a possible implementation manner, the device orientation matrix is obtained through an inertial measurement unit.

According to the first aspect, in a possible implementation manner, the determining a device orientation matrix corresponding to the first image further includes: determining a first moment at which the first image is acquired; , and the closest IMU data to the first moment is used as the device orientation matrix corresponding to the first image.

According to the first aspect, in a possible implementation, the projecting the first image onto a spatial cubemap according to the device orientation matrix and the camera projection matrix, further comprising: projecting the first image through the camera projection matrix , transform the first coordinate of the first pixel into a first direction vector; wherein, the first pixel is a pixel in the first image; through the inverse matrix of the device orientation matrix, convert the first pixel A direction vector is transformed into a second coordinate.

According to the first aspect, in a possible implementation manner, according to the device orientation matrix and the space cube map, using a specular sphere mapping projection to obtain a scene environment map, further comprising: in the cube map composed of Rendering a first specular sphere in a cube environment; determining an environment direction reflected by a second pixel on the spherical surface of the first specular sphere according to the device orientation matrix; obtaining corresponding pixels on the cube map according to the environment direction color or brightness; fill the color or brightness into the second pixel.

According to the first aspect, in a possible implementation manner, the determining a high dynamic range environment map according to the first image and the scene environment map, further includes: using a deep convolutional neural network to determine the High dynamic range environment map.

According to the first aspect, in a possible implementation manner, the deep convolutional neural network includes an image feature extraction network, an observed environment map feature extraction network, a feature fusion and a lighting generation network, wherein the image features The extraction network is used to extract the first feature vector from the first image; the observed environment map feature extraction network is used to extract the second feature vector from the scene environment map; the feature fusion and illumination generation The network is configured to generate the high dynamic range environment map according to the first feature vector and the second feature vector.

According to the first aspect, in a possible implementation manner, the extracting the second feature vector from the scene environment map further includes: marking observed pixels and unobserved pixels in the scene environment map point; according to the observed pixel point, determine the second feature vector.

According to the first aspect, in a possible implementation manner, the generating the high dynamic range environment map according to the first feature vector and the second feature vector further includes: converting the first feature The vector and the second feature vector are spliced to generate a third feature vector; and the high dynamic range environment map is generated according to the third feature vector.

According to the first aspect, in a possible implementation manner, after determining the high dynamic range environment map according to the first image and the scene environment map, the method further includes: according to the high dynamic range environment map , and render the virtual object; and fuse the rendered virtual object into the first image.

The illumination estimation method provided in the first aspect utilizes the sensor in the electronic device to obtain the device orientation information, and combines with the camera to record the scene content observed by the user during the use of the electronic device, so as to provide more effective input information to improve. The quality of the scene lighting distribution estimate. In addition, in the process of using the related application, the user usually changes the observation angle constantly. Therefore, the above method can continuously acquire more scene information as the application session progresses, and further improve the accuracy of the illumination distribution. Since this method maintains a global scene information, compared with the current illumination estimation method that only uses the so-called input of a single image, this method can provide better continuity constraints and reduce flicker.

In a second aspect, there is provided an electronic device comprising one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to The one or more programs are executed by the one or more processors, and the one or more programs include instructions, and the instructions cause the electronic device to perform each implementation method according to the first aspect.

In a third aspect, a computer-readable medium is provided for storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs The program includes instructions that cause the electronic device to perform each implementation method of the first aspect.

It should be understood that the description of technical features, technical solutions, advantages or similar language in the specification does not imply that all features and advantages can be realized in any single embodiment. Rather, it is to be understood that a description of a feature or advantage is meant to include a particular technical feature, technical solution or advantage in at least one embodiment. Therefore, descriptions of technical features, technical solutions or advantages in this specification do not necessarily refer to the same embodiments. Furthermore, the technical features, technical solutions and advantages described in each of the following embodiments can also be combined in any appropriate manner. Those skilled in the art will appreciate that an embodiment may be implemented without one or more of the specific technical features, technical solutions or advantages of a specific embodiment. In other embodiments, additional technical features and advantages may also be identified in specific embodiments that do not embody all embodiments.

Description of drawings

FIG. 1 is a flowchart of the illumination estimation method provided by the present application.

Figure 2 is a schematic diagram of the cycle of data reported by the camera and the IMU.

FIG. 3 is a schematic diagram of a cubemap provided in an embodiment of the present application.

FIG. 4 is a schematic diagram of a deep convolutional neural network provided in an embodiment of the present application.

FIG. 5 is a flowchart of illumination estimation performed by the deep convolutional neural network provided in the embodiment of the present application.

FIG. 6 is a SSIM comparison diagram of the present invention and the prior art illumination estimation method provided in the embodiment of the present application.

FIG. 7 is a comparison diagram of PSNR between the illumination estimation method of the present invention and the prior art illumination estimation method provided in the embodiment of the present application.

Detailed ways

The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations of the present application. As used in the specification of this application and the appended claims, the singular expressions "a," "an," "the," "above," "the," and "the" are intended to also Plural expressions are included unless the context clearly dictates otherwise. It will also be understood that, as used in this application, the term "and/or" refers to and includes any and all possible combinations of one or more of the listed items.

或者其它操作系统的便携式电子设备。上述便携式电子设备也可以是其它便携式电子设备，诸如具有触敏表面或触控面板的膝上型计算机(Laptop)等。还应当理解的是，在其他一些实施例中，上述电子设备也可以不是便携式电子设备，而是具有触敏表面或触控面板的台式计算机。Electronic devices, graphical user interfaces for such electronic devices, and embodiments for using such electronic devices are described below. In some embodiments, the electronic device may be a portable electronic device such as a cell phone, tablet, wearable electronic device (eg, virtual reality glasses, augmented reality glasses, etc.) )Wait. Exemplary embodiments of portable electronic devices include, but are not limited to, carry-on

Or portable electronic devices with other operating systems. The portable electronic device described above may also be other portable electronic devices, such as a laptop computer (Laptop) with a touch-sensitive surface or a touch panel, or the like. It should also be understood that, in some other embodiments, the above-mentioned electronic device may not be a portable electronic device, but a desktop computer having a touch-sensitive surface or a touch panel.

The following embodiments of the present application provide a scene illumination estimation method, which can improve the quality of scene illumination distribution estimation by combining the orientation information of the device and the data obtained by the camera.

The scene illumination estimation method of the present application will be described in detail below with reference to specific embodiments. Figure 1 shows the scene illumination estimation method of the present application.

S102: The electronic device obtains the image It and the device orientation matrix M _{view,t at time t .}

Specifically, the electronic device can obtain the image It through the camera in response to the user's operation. Wherein, the camera may be a front camera or a rear camera installed on the electronic device. Therefore, the above acquired images are real-world images. In addition, the electronic device can obtain the above-mentioned device orientation matrix M _view,t through an inertial measurement unit (IMU), which is also called a viewing angle matrix, and represents the attitude information of the electronic device relative to the reference coordinate system at time t. The above-mentioned reference coordinate system may be an earth coordinate system. The device orientation matrix M _view,t can be a 3*3 or 4*4 matrix. Specifically, M _view,t can be expressed in the following two ways:

The device orientation matrix M _{view, t represents} the pose of the camera when taking the image It. In some embodiments, in an electronic device such as a mobile phone, the orientation of the camera and the screen of the mobile phone are generally the same. Therefore, the attitude information of the mobile _phone when the image It is captured can obtain the angle of view of the camera relative to the earth coordinate system. The device is oriented towards the matrix M _view,t . The electronic device can obtain the above posture information through system-related APIs. Taking Android as an example, the attitude information of the mobile phone screen relative to the earth coordinate system can be monitored through the rotation vector sensor (RV-sensor), such as the Euler angle represented by the three variables of roll, yaw and pitch. In some other embodiments, three variables of roll, pitch and azimuth can also be used to represent the posture of the electronic device. In some other embodiments, if the camera and the screen of the mobile phone have a certain positional relationship, the device orientation matrix M _view,t can also be obtained by converting the relative positional relationship.

When using the camera to acquire an image, the camera can be triggered to capture an image by a user's operation. For example, the user can capture an image by touching a capture button displayed in a graphical user interface of a mobile phone or pressing a physical button. In some other embodiments, the camera can be set to automatically acquire multiple images, for example, the camera can be set to shoot images according to a certain period of time, or the user can be prompted to move the mobile phone on the graphical user interface. During the process of the mobile phone, the mobile phone can automatically acquire multiple images. The camera may be set to acquire one image, or the camera may be set to acquire an image sequence including multiple images. During the process of acquiring an image, the electronic device may be kept facing one direction to shoot, or the electronic device may be directed to shoot in different directions. When the electronic device is photographed in one direction, the device orientation matrix can remain unchanged, and when the electronic device is photographed in different directions, the device orientation matrix will change with the change in the orientation of the electronic device.

When the camera is set to automatically acquire images, the camera and the inertial measurement unit will periodically provide image data and device attitude data respectively. Since the camera and the inertial measurement unit are independent hardware devices, the above two data need to be synchronized when the electronic device collects the image and the device orientation matrix. Generally speaking, the inertial measurement unit is highly sensitive and has a low delay in collecting data, while the camera provides a large amount of data and requires a certain amount of processing, so the delay in collecting data is high. In some embodiments, as shown in FIG. 2 , the period in which the camera collects image data is greater than the period in which the inertial measurement unit reports data. Therefore, when the electronic device collects the camera image, the previous IMU data that is closest to the time when the camera image is collected in the system time is used as the device orientation matrix corresponding to the collected image. Specifically, the data of the inertial measurement unit can be collected through the rotation vector Rotation_Vector() function in the sensor management component SensorManager in the system API.

S104: The electronic device obtains the camera projection matrix M _proj , and projects the image It onto a spatial cubemap (Cubemap) in combination with the device orientation matrix M _{view,t .}

The camera projection matrix is used to represent the hardware performance of the camera, including but not limited to parameters such as a field of view (Field of View, FOV) and an aspect ratio of the screen. Generally speaking, in the case of optical zooming without using a lens, the projection matrix M _proj of the camera is usually a fixed value, which can be obtained directly or indirectly through device parameters. For example, the electronic device can obtain the information required by the above-mentioned camera projection matrix by calling a third-party API, such as Huawei AR Engine, to generate the camera projection matrix M _proj . You can also obtain the physical size of the camera sensor through the cameraCharacteristics function in the system API to obtain the SENSOR_INFO_PHYSICAL_SIZE parameter to obtain the camera projection matrix. However, when the camera of the electronic device can perform optical zooming, the projection matrix M _proj of the camera will change as the user adjusts.

In some embodiments, the electronic device may acquire the information required by the camera projection matrix before step S102 or when step S102 is performed.

FIG. 3 exemplarily shows the cubemap mentioned in this application. Cubemap contains six two-dimensional texture maps, which correspond to each face of a regular hexahedron. When rendering, cubemaps are often used to represent content in all directions of the scene environment (for example, front, back, left, right, up, and down). When the current viewing angle image, the device orientation matrix and the camera projection matrix are known, the spatial direction corresponding to each pixel in the two-dimensional image can be obtained, thereby projecting the two-dimensional image onto the cubemap.

Specifically, projection can be done by the following steps:

将其变换至相机坐标系中得到相对应的方向向量(xb，yb，zb)，再通过视角矩阵M_view的逆变换

将其变换到世界坐标系中的方向向量(xc，yc，zc)，进而得到该像素点在立方体贴图上对应的位置。其中，视角矩阵为在步骤S102中获得的，投影矩阵是在步骤S104中获得的。若在获取图像的同时能够获得该像素点对应的深度信息za，则可以使用像素点的坐标(xa，ya，za)进行上述变换操作，从而得到该像素点在立方体贴图上对应的位置。For the pixel point (xa, ya) in the image coordinate system, when it has no depth information, the depth information can be added to the coordinate information of the pixel point, for example, it is transformed into (xa, ya, 1), and then, by The inverse of the projection matrix M _proj

Transform it into the camera coordinate system to get the corresponding direction vector (xb, yb, zb), and then through the inverse transformation of the viewing angle matrix M _view

Transform it to the direction vector (xc, yc, zc) in the world coordinate system, and then get the corresponding position of the pixel on the cubemap. The viewing angle matrix is obtained in step S102, and the projection matrix is obtained in step S104. If the depth information za corresponding to the pixel can be obtained while acquiring the image, the above transformation operation can be performed by using the coordinates (xa, ya, za) of the pixel to obtain the corresponding position of the pixel on the cube map.

In the above conversion process, the pixels in the image It obtained in step S102 may correspond to the same face (for example, the front) in the _cubemap , or may correspond to multiple faces (for example, the front and the front) in the cubemap. left). In addition, the more images corresponding to different device orientations are obtained in step S102, the richer the pixels in the cubemap are filled. The color of an unfilled pixel can also be represented by black, ie, its RGB value is (0, 0, 0).

During the continuous use of the user, the continuous image sequence and the corresponding viewing angle matrix can be obtained, and the observed scene content can be maintained by merging the sequence information. For consecutive image sequences, there may be overlapping regions between adjacent image frames. Taking into account the accuracy and error of the sensor on the actual device, for the color of the pixels in the overlapping area, in some embodiments, a weighted average is used to calculate the historical data and the current frame data, namely:

C _t =(1-w)*C _t-1 +w*C _current

In the above formula, t represents the number of frames of the image, C _t-1 is the color value of the pixel after the t-1 frame image is merged and calculated, and C _current is the current frame, that is, the corresponding pixel in the t-th frame image The color value of , C _t is the color value of the pixel after t frames of images are merged and calculated, w is the weight, 0≤w≤1. The value of w may be a fixed value, for example, w may be 0.2 or 0.5. In some other embodiments, w may also be other values, which is not limited in this application.

In some embodiments, steps S102 and S104 may be performed iteratively to form a cubemap with richer lighting information. After acquiring the image through S102, the electronic device can further enrich the lighting information on the basis of the historical cubemap, and project the newly acquired image into the historical cubemap.

S106: Based on the device orientation matrix M _{view, t} at the current moment and the observed cubemap Cubemap _{t of the scene, obtain the scene environment map Envmap t mapped} by the specular sphere.

In some embodiments, a mirror-ball mapping projection may be used to obtain the currently observed mirror-ball mapped scene environment map Envmap _t .

The scene environment map of specular sphere mapping can be equivalent to placing a specular sphere in the scene, and all the environmental content in the scene that can be observed on its surface except the part occluded by the sphere. This sphere is used to collect the lighting distribution of the scene environment, and the corresponding specular sphere map is used to represent the lighting distribution of the scene environment.

In this step, the observed specular sphere mapping environment map under the current viewing angle is generated by rendering the cube environment map obtained from step S104. The specific operation steps are as follows:

S202: Render a specular sphere in the cube environment formed by the cube environment map; the rendering of the sphere may be performed by any rendering method in the prior art.

S204: Determine the environmental direction reflected by each pixel on the spherical surface according to the current viewing angle matrix.

S206: Acquire corresponding pixels and their colors or brightnesses on the cube map according to the environment direction.

Specifically, according to the environment direction determined in step S204, one or more corresponding pixels may be acquired on the cube map. If the acquired pixel is one, the color or brightness corresponding to the pixel can be acquired. If there are multiple acquired pixels, the average value of the colors or brightness of the multiple pixels is determined.

S208: Fill the corresponding pixels with the color or brightness determined in S206.

After the above steps, the scene environment map mapped by the specular sphere is obtained. The scene environment map may be a circular two-dimensional image. Among them, each pixel in the image corresponds to the direction in space.

S108: Using a deep convolutional neural network, using It and Envmap _t as inputs to generate a complete high-dynamic range (High-Dynamic Range, HDR) specular sphere-mapped environment map L _t under the current viewing angle _, as the scene illumination distribution.

Specifically, feature extraction _{is performed on the camera image It and the observed environment map Envmap t respectively} , and the extracted features are fused to generate the HDR environment map. In some embodiments, as shown in Figure 4, the deep convolutional neural network structure can be divided into three parts: an image feature extraction network, an observed environment map feature extraction network, and a feature fusion and illumination generation network. FIG. 5 exemplarily shows the working steps of the deep convolutional neural network in this application.

Among them, the image feature extraction network is used to extract feature vectors related to illumination from the camera image of the current perspective. For example, a network structure based on MobileNetV2 can be used. The network structure has high computational performance on current mobile devices and can effectively extract the required features from images. The input of this network can be an image with a resolution of 240*320, and the output can be a feature vector of length 512. In some other embodiments, the format of the input image and the length of the output feature vector can be set according to specific requirements.

The observed environment map feature extraction network extracts features from the input environment map through several convolutional layers, and at the same time uses a visual attention module to fuse the mask information, so that the feature extraction pays more attention to the observed effective pixels. Among them, the mask information is used to mark the observed and unobserved pixels. Specifically, in the scene environment map obtained in S106, if a pixel does not obtain any color information or brightness information from the cubemap in S104, the pixel is an unobserved pixel, otherwise, the pixel is is the observed pixel. Therefore, in some embodiments, the above-mentioned mask information may be generated in S106. The visual attention module uses the mask information to determine the observed pixels from the scene environment map to enable the feature extraction network to extract image features from the above observed pixels. The network can receive a 4-channel (RGB+Mask) observed environment map with a resolution of 256*256 and output a feature vector of length 512. In some other embodiments, the format of the input image and the length of the output feature vector can be set according to specific requirements.

The feature fusion and illumination generation network first splices the feature vectors output by the image feature extraction network and the observed environment map feature extraction network, and then generates the HDR environment map through several fully connected layers, upsampling and convolution layers. In some embodiments, the resolution of the above-mentioned HDR environment map may be 64*64.

In some embodiments, the lengths of the feature vectors output by the image feature extraction network and the observed environment map feature extraction network may or may not be the same.

In some embodiments, when splicing the feature vector A output by the image feature extraction network and the feature vector B output by the observed environment map feature extraction network, the splicing can be performed in a predetermined order, such as filling the feature vector B with After the eigenvector A, the eigenvector A can also be filled behind the eigenvector B, or the eigenvector A and the eigenvector B can be cross-spliced according to certain rules. This application does not make any restrictions on this.

In the process of generating the HDR environment map, the unobserved pixels in the scene environment map are filled through illumination estimation.

In some embodiments, prior to using the neural network, the neural network may be trained to improve the accuracy of the neural network for lighting estimation operations. For example, an image of the environment captured by a panoramic camera can be used as training data. During training, an L1 loss function is used on the HDR environment map output by the network, and an adversarial training loss is used for supervision.

S110: Render the virtual object according to the HDR environment map, and fuse the rendered virtual object into the real image. During the rendering process, the geometry and surface material of the virtual object can be combined for rendering. The rendering mode may use any rendering mode in the prior art, which is not limited in this application.

Using the illumination estimation method provided by the present application can make the visual effect more realistic when the virtual object is rendered. This application uses 100 HDR environment maps to generate 400 sets of data, including the current perspective image, the observed scene environment map, and the HDR environment map. The illumination estimation method provided in this application is evaluated and compared with the existing technology. Compare the illumination estimation methods in . In the process of rendering, this application uses two geometric models, Sphere and Venus, and combines them with three surface materials, rough, glossy and specular, to obtain six virtual object models. The present application uses the estimated illumination and the reference illumination to render the above-mentioned six virtual object models, and fuse them into the image of the current viewing angle. Figures 6 and 7 show that using the illumination estimation method provided by the present application, the rendering results of the estimated illumination and the rendering results of the reference illumination have the structural similarity (SSIM) and the peak signal-to-noise ratio (Peak Signal-to-Noise ratio). Ratio, PSNR). It can be seen from the figure that, compared with the prior art, the illumination estimation method provided by the present application is more consistent with the reference results in terms of visual effect.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server, or data center Transmission to another website site, computer, server, or data center by wire (eg, coaxial cable, optical fiber, digital subscriber line) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes an integration of one or more available media. The usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state drives), and the like.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented. The process can be completed by instructing the relevant hardware by a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed , which may include the processes of the foregoing method embodiments. The aforementioned storage medium includes: ROM or random storage memory RAM, magnetic disk or optical disk and other mediums that can store program codes.

Claims (12)

1. A method of illumination estimation, comprising the steps of:

acquiring a first image;

determining a device orientation matrix and a camera projection matrix corresponding to the first image;

projecting the first image onto a spatial cube map according to the equipment orientation matrix and the camera projection matrix;

according to the equipment orientation matrix and the space cube map, using a mirror surface ball to map and project to obtain a scene environment map;

and determining an environment map with a high dynamic range according to the first image and the scene environment map.

2. The method of claim 1, wherein the device orientation matrix is obtained by an inertial measurement unit.

3. The method of claim 1 or 2, wherein the determining a device orientation matrix corresponding to the first image further comprises:

Determining a first moment of acquiring a first image;

and taking the inertial measurement unit data which is closest to the first moment before the first moment as a device orientation matrix corresponding to the first image.

4. The method of any of claims 1-3, wherein the projecting the first image onto a spatial cube map according to the device orientation matrix and the camera projection matrix, further comprises:

transforming a first coordinate of a first pixel to a first direction vector by projecting an inverse matrix of a matrix through the camera; wherein the first pixel is a pixel in the first image;

transforming, by the device, the first direction vector into a second coordinate by an inverse of a direction matrix of the device.

5. The method of any one of claims 1-4, wherein the obtaining the scene environment map using a mirror ball mapping projection based on the device orientation matrix and the spatial cube map, further comprises:

rendering a first mirror sphere within a cubic environment formed by the cube map;

determining an ambient direction of reflection of a second pixel on a sphere of the first specular sphere from the device orientation matrix;

Acquiring the color or brightness of a corresponding pixel on the cube map according to the environment direction;

filling the color or brightness into the second pixel.

6. The method of any of claims 1-5, wherein determining a high dynamic range environment map from the first image and the scene environment map further comprises:

determining the high dynamic range environment map using a deep convolutional neural network.

7. The method of claim 6, wherein the deep convolutional neural network comprises an image feature extraction network, an observed environment map feature extraction network, a feature fusion, and a lighting generation network, wherein,

the image feature extraction network is used for extracting a first feature vector from the first image;

the observed environment map feature extraction network is used for extracting a second feature vector from the scene environment map;

the feature fusion and illumination generation network is used for generating the environment map with the high dynamic range according to the first feature vector and the second feature vector.

8. The method of claim 7, wherein said extracting a second feature vector from said scene environment map further comprises:

Marking observed pixel points and unobserved pixel points in the scene environment image;

and determining the second feature vector according to the observed pixel points.

9. The method of claim 7 or 8, wherein the generating the high dynamic range environment map from the first eigenvector and the second eigenvector, further comprises:

splicing the first feature vector and the second feature vector to generate a third feature vector;

and generating the environment map with the high dynamic range according to the third feature vector.

10. The method of any one of claims 1-9, further comprising,

rendering the virtual object according to the environment diagram with the high dynamic range;

fusing the rendered virtual object into the first image.

11. An electronic device includes a first electronic component having a first electronic component,

one or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured for execution by the one or more processors, the one or more programs including instructions for causing the electronic device to perform the method of any of claims 1-10.

12. A computer readable medium storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs comprising instructions for causing an electronic device to perform the method of any of claims 1-10.

Images