JPWO2019226366A5

JPWO2019226366A5 -

Info

Publication number: JPWO2019226366A5
Application number: JP2020564910A
Authority: JP
Publication date: 2022-04-22
Anticipated expiration: 2039-05-13

Claims

It's an electronic device:
With processing unit;
A memory that is coupled to the processing unit and stores instructions for execution by the processing unit, the instructions to the device when executed by the processing unit:
Acquiring an input image for a real object and a depth map corresponding to the input image;
Determining the scene in the input image based on the depth map;
Based on the scene, determining the shading and shadow information of the real object caused by the light source in the scene, the light source being represented by an image of a predetermined shape, the image being of the light source. A plurality of pixels corresponding to a plurality of portions are included, and the pixel value of each of the plurality of pixels represents the emission intensity of each of the plurality of portions of the light source.
Determining the shading and shadow information is:
To initialize the pixel value of each of the plurality of pixels in the image with a predetermined value;
Based on the scene and the initialized image, it is to generate a set of first irradiance maps corresponding to the plurality of pixels, each of which is a corresponding portion of the light source. Representing, generating, representing the corresponding shading and shadow information of said real object caused by;
Determining shading and shadow information ;
Determining the lighting conditions in the scene caused by the light source based on the input image and the shading and shadow information, based on the input image and the set of the first radiation illuminance map. Determining the lighting conditions in the scene, including determining the pixel values of each of the plurality of pixels in the image ;
To perform operations including, with memory;
Have an electronic device.

The above operation is further:
The output image comprises generating an output image including the real object and the virtual object based on the scene and the lighting condition, and the output image synthesizes the virtual object in the scene of the input image.
The device according to claim 1.

The scene includes a surface for the placement of the real object, and determining the scene is:
Converting the depth map into a point cloud;
To adapt a plurality of points in the point cloud to a surface, wherein the plurality of points represent the surface.
Reconstructing the plurality of points in the point cloud located on the surface into a three-dimensional mesh, wherein the surface and the three-dimensional mesh together represent the scene in the input image. To configure and: including,
The device according to claim 1.

The image representing the light source is a cross-shaped image.
The device according to claim 1 .

Determining the pixel value of each of the plurality of pixels in the image is:
A trained neural network is used to determine the pixel value of each of the plurality of pixels in the image based on the input image and the set of the first irradiance map.
The device according to claim 1 .

Determining the pixel value of each of the plurality of pixels in the image is:
It comprises progressively determining the pixel value of each of the plurality of pixels in the image using a plurality of trained neural networks based on the input image and the set of the first irradiance map. ,
The device according to claim 1 .

The plurality of trained neural networks include at least a first neural network and a second neural network, and determining the pixel value of each of the plurality of pixels in the image is:
Based on the input image and the set of the first irradiance map, the first neural network is used to the first pixel value of the first pixel of the plurality of pixels and the first pixel. Determining the pixel value of at least one adjacent pixel;
Generating a second set of irradiance maps based on the first pixel, the at least one pixel, and the scene, wherein the second set of irradiance maps is of the first pixel. To generate, corresponding to each of a plurality of sub-pixels, the at least one pixel, and a combination of the first pixel of the plurality of pixels and pixels other than the at least one pixel;
To determine whether the first pixel should be split using the second neural network based on the input image and the set of the second irradiance map;
In response to determining that the first pixel value should not be split, the use of the first pixel value as the final pixel of the first pixel;
The device according to claim 6 .

Determining the pixel value of each of the plurality of pixels in the image is further:
To divide the first pixel into a plurality of subpixels in response to determining that the first pixel should be divided;
To determine the subpixel value of each of the plurality of subpixels;
The device according to claim 7 .

A method implemented on a computer:
To acquire an input image related to a real object and a depth map corresponding to the input image;
Determining the scene in the input image based on the depth map;
Based on the scene, determining the shading and shadow information of the real object caused by the light source in the scene, the light source being represented by an image of a predetermined shape, the image being of the light source. A plurality of pixels corresponding to a plurality of portions are included, and the pixel value of each of the plurality of pixels represents the emission intensity of each of the plurality of portions of the light source.
Determining the shading and shadow information is:
To initialize the pixel value of each of the plurality of pixels in the image with a predetermined value;
Based on the scene and the initialized image, it is to generate a set of first irradiance maps corresponding to the plurality of pixels, each of which is a corresponding portion of the light source. Representing, generating, representing the corresponding shading and shadow information of said real object caused by;
Determining shading and shadow information ;
Determining the lighting conditions in the scene caused by the light source based on the input image and the shading and shadow information, based on the input image and the set of the first radiation illuminance map. Includes determining the lighting conditions in the scene, including determining the pixel value of each of the plurality of pixels in the image .
Method.

Further including generating an output image including the real object and the virtual object based on the scene and the lighting condition, the output image synthesizes the virtual object in the scene of the input image.
The method according to claim 9 .

The scene includes a surface for the placement of the real object, and determining the scene is:
Converting the depth map into a point cloud;
Adapting a plurality of points in the point cloud to a surface, wherein the plurality of points represent and adapt to the surface;
Reconstructing a plurality of points in the point cloud located on the surface into a three-dimensional mesh, wherein the surface and the three-dimensional mesh together represent the scene in the input image. To do and; including;
The method according to claim 9 .

The image representing the light source is a cross-shaped image.
The method according to claim 9 .

Determining the pixel value of each of the plurality of pixels in the image is:
A trained neural network is used to determine the pixel value of each of the plurality of pixels in the image based on the input image and the set of the first irradiance map.
The method according to claim 9 .

A machine-readable medium that stores instructions, said to the one or more machines when executed by one or more machines:
To acquire an input image related to a real object and a depth map corresponding to the input image;
Determining the scene in the input image based on the depth map;
Based on the scene, determining the shading and shadow information of the real object caused by the light source in the scene, the light source being represented by an image of a predetermined shape, the image being of the light source. A plurality of pixels corresponding to a plurality of portions are included, and the pixel value of each of the plurality of pixels represents the emission intensity of each of the plurality of portions of the light source.
Determining the shading and shadow information is:
To initialize the pixel value of each of the plurality of pixels in the image with a predetermined value;
Based on the scene and the initialized image, it is to generate a set of first irradiance maps corresponding to the plurality of pixels, each of which is a corresponding portion of the light source. Representing, generating, representing the corresponding shading and shadow information of said real object caused by;
Determining shading and shadow information;
Determining the lighting conditions in the scene caused by the light source based on the input image and the shading and shadow information, based on the input image and the set of the first radiation illuminance map. To perform an operation, including determining the illumination conditions in the scene, including determining the pixel value of each of the plurality of pixels in the image.
Machine-readable medium.

The operation is:
Further including generating an output image including the real object and the virtual object based on the scene and the lighting condition, the output image synthesizes the virtual object in the scene of the input image.
The machine-readable medium according to claim 14.

The scene includes a surface for the placement of the real object, and determining the scene is:
Converting the depth map into a point cloud;
Adapting a plurality of points in the point cloud to a surface, wherein the plurality of points represent and adapt to the surface;
Reconstructing a plurality of points in the point cloud located on the surface into a three-dimensional mesh, wherein the surface and the three-dimensional mesh together represent the scene in the input image. To do and; including;
The machine-readable medium according to claim 14.

The image representing the light source is a cross-shaped image.
The machine-readable medium according to claim 14.

Determining the pixel value of each of the plurality of pixels in the image is:
A trained neural network is used to determine the pixel value of each of the plurality of pixels in the image based on the input image and the set of the first irradiance map.
The machine-readable medium according to claim 14.