JP7084616B2 - Image processing device, image processing method, and image processing program - Google Patents

Description

Embodiments of the present invention relate to an image processing apparatus, an image processing method, and an image processing program.

Relighting is a technique for reproducing the appearance of a subject in a lighting environment different from that when the subject was photographed. If re-illumination of a portrait image can be realized, various applications such as correction of shadows in portrait photographs and cut-and-paste composition to support background images can be considered.
On the other hand, in order to reilluminate a person image based on the physical law, it is necessary to estimate the reflectance, shape and illumination from the image. This is a well-posed problem called reverse rendering. Such reverse rendering has been realized mainly for a person's face image by using a statistical 3D shape model or a convolutional neural network (CNN). Then, in the formulation of the lighting calculation at that time, the spherical harmonics (SH) have often been used to express the distribution of the light arriving from the surroundings.

S.Sengupta, A.Kanazawa, C.D.Castillo, and D.W.Jacobs, “SfSNet: Learning Shape, Reflectance and Illuminance of Faces‘ in the wild ’”, in Conference on Computer Vision and Pattern Recognition 2018.

In the reillumination based on SH, if the light shielding is ignored, the illumination calculation can be performed by an analytical formula from the illumination information and the normal vector of the object surface. However, if light shielding is ignored, parts that are not exposed to much light due to shielding, such as the sides and crotch of a person, will become unnaturally bright.
As a method of treating light shielding approximately, there is also a technique called ambient occlusion that calculates the geometrical shielding rate as a scalar value and multiplies it by the shadow, but this is because the recessed part is always darkened regardless of the light source. In particular, when the light source is moved during relighting, a feeling of strangeness occurs.

A better method is known to formulate SH illumination calculations, including light shielding, as was done with precomputed radiation transfer (PRT) developed for real-time rendering. ing.
However, if we try to incorporate the PRT framework into reverse rendering, we need the shape we are trying to estimate in order to calculate the light occlusion, and we also shoot a large number of rays from each point on the surface of the object. It is necessary to make a visibility judgment, and the calculation cost is high.
The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide an image processing device, an image processing method, and a program capable of improving the photorealism when generating an output image corresponding to an input image. And.

One embodiment of the present invention is estimated based on the input image as the acquisition unit for acquiring the input image, the light source information which is the information indicating the lighting state from the light source, and the reflectance of the subject included in the input image. Light estimated based on the reflectance information and visible information indicating whether or not the light from the input image and the light source reaches the subject as the transmission state of the light contained in the input image from the subject. A generation unit that generates an output image corresponding to the input image based on the transmission information and an output unit that outputs the output image generated by the generation unit are provided , and the light source and the light transmission information are based on each other. It is an image processing device expressed as a coefficient of a function, which is estimated by a machine learning model .
In one embodiment of the present invention, in the image processing apparatus described above, the light source information is estimated based on the input image.
In one embodiment of the present invention, in the image processing apparatus described above, the output image is a reilluminated image in which the subject included in the input image is reilluminated based on the illumination state indicated by the light source information. ..
In one embodiment of the present invention, in the above-mentioned image processing apparatus, the reflectance information and the light transmission information are estimated based on the input image by a machine learning model.
In one embodiment of the present invention, in the image processing apparatus described above, the machine learning model includes light source information estimated based on the input image by the machine learning model, reflectance information, and light transmission information. It is learned by comparing an image obtained from any one of the above or a combination of any of the above with a predetermined correct answer image.
In one embodiment of the present invention, in the image processing apparatus described above, whether or not the visible information reaches each of the plurality of points on the surface of the object for each direction of light. Created based on.

One embodiment of the present invention is estimated based on the input image as a step of acquiring an input image, light source information which is information indicating an illumination state from the light source, and a reflectance of a subject included in the input image. Light transmission estimated based on the reflectance information and visible information indicating whether or not the light from the input image and the light source reaches the subject as the transmission state of the light contained in the input image from the subject. Based on the information, it has a step of generating an output image corresponding to the input image and a step of outputting the output image generated in the step of generating , and the light source and the light transmission information are based on each other. Expressed as a coefficient of a function, the coefficient is an image processing method performed by a computer, estimated by a machine learning model .

One embodiment of the present invention is based on a step of acquiring an input image to a computer, light source information which is information indicating an illumination state from the light source, and a reflectance of a subject included in the input image based on the input image. It is estimated based on the estimated reflectance information and visible information indicating whether or not the light from the input image and the light source reaches the subject as the transmission state of the light contained in the input image from the subject. Based on the light transmission information, the step of generating the output image corresponding to the input image and the step of outputting the output image generated in the generation step are executed, and the light source and the light transmission information are combined with each other. Is an image processing program expressed as a coefficient of a base function, the coefficient being estimated by a machine learning model .

According to an embodiment of the present invention, it is possible to provide an image processing device, an image processing method, and an image processing program capable of improving the photorealism when generating an output image corresponding to an input image.

It is a figure which shows an example of the image processing system of an embodiment. It is a block diagram which shows an example of the image processing apparatus of embodiment. It is a figure which shows an example of the process which derives a visible function. It is a figure which shows an example of the machine learning model used in the image processing apparatus of embodiment. It is a flowchart which shows an example of the operation of the image processing apparatus of an embodiment. It is a flowchart which shows an example of the operation of the image processing apparatus of an embodiment. It is a figure which shows an example of a CG person data set. It is a figure which shows the comparative example of the estimation result by the image processing apparatus of embodiment, and the conventional method. It is a figure which shows an example of the estimation result by the image processing apparatus of embodiment. It is a figure which shows an example of the estimation result by the image processing apparatus of embodiment. It is a figure which shows an example of the estimation result by the image processing apparatus of embodiment.

Next, the image processing apparatus, the image processing method, and the image processing program of the present embodiment will be described with reference to the drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the following embodiments.
In all the drawings for explaining the embodiment, the same reference numerals are used for those having the same function, and the repeated description will be omitted.
Further, "based on XX" as used in the present application means "based on at least XX", and includes a case where it is based on another element in addition to XX. Further, "based on XX" is not limited to the case where XX is directly used, but also includes the case where it is based on a case where calculation or processing is performed on XX. "XX" is an arbitrary element (for example, arbitrary information).

(Embodiment)
(Image processing system)
Hereinafter, the image processing system according to the embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram showing an example of an image processing system of an embodiment. In the image processing system of the present embodiment, the terminal device accesses the server device and transmits a full-body image of a person with a mask image (hereinafter referred to as "teacher image") to the server device. The image processing device acquires a teacher image with a mask image transmitted by the terminal device from the server device, and performs supervised learning based on the acquired teacher image.
Further, the terminal device accesses a website such as a shopping site, and transmits an image (a full-body image with a mask in which the subject to be relighted is captured) (hereinafter referred to as "input image") to the server device. The image processing device 100 acquires an input image transmitted by the terminal device from the server device. The image processing device 100 extracts a feature amount from the acquired input image, and generates an output image corresponding to the input image based on the extracted feature amount and the learning result of supervised learning.
The image processing system 1 of the embodiment includes a terminal device 10-1 to a terminal device 10-n (n is an integer of n> 0), a server device 70, and an image processing device 100. These devices are connected to each other via the network 50. The network 50 includes, for example, a radio base station, a Wi-Fi access point, a communication line, a provider, the Internet, and the like. It should be noted that not all combinations of the devices shown in FIG. 1 need not be able to communicate with each other, and the network NW may include a local network in part.
Hereinafter, any terminal device among the terminal devices 10-1 to 10-n will be referred to as a terminal device 10.

The terminal device 10 is a device used by the user. The terminal device 10 is, for example, a computer device such as a mobile phone such as a smartphone, a tablet terminal, or a personal computer. For example, the terminal device 10 is used to register a user ID on a website such as a shopping site, a mail service, an SNS service, an information providing service, or the like, and transmit an image in association with the registered user ID.
The server device 70 provides various services. For example, the server device 70 may be a web server device that provides a website for providing various services via a web browser activated by the terminal device 10. Further, the server device 70 may be an application server device that exchanges various information by communicating with the terminal device 10 in which a predetermined application program is started (executed). The terminal device 10 in which the predetermined application program is activated displays a screen capable of providing various services by communicating with the server device 70. Hereinafter, for the sake of brevity, the case where the server device 70 is a web server device will be continued.
For example, the server device 70 authenticates the user ID and confirms the user before providing the service. As a result of authentication, the server device 70 provides various services if the user has already registered the user ID, and if the user has not registered the user ID, notifies that the user ID has not been registered. , Prompt the registration of user ID. When the user newly registers the user ID in response to the fact that the user ID has not been registered, the server device 70 issues the newly registered user ID. As a result, the user can newly acquire the user ID. By operating the terminal device 10, the user transmits an image including a subject to the server device 70 in association with the registered user ID.

The image processing device 100 performs supervised learning. For example, the image processing device 100 acquires a full-body image of a person with a mask image as a teacher image. For example, the image processing device 100 acquires the teacher image transmitted by the terminal device 10 from the server device 70. The image processing apparatus 100 (automatically) estimates the reflectance, the light transfer vector, and the light source information for each pixel from the acquired teacher image. Here, the light transfer vector includes light shielding information. Hereinafter, the case where the light transmission vector includes the light shielding information will be described.
The image processing device 100 creates a reflectance map Λ from the reflectance estimated for each pixel, and creates a light transfer map Ψ from the light transfer vector estimated for each pixel. The image processing apparatus 100 uses the created reflectance map Λ, the created light transmission map Ψ, and the estimated light source information Π as feature quantities, and learns that the feature quantities correspond to the teacher image.

The image processing device 100 communicates with the server device 70 to acquire a user ID of a user who uses the service provided by the server device 70, and an input image (subject to be relighted) transmitted in association with the user ID. Acquires a full-body image with a mask captured by. The image processing apparatus 100 (automatically) estimates the reflectance, the light transfer vector, and the light source information for each pixel from the acquired input image.
The image processing device 100 creates a reflectance map Λ from the reflectance estimated for each pixel, and creates a light transfer map Ψ from the light transfer vector estimated for each pixel. The image processing device 100 uses the created reflectance map Λ, the created light transmission map Ψ, and the estimated light source information Π as feature quantities, and generates an output image corresponding to the input image based on the feature quantities. .. The image processing device 100 outputs the generated output image.

(Image processing device 100)
FIG. 2 is a block diagram showing an example of the image processing apparatus of the embodiment.
The image processing device 100 electrically connects the communication unit 105, the storage unit 110, the operation unit 120, the information processing unit 130, the display unit 140, and each of the above components as shown in FIG. It is equipped with a bus line 150 such as an address bus and a data bus for information processing.
The communication unit 105 is realized by a communication module. The communication unit 105 communicates with an external communication device such as the server device 70 via the network 50. Specifically, the communication unit 105 receives the teacher image transmitted by the server device 70, and outputs the received teacher image to the information processing unit 130. Here, an example of the teacher image is an image created by rendering a mesh model of a commercially available 3D-scanned person. Most of the commercially available 3D human models do not contain a gloss component as a texture and have only a diffuse reflection component. Therefore, in the present embodiment, a case where a diffuse reflection component is targeted as the reflectance will be described. The number of 3D human models on the market is limited, and the number of models used in the embodiments is only a few hundred even if the training data and the test data are combined. Considering that there are innumerable variations of clothes, it seems that training data is insufficient, but in reality, by learning using the data prepared in this embodiment, wrinkles, sides, crotch, etc. of clothes can be used. Shadow calculation can be performed in consideration of light shielding even for areas where light shielding is likely to occur.
Further, the communication unit 105 receives the input image transmitted by the server device 70, and outputs the received input image to the information processing unit 130. Further, the communication unit 105 acquires the output image output by the information processing unit 130, and transmits the acquired output image to the server device 70.

The storage unit 110 is realized by, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), an HDD (Hard Disk Drive), a flash memory, or a hybrid storage device in which a plurality of these are combined. The storage unit 110 stores the program 111 executed by the information processing unit 130, the application 112, the reflectance map Λ113, the light source information Π114, and the light transmission map Ψ115.

The application 112 causes the image processing device 100 to acquire the teacher image transmitted by the server device 70. The application 112 causes the image processing device 100 to estimate the reflectance, the light transfer vector, and the light source information for each pixel from the acquired teacher image. The application 112 causes the image processing device 100 to create a reflectance map Λ from the reflectance estimated for each pixel, and to create a light transmission map Ψ from the light transfer vector estimated for each pixel. The application 112 causes the image processing device 100 to use the created reflectance map Λ, the created light transmission map Ψ, and the estimated light source information Π as feature quantities, and the feature quantities correspond to the teacher image. Let them learn that.

The application 112 causes the image processing device 100 to acquire the input image transmitted by the server device 70. The application 112 causes the image processing device 100 to estimate the reflectance, the light transfer vector, and the light source information for each pixel from the acquired input image. The application 112 causes the image processing device 100 to create a reflectance map Λ from the reflectance estimated for each pixel, and to create a light transmission map Ψ from the light transfer vector estimated for each pixel. The application 112 makes the image processing device 100 feature the reflectance map Λ created for each pixel, the light transmission map Ψ created for each pixel, and the estimated light source information Π, and the feature quantity thereof. Is to generate an output image corresponding to the input image.
The reflectance map Λ113 is a map showing the reflectance of the teacher image for each pixel.
The light source information Π114 is information indicating the illumination state of the teacher image.
The light transfer map Ψ115 is a map representing a light transfer vector in which geometric information of an object is recorded for each pixel. The light transfer map Ψ115 and the light transfer vector are separated from the information of the light source and do not depend on the light source. The geometric information of the object includes the light shielding information which is the information indicating whether or not the light is shielded. Since the light transfer map Ψ115 and the light transfer vector are separated from the information of the light source, the illumination calculation can be performed correctly.

The operation unit 120 is configured by, for example, a touch panel or the like, detects a touch operation on the screen displayed on the display unit 140, and outputs the detection result of the touch operation to the information processing unit 130.

All or part of the information processing unit 130 is, for example, a functional unit (hereinafter referred to as a software functional unit) realized by executing a program 111 stored in the storage unit 110 by a processor such as a CPU (Central Processing Unit). ). All or part of the information processing unit 130 may be realized by hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), or FPGA (Field-Programmable Gate Array), and is a software function. It may be realized by a combination of a unit and hardware.
The information processing unit 130 includes, for example, an acquisition unit 131, an analysis unit 132, a machine learning unit 133, a generation unit 134, and an output unit 135.

The acquisition unit 131 acquires the teacher image output by the communication unit 105, and outputs the acquired teacher image to the analysis unit 132. Further, the acquisition unit 131 acquires the input image output by the communication unit 105, and outputs the acquired input image to the analysis unit 132.
The analysis unit 132 acquires the teacher image output by the acquisition unit 131 and analyzes the acquired teacher image to estimate the reflectance, the light transfer vector, and the light source information for each pixel. The analysis unit 132 creates a reflectance map Λ from the reflectance estimated for each pixel, and creates a light transfer map Ψ from the light transfer vector estimated for each pixel. The analysis unit 132 stores the created reflectance map Λ in the reflectance map Λ113, stores the created light transmission map Ψ in the light transmission map Ψ115, and stores the estimated light source information Π in the light source information Π114. The analysis unit 132 outputs the reflectance map Λ, the light transmission map Ψ, and the light source information Π to the machine learning unit 133.
Here, the process of creating the optical transfer map Ψ by the analysis unit 132 will be described. The analysis unit 132 performs the illumination calculation based on the spherical harmonics (SH) in consideration of the light shielding.
First, a case where the lighting calculation is performed without considering the light shielding will be described.
(1) When light shielding is not considered When light shielding and mutual reflection are not considered, irradiance as shown in equation (1), where n is the unit normal vector at the point p on the object surface. E (n) is calculated by integrating the irradiance L (ω _i ) inserted from all directions ω _i of the hemisphere Ω (n) defined by the normal n.

The dependency on position p is omitted for the sake of simplicity. The distribution L (ω _i ) of the incident radiance multiplied by the cosine attenuation term max (n, ω _i , 0) is projected onto the spherical harmonics. When ω = (θ, φ) is displayed as a parameter using the direction vector ω, the elevation angle θ, and the azimuth angle φ, equations (2) and (3) are obtained.

In equation (2), Y _{l and m} are spherical harmonics (where l ≧ 0, −l ≦ m ≦ l, and m ≦ 2). In equations (2) and (3), L _{l , m} and All are coefficients of the illumination and cosine attenuation terms, respectively. When the integral of the equation (1) is rewritten, the equation (4) is obtained.

Here, ^ _All is expressed as follows.

Y _{l, m} can be expressed by a polynomial of each component x, y, z of the normal vector n = (x, y, z) ^T. If the coefficient sequence {L _{l, m} } is represented by the vector L and the basis function sequence {^ A _l Y _lm } is represented by the vector ^ Y, the irradiance E can be calculated by the inner product of the vectors.

E = ^ ^YTL (5)

Next, a case where the lighting calculation is performed in consideration of the light shielding will be described.
(2) When considering light shielding (1) As described in the case where light shielding is not considered, the place where light is originally blocked and should be dark becomes unnaturally bright. In this embodiment, the visible function V (ω _i ) is introduced in order to consider the light shielding in the equation (1).

In the formula (6), V (ω _i ) is 0 if the light in the incident direction ω _i is blocked, and 1 if the light in the incident direction ω i is not blocked.
FIG. 3 is a diagram showing an example of a process for deriving the visible function V (ωi). As shown in FIG. 3, consider radiating light to an object from a sufficiently distant place (infinity). In this case, light arriving from a sufficiently distant place (infinity) can be regarded as parallel light. The light that arrives from the surroundings (environmental light source) consists of innumerable parallel lights. In the present embodiment, for the incident direction ωi of parallel light, the visible function V (ωi, p) with respect to the direction ωi at all points p (all pixels of the optical transmission map) on the object surface is obtained in parallel. That is, each time one light source direction is selected, all visible functions for that direction are obtained at once at all points on the surface of the object. This is repeated while changing the direction of the light source. The light source direction may be randomly selected based on a random number. In the example shown in FIG. 3, the light source direction LD-1 is selected, and the light source direction LD- is selected for each of the plurality of points p-1, p-2, p-3, p-4, and p-5 on the surface of the object. A visible function indicating whether or not the light from 1 reaches each of the plurality of points p-1, p-2, p-3, p-4, and p-5 is required.
In the conventional method using graphics hardware (GPU), it is necessary to draw an object shape in order to calculate a visible function at each point on the object surface. In this method, when the number of points on the surface of the object becomes enormous, the number of times the object shape is drawn increases, and it takes time to calculate. Specifically, if you calculate with each pixel (in the mask) of 1024 x 1024 pixels, even if the mask covers half of the image, there are 524288 pixels, and it is necessary to draw the object shape by the number of pixels. be. In addition, when ray tracing is performed at each point on the surface of the object in a random direction to determine visibility, the efficiency is poor.
On the other hand, in the present embodiment, the object shape may be drawn for each light source direction in order to evaluate the visible function. Assuming that the number of light source directions actually considered is 642, the calculation time can be shortened because the number is much smaller than the above. Returning to FIG. 2, the explanation will be continued.
In the framework of PRT, the visible function V (ωi) and the cosine attenuation term are pre-calculated and projected onto the basis function of SH to realize high-speed lighting conditions by the inner product of the coefficient vectors. If the vector T is a vector consisting of the coefficients when the visible function and the cosine attenuation term are projected onto the basis function, the irradiance E can be calculated as follows.

E = ^TTL (7)

Here, the light transfer vector is a vector T, and an image having a light transfer vector in each pixel is called a light transfer map Ψ.
Further, the analysis unit 132 acquires the input image output by the acquisition unit 131 and analyzes the acquired input image to estimate the reflectance, the light transfer vector, and the light source information for each pixel. The analysis unit 132 creates a reflectance map Λ from the estimated reflectance, and creates a light transfer map Ψ from the estimated light transfer vector. The analysis unit 132 outputs the reflectance map Λ, the light transmission map Ψ, and the light source information Π to the generation unit 134.

The machine learning unit 133 acquires the reflectance map Λ, the light transmission map Ψ, and the light source information Π output by the analysis unit 132, and uses the acquired reflectance map Λ, the light transmission map Ψ, and the light source information Π as feature quantities. , Learn that the feature quantity is a teacher image. In this embodiment, the case where a convolutional neural network (CNN) is used as an example of a machine learning model will be described.

Here, the loss function used by the machine learning unit 133 when performing machine learning by CNN will be described.
The person image data set _DH is composed of a binary mask representing a part of the image in which a person is shown, a reflectance map Λ, and a light transmission map Ψ for each person model. The binary mask, the reflectance map Λ, and the light transfer map Ψ are expressed as follows.

On the other hand, the light source data set _DL has lighting information for each light source, which consists of 9 coefficients of SH for each RGB channel. Lighting information is expressed as follows.

The binary mask multiplies the correct data and the output of CNN before calculating the loss function (for example, M ³ _j * Λ _j or M ⁹ _j * Ψ _j , where * represents the multiplication for each element). In the following explanation, the multiplication of the binary mask is omitted for the sake of simplicity.
The CNN used in this embodiment estimates the reflectance map Λ, the light transmission map Ψ, and the light source information Π. The input to the CNN is a mask-multiplied human whole-body image I _{j, k} = Λ _j * (Ψ _j Π _k ). In the present embodiment, the machine learning unit 133 has, as a loss function, the correct answer and the L2 loss for each of the reflectance map Λ, the normal map, and the light source information Π, and the estimated value of the whole body image of the person which is the product thereof. For the four types of loss functions, the correct answer and the L2 loss, the L1 loss is derived, and a total of 15 types of loss functions are calculated, including the following loss functions.

L1 total variation (TV) loss for reflectance map Λ and optical transfer map Ψ (2 ways)
-For the shadow map (that is, Ψ _j Π _k ), the L1 loss (3 ways) when the light transmission map Ψ and the light source information Π are used as the estimated value for one and the correct answer for the other, and when both are estimated values.
-L1 loss (L1 loss) when one or two of the three products (that is, Λ _j * (Ψ _j Π _k )) of the reflectance map Λ, the light transmission map Ψ, and the light source information Π are used as estimated values. 6 ways)
The weight of each loss function was set to 1.
In this way, calculating the L1 loss for the product including not only the estimated value but also the correct answer corresponds to weighting the estimated value. By increasing the types of loss functions, clearer estimation results can be obtained.

The CNN used in this embodiment is the same as the correct answer even for the data in the process of being multiplied without multiplying the estimated reflectance (albedo) map Λ, the optical transmission map Ψ, and the light source Π. You can measure whether you are doing it. For example, in CNN, by multiplying the light transmission map Ψ and the light source Π, a shadow map that is purely dependent on the shape and has no clothes or skin color or pattern is obtained.
By creating this acquired shadow map with the correct light transmission map Ψ and the correct light source Π, the correct shadow map can be obtained. When comparing this with the estimated data, the error between the correct answer and the estimated data of either one of the optical transmission map Ψ and the light source Π is correct and the other is estimated, or both are estimated. By measuring, three kinds of loss functions can be obtained. Considering the combinations in this way, three combinations can be obtained for the shadow map and six combinations (except when all three are estimated) for the restoration of the input image.
In addition to this, in the case of restoring a total of two, the reflectance map Λ, the light transfer map Ψ, the light source, and the input image with estimated data in the L1 TV loss for the light transmission map Ψ and the reflectance map Λ. Considering the above, a total of four loss functions, a total of 15 types, are derived.

The generation unit 134 acquires the reflectance map Λ, the light transmission map Ψ, and the light source information Π output by the analysis unit 132, and features the acquired reflectance map Λ, the light transmission map Ψ, and the light source information Π. An output image is generated based on the feature amount. The generation unit 134 outputs the generated output image to the output unit 135.
The output unit 135 acquires the output image output by the generation unit 134, and outputs the acquired output image to the communication unit 105.

Next, the machine learning model used in the image processing apparatus 100 of the present embodiment will be described.
FIG. 4 is a diagram showing an example of a machine learning model used in the image processing apparatus of the present embodiment.
As described above, an example of the machine learning model 200 used in the image processing apparatus 100 of the present embodiment is a CNN.
The machine learning model 200 is represented by a multiplication unit 201, an encoder 202, a reflectance map decoder 203, an optical transfer map decoder 206, and a convolution unit (Conv.) 210. The reflectance map decoder 203 includes a ResNet block 204 and a deconv. 205. The optical transfer map decoder 206 includes a ResNet block 207 and a deconv. 208.
Immediately after each of the convolution unit 210, the ResNet block 204, the deconvolution unit 205, the ResNet block 207, and the deconvolution unit 208, batch normalization is performed except for the first and last layers to which data is input and output. (Batch normalization) and ReLU are applied. Further, a dropout with a probability of 0.5 is applied immediately after each of the first three deconvolution layers of the deconvolution unit 205 and the deconvolution unit 208.

As the input image 211, the person whole body image I212 and the binary mask M213 are output to the multiplication unit 201. The multiplication unit 201 multiplies the person whole body image I212 and the binary mask M213, and outputs the result of the multiplication to the encoder 202. The encoder 202 and the deconvolution unit 205 and the encoder 202 and the deconvolution unit 208 are connected by a skip connection (skip-connection).
An example of the encoder 202 is composed of seven convolutional layers. The encoder 202 applies a filter to the result of multiplying the person whole body image I212 output by the multiplication unit 201 and the binary mask M213. The encoder 202 outputs the result of applying the filter to the reflectance map decoder 203, the optical transfer map decoder 206, and the connecting unit 209.

In the reflectance map decoder 203, the result of applying the filter output by the encoder 202 is output to the ResNet block 204. The ResNet block 204 includes two ResNet blocks, and each of the two ResNet blocks performs a convolution calculation. The ResNet block 204 outputs the result of the convolution calculation to the connecting unit 209 and the deconvolution unit 205.
The deconvolution unit 205 acquires the result of the convolution calculation output by the ResNet block 204, and derives the reflectance map ^to Λ221 by performing deconvolution of 7 layers with respect to the acquired convolution calculation result. do. With this configuration, the reflectance map decoder 203 can estimate the reflectance map Λ.

In the optical transfer map decoder 206, the result of applying the filter output by the encoder 202 is output to the ResNet block 207. The ResNet block 207 includes two ResNet blocks, and each of the two ResNet blocks performs a convolution calculation. The ResNet block 207 outputs the result of the convolution calculation to the connecting unit 209 and the deconvolution unit 208.
The deconvolution unit 208 acquires the result of the convolution calculation output by the ResNet block 207, and derives the optical transmission map ^to Ψ223 by performing deconvolution of 7 layers with respect to the acquired convolution calculation result. do. With this configuration, the light transfer map decoder 206 can estimate the light transfer map Ψ.

The connecting unit 209 acquired and acquired the result of applying the filter output by the encoder 202, the result of performing the convolution calculation output by the ResNet block 204, and the result of performing the convolution calculation output by the ResNet block 207. The result of applying the filter, the result of the convolution calculation, and the result of the convolution calculation are concatenated. The connecting unit 209 outputs the result of applying the filter, the result of performing the convolution calculation, and the result of performing the convolution calculation to the convolution unit 210.
The convolution unit 210 convolves the connection result output by the connection unit 209. Specifically, the convolution unit 210 derives the light source information ^to Π222 by convolving until the spatial resolution is 1 × 1 and the number of channels is 27.

The deconvolution unit 205 outputs the derived reflectance map ^to Λ221 to the multiplication unit 224. The convolution unit 210 outputs the derived light source information ^to Π222 to the multiplication unit 224. The deconvolution unit 208 outputs the derived optical transmission map ^to Ψ223 to the multiplication unit 224. The multiplication unit 224 acquires and acquires the reflectance map ^- Λ output by the deconvolution unit 205, the light source information ^- Π222 output by the convolution unit 210, and the optical transmission map ^- Ψ223 output by the deconvolution unit 208. By multiplying the reflectance map ^- Λ221, the light source information ^- Π222, and the light transmission map ^- Ψ223, a whole-body image of a person ^- I is derived. With this configuration, the image processing apparatus 100 can derive an image that reproduces the input image 211.

(Operation of image processing device 100)
The operation of the image processing device 100 of the present embodiment will be described separately for the case of performing supervised learning and the case of generating an output image corresponding to the input image based on the result of supervised learning.
FIG. 5 is a flowchart showing an example of the operation of the image processing apparatus of the present embodiment. In the example shown in FIG. 5, a process of performing supervised learning is shown.
(Step S1)
The communication unit 105 of the image processing device 100 receives the teacher image transmitted by the server device 70, and outputs the received teacher image to the information processing unit 130. The acquisition unit 131 of the information processing unit 130 acquires the teacher image output by the communication unit 105, and outputs the acquired teacher image to the analysis unit 132.
(Step S2)
The analysis unit 132 acquires the teacher image output by the acquisition unit 131, and estimates the reflectance, the light transfer vector, and the light source information for each pixel from the acquired teacher image.
(Step S3)
The analysis unit 132 creates a reflectance map Λ from the reflectance estimated for each pixel, and creates a light transfer map Ψ from the light transfer vector estimated for each pixel. The analysis unit 132 outputs the created reflectance map Λ, the light transmission map Ψ, and the estimated light source information Π to the machine learning unit 133.
(Step S4)
The machine learning unit 133 acquires the reflectance map Λ output by the analysis unit 132, the light transmission map Ψ, and the light source information Π, and the acquired reflectance map Λ, the light transmission map Ψ, and the light source information Π. Is set as a feature quantity, and it is learned that the feature quantity corresponds to a teacher image.

FIG. 6 is a flowchart showing an example of the operation of the image processing apparatus of the present embodiment. In the example shown in FIG. 6, a process of generating an output image corresponding to an input image based on the result of supervised learning is shown.
(Step S11)
The communication unit 105 of the image processing device 100 receives the input image transmitted by the server device 70, and outputs the received input image to the information processing unit 130. The acquisition unit 131 of the information processing unit 130 acquires the input image output by the communication unit 105, and outputs the acquired input image to the analysis unit 132.
(Step S12)
The analysis unit 132 acquires the input image output by the acquisition unit 131, and estimates the reflectance, the light transfer vector, and the light source information for each pixel from the acquired input image.
(Step S13)
The analysis unit 132 creates a reflectance map ^- Λ from the reflectance estimated for each pixel, and creates a light transfer map ^- Ψ from the light transfer vector estimated for each pixel. The analysis unit 132 outputs the created reflectance map ^- Λ, the light transmission map ^- Ψ, and the estimated light source information ^- Π to the generation unit 134.
(Step S14)
The generation unit 134 acquires the reflectance map ^- Λ, the light transmission map ^- Ψ, and the light source information ^- Π output by the analysis unit 132, and the acquired reflectance map ^- Λ, the light transmission map ^- Ψ, and the like. Generates an output image corresponding to the input image based on the light source information ^~ Π.

(Comparison between the method of this embodiment and the conventional method)
A comparison was made between the case where the light transmission vector included the light shielding information (hereinafter referred to as "this method") and the case where it was not included (hereinafter referred to as "conventional method").
First, a CG person image data set and an environmental light source data set using a 3D person model used as training data (teacher image) and test data (input image) were created.
The CG portrait data set was generated by the GPU-based renderer. The CG portrait image database consists of a binary mask, a reflectance map, a normal map, and a light transmission map for each 3D portrait model. This method does not require a normal map, but the normal map is used for training data and test data in the conventional method.
FIG. 7 is a diagram showing an example of a CG person data set. FIG. 7 (1) shows an example of a reflectance map, FIG. 7 (2) shows an example of a binary mask, FIG. 7 (3) shows an example of a normal map, and FIG. 7 (4) shows. ) Shows an example of an optical transmission map.
Examples of data sources are BUFF datasets and commercial websites. Here, 345 bodies were prepared, of which 276 were used as training data and 69 were used as test data. Since most of the obtained 3D human models do not provide the texture of the glossy reflection component, the case where diffuse reflection is dealt with will be described here. However, since shadows due to fine wrinkles and the like are not completely removed during 3D scanning, shadows due to shielding are also included in the correct reflectance map.

In the present embodiment, in order to suppress the variation in the training data and improve the estimation accuracy of the CNN, the 3D person model is in a standing posture, and the sitting posture is excluded. Then, the drawing position was set to the center of the image, and the image was drawn with a margin of 5% in the vertical width of the image at the top and bottom. Also, when performing estimation, the binary mask of the input image is used to shape the image so that there is a margin of 5% in the vertical width of the image at the top and bottom, and then input to the CNN.
The light source dataset was created by converting a panoramic HDR format environment map obtained from the Laval Indoor HDR dataset into SH coefficients. Some environment maps are too dark or too bright, so I adjusted the brightness. First, the optical transfer vector is analytically calculated from the normal vector (0, 0, 1) ^T of the viewpoint coordinate system, and this is multiplied by the SH coefficient of each environment map to calculate the luminance value for reference. After excluding the environment map whose reference brightness value is too dark than the threshold value 0.2, the brightness value of the environment map was scaled so as to fall within the range of [0.7, 0.9]. Furthermore, in order to increase the variation, the environment map was rotated 35 times by 10 degrees about the vertical direction. Then, in order to reduce the redundancy, the number was reduced by k-means clustering to remove the unnatural light source. Finally, a total of 50 light sources were prepared, of which 40 were randomly selected as training data and 10 were used as test data.

This method was implemented using Python and chainer, supervised learning was performed, and an output image corresponding to the input image was generated by inferring based on the result of supervised learning. For optimization, Adam was used, the learning rate was fixed at 0.0002, and the batch size was 1. The time required for training was about 3 hours per epoch when data of 1024 × 1024 pixels was input using one GPU. As this method, the one learned up to 20 epochs was used. The time required for inference is about 0.43 seconds per input of 1024 × 1024 pixels.

In order to clarify the effect of including the light shielding information in the light transfer vector, we compared this method with the conventional method that estimated the normal map instead of the light transfer map.
The machine learning model of the conventional method is different from the machine learning model of this method, but the machine learning model of this method is the machine of the conventional method in terms of the resolution of the input image, the number of layers, and the number of channels in the intermediate layer. It is assumed that the scale is larger than the learning model and the estimation accuracy is not inferior.
In addition, as a second comparison target, we also prepared a loss function optimized by this method reduced from 15 types to 4 types. As a quantitative comparison, FIG. 8 shows the root-mean square error square root (RMSE) of the estimation results of each method for all test data. The RMSE for the normal map is calculated based on the result of the conventional method.

Except for the light source information, this method (4 types of loss functions) is better than the result of the conventional method in all common elements, and this method (15 types of loss functions) is the best. As a qualitative comparison, FIG. 9 shows the estimation results of each method for the test data.
FIG. 9 is a diagram showing an example of an estimation result by the image processing apparatus of the embodiment. In the example shown in FIG. 9, a comparison between the correct answer and the estimation result is shown in each method when the CG test data is used.
(1) of FIG. 9 is an example of an input image, (2) of FIG. 9 is an example of a correct reflectance map, and (3) of FIG. 9 is an example of a reflectance map by a conventional method. (4) is an example of a reflectance map by this method (4 types of loss functions), and (5) of FIG. 9 is an example of a reflectance map by this method (15 types of loss functions). FIG. 9 (6) is a light source, which is a correct answer in order from the top, a light source by the conventional method, a light source by the present method (4 types of loss functions), and a light source by the present method (15 types of loss functions). FIG. 9 (7) is an example of a correct answer normal map, FIG. 9 (8) is an example of a normal map by a conventional method, and FIG. 9 (9) is an example of a correct shadow map. 9 (10) is an example of a shadow map by the conventional method, FIG. 9 (11) is an example of a shadow map by this method (4 types of loss functions), and FIG. 9 (12) is an example of this method (loss). This is an example of a shadow map using 15 types of functions).
According to FIG. 9, in the reflectance map and the shading map, the conventional method, the present method (4 types of loss functions), and the present method (15 types of loss functions) are improved in this order so as to be coded as a quantitative comparison. You can see that there is. In the shadow map of the conventional method, the corresponding part of the reflectance map is darkened because the occlusion of the neck and armpits cannot be reproduced by the normal map. Comparing this method (4 types of loss functions) with this method (15 types of loss functions), it can be seen that the shadow map is sharper in this method (15 types of loss functions).

Similarly, FIG. 10 shows the results of estimation using the above-mentioned three methods with a live-action person image as an input.
FIG. 10 is a diagram showing an example of an estimation result by the image processing apparatus of the embodiment. In the example shown in FIG. 10, a comparison between the correct answer and the estimation result is shown in each method when a live-action image is used. FIG. 10 (1) is an input image, FIG. 10 (2) is an example of a reflectance map by the conventional method, and FIG. 10 (3) is a reflectance map by this method (4 types of loss functions). As an example, (4) in FIG. 10 is an example of a reflectance map by this method (15 types of loss functions). FIG. 10 (5) is a light source, which is a light source according to the conventional method, a light source according to the present method (4 types of loss function), and a light source according to the present method (15 types of loss function) in order from the top. FIG. 10 (6) is an example of a shadow map by the conventional method, FIG. 10 (7) is an example of a shadow map by this method (4 types of loss functions), and FIG. 10 (8) is an example of this method (8). This is an example of a shadow map based on the loss function (15 types).
As for the result of FIG. 10, similar to the result of FIG. 9 using the test data, the shading map estimated by the conventional method seems to have unevenness on the flat relief, while this method (4 types of loss functions) is used. With this method (15 types of loss functions), shadows due to shielding can be observed.
In addition, simultaneous estimation was performed for each of the two person images, and the light source information was exchanged and re-illuminated to reproduce the appearance under each other's lighting environment.
FIG. 11 is a diagram showing an example of an estimation result by the image processing apparatus of the embodiment. The example shown in FIG. 10 shows the result of reilluminating by exchanging the estimated light source information with each other. According to FIG. 10, since the accuracy of the estimated reflectance map, the light transmission map, and the light source information can be improved as compared with the conventional case, there is not much difference even when compared with the results calculated using the correct answer data.

In the above-described embodiment, the case where the image is transmitted from the terminal device 10 to the image processing device 100 via the server device 70 has been described, but the present invention is not limited to this. For example, the image may be directly input to the image processing device 100.
In the above-described embodiment, the terminal device 10 and the image processing device 100 may be the same device, or the server device 70 and the image processing device 100 may be the same device.
In the above-described embodiment, the case where the image processing device 100 acquires the input image (a full-body image with a mask in which the subject to be reilluminated is captured) transmitted by the terminal device 10 has been described, but the present invention is not limited to this. For example, the terminal device 10 transmits a full-body image of the subject to be relit, and the image processing device 100 acquires and acquires a full-body image of the subject to be relighted transmitted by the terminal device 10. The mask may be estimated based on the whole body image.
In the above-described embodiment, the case where the diffuse reflection component is targeted as the reflectance has been described, but the present invention is not limited to this. For example, it can be applied not only to a diffuse reflection component but also to a specular reflection component including gloss. In this case, supervised learning is performed on the reflectance map for specular reflection, and the specular reflection component of the input image is estimated based on the learning result of supervised learning.
In the above-described embodiment, the case where a convolutional neural network is used as an example of the machine learning model has been described, but the present invention is not limited to this example. For example, a general machine learning model such as a recurrence type neural network may be used.
In the above-described embodiment, the case where the quadratic 9 coefficient of SH is used as the expression method of the light source and the light transfer vector has been described, but the present invention is not limited to this example. For example, the first order or less and the third order or more of SH may be used, all of the coefficients may be used, or only a part of the coefficients may be used.

In the above-described embodiment, SH has been described as a method of expressing the light source and the light transfer vector, but the present invention is not limited to this. For example, a Haar wavelet, a spherical Gaussian function, or the like may be used to represent a light source and a light transfer vector.
In the above-described embodiment, the visible function V (ωi) is from the light source direction LD-1 for each of the plurality of points p-1, p-2, p-3, p-4, and p-5 on the surface of the object. The case where the light is derived based on whether or not the light reaches each of the plurality of points p-1, p-2, p-3, p-4, and p-5 has been described, but the present invention is not limited to this example. .. For example, the visible function V (ωi) may be derived by determining whether or not the light reaches each point (pixel) on the surface of the object and each direction of the light emitted by the light source LS.
In the above-described embodiment, the case where the light source information ^to Π is estimated based on the subject included in the input image has been described, but the present invention is not limited to this example. For example, the light source information ^to Π may be estimated based on the background information (such as the shadow of the building). That is, the light source information ^to Π may be estimated based on the input image including the background.
In the above-described embodiment, the machine learning model is a combination of the light source information Π estimated based on the input image by the machine learning model, the reflectance map Λ, and the light transfer map Ψ. Although the case where the image obtained from the above is learned by comparing the input image with the input image is described, the present invention is not limited to this example. For example, an image obtained from a combination of light source information Π estimated based on an input image by a machine learning model, a reflectance map Λ, and an optical transmission map Ψ, and an image other than the input image. It may be learned by comparing with a predetermined correct image. Specifically, by multiplying the light source and the light transmission map, a shadow map (not including the colors and patterns included in the reflectance map) is obtained, which is generated by CG, and the output of the machine learning model (same). A light source estimated as described above multiplied by a light transfer map) may be compared.
In the above-described embodiment, the case where the image processing apparatus 100 estimates the reflectance, the light transfer vector, and the light source information for each pixel from the acquired teacher image has been described, but the present invention is not limited to this example. For example, the image processing device 100 may estimate the reflectance, the light transfer vector, and the light source information for each of a plurality of pixels from the acquired teacher image.
In the above-described embodiment, the case where the image processing apparatus 100 estimates the light source information from the acquired input image has been described, but the present invention is not limited to this example. For example, the image processing device 100 may acquire light source information from the outside and generate an output image based on the acquired light source information. With this configuration, it is possible to generate a reilluminated image in which the subject included in the input image is reilluminated.

In the above-described embodiment, 15 types of loss functions have been described, but the present invention is not limited to this example. For example, in addition to the above-mentioned 15 types of loss functions, or instead of the above-mentioned 15 types of loss functions, the following loss functions may be used. Specifically, instead of TV loss, when the estimated light transmission map and reflectance map are expressed with', the gradient operator in the x direction and the gradient operator in the y direction are used. Expressed by ∇x and ∇y, a loss function may be used so as to match the correct answer with respect to the gradient, as in equations (A) and (B).

｜ ∇xΛ－∇xΛ´ ｜ ＋ ｜ ∇yΛ－∇yΛ´ ｜ (A)

｜ ∇xψ－∇xψ´ ｜ ＋ ｜ ∇yψ－∇yψ´ ｜ (B)

Further, before calculating ∇x or ∇y, ∇x or ∇y may be calculated after applying a smoothing filter such as 3x3. Specifically, when the smoothing filter is H, the equation (C) and the equation (D) are derived.

｜ ∇xHΛ－∇xHΛ´ ｜ ＋ ｜ ∇yHΛ－∇yHΛ´ ｜ (C)

｜ ∇xHψ－∇xHψ´ ｜ ＋ ｜ ∇yHψ－∇yHψ´ ｜ (D)

In this case, the actual order of calculation is the product of ∇xH and the like, that is, the derivative of the smoothing kernel in the x and y directions is calculated and then convolved.
According to the image processing apparatus 100 of the present embodiment, the image processing apparatus 100 can reilluminate the whole body image of a person taken from a single viewpoint in addition to the reflectance and the light source information in order to solve the new problem of reilluminating the whole body image. It is possible to estimate the light shielding information in the form. Specifically, the light shielding information is expressed as a coefficient of the spherical harmonic function (SH), and the light shielding information expressed as the coefficient of the spherical harmonic function is estimated. With such a configuration, it is possible to realize realistic reillumination as compared with the case where the light shielding is not taken into consideration.
In addition, it is fast because it only calculates the internal product of the 9-dimensional vector for each color channel on a pixel-by-pixel basis. It is possible to perform plausible re-illumination even when a live-action portrait image is input.

<Configuration example>
As a configuration example, an acquisition unit for acquiring an input image, light source information (in the embodiment, light source information Π) which is information indicating an illumination state from a light source, and a reflectance of a subject included in the input image are used in the input image. Reflectivity information estimated based on (in the embodiment, reflectance map Λ) and visibility indicating whether or not the light from the input image and the light source reaches the subject as the transmission state of the light from the subject included in the input image. A generator that generates an output image corresponding to an input image based on light transmission information (light transmission map Ψ in the embodiment) estimated based on information (visible information V (ωi) in the embodiment). An image processing device including an output unit that outputs an output image generated by the generation unit.
As an example of the configuration, the light source information is estimated based on the input image.
As a configuration example, the output image is a reilluminated image in which the subject included in the input image is reilluminated based on the illumination state indicated by the light source information.
As a configuration example, the reflectance information and the light transmission information are estimated based on the input image by a machine learning model (CNN in the embodiment).
As a configuration example, a machine learning model is an image obtained from a combination of light source information estimated based on an input image, reflectance information, and light transmission information by a machine learning model. It was learned by comparing with a predetermined correct image.
As an example of configuration, visible information is created for each of a plurality of points on the surface of an object based on whether or not the light reaches each of the plurality of points in each direction of light.

Although the embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other embodiments, and various omissions, replacements, changes, and combinations can be made without departing from the gist of the invention. These embodiments and variations thereof are included in the scope and gist of the invention, and at the same time, are included in the scope of the invention described in the claims and the equivalent scope thereof.
The image processing device 100 described above has a computer inside. The process of each process of each device described above is stored in a computer-readable recording medium in the form of a program, and the process is performed by the computer reading and executing this program. Here, the computer-readable recording medium refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like. Further, the computer program may be distributed to the computer via a communication line, and the computer receiving the distribution may execute the program.
Further, the above program may be for realizing a part of the above-mentioned functions.
Further, it may be a so-called difference file (difference program) that can realize the above-mentioned function in combination with a program already recorded in the computer system.

1 ... Image processing system, 10, 10-1, 10-2, ... 10-n ... Terminal device, 50 ... Network, 70 ... Server device, 100 ... Image processing device, 105 ... Communication unit, 110 ... Storage Department, 111 ... Program, 112 ... App, 113 ... Reflection map Λ, 114 ... Light source information Π, 115 ... Light transmission map Ψ, 120 ... Operation unit, 130 ... Information processing department, 131 ... Acquisition unit, 132 ... Analysis department , 133 ... Machine learning unit, 134 ... Generation unit, 135 ... Output unit, 140 ... Display unit, 202 ... Encoder, 203 ... Reflection map decoder, 204 ... ResNet block, 205 ... Reverse convolution unit, 206 ... Optical transmission Map decoder, 207 ... ResNet block, 208 ... Reverse folding part, 209 ... Connecting part, 210 ... Folding part, 211 ... Input image, 212 ... Person whole body image I, 213 ... Binary mask M, 214 ... Correct data, 215 ... Reflection map Λ, 216 ... Light transmission map Π, 217 ... Light transmission map Ψ, 220 ... Estimated value, 221 ... Reflection map Λ, 222 ... Light source information Π, 223 ... Light transmission map Ψ, 224 ... Multiplying part, 225 ... Full-body image of a person I

Claims (8)

The acquisition unit that acquires the input image and
Light source information that indicates the lighting state from the light source, reflectance information estimated based on the input image as the reflectance of the subject included in the input image, and light from the subject included in the input image. An output image corresponding to the input image is generated based on the light transmission information estimated based on the input image and the visible information indicating whether or not the light from the light source reaches the subject as the transmission state of. And the generation part to do
It is provided with an output unit that outputs the output image generated by the generation unit.
The light source and the light transmission information are expressed as coefficients of a basis function.
The coefficient is an image processing device estimated by a machine learning model . The image processing apparatus according to claim 1, wherein the light source information is estimated based on the input image. The image processing apparatus according to claim 1, wherein the output image is a reilluminated image in which the subject included in the input image is reilluminated based on the illumination state indicated by the light source information. The image processing apparatus according to any one of claims 1 to 3, wherein the reflectance information and the light transmission information are estimated based on the input image by a machine learning model. The machine learning model is an image obtained from a combination of light source information estimated based on the input image, reflectance information, and light transmission information by the machine learning model. The image processing apparatus according to claim 4, which is learned by comparing with a predetermined correct image. The visible information is created from claim 1 to claim 5 based on whether or not the light reaches each of the plurality of points in each of the plurality of points on the surface of the object for each direction of light. The image processing apparatus according to any one of the following items. Steps to get the input image and
Light source information that indicates the lighting state from the light source, reflectance information estimated based on the input image as the reflectance of the subject included in the input image, and light from the subject included in the input image. An output image corresponding to the input image is generated based on the light transmission information estimated based on the input image and the visible information indicating whether or not the light from the light source reaches the subject as the transmission state of. Steps to do and
It has a step of outputting the output image generated in the step of generating, and has a step of outputting the output image.
The light source and the light transmission information are expressed as coefficients of a basis function.
The coefficient is an image processing method performed by a computer, which is estimated by a machine learning model . On the computer
Steps to get the input image and
Light source information that indicates the lighting state from the light source, reflectance information estimated based on the input image as the reflectance of the subject included in the input image, and light from the subject included in the input image. An output image corresponding to the input image is generated based on the light transmission information estimated based on the input image and the visible information indicating whether or not the light from the light source reaches the subject as the transmission state of. Steps to do and
To execute the step of outputting the output image generated in the above-mentioned generation step ,
The light source and the light transmission information are expressed as coefficients of a basis function.
An image processing program in which the coefficients are estimated by a machine learning model .

Images