JP2020173590A - Processing device, processing system, imaging apparatus, processing method, and program - Google Patents

Abstract

To provide a processing device capable of generating a high-quality rendering image.SOLUTION: A processing device (400) includes: a normal information acquisition unit (104b) that acquires a piece of normal information of a subject; a light source condition acquisition unit (104c) that acquires either of virtual light source angle or virtual light source intensity as a first light source condition when rendering, determines a maximal value of the other of the virtual light source angle or the virtual light source intensity as the maximal value of a second light source condition based on the first light source condition, and acquires a second light source condition smaller than the maximal value; and a rendering unit (104d) that generates a rendering image based on the first light source condition and the second light source condition. The light source condition acquisition unit acquires the larger the first light source condition, the smaller the maximal value of the second light source condition.SELECTED DRAWING: Figure 3

Description

The present invention relates to a processing device that produces a rendered image.

By acquiring more physical information about the subject, it is possible to generate an image based on the physical model in the image processing after imaging. For example, it is possible to generate an image in which the appearance of the subject is changed by a rendering process. The appearance of the subject is determined by the shape information of the subject, the reflectance information of the subject, the light source information, and the like.

Patent Document 1 discloses an illuminance difference stereo method as a method for acquiring (measuring) a normal (plane normal) as shape information of a subject.

JP-A-2010-122158

When acquiring the normal of a subject by the illuminance difference stereo method in an imaging device such as a digital camera, it is necessary to irradiate the subject with light from a plurality of light source positions having different positions from each other. When the position of the light source is fixed, the farther the subject is, the smaller the angle (irradiation angle) between the optical axis (or line of sight) of the imaging optical system included in the imaging device and the light from the light source to the subject. In the illuminance difference stereo method in which the normal of the subject is determined from the brightness changes at a plurality of light source positions, the brightness change decreases as the irradiation angle decreases, and the influence of noise in the image pickup apparatus becomes stronger. As a result, the calculated normals vary.

Further, when the normal is calculated by the illuminance difference stereo method, a normal error occurs for a subject having a reflection characteristic different from the assumed reflection characteristic. Therefore, if a rendered image is generated using the normal with an error, an error also occurs in the rendered image. For example, when the normal of a subject having little diffuse reflection such as a metal or a transparent body is calculated by the illuminance difference stereo method assuming a Lambert diffuse reflection model, a particularly large normal error occurs. Therefore, when a rendered image is generated using the acquired normals, the rendered image breaks down. Further, when the angle of the virtual light source when generating the rendered image becomes large, the change in appearance under the virtual light source becomes large, but the influence of the normal error also becomes large.

Therefore, an object of the present invention is to provide a processing device, a processing system, an imaging device, a processing method, and a program capable of generating a high-quality rendered image.

The processing apparatus as one aspect of the present invention acquires either a virtual light source angle or a virtual light source intensity as the first light source condition at the time of rendering, and the normal light information acquisition unit that acquires the normal line information of the subject. Based on one light source condition, the virtual light source angle or the other maximum value of the virtual light source intensity is determined as the maximum value of the second light source condition at the time of rendering, and the second light source condition equal to or less than the maximum value is set. The light source condition acquisition unit to acquire has a light source condition acquisition unit and a rendering unit that generates a rendered image based on the first light source condition and the second light source condition, and the light source condition acquisition unit has the first light source condition. The larger the value, the smaller the maximum value of the second light source condition.

A processing system as another aspect of the present invention includes a light source unit and the processing apparatus.

An imaging device as another aspect of the present invention includes an imaging unit that images a subject and the processing device.

In the processing method as another aspect of the present invention, one of the normal light source information acquisition step for acquiring the normal line information of the subject and the virtual light source angle or the virtual light source intensity is acquired as the first light source condition at the time of rendering. Based on the first light source condition, the virtual light source angle or the other maximum value of the virtual light source intensity is determined as the maximum value of the second light source condition at the time of rendering, and the second light source condition equal to or less than the maximum value. It has a light source condition acquisition step for acquiring the above, and an image generation step for generating a rendered image based on the first light source condition and the second light source condition. In the light source condition acquisition step, the first light source The larger the condition, the smaller the maximum value of the second light source condition.

A program as another aspect of the present invention causes a computer to execute the processing method.

Other objects and features of the present invention will be described in the following examples.

According to the present invention, it is possible to provide a processing device, a processing system, an imaging device, a processing method, and a program capable of generating a high-quality rendered image.

It is an external view of the image pickup apparatus in Example 1. FIG. It is a block diagram of the image pickup apparatus in Example 1. FIG. It is a block diagram of the processing system in Example 1. FIG. It is a flowchart which shows the processing method in Example 1. FIG. It is a figure which shows the relationship between the 1st light source condition and the maximum value of the 2nd light source condition in Example 1. FIG. It is a block diagram of the image pickup apparatus in Example 2. FIG. It is a flowchart which shows the processing method in Example 2. It is a figure which shows the relationship between the light receiving part of the image pickup device in Example 2 and the pupil of an image pickup optical system. It is a schematic diagram of the imaging unit in Example 2. It is a figure which shows another example of the image pickup apparatus in Example 2. FIG. It is a figure which shows the relationship between the photographing condition in Example 2 and the maximum value of the first light source condition. It is a block diagram of the image pickup apparatus in Example 3. FIG. It is a flowchart which shows the processing method in Example 3. FIG. It is a figure which shows the relationship between the normal reliability in Example 3 and the maximum value of the first light source condition. It is an external view of the processing system in Example 4. FIG. It is explanatory drawing of the Torrance-Sparrow model.

Hereinafter, examples of the present invention will be described in detail with reference to the drawings. In each figure, the same member is given the same reference number, and duplicate description is omitted.

The illuminance difference stereo method assumes the subject's normal (surface normal) and the subject's reflection characteristics based on the direction from the subject to the light source, and the normal is based on the reflection characteristics assumed to be the subject's brightness information at multiple light source positions. Is a method of calculating. The reflection characteristics can be approximated by a Lambertian reflection model that follows Lambert's cosine law if the reflectance is not uniquely determined given a given normal and light source position. FIG. 15 is an explanatory diagram of the Torrance-Sparrow model. As shown in FIG. 15, the specular reflection component depends on the bisector of the light source vector s and the line-of-sight direction vector v and the angle α formed by the normal line n. Therefore, the reflection characteristic may be a characteristic based on the line-of-sight direction. Further, the luminance information may be obtained by photographing each subject when the light source is on and when the light source is off and taking the difference between them to remove the influence of a light source other than the light source such as ambient light.

Hereinafter, the case where the reflection characteristics are assumed in the Lambertian reflection model will be described. The brightness value of the reflected light is i, the Lambert diffuse reflectance of the object is ρ _d , the intensity of the incident light is E, the unit vector (light source direction vector) indicating the direction from the object to the light source (light source direction) is s, and the object Let n be the unit normal vector. At this time, the luminance value i is expressed by the following equation (1) from Lambert's cosine law.

Here, the respective components of the different M (M ≧ 3) light source vectors are s ₁ , s ₂ , ..., S _M , and the luminance values of each component of the light source vector are i ₁ , i ₂ , ... i _M. Then, the equation (1) is expressed as the following equation (2).

In equation (2), the left side is the luminance vector of M rows and 1 column, the right side [s ₁ ^T , ... s _M ^T ] is the incident light matrix S indicating the light source direction of M rows and 3 columns, and n is the 3 rows and 1 column. It is an identity normal vector. When M = 3, Eρ _d n is expressed by the following equation (3) by using the inverse matrix S ^-1 of the incident light matrix S.

The norm of the vector on the left side of the equation (3) is the product of the intensity E of the incident light and the Lambert diffuse reflectance ρ _d , and the normalized vector is calculated as the normal vector of the object. That is, since the intensity E of the incident light and the Lambert diffuse reflectance ρ _d appear in the conditional equation only in the form of a product, if Eρ _d is one variable, the equation (3) has two degrees of freedom of the unit normal vector n. Together with, it can be regarded as a simultaneous equation that determines three unknown variables. Therefore, each variable can be determined by acquiring the luminance information using at least three light sources. If the incident light matrix S is not a regular matrix, there is no inverse matrix. Therefore, it is necessary to select each component s _{1 to} s ₃ of the incident light matrix S so that the incident light matrix S becomes a regular matrix. That is, it is preferable to select the component s ₃ linearly independently of the components s ₁ and s ₂ .

When M> 3, more conditional expressions than the unknown variables to be obtained can be obtained. Therefore, the unit normal vector n may be calculated from the three conditional expressions arbitrarily selected by the same method as in the case of M = 3. When four or more conditional equations are used, the incident light matrix S is no longer an invertible matrix. Therefore, for example, the Moore-Penrose pseudo-inverse matrix may be used to calculate the approximate solution. Further, the unit normal vector n may be calculated by a fitting method or an optimization method.

Assuming that the reflection characteristics of the subject are different from the Lambertian reflection model, the conditional equation may differ from the linear equation for each component of the unit normal vector n. In this case, if a conditional expression equal to or greater than the unknown variable is obtained, a fitting method or an optimization method can be used.

Further, when M> 3, since a plurality of conditional expressions of 3 or more and M-1 or less can be obtained, a plurality of solution candidates of the unit normal vector n can be obtained. In this case, a solution may be selected from a plurality of solution candidates using yet another condition. For example, the continuity of the unit normal vector n can be used as a condition. When calculating the unit normal n for each pixel of the image pickup apparatus, the normal at the pixel (x, y) is n (x, y), and if n (x-1, y) is known, the following The solution that minimizes the evaluation function represented by Eq. (4) may be selected.

If n (x + 1, y) and n (x, y ± 1) are also known, the solution in which the following equation (5) is minimized may be selected.

If there is no known normal and there is indefiniteness of the normal at all pixel positions, the sum of all pixels in equation (5) should be minimized, as shown in equation (6) below. You may choose a solution.

It should be noted that the normals of pixels other than the nearest neighbor may be used, or an evaluation function weighted according to the distance from the pixel position of interest may be used. Further, as another condition, the luminance information at an arbitrary light source position may be used. In the diffuse reflection model represented by the Lambertian reflection model, the closer the unit normal vector and the light source direction vector are, the greater the brightness of the reflected light. Therefore, the unit normal vector can be determined by selecting a solution close to the light source direction vector having the largest brightness value among the brightness values in a plurality of light source directions.

Further, in the specular reflection model, if the light source vector is s and the unit vector in the direction from the object to the camera (camera line-of-sight vector) is v, the following equation (7) holds.

As represented by the equation (7), if the light source direction vector s and the camera line-of-sight vector v are known, the unit normal vector n can be calculated. When the surface is rough, the specular reflection also has a widening of the emission angle, but it spreads near the solution obtained as a smooth surface, so it is sufficient to select the candidate closest to the solution for the smooth surface from among the multiple solution candidates. .. In addition, the true solution may be determined by averaging a plurality of solution candidates.

When the normal line n and the reflectance ρ (= Eρ _d ) are obtained by the above illuminance difference stereo method, the brightness value i under an arbitrary light source can be obtained by giving an arbitrary light source vector s to the equation (1). Can be calculated. That is, it is possible to generate a rendered image that reproduces the appearance under an arbitrary light source. In equation (1), a rendered image with Lambert diffuse reflection is generated, but a rendered image with other diffuse reflection characteristics and specular reflection characteristics can also be generated.

Next, the image pickup apparatus according to the first embodiment of the present invention will be described with reference to FIGS. 1 and 2A. FIG. 1 is an external view of the image pickup apparatus 1 in this embodiment. FIG. 2A is a block diagram of the image pickup apparatus 1.

The image pickup apparatus 1 performs a rendering process to generate a rendered image (rewriting image). As shown in FIG. 1, the image pickup apparatus 1 has an image pickup unit 100 and a light source unit 200 for photographing a subject. As shown in FIG. 2A, the image pickup unit 100 includes an image pickup optical system 101 and an image pickup element 102. The light source unit 200 is configured to be able to irradiate the subject with light from a plurality of positions different from each other. In this embodiment, the light source unit 200 has eight light sources 200a to 200h, but is not limited thereto. Since at least three light sources are required when performing the illuminance difference stereo method, it is sufficient to provide at least three or more light sources for acquiring an input image. Further, in the present embodiment, eight light sources are arranged concentrically at equal distances from the optical axis OA of the imaging optical system constituting the imaging unit 100, but the present invention is not limited to this. Further, in this embodiment, the light source unit 200 is built in the image pickup apparatus 1, but the light source unit 200 is not limited thereto. The light source unit 200 may be configured to be detachably attached to the image pickup apparatus 1.

The image pickup optical system 101 includes a diaphragm 101a, and forms an image of light from a subject on the image pickup element 102. The image sensor 102 is composed of a photoelectric conversion element such as a CCD sensor or a CMOS sensor, and images a subject. That is, the image sensor 102 photoelectrically converts the image (optical image) of the subject formed by the image pickup optical system 101 to generate an analog electric signal (image data corresponding to the input image). The A / D converter 103 converts the analog signal generated by the photoelectric conversion of the image sensor 102 into a digital signal, and outputs the digital signal to the image processing unit 104.

The image processing unit (processing device) 104 performs various image processing on the digital signal input from the A / D converter 103. Further, in this embodiment, the image processing unit 104 calculates the normal information of the subject and generates a rendered image under an arbitrary light source. The image processing unit 104 includes an input image acquisition unit 104a, a normal information acquisition unit 104b, a light source condition acquisition unit 104c, and a rendering unit 104d.

The output image processed by the image processing unit 104 is stored in an image recording unit 109 such as a semiconductor memory or an optical disk. Further, the output image may be displayed on the display unit (display) 105. In this embodiment, the input image acquisition unit 104a, the normal information acquisition unit 104b, the light source condition acquisition unit 104c, and the rendering unit 104d are built in the image pickup apparatus 1. However, the present invention is not limited to this, and at least a part of each of the above-mentioned parts may be provided separately from the image pickup apparatus 1.

The information input unit 108 supplies the shooting conditions (aperture value, exposure time, ISO sensitivity, focal length, etc.) selected by the user to the system controller 110. The image pickup control unit 107 acquires an image under desired shooting conditions selected by the user based on the information from the system controller 110. The irradiation light source control unit 106 controls the light emitting state of the light source unit 200 in response to a control instruction from the system controller 110. Further, the information input unit 108 supplies the system controller 110 with the light source conditions (virtual light source angle, virtual light source intensity, virtual light source color, etc.) selected by the user. The image processing unit 104 generates a rendered image (rewriting image) under desired virtual light source conditions selected by the user based on the information from the system controller 110. In this embodiment, the imaging optical system 101 is integrally configured with the imaging device 1, but the present invention is not limited to this. The present invention can also be applied to a camera system such as a single-lens reflex camera or a mirrorless camera, which is composed of an image pickup device main body having an image pickup element and an image pickup optical system (interchangeable lens) attached to and detachable from the image pickup device main body. is there.

Next, the rendering process (processing method) in this embodiment will be described with reference to FIGS. 3 and 4. FIG. 3 is a flowchart showing the rendering process. The rendering process of this embodiment is executed by the system controller 110 and the image processing unit 104 according to a processing program as a computer program. The processing program is stored in, for example, a computer-readable storage medium (such as the internal memory of the system controller 110).

First, in step S101, the input image acquisition unit 104a acquires a plurality of input images acquired by the imaging unit 100 by imaging the subject at a plurality of light source positions having different positions from each other. A plurality of input images can be acquired by sequentially irradiating light from a single light source while changing the position of a single light source (using a drive unit or the like). Alternatively, the plurality of input images may be acquired by sequentially irradiating light from a plurality of light sources having different positions (for example, the eight light sources 200a to 200h shown in FIG. 1).

Further, when the normal information is acquired by the illuminance difference stereo method assuming a diffuse reflection model such as Lambertian reflection in step S102 described later, a plurality of diffuse reflection images obtained by removing the specular reflection component from the captured image are used as the input image. You may. In order to acquire a diffuse reflection image in which the specular reflection component is removed from the image, for example, a method using a bicolor reflection model can be used. However, the method for removing the specular reflection component from the image is not limited to this, and various methods can be used.

Subsequently, in step S102, the normal information acquisition unit 104b acquires the normal information. In this embodiment, specifically, the normal information acquisition unit 104b uses a plurality of input images acquired by imaging the subject at a plurality of light source positions having different positions from each other, and uses the normal information of the subject. And get the reflectance. The normal information and the reflectance are calculated based on the change in the luminance information depending on the position of the light source by using the illuminance difference stereo method. In this embodiment, the normal information acquisition unit 104b calculates the normal information and the reflectance based on a plurality of input images, but the normal information and the reflectance calculated by another unit may be acquired. , It is not necessary to acquire the input image.

Here, the acquired normal information has an error due to various factors as described above. For example, as the distance to the subject increases, the angle (irradiation angle) formed by the optical axis (or line of sight) of the imaging optical system 101 included in the imaging device 1 and the light from the light source to the subject becomes smaller. In the illuminance difference stereo method in which the normal of the subject is determined from the brightness changes at a plurality of light source positions, the brightness change decreases as the irradiation angle becomes smaller, and the influence of noise in the image pickup apparatus 1 becomes stronger. As a result, the calculated normals vary. Further, when the surface normal is calculated by the illuminance difference stereo method, a normal error occurs for a subject having a reflection characteristic different from the assumed reflection characteristic. Therefore, when a rendered image is generated using the surface normal in which an error occurs, an error also occurs in the rendered image. For example, when the surface normal of a subject having little diffuse reflection such as a metal or a transparent body is calculated by the illuminance difference stereo method assuming a Lambert diffuse reflection model, a particularly large normal error occurs. Therefore, when a rendered image is generated using the acquired surface normal, the rendered image is broken.

Further, when the virtual light source angle at the time of rendering becomes large, the change in appearance under the virtual light source becomes large, but the influence of the normal error on the rendered image becomes large. As shown by the equation (1), the rendered image changes according to the inner product of the normal vector and the virtual light source vector, that is, the cosine of the angle formed by the normal vector and the virtual light source vector. When an error occurs in the normal vector, the error in the rendered image increases as the angle formed by the normal vector and the virtual light source vector approaches 90 °. Further, in the illuminance difference stereo method, since the normal information is calculated from the brightness changes at a plurality of different light source positions, the error in the rendered image increases as the virtual light source at the time of rendering moves away from the light source position at the time of shooting. ..

Subsequently, in step S103, the light source condition acquisition unit 104c acquires the first light source condition (at the time of rendering) for generating the rendered image. Here, the first light source condition is one of the virtual light source angle and the virtual light source intensity. The first light source condition can be a virtual light source angle or a virtual light source intensity specified by the user, and may be, for example, a virtual light source condition predetermined for each shooting scene or the like. The first light source condition selected by the user is supplied from the information input unit 108 to the system controller 110, and the light source condition acquisition unit 104c acquires the first light source condition based on the information from the system controller 110.

Here, the virtual light source angle is an angle formed by the line-of-sight vector or the optical axis and the virtual light source vector at the time of rendering. The virtual light source intensity is the intensity of the rendered image that reproduces the appearance under the virtual light source, and is the ratio of the rendered image to be generated when the rendered image generated by the equation (1) is considered as 1. That is, the virtual light source intensity is determined based on the proportion of the base image in the rendered image. For example, when the virtual light source intensity is 0.5, the rendered image generated by using the equation (1) multiplied by 0.5 is generated as a rendered image at the set virtual light source intensity. In addition, when a rendered image that reproduces the appearance under the virtual light source is added to or weighted averaged to the base image such as the input image to generate the rendered image, the appearance under the virtual light source of the final rendered image is reproduced. The ratio of the rendered image may be used as the virtual light source intensity. Therefore, the greater the intensity of the virtual light source, the greater the change in appearance under the virtual light source, but the greater the effect of the normal error on the rendered image.

Subsequently, in step S104, the light source condition acquisition unit 104c has a range (maximum value) of the second light source condition (at the time of rendering) for generating a rendered image based on the first light source condition acquired in step S103. ) Is determined. Here, the second light source condition is the other of the virtual light source angle or the virtual light source intensity acquired as the first light source condition. That is, when the first light source condition is the virtual light source angle, the second light source condition is the virtual light source intensity. When the first light source condition is the virtual light source intensity, the second light source condition is the virtual light source angle.

As described above, when the virtual light source angle (first light source condition) at the time of rendering becomes large, the change in appearance under the virtual light source becomes large, but the influence of the normal error on the rendered image becomes large. Therefore, the larger the virtual light source angle, the smaller the virtual light source intensity. That is, the larger the virtual light source angle, the smaller the maximum value of the virtual light source intensity (maximum intensity of the entire image). By limiting the range of the virtual light source intensity to be smaller as the virtual light source angle is larger, it is possible to reduce the influence of the normal error on the rendered image when the virtual light source angle is increased.

When the virtual light source intensity is acquired as the first light source condition in step S103, the larger the virtual light source intensity, the smaller the maximum value of the virtual light source angle is limited. By limiting the range of the virtual light source angle to a smaller value as the virtual light source intensity increases, the influence of the normal error on the rendered image can be reduced. Here, the virtual light source angle when determining the range of the light source condition is the angle formed by the line-of-sight vector or the optical axis and the virtual light source vector at the time of rendering, or the light source vector at the time of shooting and the virtual light source vector at the time of rendering. It is a horn. Alternatively, the virtual light source angle may be the minimum value of the angle formed by the plurality of light source vectors at the time of shooting and the virtual light source vector at the time of rendering.

FIG. 4 is a diagram showing the relationship between the first light source condition and the maximum value of the second light source condition. In FIG. 4, the horizontal axis represents the maximum value of the first light source condition, and the vertical axis represents the maximum value of the second light source condition. For example, as shown in FIG. 4, the larger the first light source condition, the smaller the maximum value of the second light source condition. Although it is determined in FIG. 4 that the maximum value of the second light source condition is linear with respect to the first light source condition, the present invention is not limited to this. The maximum value of the second light source condition may be non-linear with respect to the first light source condition. Further, the relationship between the maximum value of the second light source condition and the first light source condition can be stored as a function, or the maximum value of the second light source condition with respect to the first light source condition can be discretely tabled. You may remember as.

Subsequently, in step S105 of FIG. 3, the light source condition acquisition unit 104c acquires the second light source condition based on the range (maximum value) of the second light source condition determined in step S104. When the virtual light source angle is acquired as the first light source condition in step S103, the range of the virtual light source intensity as the second light source condition is determined in step S104, and the virtual light source intensity is acquired based on the range. The virtual light source intensity may be set within the determined virtual light source intensity range. Alternatively, when the virtual light source intensity outside the range is set (when the light source condition acquisition unit 104c acquires the value of the second light source condition larger than the maximum value of the second light source condition), a warning is displayed. May be good.

When the virtual light source intensity is acquired as the first light source condition in step S103, the range of the virtual light source angle as the second light source condition is determined in step S104, and the virtual light source angle is acquired based on the range. The light source angle may be set within the determined range of the light source angle, or a warning may be displayed when a virtual light source angle outside the range is set. The acquisition of the second light source condition may be the virtual light source angle or the virtual light source intensity specified by the user as in the case of the first light source condition, or is automatically determined by setting the maximum virtual light source condition that can be set. It may be a virtual light source condition. The second light source condition selected by the user is also supplied from the information input unit 108 to the system controller 110 in the same manner as the first light source condition, and the light source condition acquisition unit 104c is the second based on the information from the system controller 110. Get the light source condition.

Subsequently, in step S106, the rendering unit 104d generates a rendered image based on the first light source condition and the second light source condition acquired in step S103 and step S105. The rendered image may be only a rendered image that reproduces the appearance under a virtual light source generated based on the light source condition, or may be an addition or weighted averaging of a base image such as an input image. In either case, a rendered image having the virtual light source intensity acquired in step S105 is generated. As the base image, an input image (an image based on an input image which is a plurality of captured images taken at at least three different light source positions) obtained by changing the light source position as described above can be used. Alternatively, as the base image, an image having a different light source environment, such as an ambient light image obtained by taking a picture without irradiating the light source, may be used. Further, in the generation of the rendered image, the reflection characteristics assumed in the calculation of the normal information in step S102 may not be used, or the rendered image with other diffuse reflection characteristics and in addition to the specular reflection characteristics may be generated. Good. Further, the rendered image may be generated by regarding the input image or the base image as the reflectance.

As described above, in this embodiment, the processing device (image processing unit 104) has a normal information acquisition unit 104b, a light source condition acquisition unit 104c, and a rendering unit 104d. The normal information acquisition unit acquires the normal information of the subject. The light source condition acquisition unit acquires either the virtual light source angle or the virtual light source intensity as the first light source condition at the time of rendering. Then, the light source condition acquisition unit determines the virtual light source angle or the other maximum value of the virtual light source intensity as the maximum value of the second light source condition at the time of rendering based on the acquired first light source condition, and is equal to or less than the maximum value. Get the second light source condition. The rendering unit generates a rendered image based on the first light source condition and the second light source condition. Further, the light source condition acquisition unit reduces the maximum value of the second light source condition as the first light source condition becomes larger.

In this embodiment, the image pickup apparatus 1 is used to calculate the surface normal of the subject to generate a rendered image, but the present embodiment is not limited to this. For example, as shown in FIG. 2B, processing may be performed using a processing system 2 different from that of the imaging device 1. FIG. 2B is a block diagram of the processing system 2. The processing system 2 includes a processing device 500, an imaging unit 501, and a light source unit 502. The processing device 500 includes an input image acquisition unit 500a, a normal information acquisition unit 500b, a light source condition acquisition unit 500c, and a rendering unit 500d.

When a rendered image is generated using the processing system 2, first, the input image acquisition unit 500a acquires a plurality of input images acquired by imaging a subject at a plurality of light source positions having different positions from each other. Next, the normal information acquisition unit 500b calculates the normal information and the reflectance of the subject based on the plurality of input images. Then, the light source condition acquisition unit 500c acquires the first light source condition for generating the rendered image, determines the range (maximum value) of the second light source condition based on the first light source condition, and the second Obtain a second light source condition based on the range of light source conditions. Then, the rendering unit 500d generates a rendered image based on the acquired light source conditions. The image pickup unit 501 and the light source unit 502 may be individual devices, or the light source unit 502 may be built in the image pickup unit 501.

According to this embodiment, a high-quality rendered image can be generated by determining the range (maximum value) of the second light source condition based on the first light source condition.

Next, Example 2 of the present invention will be described. The imaging device 1a of the present embodiment acquires the normal of the subject and generates a high-quality rendered image, similarly to the imaging device 1 of the first embodiment. Further, the image pickup apparatus 1a of the present embodiment determines the range (maximum value) of the first light source condition based on the imaging conditions, and acquires the first light source condition based on the determined range.

FIG. 5 is a block diagram of the image pickup apparatus 1a in this embodiment. FIG. 6 is a flowchart showing a rendering process (processing method) in this embodiment. The rendering process of this embodiment is executed by the system controller 110 and the image pickup control unit 107 according to the image processing program as a computer program, as in the first embodiment.

The image processing unit 104 acquires the normal information of the subject and generates a rendered image in addition to the image processing generally performed on the digital signal. As shown in FIG. 5, the image processing unit 104 includes an input image acquisition unit 104a, a normal information acquisition unit 104b, a light source condition acquisition unit 104c, a rendering unit 104d, and a shooting condition acquisition unit 104e. Since steps S201, S203, and S206 to S208 in FIG. 6 are the same as steps S101, S102, and S104 to S106 of the first embodiment, detailed description thereof will be omitted.

In step S202 of FIG. 6, the shooting condition acquisition unit 104e acquires the shooting conditions when the input image is acquired. The shooting condition acquisition unit 104e acquires distance information (subject distance) to the subject as shooting conditions, for example. In this embodiment, the distance information is acquired from the autofocus when the input image acquired in step S201 is taken, or the position of the focus lens when the user manually focuses. Further, the distance information may be acquired by the stereo method of acquiring a plurality of parallax images taken from different viewpoints. In the stereo method, the depth is acquired by triangulation from the amount of parallax of the corresponding point of the subject in the acquired plurality of parallax images, the position information of each viewpoint taken, and the focal length of the optical system. The distance information may be the average value of the depths calculated at the corresponding points of the subject, or may be the depth at a specific point of the subject.

When acquiring distance information from a parallax image, a plurality of image pickup units (imaging systems) 100 of the plurality of parallax images have passed through different regions P1 and P2 of the pupils of the image pickup optical system 101, as shown in FIG. The luminous flux of the image sensor 102 is guided to different light receiving units (pixels) of the image sensor 102 to perform photoelectric conversion.

FIG. 7 is a diagram showing the relationship between the light receiving portion of the image sensor 102 and the pupil of the image pickup optical system 101. A plurality of pairs (pixel pairs) of G1 pixels and G2 pixels, which are light receiving units, are arranged on the image sensor 102. A plurality of G1 pixels are collectively referred to as a G1 pixel group, and a plurality of G2 pixels are collectively referred to as a G2 pixel group. The pair of G1 pixels and the G2 pixel have a conjugate relationship with the exit pupil EXP of the imaging optical system 101 via a common microlens ML (that is, one is provided for each pixel pair). Further, a color filter CF is provided between the microlens ML and the light receiving portion.

FIG. 8 is a schematic view of an imaging unit (imaging system) assuming that a thin-walled lens is located at the position of the exit pupil EXP. The G1 pixel receives the luminous flux that has passed through the region P1 of the exit pupil EXP, and the G2 pixel receives the luminous flux that has passed through the region P2 of the exit pupil EXP. An object does not necessarily have to exist in the object point OSP being imaged, and the luminous flux passing through the object point OSP is incident on the G1 pixel or the G2 pixel depending on the region (position) in the pupil through which the object is being imaged. The passage of the luminous flux through different regions in the pupil corresponds to the separation of the incident light from the object point OSP by the angle (parallax). That is, among the G1 pixels and G2 pixels provided for each microlens ML, the image generated by using the output signal from the G1 pixel and the image generated by using the output signal from the G2 pixel are separated from each other. It becomes a plurality of (here, a pair) parallax images having. In the following description, receiving light flux passing through different regions in the pupil by different light receiving portions (pixels) is referred to as pupil division.

In FIGS. 7 and 8, even when the above-mentioned conjugate relationship is not perfect due to the position of the exit pupil EXP being displaced, or when the regions P1 and P2 partially overlap, the plurality of obtained regions are obtained. Image can be treated as a parallax image.

FIG. 9 is a diagram showing another example of the imaging device. As shown in FIG. 9, a parallax image can be acquired by providing a plurality of imaging optical systems OSj (j = 1, 2) in one imaging device. A parallax image can also be obtained when the same subject is imaged using a plurality of cameras.

Further, in this embodiment, ISO sensitivity and noise amount may be acquired as imaging conditions. Alternatively, variations in light emission (light intensity) from a plurality of different light source positions when the input image is acquired may be acquired as an imaging condition.

In step S204 of FIG. 6, the light source condition acquisition unit 104c determines the range (maximum value) of the first light source condition based on the shooting conditions acquired in step S202. When the position of the light source is fixed, the farther the subject is, the smaller the angle (irradiation angle) formed by the optical axis (or line of sight) of the imaging optical system 101 and the direction of the light source. In the illuminance difference stereo method in which the normal of the subject is calculated from the brightness changes at a plurality of light source positions, the brightness change decreases and the influence of noise becomes stronger as the irradiation angle becomes smaller. When the influence of noise becomes strong, the calculated normals vary. When a rendered image is generated using scattered normals, the noise is amplified. Therefore, when the distance to the subject (subject distance) is long and the influence of noise becomes strong, the range (maximum value) of the first light source condition is reduced.

When the first light source condition is the virtual light source angle, the maximum value of the virtual light source angle is reduced as the distance to the subject increases. When the first light source condition is the virtual light source intensity, the maximum value of the virtual light source intensity is reduced as the distance increases. By limiting the range of the first light source condition as the distance to the subject increases, it is possible to reduce the influence of noise on the normal and the rendered image when the distance to the subject increases. Further, when the ISO sensitivity and the amount of noise are acquired as the shooting conditions, the range of the first light source condition is reduced as the amount of noise increases. That is, the higher the ISO sensitivity, the smaller the range of the first light source condition. Further, when the emission variation of a plurality of light sources when the input image is acquired as the shooting condition is acquired, the larger the emission variation is, the smaller the range of the first light source condition is.

When a photographed image obtained by imaging a subject at a plurality of light source positions having different positions is acquired as an input image, the greater the variation in the amount of light from the plurality of light source positions, the greater the normal error. The range of the light source condition of is reduced. By limiting the range of the first light source condition as the above-mentioned shooting condition is larger, it is possible to reduce the influence of noise and emission variation on the normal and the rendered image.

FIG. 10 is a diagram showing the relationship between the photographing conditions and the maximum value of the first light source condition in this embodiment. In FIG. 10, the horizontal axis represents the shooting conditions (distance, ISO sensitivity, noise amount, emission variation), and the vertical axis represents the maximum value of the first light source condition. For example, as shown in FIG. 10, the maximum value of the first light source condition is reduced as the shooting conditions such as the above-mentioned distance information become larger. In FIG. 10, it is determined that the maximum value of the first light source condition is linear with respect to the photographing condition, but this embodiment is not limited to this. The maximum value of the first light source condition may be non-linear with respect to the shooting condition. Further, the relationship between the maximum value of the first light source condition with respect to the shooting condition may be stored as a function, or the maximum value of the first light source condition with respect to the shooting condition may be stored discretely as a table. May be good. Further, the range of the first light source condition may be determined for each region in the rendered image. For example, the farther the distance to the subject is, the smaller the maximum value of the virtual light source intensity is.

Subsequently, in step S205 of FIG. 6, the light source condition acquisition unit 104c acquires the first light source condition based on the range (maximum value) of the first light source condition determined in step S204. When acquiring the virtual light source angle as the first light source condition, the range (maximum value) of the virtual light source angle is determined in step S204, and the virtual light source angle is acquired based on the range. The virtual light source angle may be set within the range of the determined virtual light source angle, or a warning may be displayed when the virtual light source angle outside the range is set. When acquiring the virtual light source intensity as the first light source condition, the range (maximum value) of the virtual light source intensity is determined in step S204, and the virtual light source intensity is acquired based on the range. The virtual light source intensity may be set within the determined virtual light source intensity range, or a warning may be displayed when the virtual light source intensity outside the range is set.

The first light source condition may be a virtual light source angle or a virtual light source intensity specified by the user, or may be a light source condition automatically determined by setting the maximum light source condition that can be set. The first light source condition selected by the user is supplied from the information input unit 108 to the system controller 110, and the light source condition acquisition unit 104c acquires the first light source condition based on the information from the system controller 110.

As described above, in this embodiment, the shooting condition acquisition unit 104e acquires the shooting conditions when the input image is shot. The light source condition acquisition unit 104c determines the maximum value of the first light source condition based on the photographing condition. Preferably, the shooting condition is the subject distance when the input image is shot. More preferably, the light source condition acquisition unit reduces the maximum value of the first light source condition as the subject distance increases. Further, preferably, the shooting conditions are at least one of the amount of noise in the input image, the ISO sensitivity when the input image is shot, and the emission variation when the input image is shot. Further, preferably, the light source condition acquisition unit determines the maximum value of the first light source condition for each region of the rendered image.

According to this embodiment, a high-quality rendered image can be generated by determining the range of the second light source condition based on the first light source condition. Further, according to this embodiment, a high-quality rendered image is generated by determining the range of the first light source condition based on the shooting conditions and acquiring the first light source condition based on the determined range. Can be done.

Next, Example 3 of the present invention will be described. The image pickup apparatus 1b of the present embodiment acquires the normal of the subject and generates a high-quality rendered image, similarly to the image pickup apparatus 1 of the first embodiment. Further, the image pickup apparatus 1b of the present embodiment determines the range (maximum value) of the first light source condition based on the normal reliability, and acquires the first light source condition based on the determined range.

FIG. 11 is a block diagram of the image pickup apparatus 1b in this embodiment. FIG. 12 is a flowchart showing a rendering process (processing method) in this embodiment. The rendering process of this embodiment is executed by the system controller 110 and the image pickup control unit 107 according to the image processing program as a computer program, as in the first embodiment.

The image processing unit 104 acquires the normal information of the subject and generates a rendered image in addition to the image processing generally performed on the digital signal. As shown in FIG. 11, the image processing unit 104 includes an input image acquisition unit 104a, a normal information acquisition unit 104b, a light source condition acquisition unit 104c, a rendering unit 104d, and a normal reliability acquisition unit 104f. Since steps S301, S302, and S306 to S308 in FIG. 12 are the same as steps S101, S102, and S104 to S106 of the first embodiment, detailed description thereof will be omitted.

In step S303, the normal reliability acquisition unit 104f produces a reproduced rendered image based on the normal information and reflectance acquired in step S302 and the light source conditions when the input image acquired in step S301 is captured. Generate. That is, the normal reliability acquisition unit 104f uses the acquired normal information and reflectance to generate a rendered image that reproduces the input image. The reproduction rendered image is generated by the rendering process using the reflection characteristic assumed when calculating the normal information in step S302. When the normal information is calculated assuming Lambert diffusion, a plurality of reproduced rendered images are generated according to the equation (1). Further, the reproduced rendered image may be a rendered image corresponding to a part of the input images among the plurality of input images, or may be a plurality of rendered images corresponding to all of the plurality of input images.

Further, the normal reliability acquisition unit 104f acquires the normal reliability, which is the reliability of the normal information, based on the input image acquired in step S301 and the reproduced rendered image acquired in step S303. As described above, when the surface normal is calculated by the illuminance difference stereo method, a normal error occurs for a subject having a reflection characteristic different from the assumed reflection characteristic. Therefore, when a rendered image is generated using the surface normal in which an error occurs, an error also occurs in the rendered image.

Therefore, the normal reliability acquisition unit 104f acquires the normal reliability based on the difference between the input image acquired in step S301 and the reproduced rendered image acquired in step S303. The larger the difference, the lower the normal reliability. The normal reliability may be the difference between the input image and the reproduced rendered image acquired in step S303. Alternatively, the normal reliability may be a normalized difference obtained by dividing the difference by the reflectance as the normal reliability, or the difference is based on an input image such as an input image or an average value or a median value of a plurality of input images. The normalized difference divided by the image may be used as the normal reliability. By dividing the difference by the reflectance or the input image, the influence of brightness can be excluded from the calculated difference.

Alternatively, the maximum difference, which is the maximum value in the plurality of differences between the plurality of input images and the plurality of reproduced rendered images, or the average difference, which is the average value, may be used as the normal reliability. Further, the normalized maximum difference or the normalized average difference obtained by dividing the maximum difference or the average difference by the reflectance may be used as the normal reliability. Alternatively, the normal maximum difference or the normalized average difference obtained by dividing the maximum difference or the average difference by an image based on the input image such as the input image or the average value or the median value of the input image or a plurality of input images may be used as the normal reliability.

In addition, the rendering process using the normal information cannot reproduce the shadow generated by blocking the light. Therefore, when the difference between the input image and the reproduced rendered image is taken, the difference becomes large in the shadow area in the input image, and even if the normal information is accurate, it is detected as an error area. Therefore, it is preferable to generate a reproduced rendered image that reproduces shadows by using shape information in addition to normal information. When there is no shape information, it is preferable to detect a shadow area in the input image and not to take a difference in the detected shadow area. The shadow area in the input image may be an area where the brightness value is equal to or less than the threshold value. Alternatively, the normal reliability may be obtained based on the difference and the code between the input image and the reproduced rendered image. For example, in the shadow area of the input image, the value obtained by subtracting the reproduction rendered image from the input image is negative, so that the difference is not large (reliability is low) in the negative difference area.

Further, the normal reliability obtained by threshold-processing the normal reliability based on the difference between the input image and the reproduced rendered image acquired in step S303 may be used as the normal reliability. Specifically, with respect to the difference between the input image and the reproduced rendered image, the area where the difference is greater than or equal to the threshold value is regarded as the unreliable area, and the area where the difference is smaller than the threshold value is defined as the area with reliability. .. In this embodiment, the normal reliability is acquired based on the difference between the input image and the reproduced rendered image. However, this embodiment is not limited to this, and the normal information is acquired by fitting the luminance values in a plurality of input images, and the obtained fitting error and the threshold value processing of the obtained fitting error are obtained. It may be acquired as normal reliability.

Subsequently, in step S304, the light source condition acquisition unit 104c determines the range (maximum value) of the first light source condition based on the normal reliability acquired in step S303. If the normal reliability is low, reduce the range of the first light source condition. Specifically, the average value, the median value, and the maximum value of the difference between the input image and the reproduced rendered image, which are the normal reliability, are large, and the lower the reliability, the smaller the range of the first light source condition. Alternatively, the larger the number of pixels in the region without normal reliability, the smaller the range of the first light source condition. For example, when the first light source condition is the virtual light source angle, the lower the normal reliability, the smaller the maximum value of the virtual light source angle. Alternatively, when the first light source condition is the virtual light source intensity, the lower the normal reliability, the smaller the maximum value of the virtual light source intensity. By limiting the range of the first light source condition as the normal reliability is lower, the influence of the normal error on the rendered image can be reduced.

FIG. 13 is a diagram showing the relationship between the normal reliability and the maximum value of the first light source condition in this embodiment. In FIG. 13, the horizontal axis represents the normal reliability and the vertical axis represents the maximum value of the first light source condition. For example, as shown in FIG. 13, the lower the normal reliability, the smaller the maximum value of the first light source condition. In FIG. 13, it is determined that the maximum value of the first light source condition is linear with respect to the normal reliability, but this embodiment is not limited to this. The maximum value of the first light source condition may be non-linear with respect to the normal reliability. Further, the relationship between the maximum value of the first light source condition with respect to the normal reliability may be stored as a function, or the maximum value of the first light source condition with respect to the normal reliability may be stored discretely as a table. You may leave it.

Subsequently, in step S305, the light source condition acquisition unit 104c acquires the first light source condition based on the range (maximum value) of the first light source condition determined in step S304. When acquiring the virtual light source angle as the first light source condition, the range (maximum value) of the virtual light source angle is determined in step S304, and the virtual light source angle is acquired based on the range. The virtual light source angle may be set within the range of the determined virtual light source angle, or a warning may be displayed when the virtual light source angle outside the range is set. When acquiring the virtual light source intensity as the first light source condition, the range (maximum value) of the virtual light source intensity is determined in step S304, and the virtual light source intensity is acquired based on the range. The virtual light source intensity may be set within the determined virtual light source intensity range, or a warning may be displayed when the virtual light source intensity outside the range is set.

The first light source condition may be a virtual light source angle or a virtual light source intensity specified by the user, or may be a light source condition automatically determined by setting the maximum light source condition that can be set. The first light source condition selected by the user is supplied from the information input unit 108 to the system controller 110, and the light source condition acquisition unit 104c acquires the first light source condition based on the information from the system controller 110.

In this embodiment, the range (maximum value) of the first light source condition was determined based on the normal reliability, and the first light source condition was acquired based on the determined range, which was described in Example 2. The range of the first light source condition may be determined based on such shooting conditions. When determining the first light source condition based on the normal reliability and the imaging condition, the range of the first light source condition may be reduced as the imaging condition is larger and the normal reliability is lower. Alternatively, a smaller range is adopted or an average range is adopted between the range of the first light source condition determined based on the shooting conditions and the range of the first light source condition determined based on the normal reliability. It may be adopted.

As described above, in this embodiment, the normal reliability acquisition unit 104f acquires the normal reliability, which is the reliability of the normal information. The light source condition acquisition unit 104c determines the maximum value of the first light source condition based on the normal reliability. Preferably, the light source condition acquisition unit determines the maximum value of the first light source condition based on the region determined to have no normal reliability. More preferably, the light source condition acquisition unit reduces the maximum value of the first light source condition as the number of regions determined to have no normal reliability increases. Further, preferably, the light source condition acquisition unit reduces the maximum value of the first light source condition as the normal reliability becomes smaller. Further, preferably, the normal reliability acquisition unit reproduces the input image, which is a plurality of captured images taken at at least three different light source positions, and the appearance under the same light source conditions as at least one of the input images. Obtain the normal reliability based on the image.

According to this embodiment, a high-quality rendered image can be generated by determining the range of the second light source condition based on the first light source condition. Further, according to this embodiment, a high-quality rendered image is generated by determining the range of the first light source condition based on the normal reliability and acquiring the first light source condition based on the determined range. can do.

Next, Example 4 of the present invention will be described with reference to FIG. In Examples 1 to 3, the image pickup apparatus having a built-in light source has been described, but in this embodiment, a processing system including the image pickup apparatus and the light source unit will be described. FIG. 14 is an external view of the processing system 3. The processing system 3 includes an image pickup device 301 that captures an image of the subject 303, and a plurality of light source units 302. The image pickup device 301 of this embodiment has the same configuration as the image pickup device 1 of Example 1, but it is not necessary to incorporate a plurality of light sources for the illuminance difference stereo method as a light source unit.

It is preferable that the light source unit 302 is connected to the image pickup device 301 by wire or wirelessly and can be controlled based on the information from the image pickup device 301. Further, in the illuminance difference stereo method, an image captured by sequentially irradiating at least three light sources is required, but when a light source unit configured so that the light source can be moved is used, at least one light source unit should be provided. Just do it. However, it is necessary to move the light sources and perform imaging at at least three different light source positions. If the light source unit 302 cannot automatically change the light source position or if the light source unit 302 cannot be controlled by the image pickup device 301, the light source unit 302 is provided to the user so as to be located at the light source position displayed on the display unit of the image pickup device 301. It may be adjusted. Since the normal calculation process of this embodiment is the same as the process of Examples 1 to 3, detailed description thereof will be omitted.

In this embodiment, a high-quality rendered image can be generated by determining the range (maximum value) of the second light source condition based on the first light source condition.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

According to each embodiment, it is possible to provide a processing device, a processing system, an imaging device, a processing method, and a program capable of generating a high-quality rendered image.

Although preferable examples of the present invention have been described above, the present invention is not limited to these examples, and various modifications and modifications can be made within the scope of the gist thereof. For example, the range of the second light source condition or the like is a range between the maximum value and the minimum value, but may be a case where the maximum value and the minimum value are equal (have only one value).

104 Image processing unit (processing device)
104b Normal information acquisition unit 104c Light source condition acquisition unit 104d Rendering unit

Claims (25)

The normal information acquisition unit that acquires the normal information of the subject,
Obtain either the virtual light source angle or the virtual light source intensity as the first light source condition at the time of rendering, and based on the first light source condition, the virtual light source angle or the above as the maximum value of the second light source condition at the time of rendering. A light source condition acquisition unit that determines the other maximum value of the virtual light source intensity and acquires the second light source condition equal to or less than the maximum value.
It has a rendering unit that generates a rendered image based on the first light source condition and the second light source condition.
The light source condition acquisition unit is a processing device characterized in that the larger the first light source condition is, the smaller the maximum value of the second light source condition is. It also has an input image acquisition unit that acquires a plurality of captured images taken at at least three different light source positions as input images.
The processing device according to claim 1, wherein the normal information acquisition unit acquires the normal information of the subject using the input image. The processing apparatus according to claim 1 or 2, wherein the first light source condition is the virtual light source angle at the time of rendering. The processing apparatus according to claim 3, wherein the virtual light source angle is an angle formed by an optical axis or a line-of-sight vector and a virtual light source vector at the time of rendering. The processing apparatus according to claim 3, wherein the virtual light source angle is an angle formed by a light source vector when an input image is taken and a virtual light source vector when rendering. The processing apparatus according to claim 5, wherein the virtual light source angle is the minimum value of an angle formed by a plurality of light source vectors when an input image is taken and a virtual light source vector at the time of rendering. The processing apparatus according to claim 1 or 2, wherein the first light source condition is a virtual light source intensity at the time of rendering. The processing apparatus according to claim 7, wherein the virtual light source intensity is determined based on the ratio of the base image in the rendered image. The processing apparatus according to claim 8, wherein the base image is an image based on an input image which is a plurality of captured images captured at at least three different light source positions. The processing apparatus according to claim 8, wherein the base image is an ambient light image. It has a shooting condition acquisition unit that acquires the shooting conditions when the input image is shot.
The processing apparatus according to any one of claims 1 to 10, wherein the light source condition acquisition unit determines the maximum value of the first light source condition based on the photographing condition. The processing device according to claim 11, wherein the shooting condition is a subject distance when the input image is shot. The processing apparatus according to claim 12, wherein the light source condition acquisition unit reduces the maximum value of the first light source condition as the subject distance increases. The claim is characterized in that the shooting condition is at least one of a noise amount in the input image, an ISO sensitivity when the input image is shot, and a light emission variation when the input image is shot. 11. The processing apparatus according to 11. The processing device according to claim 11, wherein the light source condition acquisition unit determines the maximum value of the first light source condition for each region of the rendered image. It has a normal reliability acquisition unit that acquires the normal reliability, which is the reliability of the normal information.
The processing apparatus according to any one of claims 1 to 15, wherein the light source condition acquisition unit determines a maximum value of the first light source condition based on the normal reliability. The processing apparatus according to claim 16, wherein the light source condition acquisition unit determines the maximum value of the first light source condition based on a region determined to have no normal reliability. The processing apparatus according to claim 17, wherein the light source condition acquisition unit reduces the maximum value of the first light source condition as the number of regions determined to have no normal reliability increases. The processing apparatus according to claim 16, wherein the light source condition acquisition unit reduces the maximum value of the first light source condition as the normal reliability becomes smaller. The normal reliability acquisition unit includes an input image which is a plurality of captured images taken at at least three different light source positions, and a reproduced rendered image which reproduces the appearance under the same light source conditions as at least one of the input images. The processing apparatus according to any one of claims 16 to 19, wherein the normal reliability is acquired based on the above. Light source and
A processing system comprising the processing apparatus according to any one of claims 1 to 20. The processing system according to claim 21, wherein the light source unit includes three or more light sources having different positions. An imaging unit that captures the subject and
An imaging device comprising the processing device according to any one of claims 1 to 20. The normal information acquisition step to acquire the normal information of the subject, and
Obtain either the virtual light source angle or the virtual light source intensity as the first light source condition at the time of rendering, and based on the first light source condition, the virtual light source angle or the above as the maximum value of the second light source condition at the time of rendering. A light source condition acquisition step of determining the other maximum value of the virtual light source intensity and acquiring the second light source condition equal to or less than the maximum value.
It has an image generation step of generating a rendered image based on the first light source condition and the second light source condition.
In the light source condition acquisition step, the processing method is characterized in that the larger the first light source condition is, the smaller the maximum value of the second light source condition is. A program comprising causing a computer to execute the processing method according to claim 24.