JP7494031B2 - IMAGE PROCESSING APPARATUS, IMAGING APPARATUS, AND IMAGE PROCESSING METHOD - Google Patents

Description

The present invention relates to image processing technology for multi-color images that contain polarization information.

It is known that by observing the polarization state of light from a subject, it is possible to highlight and detect certain characteristics of the subject. For example, by changing the polarization direction of light passing through a polarizing filter attached to the front of the lens of a single-lens reflex camera, it is possible to obtain effects that highlight the texture of the subject, such as its color and contrast, and to emphasize or reduce the appearance of reflected light from the surface of water, etc.

Patent Document 1 discloses a method for calculating polarization information from multiple input images generated by imaging with different polarization states, and reducing coloring that occurs in the generated output image when generating an output image from the polarization information. Specifically, the coloring of the composite image is corrected based on color information of a reference image selected from the multiple input images.

JP 2017-191986 A

However, the method disclosed in Patent Document 1 has the problem that if the color information of the reference image is not acquired correctly, the color correction of the output image is not performed correctly.

The present invention provides an image processing device and method capable of reducing coloring in an output image when generating an output image that differs from an input image.

The image processing device according to one aspect of the present invention includes: an acquisition unit that acquires a plurality of input images obtained by imaging using light having a plurality of different polarization states, acquires first angle information for each first pixel based on a first luminance value of each of a plurality of first pixels corresponding to a first wavelength in each of the plurality of input images, and acquires second angle information for each second pixel based on a second luminance value of each of a plurality of second pixels corresponding to a second wavelength in a specific region in each of the plurality of input images ; a correction unit that acquires corrected polarization information obtained by replacing the first angle information of at least one pixel in the specific region among the plurality of first pixels with second angle information ; and a generation unit that generates an output image using the corrected polarization information . The first angle information indicates a first polarization angle that maximizes the first luminance value, and the second angle information indicates a second polarization angle that maximizes the second luminance value. Note that an imaging device including the above image processing device also constitutes another aspect of the present invention.

An image processing method according to another aspect of the present invention includes the steps of: acquiring a plurality of input images obtained by imaging using light having a plurality of different polarization states; acquiring first angle information for each first pixel based on a first luminance value of each of a plurality of first pixels corresponding to a first wavelength in each of the plurality of input images; acquiring second angle information for each second pixel based on a second luminance value of each of a plurality of second pixels corresponding to a second wavelength in a specific region in each of the plurality of input images; acquiring corrected polarization information obtained by replacing the first angle information of at least one pixel in the specific region among the plurality of first pixels with second angle information ; and generating an output image using the corrected polarization information . The first angle information indicates a first polarization angle that maximizes the first luminance value, and the second angle information indicates a second polarization angle that maximizes the second luminance value . Note that a program for causing a computer to execute a process according to the above image processing method also constitutes another aspect of the present invention.

According to the present invention, when generating an output image that differs from an input image, coloring in the output image can be reduced.

1 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention; FIG. 4 is a diagram showing the relationship between pixel luminance and polarization information. 11 is a flowchart showing image processing in the first embodiment (and the second embodiment). FIG. 4 is a diagram showing a captured image in the first embodiment. FIG. 4 is a diagram showing a histogram of the degree of polarization in Example 1. FIG. 4 is a diagram showing a threshold region in the first embodiment. FIG. 4 is a diagram showing a composite image in the first embodiment. FIG. 4 is a diagram showing extraction regions in Example 1 and a comparative example. FIG. 2 is a diagram showing the configuration of a polarizing element in the first embodiment. FIG. 11 is a diagram showing the configuration of an image processing system according to a third embodiment of the present invention. FIG. 11 is a diagram showing the configuration of an imaging apparatus according to a third embodiment.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 shows the configuration of an image processing device 101 according to an embodiment of the present invention. The image processing device 101 has a polarization calculation unit (acquisition means) 11, a polarization correction unit (correction means) 12, and an image synthesis unit (generation means) 13.

The polarization calculation unit 11 acquires an RGB image (hereinafter, referred to as a captured image) as an input image generated by capturing light of multiple colors (R, G, and B in this embodiment) in different polarization states using an imaging device described later in Example 3. At this time, one or multiple images are acquired as the captured image. The polarization calculation unit 11 also calculates first polarization information for each of R, G, and B at each pixel from the luminance information of the acquired captured image.

The polarization correction unit 12 calculates second polarization information using only information on a specific color among R, G, and B in a specific region of the captured image, and generates corrected polarization information by replacing the first polarization information calculated by the polarization calculation unit 11 with this calculated second polarization information. The polarization synthesis unit 13 generates (synthesizes) a synthetic image as an output image using at least the corrected polarization information generated by the polarization correction unit 12.

The following describes in detail the first polarization information acquired by the polarization calculation unit 11 and the generation of the corrected polarization information by the polarization correction unit 12.

When an image is acquired by imaging through an element (e.g., a polarizing plate) that transmits only linearly polarized light having a specific polarization direction, the angle α of the transmitted linearly polarized light (hereinafter referred to as the polarization angle) and the luminance value (luminance information) I _j (α) of each pixel of the acquired image have the relationship shown in the following equation (1).
_Ij (α)= _Ajcos2 ⁽ α- _θj )+ _Bj (1)
FIG. 2 shows the luminance value I _j (α) for a certain color channel at a certain pixel with respect to the polarization angle α. Here, j indicates one of the colors R, G, and B, and for example, I _R indicates the luminance value at a certain pixel of R. A _j indicates the luminance component that changes according to the polarization angle α (hereinafter, referred to as the polarized component), and B _j indicates the luminance component that does not change according to the polarization angle α (hereinafter, referred to as the non-polarized component). Furthermore, θ _j indicates the polarization angle (hereinafter, referred to as the angle component) at which the luminance value I (α) is maximum. A _j , B _j , and θ _j are constants calculated for each pixel and for each color. The angle component θ _j corresponds to the angle information.

By acquiring images while changing the polarization angle α to three or more different angles, A _j , B _j and θ _j in equation (1) can be obtained, thereby obtaining the relationship between the polarization angle α and the luminance value I _j (α) for each color channel for each pixel.

Here, the angle component _θj does not usually change depending on the color. This is because the polarization component _Aj is mainly derived from the specular reflection component. Since the specular reflection component follows Fresnel's law, the reflectance of s-polarized light is generally higher than that of p-polarized light. Therefore, the direction of s-polarized light and the direction of the angle component _θj match. The direction of s-polarized light is determined by the plane of incidence and the plane of emergence, and does not depend on the wavelength (color).

However, due to some error, the angle component _θj may be calculated differently depending on the color. In particular, when the polarization angle α is controlled using a polarizing element including a variable retardation plate as disclosed in Patent Document 1, the influence of the wavelength (color) becomes significant because of the influence of the angle characteristics of the variable retardation plate and the wavelength dependency of the retardation.

Figure 9(a) shows the configuration of the polarizing element 200 disclosed in Patent Document 1, and Figures 9(b) to (d) respectively show the λ/4 plate 21, the variable retardation plate 22 made of liquid crystal, and the polarizing plate 23, which are elements that make up the polarizing element 200. The arrows in Figures 9(b) to (d) indicate any of the axial directions of the fast axis, the slow axis, and the transmission axis. The z-axis indicates the optical axis direction during imaging, and the x-axis and y-axis indicate axes that are perpendicular to each other and lie in a plane perpendicular to the optical axis (z-axis). The angles that the axial directions of the λ/4 plate 21, the variable retardation plate 22, and the polarizing plate 23 make with respect to the x-axis are 90°, 45°, and 90°, respectively.

The λ/4 plate 21 imparts a relative phase difference of π/2 (rad) between the orthogonal polarization components of the incident light. The relative phase difference of π/2 imparted by the λ/4 plate 21 is invariant (fixed). The variable retardation plate 22 is an element that uses liquid crystal, and imparts a relative phase difference that can be changed depending on the voltage applied between the orthogonal polarization components. The polarizing plate 23 transmits the polarization component in the direction of its transmission axis.

When RGB light passes through the polarizing element 200 having such a configuration, the peak positions of the polarization angle component cos ² (α _c - θ _j ) at the specific polarization angle α _c shift between R, G, and B, and an unnatural coloring (unwanted color) that does not occur in a normal image occurs in a composite image generated using the polarization angle component cos ² (α _c - θ _j ) at the specific polarization angle α _c . Moreover, since this coloring occurs due to the shift in the peak positions of the polarization angle component cos ² (α _c - θ _j ) between R, G, and B, the appearance of the color changes depending on the value of the specific polarization angle α _c used when generating the composite image.

In the embodiment, in order to reduce the coloring that changes depending on the value of this specific polarization angle _αc , the polarization correction unit 12 generates a correction angle component (information) as corrected polarization information, and the polarization synthesis unit 13 generates a synthetic image using the correction angle component.

The correction angle component is generated by replacing the angle component (first or third angle information) for each color channel calculated for each pixel by the polarization calculation unit 11 in a specific region of the captured image with a separately calculated angle component (second angle information). The separately calculated angle component is an angle component calculated using only information on a specific color among R, G, and B. For example, when G is selected as the specific color, the θ _G information of all pixels is calculated using the luminance value of G in the specific region and formula (1), and the θ _R , θ _G , and θ _B calculated by the polarization calculation unit 11 are replaced with this θ _G. This process makes θ _R , θ _B , and θ _G equal to each other, and the coloring that changes depending on the value of α _c described above can be reduced.

In addition, if the polarization calculation unit 11 has calculated θR, θG, and θB in advance using only the luminance values of _R , _G , and _B , it is only necessary to perform the process of replacing _θR and _θB with _θG for each pixel in the specific region. On the other hand, if the luminance distribution of R and G is also taken into consideration when calculating _θG , the polarization correction unit 12 again calculates _θG " taking into consideration only the luminance value of G, and performs the process of replacing _θR , _θG , and _θB with this _θG ".

The specific color used to calculate the correction angle component can be selected from R, G, and B. For example, when using a captured image captured using an image sensor with a Bayer array, G, which has a greater number of pixels than other colors, can be selected as the specific color.

In addition, the specific color may be selected according to the polarization information and color information of the subject image in the specific region. For example, the accuracy of the polarization information tends to be higher when the degree of polarization (DOP) defined by the following formula (2) is high than when the degree of polarization is low.
_DOPj = ( _Ijmax - _Ijmin ) / ( _Ijmax + _Ijmin )
= Aj _/ ( _Aj + 2Bj ₎ (2)
In equation (2), I _jmax and I _jmin respectively represent the maximum and minimum values of the luminance value I _j (α) when the polarization angle α is changed in any one of the R, G, and B color channels.

By calculating the degree of polarization within a specific region from equation (2) and the polarization information calculated by the polarization calculation unit 11, and selecting the color with the highest degree of polarization among R, G, and B as the specific color, it is possible to obtain a correction angle component with higher accuracy.

Another method is to compare the luminance values of R, G, and B of the subject (the area in which the subject is shown) in a specific area and select the color with the lowest luminance value (for example, B for a yellow subject) as the specific color. Since the degree of polarization increases as the diffuse component decreases, the luminance values of R, G, and B of the subject in the captured image tend to be inversely proportional to the degree of polarization of R, G, and B. Therefore, with this method as well, it is possible to indirectly select a color with a high degree of polarization as the specific color.

So far, the case where one of R, G, and B is selected as the specific color has been described, but two colors (for example, B and G, or R and B, etc.) may be selected. In that case, too, the two colors may be selected taking into consideration the degree of polarization, the color of the subject, the number of pixels, etc., as described above. In addition, in the above description, the correction angle components are generated by replacing θ _B , θ _G , and θ _R with the angle components of a specific color, but the correction angle components may also be generated by replacing α _c - θ _j or cos ² (α _c - θ _j ).

By performing the above-mentioned angle component replacement process only within the specific region, the amount of calculation can be reduced compared to processing the entire image. For this reason, it is preferable to select an area where angle component replacement process is required as the specific region, that is, an area where coloring is likely to occur. For example, it is preferable to select an area where the degree of polarization is greater than a predetermined value (threshold value) or an area where the difference in angle component θ for R, G, and B is greater than a threshold value.

The number of specific regions is not limited to one, and multiple specific regions may be selected from the entire image. When multiple specific regions are selected, any classification method or technique may be used to select or separate specific regions, such as specifying a specific region for each specific color used to calculate the above-mentioned correction angle component, separating specific regions for each subject, or separating specific regions for the background and main subject. Alternatively, the entire image may be divided into multiple regions in advance according to the subject, color, degree of polarization, etc., and a specific region may be selected from the multiple divided regions.

The flowchart in FIG. 3 shows Example 1 as a process (image processing method) executed by the image processing device 101 shown in FIG. 1. The image processing device 101 is configured by a computer, and executes this process according to a computer program.

In step S101, the image processing device 101 reads in polarization image data as a captured image generated by capturing images in different polarization states for R, G, and B. The polarization image data to be read is a color image having luminance information for each of R, G, and B, and is a RAW image that has not been subjected to compression processing or gamma processing. In the embodiment, an image capturing device equipped with an image sensor having a Bayer array captures images using a polarizing element 200 shown in FIGS. 9(a) to 9(d). The polarizing element 200 can modulate the polarization angle α, which is the polarization direction of the transmitted light, by changing the in-plane phase difference Δ that the variable retardation plate 21 imparts to the transmitted light. The relationship between the polarization angle α (deg) of the transmitted light and the in-plane phase difference Δ (nm) is expressed by the following formula (3).
α = Θ - 180 Δ / λ (3)
In formula (3), Θ is the transmission axis angle (deg) of the polarizing plate 23 disposed in the polarizing element 200, and λ is the wavelength (nm) of the transmitted light. From formula (3), the captured image obtained by imaging through the polarizing element 200 using the variable retarder 21 has a polarization angle α that differs depending on the wavelength λ.

Figures 4(a) to (c) show examples of three captured images (polarized image data) acquired in this embodiment. The captured images in Figures 4(a) to (c) were obtained by imaging with the transmission axis angle Θ of the polarizing plate 23 set to 90° and the in-plane phase differences Δ set to 10 nm, 133 nm, and 265 nm, respectively. The polarization angles α (deg) of the R, G, and B color channels at this time are shown in Table 1.

In Table 1, the wavelengths used to calculate the transmission angles of the R, G, and B color channels are 600 nm, 530 nm, and 470 nm, respectively, and the wavelengths at which the amount of transmitted light through the color filters arranged in the R, G, and B pixels is maximized were selected.

Next, in step S102, the image processing device 101 (polarization calculation unit 11) plots the luminance value of each pixel against the polarization angle α from the luminance value data of the three captured images shown in FIGS. 4(a) to 4(c), and calculates the polarization information A _j , B _j , and θ _j of each pixel. Specifically, since the luminance value of any of R, G, and B is acquired for each pixel of the captured image, the value of the transmission angle α in Table 1 is selected according to the R, G, or B of the pixel, and the polarization information A _j , B _j , and θ _j of any of R, G, and B is calculated according to the pixel. Then, the captured image is subjected to demosaic processing to obtain the polarization information A _j , B _j , and θ _j for R, G, and B of all pixels. Note that a general method may be used for the demosaic processing. However, in this embodiment, in consideration of the subsequent processing, the demosaic processing is performed using only the luminance value of each color for each color channel.

Here, the polarization information is acquired and then demosaic processing is performed to obtain the R, G, and B polarization information for all pixels, but it is also possible to perform demosaic processing on the luminance values of the captured image and then calculate the polarization information for each pixel and each color. However, it is preferable to calculate the polarization information before performing demosaic processing, as this reduces the amount of calculation required.

Furthermore, before calculating polarization information from the three captured images, it is preferable to carry out a subject alignment process as necessary.
After calculating the polarization information _Aj , Bj, _{and θj} for R, G, and B of all pixels in this way, in step S103, the image processing device 101 (polarization correction unit 12) selects a specific region from the entire region of the captured image. As described above, the specific region is selected by comparing the degree of polarization or the difference in _θj due to RGB with a threshold value, and a region where the degree of polarization is greater than the threshold value or a region where the difference in _θj is greater than the threshold value is selected. In this embodiment, a case where a region where the degree of polarization is greater than the threshold value is selected as the specific region will be described. Also, in this embodiment, a case where the threshold value is determined using a histogram of the degree of polarization will be described.

Figures 5(a) and (b) show histograms of the degree of polarization used to determine the threshold. Figure 5(a) shows a histogram for degrees of polarization from 0 to 1, and Figure 5(b) shows a histogram for degrees of polarization from 0.05 to 1. From Figures 5(a) and (b), it can be estimated that areas with a degree of polarization less than 0.1 are background images, and areas with a degree of polarization around 0.6 are areas where the degree of polarization of the subject is at its peak. Therefore, in this embodiment, in order to eliminate background areas and sufficiently correct colored areas of the subject, a degree of polarization of 0.45, which is around 0.6, is determined as the threshold, and areas with a degree of polarization of 0.45 or more are selected as specific areas.

Figure 6 shows a specific area selected from a captured image using the above method. In Figure 6, areas with a polarization degree of 0.45 or more are shown in white, and areas with a polarization degree of less than 0.45 are shown in black. Comparing Figure 6 with Figures 4(a) to (c), it can be seen that by selecting an area with a polarization degree of 0.45 or more as a specific area, it is possible to select an area of the subject where the brightness value changes depending on the polarization state.

The threshold value for the degree of polarization is not limited to the above-mentioned method using the histogram of the degree of polarization, and any threshold value may be used. For example, although it varies depending on the captured image, it is usually preferable to set it to 0.1 or more, and more preferably to set it to 0.3 or more. In addition, the same process as the case of using the degree of polarization can be used to select a specific region where the difference in _θj is larger than the threshold value.

Next, in step S104, the image processing device 101 (polarization correction unit 12) selects a specific color to be used to generate corrected polarization information. In this embodiment, the subject is assumed to be almost achromatic, and G, which has a large number of pixels, is selected as the specific color.

When selecting a specific color using the hue of a captured image, it is preferable to determine the hue of the captured image using the image in which the absolute value of the phase difference of the variable retardation plate is the smallest (the captured image shown in FIG. 4(a) in this embodiment). This is because as the absolute value of the phase difference of the variable retardation plate increases, the wavelength dispersion of the polarization angle α increases, and as a result, coloring due to the wavelength dispersion of the polarization angle _αj occurs in the captured image.

In step S105, the polarization correction unit 12 that has selected the specific color generates (calculates) corrected polarization information using information on the specific color G. In this embodiment, the polarization information of R, G, and B of each pixel is calculated using only the luminance values of R, G, and B when calculating the polarization information in step S102. Therefore, in this embodiment, the corrected polarization information is generated by replacing θ _R and θ _B of each pixel in the specific region with θ _G.

Next, in step S106, the image processing device 101 (image synthesis unit 13) generates a synthetic image using the corrected polarization information and the polarization information. In this embodiment, the luminance value I _j ' of each pixel of the synthetic image is calculated using the following equation (4).
I _j '(α _c ) = k ₁ A _j cos ² [k ₂ (α _c - _{θ j} ')] + k ₃ B _j (4)
In equation (4), _k1 to _k3 are arbitrary coefficients, _αc is an arbitrary polarization angle, _θj ' is the corrected polarization information (corrected angle component) generated in step S105, and _Aj and _Bj are the polarized component and the non-polarized component calculated in step S102, respectively.

The coefficients _k1 to _k3 and the polarization angle _αc are constant values regardless of R, G, and B or pixels. By setting these coefficients and polarization angles to constant values regardless of color or pixel, the texture imparted to the composite image by the polarization information can be unified within the image. However, the coefficients do not necessarily need to be the same throughout the image, and may be set for each area in which the texture is desired to be unified, such as for each area in the image or for each subject.

In this embodiment, k ₁ = k ₂ = k ₃ = 1 across the entire image, and four composite images are generated with α _c = 0°, 45°, 90°, and 135°. Figures 7(a) to (d) show examples of composite images generated under these conditions. The composite images actually generated are color images, but are shown as black and white images in Figures 7(a) to (d). These figures show that composite images with different gloss positions can be generated by generating composite images with different polarization angles α _c .

Finally, in step S107, the image processing device 101 outputs the generated composite image and completes this process.

In this embodiment, the in-plane retardation Δ is a constant value for R, G, and B, but Δ may be different for R, G, and B. When Δ is different for R, G, and B, the polarization angle α may be calculated using an appropriate Δ for each color channel.
(Comparative Example)
The effect of using the corrective angle component in the first embodiment will be described in comparison with a comparative example in which the corrective angle component is not used. In this comparative example, the angle component _θj is used instead of the corrective angle component θj _' when generating a composite image, and four composite images ( _αc = 0°, 45°, 90°, 135°) similar to those in the first embodiment are generated in the same manner as in the first embodiment except for the above.

Table 2 shows the results of calculating the L*a*b* representative value of each composite image in order to confirm the coloring depending on the polarization angle _αc in the four composite images generated in the comparative example and the four composite images generated in Example 1 ^{(simply referred to as Example in the table). The L*a*b* representative value of each} composite image is the result of extracting the same partial area (the area within the white frame on the lens cap of the interchangeable lens shown in FIG ^. 8) in the four composite images and calculating the average value of the L ^* a ^* b ^* of the partial area. Table 2 also shows the average and variance of the L ^* a ^* b ^* representative value of each composite image.

Looking at the representative values of L ^* a ^* b ^* for Example 1 and Comparative Example in Table 2, it can be seen that the value of L ^* , which represents lightness, is almost the same in Example 1 and Comparative Example, whereas the values of a ^* and b ^* , which represent chromaticity, are different between Example and Comparative Example, and the variation is greater in Comparative Example than in Example. Also, when comparing the variances of a ^* and b ^* , the Comparative Example has a larger variance than the Example. From this, it can be seen that the change in color between the composite images in Example 1 is smaller than that in Comparative Example, and that the use of the correction angle component has reduced the coloring that changes depending on the value of _αc .

A second embodiment of the present invention will be described below. The second embodiment differs from the first embodiment in that, in addition to the corrected angle component _θj ', the corrected polarized component _Aj ' and the corrected unpolarized component _Bj ' (information) are calculated as the corrected polarization information, and a composite image is generated using _θj ', _Aj ', and _Bj '.

3, steps S101 to S104 are the same as those in the first embodiment. The calculation of the corrected angle component θ _j ′ in step S105 is also the same as that in the first embodiment.

In the second embodiment, in step S105, the image processing device 101 (polarization correction unit 12) further calculates a corrected polarization component A _j ' and a corrected non-polarized component B _j ' using the corrected angle component θ _j '. The corrected polarization component A _j ' and the corrected non-polarized component B _j ' are generated by replacing the polarization component and non-polarized component for each color calculated for each pixel by the polarization calculation unit 11 in a specific region of the captured image with the polarization component and non-polarized component calculated separately. The separately calculated polarization component and non-polarized component are calculated from formula (1) using the corrected angle component θ _j ' and the luminance value data of the three captured images.

Specifically, the corrected angle component _θj ' obtained by replacing _θR and _θB of each pixel in the specific region with _θG is defined as _θj in formula (1), and the polarized component _Aj and non-polarized component _Bj for each pixel are again obtained from the luminance value data of the three captured images. Then, the polarized components _AR , _AG and non-polarized components BR, _BG calculated by the polarization calculation unit 11 are replaced with these polarized components _Aj and non-polarized components _Bj , thereby obtaining the corrected polarized component _{Aj '} and corrected non-polarized component _Bj '. This process makes it possible to correct errors contained in the polarized components and non-polarized components.

In step S1076, the image processing device 101 (image synthesis unit 13) generates a synthetic image using θ _j ', A _j ', and B _j '.

In the first embodiment, a composite image is generated using the corrected angle component and the polarization information (polarized component and non-polarized component) calculated in step S102, but in the second embodiment, a composite image is generated using only the corrected polarization information (corrected polarization component, corrected non-polarized component, and corrected angle component). This makes it possible to reduce coloring resulting from errors between the polarized component and the non-polarized component.

In the above-described first and second embodiments, the polarization calculation unit 11 performs demosaic processing to calculate the polarization information of all pixels and all color channels, but the timing of the demosaic processing can be changed as appropriate. For example, when generating a composite image, the luminance value of each pixel may be calculated using the above-described formula (4) and then the demosaic processing may be performed. In that case, however, when calculating the corrected polarization information, it is necessary to generate the polarization information of each pixel in the specific region from the information of the surrounding pixels.

The polarizing element used for capturing an image to obtain a photographed image is not limited to the polarizing element 200 shown in FIG. 9. For example, various polarizing elements can be used, such as a polarizing plate, a C-PL filter that combines a polarizing plate and a 1/4 λ plate, or a polarizing sensor in which polarizers of different orientations are incorporated in each pixel of the image sensor.

Furthermore, the generation of the composite image is not limited to the method using equation (4). For example, the composite image may be generated by calculating the luminance value I _j ' of each pixel using a plurality of polarization angles α _c1 to α _cn as shown in the following equation (5).
I _j ′=Σk _1n A _j cos ² [k _2n (α _cn −θ _j )]+k ₃ B _j (5)
In formula (5), k _1n , k _2n , and k ₃ are arbitrary coefficients, and α _cn is an arbitrary polarization angle. As with formula (3), these coefficients and polarization angles use constant values regardless of R, G, and B or pixels. Also, a composite image can be generated using only a part of formula (4) or formula (5).

In addition, general image processing such as noise reduction, contrast correction, and white balance correction may be performed on the captured image, the composite image, the polarization information, and the corrected polarization information.

Furthermore, the captured image does not necessarily have to be a RAW image, and a compressed image such as a JPEG image can also be used. However, when using a captured image in which gamma correction has been performed on the luminance values, it is preferable to perform inverse correction before calculating the polarization information to return the luminance values to linearity.

Figure 10 shows the configuration of an image processing system 100 including the image processing device 101 described above.

The image processing device 101 is a computer device that is equipped with image processing software (an image processing program as a computer program) 106 and executes the image processing described in the first or second embodiment in accordance with the image processing software 106.

The imaging device 102 is a device that captures an image, such as a camera, a telescope, an endoscope, or a scanner. FIG. 11 shows an interchangeable lens camera as an example of the imaging device 102. The interchangeable lens camera is composed of an interchangeable lens 121, an adapter 122, and a digital camera body 123. The adapter 122 is provided inside the polarizing element 200 shown in FIG. 9(a).

The digital camera body 123 is equipped with an image sensor (image sensor) 123a such as a CMOS sensor, and can capture images of light that passes through the interchangeable lens 121 and the adapter 122.

In FIG. 11, a polarizing element (adapter 122) is provided between the interchangeable lens 121 and the digital camera body 123, but the polarizing element may be located closer to the object than the interchangeable lens 121. Also, instead of providing the adapter 122, the polarizing element may be provided integral with or immediately before the image sensor in the digital camera body 123.

The storage medium 103 shown in FIG. 10 stores images captured by imaging, such as a semiconductor memory, a hard disk, or a server on a network.

The image processing device 101 acquires a captured image as an input image from the imaging device 102 or the storage medium 103 by wired or wireless communication with them or by attached reading. Then, an output image is generated by the image processing described in the first or second embodiment, and the output image is output to at least one of the output device 105, the imaging device 102, and the storage medium 103. The output image can also be saved in an internal storage unit built into the image processing device 101. The output device 105 is, for example, a printer.

The image processing device 101 is also connected to a display device 104. Therefore, a user can perform image processing tasks and evaluate the generated output image through the display device 104. In addition to the image processing described in the first or second embodiment, the image processing device 101 may also perform other image processing such as development processing and image restoration processing as necessary.

In the present embodiment, the image processing device 101 is provided separately from the imaging device 102 . However, the image processing device 101 may be built into the imaging device 102 .
Other Examples
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The embodiments described above are merely representative examples, and various modifications and alterations are possible when implementing the present invention.

11 Polarization calculation unit 12 Polarization correction unit 13 Image synthesis unit 101 Image processing device

Claims (13)

an acquisition means for acquiring a plurality of input images obtained by imaging using light having a plurality of different polarization states, acquiring first angle information for each of a plurality of first pixels corresponding to a first wavelength in each of the plurality of input images based on a first luminance value of each of the plurality of first pixels, and acquiring second angle information for each of the plurality of second pixels based on a second luminance value of each of a plurality of second pixels corresponding to a second wavelength in a specific region in each of the plurality of input images;
a correction means for acquiring corrected polarization information obtained by replacing the first angle information of at least one pixel in the specific region among the plurality of first pixels with the second angle information ;
generating means for generating an output image using the corrected polarization information ;
An image processing device characterized in that the first angle information indicates a first polarization angle that maximizes the first luminance value, and the second angle information indicates a second polarization angle that maximizes the second luminance value . the acquiring means acquires third angle information for each of a plurality of third pixels based on a third luminance value of each of a plurality of third pixels corresponding to a third wavelength in each of the plurality of input images;
The image processing device according to claim 1 , characterized in that the correction means acquires the corrected polarization information by replacing the third angle information of at least one pixel within the specific region among the plurality of third pixels with the second angle information . 3. The image processing apparatus according to claim 1 , wherein the first wavelength and the second wavelength each correspond to any two of red, green, and blue. 3. The image processing apparatus according to claim 2 , wherein the first wavelength, the second wavelength and the third wavelength each correspond to any one of red, green and blue. 5 . The image processing apparatus according to claim 1 , wherein the correction means selects, as the specific region, a region in each of the plurality of input images in which the degree of polarization is greater than a predetermined value. 5. The image processing device according to claim 1, wherein the correction means selects, as the specific area, an area in each of the plurality of input images where a difference between the first polarization angle and the second polarization angle is greater than a predetermined value. The correction means divides each of the plurality of input images into a plurality of regions based on wavelength information,
7. The image processing apparatus according to claim 1, wherein the specific area is selected from the plurality of areas. 8. The image processing apparatus according to claim 1, wherein the correction means selects the second wavelength in accordance with polarization information or wavelength information in the specific region. An image processing device according to any one of claims 1 to 8;
and an image sensor for acquiring the input image. The imaging device according to claim 9, characterized in that it has a polarizing element that controls the polarization state of the transmitted light. The imaging device according to claim 10, characterized in that the polarizing element includes a variable retardation plate made of liquid crystal and a polarizing plate. acquiring a plurality of input images obtained by imaging with light having a plurality of different polarization states;
obtaining first angle information for each of a plurality of first pixels based on a first luminance value of each of the plurality of input images, the first luminance value corresponding to a first wavelength;
acquiring second angle information for each of a plurality of second pixels based on second luminance values of the second pixels corresponding to a second wavelength within a specific region in each of the plurality of input images;
acquiring corrected polarization information obtained by replacing the first angle information of at least one pixel in the specific region among the plurality of first pixels with the second angle information ;
and generating an output image using the corrected polarization information ;
An image processing method, characterized in that the first angle information indicates a first polarization angle that maximizes the first brightness value, and the second angle information indicates a second polarization angle that maximizes the second brightness value . A program for causing a computer to execute processing according to the image processing method described in claim 12.