WO2022054167A1

WO2022054167A1 - Position estimation method, position estimation device, and program

Info

Publication number: WO2022054167A1
Application number: PCT/JP2020/034113
Authority: WO
Inventors: 志織杉本; 陽光曽我部; 隆行黒住; 英明木全
Original assignee: 日本電信電話株式会社
Priority date: 2020-09-09
Filing date: 2020-09-09
Publication date: 2022-03-17
Also published as: JPWO2022054167A1; JP7445176B2

Abstract

This position estimation method is performed by a position estimation device and comprises: an image capture step for, in accordance with light reflected from a subject irradiated with polarized light having a reference polarization angle, generating both a first observed image, which is a polarized image of a channel corresponding to a first polarization angle that is inclined from a reference in a negative direction and is not orthogonal to the reference, and a second observed image, which is a polarized image of a channel corresponding to a second polarization angle that is inclined from the reference in a positive direction and is not orthogonal to the reference; and a position estimation step for estimating the three-dimensional position of the subject on the basis of the first observed image and the second observed image.

Description

Position estimation method, position estimation device and program

The present invention relates to a position estimation method, a position estimation device and a program.

As a method for estimating the three-dimensional position (depth) of the subject and the reflection parameter on the surface of the subject, for example, there are a stereo method, a pattern projection method, and a TOF (Time of Flight). In the stereo method, a compound eye camera or a plurality of cameras are used. In the stereo method, the corresponding points between the multi-viewpoint images having parallax are obtained, and the three-dimensional position is estimated by triangulation. In the pattern projection method, an illumination that projects a pattern light and a camera are used. In the TOF, a pulse wave or a modulated wave is applied to the subject, and the reflection time of the pulse wave or the modulated wave is measured.

Further, in the field of computational photography, a method called TCA (three color-filtered apertures) is used to estimate the three-dimensional position of the subject based on the image of the subject captured by the monocular camera. (See Non-Patent Document 1).

In TCA, red, green, and blue color filters (sub-openings) are placed in the vicinity of the lens of a camera equipped with an RGB (Red, Green, Blue) image sensor, with their positions offset from each other. When such a camera captures a subject, a multi-viewpoint image having a minute parallax between the red, green, and blue color channels is generated. The three-dimensional position (depth) of the subject is estimated by performing a corresponding point search similar to the stereo method for the multi-viewpoint image between each color channel.

In the field of computational optics, there is also a method called DCA (Dual off-axis color-filtered aperture). In the DCA, the sub-openings are arranged away from the optical axis so that the estimation accuracy of the three-dimensional position of the subject is improved. Further, in DCA, the number of sub-apertures is reduced to two so that the amount of light per sub-aperture increases (see Non-Patent Document 2).

In both TCA and DCA, it is necessary to execute a corresponding point search in consideration of the color shift between the color channels. Here, when the color of the subject is not known, it is difficult for the position estimation device to accurately estimate the color shift, so that there is a problem that an error in searching for the corresponding point occurs.

A red filter and a cyan filter may be prepared to prevent an error in the search for the corresponding point. In the camera, the optical image of the subject is captured in both the green and blue channels via a cyan filter. By using each optical image acquired in both the green channel and the blue channel for estimating the three-dimensional position, the estimation accuracy of the three-dimensional position of the subject is improved (see Non-Patent Document 3).

However, in these methods, the brightness of each optical image acquired in both the green channel and the blue channel is lower than that of the optical image acquired in the red channel because it passes through the cyan filter. .. Therefore, accurate color reproduction becomes difficult. Further, in these methods, only diffuse reflection is assumed as the reflection characteristic of the surface of the subject. Therefore, these methods are not suitable as a method for estimating the three-dimensional position of the surface of a subject having a specular reflection characteristic (a subject whose brightness varies greatly depending on the light reflection direction).

In recent years, there is a polarized image sensor equipped with a four-direction polarizing filter instead of the color filter of the RGB image sensor. When the subject is observed using a camera equipped with such a polarization image sensor and a sub-aperture and the polarization analysis is performed using TCA or DCA in order for the position estimation device to estimate the three-dimensional position of the surface of the subject. There is. Even in this case, it is difficult to estimate the three-dimensional position of the surface of the subject because the brightness of the surface of the subject having the specular reflection characteristic changes greatly for each polarization filter.

Ellipsometry is often used as a method for estimating the reflection parameters on the surface of a subject having specular reflection characteristics. In this method, the subject is irradiated with light from a plurality of lights, and ellipsometry is performed for each light. As a result, the reflection parameter on the surface of the subject is estimated based on the relationship between the incident angle and the reflection angle of the light.

However, the method in which ellipsometry is performed cannot estimate the three-dimensional position of the surface of the subject. Therefore, the method in which the ellipsometry is performed is often combined with a method such as a stereo method, a pattern method, or a TOF method in order to estimate the three-dimensional position of the surface of the subject.

When the 3D position (depth) of the surface of the subject is estimated using the sub-aperture, the estimation accuracy of the 3D position of the surface of the subject may decrease due to the estimation result of the color shift. Further, when the three-dimensional position of the subject is estimated using the sub-aperture, the three-dimensional position of the surface of the subject having the specular reflection characteristic cannot be estimated. Further, when the polarization analysis is performed in TCA or DCA, the brightness changes greatly for each polarization filter, so that it is not possible to estimate the three-dimensional position of the surface of the subject having specular reflection characteristics. As described above, there is a problem that the three-dimensional position of the subject may not be estimated.

In view of the above circumstances, the present invention provides a position estimation method, a position estimation device, and a program capable of suppressing a decrease in accuracy due to the influence of the subject or the environment in estimating a three-dimensional position of a subject in real space. It is an object.

One aspect of the present invention is a position estimation method executed by a position estimation device, which is a polarized image of a channel corresponding to a first polarization angle, which is a polarization angle that is tilted from a reference in the negative direction and is not orthogonal to the reference. A certain first observed image and a second observed image which is a polarized image of a channel corresponding to a second polarization angle which is inclined in the positive direction from the reference and is not orthogonal to the reference are polarized with respect to the reference. An image imaging step generated according to the light reflected from the subject irradiated with angular polarization, and a position estimation step for estimating the three-dimensional position of the subject based on the first observation image and the second observation image. It is a position estimation method including and.

One aspect of the present invention is a first observed image which is a polarized image of a channel corresponding to a first polarization angle which is a polarization angle which is inclined from a reference in a negative direction and is not orthogonal to the reference, and a positive direction from the reference. The light reflected from the subject irradiated with the polarization of the reference polarization angle with the second observation image which is the polarized image of the channel corresponding to the second polarization angle which is inclined and is not orthogonal to the reference. It is a position estimation device including an image imaging unit generated according to the above and a position estimation unit that estimates a three-dimensional position of the subject based on the first observation image and the second observation image.

One aspect of the present invention is a program for operating a computer as the above-mentioned position estimation device.

According to the present invention, it is possible to suppress a decrease in accuracy due to the influence of the subject or the environment in estimating the three-dimensional position of the subject in the real space.

It is a figure which shows the structural example of the position estimation apparatus in 1st Embodiment. It is a figure which shows the structural example of the image imaging unit in 1st Embodiment. It is a figure which shows the structural example of the image processing part in 1st Embodiment. It is a flowchart which shows the operation example of the position estimation apparatus in 1st Embodiment. It is a figure which shows the relationship between the polarization angle and the luminance in 1st Embodiment. It is a figure which shows the structural example of the position estimation apparatus in 2nd Embodiment. It is a figure which shows the structural example of the image processing part in 2nd Embodiment. It is a flowchart which shows the operation example of the position estimation apparatus in 2nd Embodiment. It is a figure which shows the hardware configuration example of the position estimation apparatus common to each embodiment.

Embodiments of the present invention will be described in detail with reference to the drawings.
(First Embodiment)
FIG. 1 is a diagram showing a configuration example of the position estimation device 1a in the first embodiment. The position estimation device 1a is a device that estimates the three-dimensional position (depth) of the surface of the subject. In FIG. 1, the position estimation device 1a estimates a three-dimensional position on the surface of the subject 100.

The position estimation device 1a includes a polarized lighting unit 10, an image imaging unit 11, and an image processing unit 12a. The polarized illumination unit 10 irradiates the subject 100 (imaging target) with polarization. The image capturing unit 11 (polarized camera) generates an image of the subject 100 by capturing the subject 100. The image capturing unit 11 outputs the image of the subject 100 to the image processing unit 12a.

The image processing unit 12a acquires an image of the subject 100 from the image imaging unit 11. The image processing unit 12a estimates the three-dimensional position (depth) of the surface of the subject 100 based on the image acquired from the image capturing unit 11.

FIG. 2 is a diagram showing a configuration example of the image capturing unit 11 in the first embodiment. The image capturing unit 11 includes a lens 110, a mask 111, and a polarized image sensor 112. The mask 111 is provided with a first opening region 113 and a second opening region 114 as each sub-opening.

The first aperture region 113 is provided with a polarization filter having a first angle with respect to the deflection angle of the polarization transmitted through the lens 110. The same applies to the second aperture region 114, and the second aperture region 114 is provided with a polarization filter having a second angle with respect to the deflection angle of the polarization transmitted through the lens 110.

In FIG. 2, “Δc _x ^eff ” represents the effective aperture of the lens 110. “Δc _x ” represents the distance between the first opening region 113 and the second opening region 114. “ _Cz ” represents the distance between the lens 110 and the mask 111. “F” represents the distance between the lens 110 and the polarized image sensor 112. “Δx” represents an image spacing. “Z” represents the distance between the polarized image sensor 112 and the subject 100. “Z ₀ ” represents the distance between the polarized image sensor 112 and the focal surface 200.

Reflected light is incident on the lens 110 from the surface of the subject 100 irradiated with polarized light. The lens 110 transmits the incident light and sends the light transmitted through the lens 110 to the mask 111. The first opening region 113 and the second opening region 114 of the mask 111 transmit a part of the light sent to the mask 111. The first aperture region 113 and the second aperture region 114 project the transmitted light onto the polarized image sensor 112.

The light projected on the polarized image sensor 112 is observed as a polarized image of the subject 100. The polarized image sensor 112 generates an image of the subject 100 according to the polarized image of the subject 100. The polarized image sensor 112 transmits an image of the subject 100 to the image processing unit 12a.

Of the part of the light transmitted through the lens 110 in FIG. 2, the light that is not blocked by any of the mask 111, the first aperture region 113, and the second aperture region 114 is projected onto the polarization image sensor 112. You may.

FIG. 3 is a diagram showing a configuration example of the image processing unit 12a in the first embodiment. The image processing unit 12a includes an image input unit 120a and a position estimation unit 121a.

The image input unit 120a acquires an image of the subject 100 from the image imaging unit 11. The image input unit 120a outputs the image of the subject 100 to the position estimation unit 121a.

The position estimation unit 121a estimates the three-dimensional position (depth) of the surface of the subject 100 based on the image of the subject 100. The position estimation unit 121a outputs the estimation result of the three-dimensional position of the surface of the subject 100 to a predetermined external device (not shown).

Next, an operation example of the position estimation device 1a will be described.
FIG. 4 is a flowchart showing an operation example of the position estimation device 1a in the first embodiment. The polarized illumination unit 10 irradiates the subject 100 to be imaged with polarized light (step S101). The polarization applied to the subject 100 is not limited to a specific polarization. In the following, the linearly polarized light is applied to the subject 100 with a single polarization angle “θ”.

The polarized light applied to the subject 100 is reflected on the surface of the subject 100 and passes through the lens 110. A part of the light transmitted through the lens 110 is absorbed by the mask 111. The unabsorbed light selectively passes through the first aperture region 113 or the second aperture region 114.

The first opening region 113 transmits polarized light having an angle component corresponding to the angle (polarization angle) of the polarizing filter provided in the first opening region 113. The second opening region 114 transmits polarized light having an angle component corresponding to the angle (polarization angle) of the polarizing filter provided in the second opening region 114.

The first opening region 113 absorbs a part of the polarized light having an angular component orthogonal to the angle of the polarizing filter provided in the first opening region 113, and reflects the remaining polarized light. The second opening region 114 absorbs a part of the polarized light having an angular component orthogonal to the angle of the polarizing filter provided in the second opening region 114, and reflects the remaining polarized light.

The angle (polarization angle) of the polarization filter in each aperture region is predetermined according to the relative positional relationship between the polarization illumination unit 10 and the polarization image sensor 112. In the following, the first aperture region 113 includes a polarizing filter having a polarization angle of “ω ₁ ”. The second aperture region 114 includes a polarizing filter having a polarization angle of “ω ₂ ”.

FIG. 5 is a diagram showing the relationship between the polarization angle and the brightness in the first embodiment. Generally, the reflected light from the surface of the subject 100 is composed of unpolarized light due to diffuse reflection and linearly polarized light due to specular reflection. The polarization angle of the linearly polarized light mirror-reflected in the subject 100 is equal to the polarization angle of the polarized light applied to the subject 100.

Generally, when the brightness of the reflected light is observed through the rotating polarizing plate, the brightness changes in a sine wave shape according to the polarization angle (rotation angle of the polarizing plate). In FIG. 5, “I _max ” represents the luminance (maximum luminance) at the angle “φ” where the specular reflection is the strongest. "I _min " represents the luminance (minimum luminance) at the angle at which the specular reflection is the weakest. That is. "I _min " represents the luminance at an angle at which only diffusely reflected polarization is observed.

In the polarization analysis, the brightness of the reflected light is observed for four types of polarization angles (for example, “0, π / 4, π / 2, −π / 4”) shifted by “π / 4” from each other. Further, based on the parameter (Stokes parameter) representing the polarization component, the maximum brightness "I _max " and the minimum brightness "I _min ", the normal direction or bidirectional reflectance distribution of the surface of the subject 100 (Bidirectional Reflectance Distribution). Function: BRDF) The function is estimated.

For this reason, the polarizing filter provided in the polarizing image sensor is often a polarizing filter having four types of polarization angles shifted by "π / 4" from each other. The polarization image sensor 112 includes four types of polarization filters [Ψ −π / 4, Ψ, Ψ + π / 4, Ψ + π / 2] with reference to the polarization angle “Ψ”.

The intensity and polarization angle of the specularly reflected light on the surface of the subject change according to Fresnel's law. Fresnel's law is that the intensity and polarization angle of light specularly reflected on the surface of a subject depend on the refractive index of air and the subject and the incident angle of light. When the light is specularly reflected, the incident angle and the reflection angle are equal, so that the specularly reflected light can be observed only from the direction of the illumination and the direction of the regular reflection across the subject surface. For example, if the lighting to the subject and the direction of the camera can be regarded as substantially the same, the specularly reflected light is observed only from the surface of the subject facing the lighting and the camera. In this case, the polarization angle and intensity of the specularly reflected light are substantially the same as the polarization angle and intensity of the polarized illumination.

In this case, when the polarization angle (polarization angle of the polarization filter of the polarization illumination unit 10) “θ” of the polarization applied to the subject 100 is “0 (radian)” as an example, the angle “φ” at which the mirror reflection is the strongest. Is 0. Therefore, the brightness observed in the polarizing image sensor 112 according to the polarization transmitted through the polarizing filter having a polarization angle of “−π / 4” shifted from 0 to “−π / 4” and “π / 4” from 0. It is equal to the brightness observed in the polarization image sensor 112 according to the polarization transmitted through the polarization filter having the shifted polarization angle “π / 4”.

Therefore, if the image capturing unit does not include the first opening region, the second opening region, and the mask, the subject generated by the polarized image sensor according to the polarization transmitted through the lens and projected on the polarized image sensor. With respect to the channels of the 100 polarized images, the image of the channel corresponding to the polarization angle “π / 4” and the image of the channel corresponding to the polarization angle “−π / 4” are substantially the same.

On the other hand, in the image capturing unit 11, among the polarized light transmitted through the lens 110, the polarized light selectively transmitted through the first aperture region 113 or the second aperture region 114 is the first aperture region 113 or the second aperture region 114. The polarized light selectively transmitted through the image is formed in the polarized image sensor 112 at positions deviated from each other according to the direction in which the polarized light is incident on the polarized image sensor 112. The image (optical image) imaged in this way passes through filters of four types of polarization angles provided in the polarization image sensor 112 and is generated by the polarization image sensor 112 as a polarization image having four channels. Will be done. The generated polarized image may be recorded in a memory.

In such a case, by providing a polarization filter having a polarization angle of "Ψ-π / 4" in the first opening region 113, the polarization of the polarization angle different from the polarization angle of "Ψ-π / 4" is the polarization of the polarization. Since the polarization angle and the polarization angle of the polarization filter are orthogonal to each other, they are blocked by the polarization filter of the first opening region 113. Therefore, a polarized image of the channel corresponding to the polarization angle “Ψ−π / 4” is generated according to the polarization transmitted through the first opening region 113. Hereinafter, the polarized image of the channel corresponding to the polarization angle “Ψ−π / 4” is referred to as a “first observed image”.

Further, since the polarization filter having the polarization angle “Ψ + π / 4” is provided in the second opening region 114, the polarization of the polarization angle different from the polarization angle “Ψ + π / 4” is the polarization angle of the polarization and the polarization of the polarization filter. Since the corners are orthogonal to each other, they are blocked by the polarizing filter of the second opening region 114. Therefore, a polarized image of the channel corresponding to the polarization angle “Ψ + π / 4” is generated according to the polarization transmitted through the second opening region 114. Hereinafter, the polarized image of the channel corresponding to the polarization angle “Ψ + π / 4” is referred to as a “second observation image”.

When “θ = Ψ”, the polarized image of the channel corresponding to the polarization angle “Ψ−π / 4” and the polarized image of the channel corresponding to the polarization angle “Ψ + π / 4” are separated from each other for the above reason. No color shift has occurred.

Returning to FIG. 4, the explanation of the operation example of the position estimation device 1a is continued. As described above, the image capturing unit 11 (polarized image sensor 112) generates the first observed image and the second observed image (step S102).

The image input unit 120a acquires the first observation image and the second observation image from the image imaging unit 11 (polarized image sensor 112). The image input unit 120a outputs the first observation image and the second observation image to the position estimation unit 121a. The image input unit 120a may record the first observation image and the second observation image in the memory (step S103).

The position estimation unit 121a estimates the position parameter of the subject 100 based on the first observation image and the second observation image. The position estimation unit 121a outputs the position parameter of the surface of the subject 100 to a predetermined external device (not shown) (step S104).

The position parameter is not limited to a specific parameter as long as it is a parameter related to the position of the subject 100. Further, the unit used for estimating the position parameter is not limited to a specific unit. In the following, the position vector indicating the three-dimensional position (depth) is determined for each pixel of the image as an example. The position parameter may be a map parameter such as a parallax map or a depth map. The position parameter may be set for each block or for each area where the image is divided.

The method of estimating the position parameter is not limited to a specific method. For example, the position estimation unit 121a estimates the position parameter by the stereo method using the first observation image and the second observation image. For example, the position estimation unit 121a estimates the parallax using the result of block matching between the images of each channel. The position estimation unit 121a may convert the parallax into a three-dimensional position by using geometric information such as the distance “Δc _x ” between the opening regions. For example, the position estimation unit 121a may estimate the position parameter of the subject 100 based on the detection result of the feature point of the subject 100. For example, the position estimation unit 121a may estimate the position parameter of the subject 100 by using a marker placed within the angle of view of the image imaging unit 11 while the subject 100 is being imaged. For example, when the polarized light applied to the subject 100 is the pattern light, the position estimation unit 121a may estimate the position parameter on the surface of the subject 100 by using the pattern light applied to the subject 100.

As described above, the first observed image corresponds to the first polarization angle (for example, Ψ−π / 4), which is a polarization angle that is tilted from the reference (for example, Ψ = 0) in the negative direction and is not orthogonal to the reference. It is a polarized image of the channel to be used. The second observed image is a polarized image of the channel corresponding to the second polarization angle (for example, Ψ + π / 4), which is a polarization angle that is tilted from the reference in the positive direction and is not orthogonal to the reference. The image capturing unit 11 generates a first observed image and a second observed image according to the light reflected from the subject 100 irradiated with the polarization of the reference polarization angle. The position estimation unit 121a estimates the three-dimensional position of the subject 100 in the real space based on the first observation image and the second observation image. The position estimation unit 121a estimates the three-dimensional position of the subject 100 in the real space by using, for example, a stereo method, a pattern projection method, or the like.

A plurality of sub-apertures (first aperture region 113, second aperture region 114) are provided on the mask 111 in the vicinity of the lens 110 of the image capturing unit 11. A polarization filter having a constant first angle with respect to the polarization angle of the polarization from the polarization illumination unit 10 is provided in the first opening region 113. Similarly, a polarization filter having a constant second angle with respect to the polarization angle of the polarization from the polarization illumination unit 10 is provided in the second opening region 114. The polarized lighting unit 10 irradiates the subject 100 with polarized light. The polarized light (reflected light) reflected from the subject 100 is observed by the polarized image sensor 112 using the lens 110 and the mask 111 (plural sub-apertures).

As a result, the image of the channel (polarized image) corresponding to the polarization angle equal to the polarization angle "Ψ-π / 4" of the polarization filter provided in the first opening region 113 and the polarization provided in the second opening region 114. It is possible to cancel the change in brightness between the image of the channel corresponding to the polarization angle equal to the polarization angle “Ψ + π / 4” of the filter (polarized image). Therefore, it is possible to easily estimate the three-dimensional position of the surface of the subject 100. This makes it possible to suppress a decrease in accuracy due to the influence of the subject or the environment in estimating the three-dimensional position of the subject in the real space.

(Second Embodiment)
The second embodiment differs from the first embodiment in that the image processing unit estimates the reflection parameter on the surface of the subject. In the second embodiment, the differences from the first embodiment will be mainly described.

FIG. 6 is a diagram showing a configuration example of the position estimation device 1b in the first embodiment. The position estimation device 1b is a device that estimates the three-dimensional position (depth) of the surface of the subject. In FIG. 6, the position estimation device 1b estimates the reflection parameter on the surface of the subject 100.

The position estimation device 1b includes a polarized lighting unit 10, an image imaging unit 11, and an image processing unit 12b. The image capturing unit 11 outputs the image of the subject 100 to the image processing unit 12b.

The image processing unit 12b acquires an image of the subject 100 from the image imaging unit 11. The image processing unit 12b estimates the three-dimensional position (depth) of the surface of the subject 100 based on the image acquired from the image capturing unit 11. The image processing unit 12b estimates the reflection parameter of the surface of the subject 100 based on the three-dimensional position of the surface of the subject 100 and the image (observation image) of the subject 100.

FIG. 7 is a diagram showing a configuration example of the image processing unit 12b in the first embodiment. The image processing unit 12b includes an image input unit 120b, a position estimation unit 121b, and a reflection component estimation unit 122.

The image input unit 120b outputs the image of the subject 100 to the position estimation unit 121b and the reflection component estimation unit 122. The position estimation unit 121b estimates the three-dimensional position (depth) of the surface of the subject 100 based on the image of the subject 100. The position estimation unit 121b outputs the estimation result of the three-dimensional position of the surface of the subject 100 to a predetermined external device (not shown) and the reflection component estimation unit 122.

The reflection component estimation unit 122 acquires an image of the subject 100 from the image input unit 120b. The reflection component estimation unit 122 acquires the estimation result of the three-dimensional position of the surface of the subject 100 from the position estimation unit 121b. The reflection component estimation unit 122 estimates the reflection parameter (reflection component) on the surface of the subject 100 based on the three-dimensional position of the subject 100 and the image of the subject 100. The reflection component estimation unit 122 outputs the reflection parameter on the surface of the subject 100 to a predetermined external device (not shown).

FIG. 8 is a flowchart showing an operation example of the position estimation device 1b in the first embodiment. The operation from step S201 to step S202 is the same as the operation from step S101 to step S102 shown in FIG. Hereinafter, the polarized image of the channel corresponding to the polarization angle “Ψ” is referred to as a “third observation image”. Hereinafter, the polarized image of the channel corresponding to the polarization angle “Ψ + π / 2” is referred to as a “fourth observation image”.

The image input unit 120b acquires the first observation image, the second observation image, the third observation image, and the fourth observation image from the image imaging unit 11 (polarized image sensor 112). The image input unit 120b outputs the first observation image, the second observation image, the third observation image, and the fourth observation image to the position estimation unit 121b and the reflection component estimation unit 122. The image input unit 120b may record the first observation image, the second observation image, the third observation image, and the fourth observation image in the memory (step S203).

The position estimation unit 121b estimates the position parameter of the subject 100 based on the first observation image and the second observation image. The position estimation unit 121b outputs the position parameter of the surface of the subject 100 to a predetermined external device (not shown) and the reflection component estimation unit 122 (step S204).

The reflection component estimation unit 122 is the surface of the subject 100 based on at least one of the first observation image, the second observation image, the third observation image, and the fourth observation image, and the position parameter of the surface of the subject 100. Estimate the reflection parameters of. The reflection component estimation unit 122 outputs the reflection parameter on the surface of the subject 100 to a predetermined external device (not shown) (step S205).

The reflection parameter is not limited to a specific parameter as long as it is a parameter related to the reflection on the surface of the subject 100. For example, the reflection parameter is a Stokes parameter. For example, the reflection parameter may be a combination of the maximum luminance "I _max " and the minimum luminance "I _min " in an image such as the fourth observation image. The reflection parameter may be a mirror reflection component and a diffuse reflection component separated in the reflected light, or may be a normal direction or bidirectional reflectance distribution (BRDF) function of the surface of the subject 100.

Generally, when a subject is observed by a polarizing camera, the reflection parameter can be obtained based on the observation results for four types of polarization angles for each pixel. In the present embodiment, the brightness of the corresponding point of the first observed image and the brightness of the corresponding point of the second observed image match. This matched brightness is the average of the brightness of the corresponding point of the first observed image and the brightness of the corresponding point of the second observed image when the polarization angle changes from "Ψ-π / 4" to "Ψ + 3π / 4". It becomes brightness.

The reflection component estimation unit 122 acquires the first observation image, the second observation image, the third observation image, and the fourth observation image from the image input unit 120b. The reflection component estimation unit 122 acquires the position parameter of the surface of the subject 100 from the position estimation unit 121b. The reflection component estimation unit 122 estimates the reflection parameter on the surface of the subject 100 for each pixel of the first observed image.

Here, the third observation image is an image generated by the polarization selectively transmitted through the first aperture region 113 or the second aperture region 114 and passing through the polarization filter having the polarization angle “Ψ” of the polarization image sensor 112. (Polarized image). That is, the third observation image includes an image corresponding to the polarization transmitted through the polarization filter of the first opening region 113 and the polarization filter of the polarization angle “Ψ” of the polarization image sensor 112, and the polarization filter of the second opening region 114. It is generated according to the superposition with the image corresponding to the polarization transmitted through the polarizing filter having the polarization angle “Ψ” of the polarization image sensor 112.

The fourth observation image is an image generated by the polarization selectively transmitted through the first opening region 113 or the second opening region 114 and transmitted through a polarizing filter having a polarization angle of “Ψ + π / 2” of the polarization image sensor 112 (. Polarized image). That is, the fourth observation image is an image corresponding to the polarization transmitted through the polarization filter of the first opening region 113 and the polarization filter of the polarization angle “Ψ + π / 2” of the polarization image sensor 112, and the polarization of the second opening region 114. It is generated according to the superposition of the image corresponding to the polarization transmitted through the filter and the polarization filter having the polarization angle “Ψ + π / 2” of the polarization image sensor 112. Further, when “θ = Ψ”, the polarization angle “Ψ + π / 2” of the polarization filter corresponding to the fourth observed image and the polarization angle “θ” of the polarization filter of the polarization illumination unit 10 are orthogonal to each other. The fourth observation image is an image containing only the diffuse reflection component.

The method for estimating the reflection parameter is not limited to a specific method as long as it is a method for estimating the parameter related to the reflection of light on the surface of the subject 100. For example, the reflection component estimation unit 122 estimates the minimum luminance “I _min ” which is the diffuse reflection component of the fourth observation image based on the fourth observation image and the position parameter. Further, based on the assumption that the brightness changes according to the polarization angle with a sine wave based on the average brightness "I _mean " of the fourth observed image, the reflection component estimation unit 122 has a case where the specular reflection component becomes the maximum. The maximum brightness "I _max " of the 4th observation image observed in 1 is estimated.

The position parameter of the surface of the subject 100 corresponding to the image corresponding to the polarization transmitted through the first opening region 113 and the position parameter of the surface of the subject 100 corresponding to the image corresponding to the polarization transmitted through the second opening region 114. The correspondence can be known about the fourth observation image. The minimum luminance "I _min ", which is the diffuse reflection component of the fourth observed image, is expressed by the equation (1). Further, the equation (2) holds.

Here, "d" represents a vector representing parallax to a polarized image at a position different from that of the first observed image, with reference to a polarized image at the same position as the first observed image. The minimum luminance "I _mix " represents the diffuse reflection component of the fourth observed image. “I _a ” is generated by the polarization transmitted through the first opening region 113 among the polarized images constituting the fourth observed image and transmitted through the polarizing filter having the polarization angle “π / 2” of the polarization image sensor 112. Represents a polarized image. “I _b ” is generated by the polarization transmitted through the second opening region 114 in the polarized image constituting the fourth observed image and transmitted through the polarizing filter having the polarization angle “π / 2” of the polarization image sensor 112. Represents a polarized image.

The method for separating the polarized image from the fourth observation image is not limited to a specific method as long as the polarized image can be separated. For example, the reflection component estimation unit 122 estimates the image “I _b ” corresponding to the polarization transmitted through the second opening region 114 by solving the optimization problem shown in the equation (3). In addition, an arbitrary regularization term such as "total variation" may be added to the equation (3).

As described above, the image capturing unit 11 is further orthogonal to the reference with the third observation image which is the polarized image of the channel corresponding to the third polarization angle which is the polarization angle equal to the reference (for example, Ψ = 0). At least one of the fourth observed image, which is a polarized image of the channel corresponding to the fourth polarization angle (for example, Ψ + π / 2), which is the polarization angle, may be generated. Even if the reflection component estimation unit 122 estimates the reflection parameter of the subject based on at least one of the first observation image, the second observation image, the third observation image, and the fourth observation image, and the position parameter. good. For example, when the fourth observation image is generated, the normal estimation unit 123 estimates the reflection parameter on the surface of the subject 100 based on the position parameter of the subject 100 and the fourth observation image.

As a result, the polarization angle is different from both the polarization angle “Ψ−π / 4” of the polarization filter provided in the first opening region 113 and the polarization angle “Ψ + π / 4” of the polarization filter provided in the second opening region 114. It is possible to estimate reflection parameters such as normal information or BRDF function using each polarized image of the channel corresponding to the angle (eg, “Ψ = 0”, “Ψ + π / 2”).

In this way, it is possible to estimate the three-dimensional position (depth) of the surface of the subject 100 and the reflection parameter of the surface of the subject 100 as the three-dimensional shape (three-dimensional geometric information) of the subject 100 in the real space. be.

Further, the image processing unit 12b may estimate the three-dimensional shape (three-dimensional geometric information) of the subject 100 based on the three-dimensional position of the surface of the subject 100 and the reflection parameter of the surface of the subject 100.

(First modification)
The position estimation device 1b may include a plurality of polarized lighting units 10 and a plurality of image capturing units 11. The plurality of polarized lighting units 10 and the plurality of image capturing units 11 may be arranged in a spherical shape centered on the subject 100 to be imaged. This makes it possible to estimate a complex bidirectional reflectance distribution (BRDF) function or the like based on a plurality of combinations of the incident angle and the reflected angle of the polarized light.

(Second modification)
The polarization illumination unit 10 may time-modulate the polarization angle of the polarization. By changing the polarization angle of the polarization applied to the subject 100 by the polarization illumination unit 10, the polarization angle of the reflected light that is not completely specular reflection is set to the polarization angle of the first opening region 113 and the polarization angle of the second opening region 114. Can be adjusted to. As a result, even if the polarization angle of the reflected light and the polarization angle of the illumination do not match because the device and the subject cannot be arranged so that complete specular reflection occurs, the three-dimensional position of the surface of the subject can be obtained. It is possible to estimate the reflection parameters.

FIG. 9 is a diagram showing a hardware configuration example of the position estimation device 1 (position estimation device 1a, position estimation device 1b) common to each embodiment. A part or all of each functional unit of the position estimation device 1 is a storage device 3 and a memory 4 in which a processor 2 such as a CPU (Central Processing Unit) has a non-volatile recording medium (non-temporary recording medium). It is realized as software by executing the program stored in. The program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, flexible disks, optomagnetic disks, portable media such as ROM (ReadOnlyMemory) and CD-ROM (CompactDiscReadOnlyMemory), and storage of hard disks built into computer systems. It is a non-temporary recording medium such as a device. The display unit 5 displays an image.

A part or all of each functional part of the position estimation device 1 uses, for example, an LSI (Large Scale Integration circuit), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field Programmable Gate Array), or the like. It may be realized by using the hardware including the electronic circuit (electronic circuit or circuitry) that has been used.

As described above, the embodiment of the present invention has been described in detail with reference to the drawings, but the specific configuration is not limited to this embodiment, and the design and the like within a range not deviating from the gist of the present invention are also included.

The present invention is applicable to a device that estimates the three-dimensional shape of an imaged subject.

1,1a, 1b ... Position estimation device, 2 ... Processor, 3 ... Storage device, 4 ... Memory, 5 ... Display unit, 10 ... Polarized illumination unit, 11 ... Image imaging unit, 12a, 12b ... Image processing unit, 100 ... Subject, 110 ... lens, 111 ... mask, 112 ... polarized image sensor, 113 ... first aperture region, 114 ... second aperture region, 120a, 120b ... image input unit, 121a, 121b ... position estimation unit, 122 ... reflection component Estimator, 200 ... Focusing surface

Claims

It is a position estimation method executed by the position estimation device.
The first observation image, which is a polarized image of the channel corresponding to the first polarization angle, which is a polarization angle that is tilted from the reference in the negative direction and is not orthogonal to the reference, and the reference, which is tilted from the reference in the plus direction. Image imaging that generates a second observation image, which is a polarized image of the channel corresponding to the second polarization angle, which is a non-orthogonal polarization angle, according to the light reflected from the subject irradiated with the polarization of the reference polarization angle. Steps and
A position estimation method including a position estimation step for estimating a three-dimensional position of the subject based on the first observation image and the second observation image.
In the image imaging step, the third observation image, which is a polarized image of the channel corresponding to the third polarization angle, which is a polarization angle equal to the reference, and the fourth polarization angle, which is a polarization angle orthogonal to the reference, are further supported. Generates a fourth observation image, which is a polarized image of the channel to be used.
Further included is a reflection component estimation step for estimating the reflection parameters on the surface of the subject based on at least one of the first observation image, the second observation image, the third observation image, and the fourth observation image. ,
The position estimation method according to claim 1.
The first observation image, which is a polarized image of the channel corresponding to the first polarization angle, which is a polarization angle that is tilted from the reference in the negative direction and is not orthogonal to the reference, and the reference, which is tilted from the reference in the plus direction. Image imaging that generates a second observation image, which is a polarized image of the channel corresponding to the second polarization angle, which is a non-orthogonal polarization angle, according to the light reflected from the subject irradiated with the polarization of the reference polarization angle. Department and
A position estimation device including a position estimation unit that estimates a three-dimensional position of the subject based on the first observation image and the second observation image.
A program for operating a computer as the position estimation device according to claim 3.