WO2022259457A1 - Shape estimation device, shape estimation method, and program - Google Patents

Abstract

This shape estimation device comprises: a polarization camera 10 for capturing a first image of a subject with no transparent refractive layer interposed between the camera 10 and the subject and a second image of the subject with a refractive layer interposed between the camera 10 and the subject; a feature value acquisition unit 20 for applying a feature point tracking method to the first image and second image and thereby acquiring a distortion vector Δ_g that is a feature value representing variation in geometric appearance resulting from refraction; a first estimation unit 30 for acquiring, from the second image, brightness value pairs respectively corresponding to at least three different polarization angles and estimating polarization feature values I_max, I_min, Ψ; a second estimation unit 40 for estimating two (nⁱ ₊, nⁱ _-) candidates for normal vectors nⁱ of the refractive layer to be estimated using the polarization feature values; and a refractive surface generation unit 50 for using the distortion vector Δ_g to select one normal vector candidate from among the normal vector candidates nⁱ ₊, nⁱ _- and generating three-dimensional refractive surface shape information representing the surface shape of the refractive layer.

Description

Shape estimation device, shape estimation method, and program

The present invention relates to a shape estimation device, a shape estimation method, and a program for estimating the three-dimensional shape of transparent refraction layers such as water surfaces and air fluctuations from images.

　Technology for estimating the 3D shape of an object in an image is particularly important in fields such as robot vision, augmented reality, and autonomous driving.

In general 3D shape estimation, multiple cameras are prepared and the 3D shape is estimated from the difference in appearance based on the difference in camera installation position. This conventional method assumes that the object to be estimated is an opaque and diffusely reflective surface. Therefore, it is not possible to estimate the shape of a transparent refracting surface such as a water surface.

In order to estimate the three-dimensional shape of such a refracting surface, it is necessary to model the propagation of light from the camera to the refracting surface to the subject behind the refracting surface. However, the three-dimensional shape of both the refracting surface and the object behind it are generally unknown. Also, the complexity of the refraction model makes it difficult to formulate as a transparent optimization problem.

Therefore, Non-Patent Document 1 theoretically and experimentally discloses that at least two cameras are required to estimate the three-dimensional shape of a scene in which one refraction occurs. In addition, Non-Patent Document 2 discloses that it is possible to estimate the three-dimensional shape of a refracting surface with a single camera by taking into account the difference in appearance of the background due to the presence or absence of refraction. Non-Patent Document 3 discloses a method of estimating the three-dimensional shape of a transparent surface using polarization information.

However, the method of Non-Patent Document 1 requires at least two cameras, and requires alignment and time synchronization between the cameras. In addition, in Non-Patent Document 2, the model formulated as the optimization problem is complicated, and shape estimation takes a long time, requiring a large calculation cost. Also, in Non-Patent Document 3, it is necessary to know a rough three-dimensional shape of an estimation target in advance.

In this way, the conventional technology requires multiple cameras, a large calculation cost, and the 3D shape is known, etc., and there is a problem that there is no suitable technology for estimating the 3D shape.

The present invention has been made in view of this problem. and to provide programs.

A shape estimating device according to an aspect of the present invention includes a polarizing camera that captures a first image of a subject when no transparent refractive layer is interposed and a second image of the subject when the refractive layer is interposed; a feature acquisition unit that acquires a distortion vector, which is a feature representing a change in geometric appearance due to refraction, by applying a feature point tracking method between the first image and the second image; and the second image. a first estimating unit that acquires sets of luminance values respectively corresponding to at least three different polarization angles from the a second estimating unit that estimates two candidates; and a refractive surface that selects one of the candidates for the normal vector using the distortion vector and generates refractive surface three-dimensional shape information representing the surface shape of the refractive layer. and a generator.

Further, a method for estimating a three-dimensional shape of a refraction surface according to an aspect of the present invention is a method for estimating a three-dimensional shape of a refraction surface, which is performed by the three-dimensional shape estimation device for a refraction surface, wherein the polarization camera has a transparent refraction layer interposed therebetween. A first image, which is an image of the subject when the refractive layer is not present, and a second image, which is an image of the subject when the refraction layer is present, are captured, and the feature amount acquisition unit obtains the first image and the second image. A distortion vector, which is a feature quantity representing a change in geometric appearance due to refraction, is obtained by applying a feature point tracking method between the two images, and a first estimating unit obtains at least three different angles and brightness from the second image. A second estimating unit estimates two candidates for the normal vector of the refraction layer to be estimated using the characteristic amount of polarization by obtaining a set of values, and a refraction surface generating unit selects one of the candidates for the normal vector using the distortion vector, and generates refractive surface three-dimensional shape information representing the surface shape of the refractive layer.

Also, a program according to one aspect of the present invention is summarized as a program for causing a computer to function as the above-described refractive surface three-dimensional shape estimation device.

According to the present invention, it is possible to estimate the three-dimensional shape of a transparent refractive surface with a single camera, low computational complexity, and no pre-learning data.

It is a block diagram showing an example of functional composition of a shape estimating device concerning an embodiment of the present invention. 2 is a diagram schematically showing the relationship between the shape estimation device shown in FIG. 1, a subject, and a transparent refractive layer; FIG. FIG. 4 is a diagram schematically showing modeling of changes in polarization, where (a) shows the normal vector and light reflected by the refraction layer, and (b) shows the relationship between the zenith angle and the degree of polarization. FIG. 4 is a diagram schematically showing the relationship between the azimuth angle and the brightness of an image; FIG. 4 schematically illustrates modeling of geometrical changes; FIG. 4 is a diagram schematically showing an example of refracting surface three-dimensional shape information; 1. It is a flowchart which shows the processing procedure of the shape estimation method which the shape estimation apparatus shown in FIG. 1 performs. 1 is a block diagram showing a configuration example of a general-purpose computer system; FIG.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The same reference numerals are given to the same items in multiple drawings, and the description will not be repeated.

FIG. 1 is a block diagram showing a functional configuration example of a shape estimation device according to an embodiment of the present invention. A shape estimation device 100 shown in FIG. 1 is a device for estimating the three-dimensional shape of a transparent refraction layer interposed between a subject.

The shape estimation device 100 includes a polarization camera 10 , a feature quantity acquisition unit 20 , a first estimation unit 30 , a second estimation unit 40 , and a refractive surface generation unit 50 . Each functional component except for the polarization camera 10 can be realized by a computer including ROM, RAM, CPU, and the like. In that case, the content of the processing is described by the program.

The polarization camera 10 is a general polarization camera. The polarization camera 10 incorporates, for example, polarizers (polarization filters) with four different polarization angles.

The polarization camera 10 captures a first image that is an image of the subject without a transparent refractive layer and a second image that is an image of the subject with a refractive layer interposed. A transparent refraction layer is a surface of water, a fluctuation layer of air, or the like.

The feature quantity acquisition unit 20 applies a feature point tracking method between the first image and the second image captured by the polarizing camera 10 to obtain a distortion vector, which is a feature quantity representing a geometric change in appearance due to refraction. get. The feature point tracking method is, for example, optical flow.

The first estimating unit 30 acquires sets of luminance values respectively corresponding to at least three different polarization angles from the second image and estimates the polarization feature amount. The characteristic quantity of polarization is the degree of polarization. Details will be described later.

The second estimation unit 40 estimates two candidates for the normal vector of the refraction layer to be estimated using the polarization feature amount. In other words, the candidates for the normal vector of the change in polarization are narrowed down to two.

The refraction surface generation unit 50 selects one of the normal vector candidates using the distortion vector, and generates refraction surface three-dimensional shape information representing the surface shape of the refraction layer. A normal vector is a vector orthogonal to a tangent to a pixel in the second image. Refraction surface three-dimensional shape information representing the surface shape of the refraction layer can be generated from the normal vectors of the pixels in which the refraction layer is reflected in the second image.

FIG. 2 is a diagram schematically showing the relationship between the shape estimation device 100, the subject (background), and the transparent refraction layer (refraction surface to be estimated). FIG. 2 shows only an image sensor and a polarizing filter that constitute the polarizing camera 10 .

The strip-shaped image sensor shown in FIG. 2 is, for example, a CMOS image sensor with millions of pixels. Pixel i on the image plane receives light through the polarizing filter from the background through the refractive layer.

A polarizing filter with four different polarization angles is placed in front of the image sensor. It is common to have four polarization angles.

The shape estimating apparatus 100 estimates the azimuth angle φ and the elevation angle θ of the normal vector n ⁱ , which is a vector orthogonal to the tangent line of the pixel i, for each pixel _i when the direction of the incident light is the optical axis Zc. Estimate the surface shape of the refractive layer. Hereinafter, each functional component of the shape estimation device 100 will be described.

When the image plane and the plane of the polarizing filter are parallel, the azimuth angle φ of the normal vector n ⁱ and the polarization angle have the same meaning geometrically. described as.

(First estimation unit)
The first estimating unit 30 obtains sets of luminance values respectively corresponding to at least three different polarization angles from the second image of the subject in the case where the refraction layer is interposed, and estimates the polarization feature amount.

The relationship shown in the following formula holds between the polarization angle and the luminance value.

Here, the polarization features are I _max , I _min , and Ψ. A polarization feature can be estimated from three or more different sets of polarization angles and luminance values.

By using the polarization feature quantities I _max , I _min , and Ψ, the Stokes vector s representing the polarization state can be expressed by the following equation.

A change in the Stokes vector s can be represented by multiplying the Stokes vector before change by the Mueller matrix M. That is, s _out =M·s _in , where _sin is the Stokes vector before change and s _out is the Stokes vector after change.

Here, Ts is the Fresnel transmission coefficient (component horizontal to the plane of incidence), and Tt is the Fresnel transmission coefficient (component perpendicular to the plane of incidence).

The degree of polarization ρ (Degree of Polarization), which indicates the degree of change in polarization, can be expressed by the following formula.

_In particular, when sin is non-polarized, it can be expressed by the following equation.

Equation (7) is seemingly complicated, but it is a monotonically increasing function and can be formulated as a convex optimization problem. Therefore, the elevation angle θ of the normal vector n ⁱ can be uniquely estimated from the observed value of the degree of polarization.

FIG. 3 is a diagram schematically showing the modeling of the change in polarization, (a) showing the normal vector n ⁱ and the light reflected by the refracting layer, and (b) the relationship between the zenith angle and the degree of polarization. It is a figure which shows.

In FIG. 3A, s _in represents light incident on the refractive surface (surface of the refractive layer), and s _out represents light captured by the polarization camera 10 .

In FIG. 3(b), the horizontal axis represents the zenith angle and the vertical axis represents the degree of polarization. As shown in FIG. 3(b), if the elevation angle θ is known, the degree of polarization ρ can be uniquely determined.

(Second estimation unit)
The second estimator 40 estimates two candidates for the normal vector n ⁱ of the refraction layer (refraction surface (surface of the refraction layer)) to be estimated using the polarization feature amount.

When the image plane and the plane of the polarizing filter are parallel, the azimuth angle φ of the normal vector n ⁱ coincides with the polarization angle Ψ (polarization angle at which the luminance value is maximized). When the polarizing filter is rotated once, the ambiguity of 180° remains because there are two angles at which the luminance value is maximized.

FIG. 4 is a diagram schematically showing the relationship between the azimuth angle φ and the luminance value. The horizontal axis of FIG. 4 indicates the azimuth angle φ, and the vertical axis indicates the image brightness I(φ). As shown in FIG. 4, the luminance value I(φ) has two maximum values.

Therefore, the candidates for the normal vector n ⁱ are narrowed down to the following two.

(Feature quantity acquisition unit)
A feature point tracking technique is applied between a first image of an object without a transparent refractive layer and a second image of the object with a refractive layer to represent the geometric change in appearance due to refraction. Acquire a distortion vector, which is a feature quantity.

FIG. 5 is a diagram schematically showing modeling of geometric changes. v _f shown in FIG. 5 represents the ray space on the polarization camera 10 side. μ is the relative refractive index. Also, _vr is the direction vector of the refracted light.

The direction vector v _r of the refracted light can be expressed by the following equation.

A distortion vector _Δg , which is a feature quantity representing a change in geometric appearance due to refraction, can be expressed by the following equation.

(Bending surface generator)
The refraction surface generator 50 selects one of the candidates n ⁱ ₊ and n ⁱ ₋ for the normal vector n ⁱ using the strain vector Δ _g and generates refraction surface three-dimensional shape information representing the surface shape of the refraction layer. do.

The refractive surface generation unit 50 generates refractive surface three-dimensional shape information by solving the optimum solution of the following equation from the candidates n ⁱ ₊ and n ⁱ ₋ of the normal vector n ⁱ narrowed down from the polarization constraint.

The calculation for solving the optimum solution of Equation (12) is performed for all pixels i.

FIG. 6 is a diagram schematically showing an example of refracting surface three-dimensional shape information. Three-dimensional shape information can be generated as shown in FIG.

(Shape estimation method)
FIG. 7 is a flow chart showing the processing procedure of the shape estimation method performed by the shape estimation device 100. As shown in FIG.

First, the polarization camera 10 captures a first image of the subject without a transparent refraction layer and a second image of the subject with a refraction layer (step S1).

Next, the first estimator 30 acquires sets of luminance values respectively corresponding to at least three different polarization angles from the second image, and estimates polarization feature quantities I _max , I _min , and Ψ (step S2). .

Next, the second estimating unit 40 estimates two (n ⁱ ₊ , n ⁱ ₋ ) candidates for the normal vector n ⁱ of the refraction layer to be estimated using the polarization feature amount (step S3).

Next, the feature amount acquisition unit 20 applies a feature point tracking method between the first image and the second image to acquire a distortion vector _Δg , which is a feature amount representing a geometric change in appearance due to refraction. (step S4).

Next, the refractive surface generator 50 selects one of the candidates n ⁱ ₊ and n ⁱ ₋ for the normal vector n ⁱ using the strain vector Δ _g , Information is generated (step S5). The processing of steps S2 to S5 is repeated until all pixels i are completed (NO in step S6).

The processing of steps S2 to S5 can be easily parallelized because each pixel is processed independently. Parallelization enables faster 3D shape estimation.

As described above, the shape estimating apparatus 100 according to the present embodiment captures a first image of a subject without a transparent refraction layer and a second image of the subject with a refraction layer. A camera 10, and a feature acquisition unit 20 that acquires a distortion vector _Δg , which is a feature representing a geometric change in appearance due to refraction, by applying a feature point tracking method between the first image and the second image. and a first estimating unit 30 that obtains sets of luminance values respectively corresponding to at least three different polarization angles from the second image to estimate the polarization feature amounts I _max , I _min , and Ψ; A second estimator 40 that estimates two (n ⁱ ₊ , n ⁱ ₋ ) candidates for the normal vector n ⁱ of the refraction layer to be estimated using the distortion vector Δ _g , and a candidate n for the normal vector n ⁱ using the distortion vector Δ g a refraction surface generator 50 that selects one from ⁱ ₊ and n ⁱ ₋ and generates refraction surface three-dimensional shape information representing the surface shape of the refraction layer.

Further, the shape estimation method according to the present embodiment is a shape estimation method performed by the shape estimation device 100, and the polarization camera 10 obtains the first image of the subject when no transparent refraction layer is interposed and the refraction layer is interposed. A second image of the subject is photographed in the case where the feature amount acquisition unit 20 applies a feature point tracking method between the first image and the second image to obtain a feature representing a change in geometric appearance due to refraction. The first estimating unit 30 obtains sets of luminance values respectively corresponding to at least three different polarization angles from the second image, and _{obtains the polarization feature quantities I max , I min , and} Ψ is estimated, and the second estimating unit 40 estimates two (n ⁱ ₊ , n ⁱ ₋ ) candidates for the normal vector n ⁱ of the refraction layer to be estimated using the polarization feature amount, and the refraction surface generation unit 50 selects one of normal vector n ⁱ candidates n ⁱ ₊ and n ⁱ ₋ using the distortion vector Δ _g to generate refractive surface three-dimensional shape information representing the surface shape of the refractive layer.

Note that the shape estimation device 100 can be realized by a general-purpose computer system shown in FIG. For example, in a general-purpose computer system including a CPU 90, a memory 91, a storage 92, a communication unit 93, an input unit 94, and an output unit 95, the CPU 90 executes a predetermined program loaded on the memory 91 to obtain a shape. Each function of the estimation device 100 is realized. The prescribed program can be recorded on computer-readable recording media such as HDD, SSD, USB memory, CD-ROM, DVD-ROM, MO, etc., or can be distributed via a network.

As described above, the shape estimating apparatus 100 and the shape estimating method according to the present embodiment use a single camera, low computational complexity, no prior knowledge of shape (no prior learning data), and transparent Allows estimation of the three-dimensional shape of the refractive surface. In other words, as an approach, by constructing a model that takes into account not only the geometrical changes in the propagation path of light due to refraction, but also the optical changes (polarization), the shape of the refraction surface can be obtained from the information obtained with only a single camera. We can obtain the necessary constraints for the three-dimensional shape of In addition, since it can be formulated as a convex optimization problem and the search range for the solution is narrowed, it is possible to estimate a three-dimensional shape with a lower amount of calculation than in the prior art.

It should be noted that the present invention includes various embodiments and the like that are not described here. Therefore, the technical scope of the present invention is defined only by the matters specifying the invention according to the valid scope of claims based on the above description.

10: Polarization camera 20: Feature amount acquisition unit 30: First estimation unit 40: Second estimation unit 50: Refraction surface generation unit 100: Shape estimation device

Claims (3)

1. a polarizing camera that captures a first image of an object without an intervening transparent refractive layer and a second image of the object with an intervening transparent refractive layer;
   a feature acquisition unit that acquires a distortion vector, which is a feature representing a change in geometric appearance due to refraction, by applying a feature point tracking method between the first image and the second image;
   a first estimating unit that obtains sets of luminance values respectively corresponding to at least three different polarization angles from the second image and estimates a polarization feature amount;
   a second estimating unit that estimates two candidates for the normal vector of the refraction layer to be estimated using the characteristic amount of polarization;
   a refraction surface generation unit that selects one of the candidates for the normal vector using the distortion vector and generates refraction surface three-dimensional shape information representing the surface shape of the refraction layer.
2. A shape estimation method performed by a shape estimation device,
   a polarization camera capturing a first image of an object without an intervening transparent refractive layer and a second image of the object with an intervening transparent refractive layer;
   A feature amount acquisition unit applies a feature point tracking method between the first image and the second image to acquire a distortion vector, which is a feature amount representing a change in geometric appearance due to refraction,
   a first estimating unit obtains sets of luminance values respectively corresponding to at least three different polarization angles from the second image to estimate a polarization feature amount;
   a second estimating unit estimating two candidates for the normal vector of the refraction layer to be estimated using the characteristic amount of polarization;
   A shape estimation method, wherein a refractive surface generation unit selects one of the candidates for the normal vector using the distortion vector, and generates refractive surface three-dimensional shape information representing a surface shape of the refractive layer.
3. A program for causing a computer to function as the shape estimation device according to claim 1 .
   

Images