JP2018066655A - Specular surface information acquisition device, specular surface measurement method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a specular surface information acquisition device and a specular surface information acquisition method for estimating a position and a posture of a specular surface.SOLUTION: A specular surface information acquisition device includes: a real image mirror image acquisition part for acquiring a feature point on a real image X in occurrence of interference between a direct light d2 directly applied from a TOF sensor 11 based on cross-correlation detection and reflected by a subject 30 and a reflection light obtained by reflection of a light d1 applied via a specular surface 20 by the subject, a feature point on a mirror image X', and an interference-time cross-correlation value in light emission and light reception of the TOF sensor; an initial value estimation part for estimating initial values of a position and a direction of a specular surface based on both the feature points; a real image acquisition part for acquiring a feature point on a real image and a non-interference-time cross-correlation value when interference does not occur; a position estimation part for estimating a position of a subject based on the non-interference-time cross-correlation value; and a specular surface information acquisition part for acquiring specular surface information for indicating a position and a direction of a specular surface based on an interference-time cross-correlation value, initial values of a position of a specular surface and a direction of a specular surface, and a position of a subject.SELECTED DRAWING: Figure 6

Description

  The present invention relates to a technique for estimating the position and orientation of a mirror surface from an image taken through a reflective object such as a mirror.

  In the field of image expression and image processing technology, research and development for acquiring a three-dimensional shape has been advanced for a long time, and is still widely recognized as one of the important research elements. In recent years, inexpensive TOF (Time of Flight) sensors have appeared on the market, making it easy to measure the three-dimensional shape of the surface of the subject. However, since the TOF sensor alone can only measure the shape of the front surface of the subject, in order to measure the three-dimensional shape of the entire periphery of the subject, a plurality of TOF sensors are installed around the subject, and shooting is performed. It is necessary to combine the three-dimensional shape data from the plurality of TOF sensors. However, the method using a plurality of TOF sensors increases the installation cost. Therefore, by installing a mirror behind the subject and assuming that the mirror image reflected on the mirror is an image taken by a plurality of pseudo TOF sensors, the TOF sensor alone has a three-dimensional shape around the subject. A method has been proposed for measuring.

  For example, there is a method of measuring a three-dimensional shape of a subject surface by a visual volume intersection method from a mirror image using a general color camera and a mirror instead of a TOF sensor (see Non-Patent Document 1). When using this method, it is necessary to estimate the position and orientation of the mirror relative to the camera. For example, before taking a picture of a subject, a calibration board with a checkered pattern is installed in the vicinity of the subject, and the position and orientation of the mirror are measured by measuring checkered lattice points that can be detected from both the mirror image and the real image. Can be estimated (see Non-Patent Document 2).

Forbes, Keith, et al. "Shape-from-Silhouette with Two Mirrors and an Uncalibrated Camera." Computer Vision-ECCV 2006. Springer Berlin Heidelberg, 2006. 165-178. Takahashi, Kosuke, Shohei Nobuhara, and Takashi Matsuyama. "A New Mirror-based Extrinsic Camera Calibration Using an Orthogonality Constraint." Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on.IEEE, 2012.

  Since the method described in Non-Patent Document 1 restores the surface shape of the object by the visual volume intersection method, there is a problem that the restoration is difficult when the surface shape of the object is complicated. Moreover, in the method described in Non-Patent Document 2, it is difficult to estimate the position and orientation of the mirror only by replacing the color camera with a TOF sensor. This is because the TOF sensor measures the distance using near-infrared light, and mutual interference occurs due to multiple irradiation of near-infrared light onto the subject by the mirror. Therefore, the calibration accuracy of the mirror is significantly lowered.

  Further, in order to measure the three-dimensional shape of the surface of the object using the TOF sensor and the mirror, it is necessary to calculate the reflectance of the mirror, but the method described in Non-Patent Document 2 obtains the reflectance of the mirror. I can't. Although it is possible to make the reflectivity of the mirror known in advance, it is difficult to measure the reflectivity in advance for all wavelengths because different wavelengths of infrared are used for each TOF sensor product. It is. In addition, the reflectance estimation accuracy can be made higher when the reflectance is estimated together with the estimation of the position and orientation of the mirror than when the reflectance of the mirror is measured in advance.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a technique for estimating the position and orientation of a mirror surface using an image photographed through a reflecting object such as a mirror.

  One embodiment of the present invention is directed to direct light, which is light that is directly irradiated to a subject from a TOF sensor based on cross-correlation detection and reflected by the subject, and to the subject via a mirror surface from the TOF sensor based on cross-correlation detection. A feature point on the real image that is a feature point of the subject in the real image that is an image based on the direct light when interference between the reflected light and the reflected light that is reflected by the subject occurs, Interference indicating a cross-correlation value in light emission and reception of the TOF sensor based on the cross-correlation detection, and a feature point on the mirror image corresponding to the feature point on the real image in the mirror image that is an image based on the reflected light A real image mirror image acquisition unit for acquiring a time cross-correlation value, a position of the mirror surface and a direction of the mirror surface based on the feature point on the real image and the feature point on the mirror image An initial value estimating unit for estimating a period value, a real image acquiring unit for acquiring the feature point on the real image when the interference does not occur, and a non-interference cross-correlation value indicating the cross-correlation value; A position estimation unit that estimates the position of the subject based on the cross-correlation value at the time of non-interference, the cross-correlation value at the time of interference, the initial value of the position of the mirror surface and the direction of the mirror surface, and the position of the subject And a mirror surface information acquisition unit that acquires mirror surface information indicating the position of the mirror surface and the direction of the mirror surface based on the mirror surface information acquisition device.

  One aspect of the present invention is the above-described mirror surface information acquisition device, wherein the mirror surface information acquisition unit includes the cross-correlation value at the time of interference, the initial value of the position of the mirror surface and the direction of the mirror surface, and the subject. Based on the position, the reflectance of the mirror surface is acquired.

  According to one aspect of the present invention, direct light, which is light that is directly irradiated onto the subject from the TOF sensor based on cross-correlation detection and reflected by the subject, and the subject is irradiated on the subject via a mirror surface from the TOF sensor based on the cross-correlation detection. A feature point on the real image that is a feature point of the subject in the real image that is an image based on the direct light in a case where interference with the reflected light that is the light reflected by the subject occurs, and the reflection A mutual point at the time of interference indicating a cross-correlation value in light emission and light reception of the TOF sensor based on the cross-correlation detection and a feature point on the mirror image corresponding to the feature point on the subject in the mirror image that is an image based on light A real image mirror image acquisition step for acquiring a correlation value, and the position of the mirror surface and the direction of the mirror surface based on the feature point on the real image and the feature point on the mirror image An initial value estimating step for estimating an initial value; a real image obtaining step for obtaining a feature point on the real image when no interference occurs; and a non-interference cross-correlation value indicating the cross-correlation value; A position estimating step of estimating the position of the subject based on the cross-correlation value at the time of non-interference, the cross-correlation value at the time of interference, the initial value of the position of the mirror surface and the direction of the mirror surface, and the position of the subject And acquiring the mirror surface information indicating the position of the mirror surface and the direction of the mirror surface based on the mirror surface information acquisition method.

  One embodiment of the present invention is a computer program for causing a computer to function as the mirror surface information acquisition apparatus.

  According to the present invention, it is possible to estimate the position and orientation of a mirror surface using an image taken through a reflective object such as a mirror.

It is a block diagram which shows the structure of a mirror surface information acquisition apparatus. It is a figure which shows the structure of a real image mirror image acquisition part. It is a figure which shows the structure of a real image acquisition part. It is a figure for demonstrating the detail of a real image mirror image acquisition part. It is a figure for demonstrating the detail of a mirror surface information acquisition part. It is a figure for demonstrating the detail of a mirror surface information acquisition part. It is a figure for demonstrating the detail of a mirror surface information acquisition part. It is a flowchart which shows operation | movement of a mirror surface information acquisition apparatus. It is a flowchart which shows operation | movement of a mirror surface information acquisition apparatus.

(Configuration of mirror surface information acquisition device)
FIG. 1 is a block diagram showing the configuration of the mirror surface information acquisition apparatus according to this embodiment.
The mirror surface information acquisition device 10 is configured by, for example, a computer device, and as shown in FIG. 1, a real image mirror image acquisition unit 101, an initial value estimation unit 102, a real image acquisition unit 103, a position estimation unit 104, and mirror surface information acquisition. Unit 105. In addition, the mirror surface information acquisition apparatus 10 is connected to the imaging unit 11 for communication. The imaging unit 11 is configured by a TOF sensor based on cross-correlation detection (in this paper, “TOF sensor based on cross-correlation detection” is simply referred to as “TOF sensor”), and as shown in FIG. And a light receiving unit 112. It should be noted that a general four bucket sampling is used as the distance measuring principle of the TOF sensor.

  Hereinafter, acquisition by the real image mirror image acquisition unit 101 will be described with reference to FIG. FIG. 2 is a schematic diagram showing an outline of acquisition by the real image mirror image acquisition unit 101.

  As shown in FIG. 2, the mirror surface information acquisition device 10 including the real image mirror image acquisition unit 101 and the imaging unit 11 (TOF sensor) are connected by communication. The light emitting unit 111 and the light receiving unit 112 included in the imaging unit 11 are directed in the direction in which the mirror 20 and the calibration board 30 are installed. Instead of the mirror 20, another object (reflective object) having a similar mirror surface (reflection surface), for example, a floor surface or the like may be used. Further, the shape of the mirror surface may not be a plane as long as it is a known shape.

The real image mirror image acquisition unit 101 of the mirror surface information acquisition device 10 is a feature coordinate group of the calibration board on the real image X reflected in the luminance image captured by the imaging unit 11 in a state of receiving interference of reflected light from the mirror 20. (U _R , v _R ), a characteristic coordinate group (u _M , v _M ) of the calibration board on the mirror image X ′, and a cross-correlation value (cross-correlation value at the time of interference) by the light emitting unit 111 and the light receiving unit 112 get.

  In addition, regarding the arrangement of the mirror 20 and the calibration board 30, the calibration unit 30 on the real image X acquired by the imaging unit 11 without passing through the mirror 20 and the mirror image X ′ acquired through the mirror 20 are used. It is assumed that the condition that the calibration board 30 and the calibration board 30 are both obtainable is satisfied.

The light emitting unit 111 repeats light emission based on an arbitrarily set modulation frequency f, and the light receiving unit 112 repeats light reception based on an arbitrarily set modulation frequency f.
The light receiving unit 112 includes a light receiving element that can acquire the wavelength of light emitted by the light emitting unit 111. In addition, when acquisition is performed by the near-infrared wavelength band generally used in the TOF sensor as in the present embodiment, for example, the light emitting unit 111 irradiates near-infrared light at regular intervals. A near infrared light capable of acquiring near infrared light is used as the light receiving unit 112. Further, when acquisition is performed using visible light, for example, a general projector is used as the light emitting unit 111, and a general color camera is used as the light receiving unit 112. The resolution of the image captured by the imaging unit 11 may be an arbitrary resolution.

  In the following description, the processing performed by the mirror surface information acquisition apparatus 10 for one pixel will be described for the sake of simplification. However, some of all the pixels or all the pixels are included. May be configured to process the above in parallel.

The description will be continued with reference to FIG. The real image mirror image acquisition unit 101 outputs the feature coordinate group (u _R , v _R ) and the feature coordinate group (u _M , v _M ) to the initial value estimation unit 102. The real image mirror image acquisition unit 101 outputs the cross-correlation value at the time of interference to a mirror surface information acquisition unit 105 described later.

The initial value estimation unit 102 acquires information indicating the luminance image captured by the imaging unit 11 from the real image mirror image acquisition unit 101. The initial value estimation unit 102 estimates the normal vector n ₀ of the mirror 20 (the mirror surface) and the distance d ₀ between the mirror 20 and the light receiving unit 112 based on the luminance image, and acquires mirror surface information described later. The initial value to be used for processing in the unit 105 is output to the specular information acquisition unit 105.

  Hereinafter, the acquisition by the real image acquisition unit 103 will be described with reference to FIG. FIG. 3 is a schematic diagram showing an outline of acquisition by the real image acquisition unit 103.

  As shown in FIG. 3, unlike the configuration shown in FIG. 2, a shielding plate 40 for shielding the mirror surface is installed on the front surface of the mirror surface portion of the mirror 20. Instead of the shielding plate 40, a cloth or the like for shielding the mirror surface may be used.

The real image acquisition unit 103 of the mirror surface information acquisition apparatus 10 is a characteristic coordinate group of the calibration board on the real image X that is reflected in the luminance image captured by the imaging unit 11 in a state where the reflected light from the mirror 20 is not interfered. (U _R , v _R ) and a cross-correlation value (non-interference cross-correlation value) by the light emitting unit 111 and the light receiving unit 112 are acquired.

The description will be continued with reference to FIG. The real image acquisition unit 103 outputs the non-interference cross-correlation value to the position estimation unit 104.
The position estimation unit 104 acquires a non-interference cross-correlation value from the real image acquisition unit 103, and estimates the position X of the calibration board 30 using the non-interference cross-correlation value. The position estimation unit 104 outputs the estimated position X to the mirror surface information acquisition unit 105 described later.

Specular information acquisition unit 105 acquires an interference when the cross-correlation value from the real mirror image acquisition unit 101, the distance from the initial value estimation unit 102 as an initial value and the normal vector n ₀ and mirror 20 of the mirror 20 and the light receiving portion 112 d ₀ is acquired, and the position X of the calibration board 30 is acquired from the position estimation unit 104. The mirror surface information acquisition unit 105 uses the acquired cross-correlation value at interference, normal vector n _0, distance d ₀ and position X, and the normal vector n of the mirror 20 and the distance d between the mirror 20 and the light receiving unit 112. Then, the reflectance ρ of the mirror 20 is obtained.

(Operation of mirror surface information acquisition device)
Hereinafter, the operation of the mirror surface information acquisition apparatus 10 will be described. 4 and 5 are flowcharts showing the operation of the mirror surface information acquisition apparatus shown in FIG.

  First, the real image mirror image acquisition unit 101 sets parameters (modulation frequency and sampling period) used for acquisition (step S001). As described above, the real image mirror image acquisition unit 101 sets the frequency at which the light emitting unit 111 emits light and the modulation frequency f that is the frequency at which the light receiving unit 112 receives light. Although the value of the modulation frequency f is arbitrary, for example, the real image mirror image acquisition unit 101 sets the value of the modulation frequency f to 10 MHz (megahertz).

In addition, the real image mirror image acquisition unit 101 sets θ ₀ , θ ₁ , θ _2, and θ ₃ as the four sampling periods acquired by the light receiving unit 112. The values of the sampling periods θ ₀ , θ ₁ , θ _2, and θ ₃ are arbitrary, but the real image mirror image acquisition unit 101 sets the values of the sampling periods θ ₀ , θ ₁ , θ _2, and θ ₃ to π / 2, for example. At intervals, θ ₀ = 0, θ ₁ = π / 2, and θ ₂ = πθ ₃ = 3π / 2 are set.

  Next, the real image mirror image acquisition unit 101 acquires a cross-correlation value (inter-correlation cross-correlation value) between the light emitting unit 111 and the light receiving unit 112 in a state where the direct light and the reflected light interfere with each other (step S002).

Hereinafter, acquisition of a cross-correlation value by the light emitting unit 111 and the light receiving unit 112 will be described with reference to FIG.
FIG. 6 schematically shows the configuration of the real image mirror image acquisition unit 101. The light receiving unit 112 and the light emitting unit 111 acquire a cross-correlation value based on four bucket sampling for each pixel according to the set parameters.

As shown in the figure, the near-infrared light emitted from the light emitting unit 111 and received by the light receiving unit 112 includes an optical path having an optical path length d ₁ that is an optical path directly reflected on the subject (calibration board 30), and the mirror 20. And an optical path having an optical path length d ₂ , which is an optical path reflected again to the subject (calibration board 30), and two optical paths. Hereinafter, the light passing through the optical path having the optical path length d ₁ is referred to as direct light, and the light passing through the optical path having the optical path length d ₂ is referred to as reflected light.

Hereinafter, an expression representing a cross-correlation value C (τ) acquired in a state where direct light and reflected light interfere with each other at a certain time τ is shown in Expression (1). Here, A ₁ and A ₂ are the amplitudes of the amplitudes of the direct light and the reflected light, respectively, and Ψ ₁ and Ψ ₂ are phase differences represented by the equation (2). In equation (2), c is the speed of light.

Since the cross-correlation function represented by Expression (1) is a continuous function and it is difficult to acquire the cross-correlation values C at all times, the real image mirror image acquisition unit 101 uses the sampling period θ set in step S001. _The cross-correlation value C is acquired only at ₀ , θ ₁ , θ ₂ and θ ₃ (that is, C ₀ , C ₁ , C ₂ and C ₃ are acquired). The cross-correlation value obtained here is expressed by the following formula (3), assuming that 2πfτ ₀ = θ ₀ , 2πfτ ₁ = θ ₁ , 2πfτ ₂ = θ ₂ , 2πfτ ₃ = θ ₃ .

Returning to FIG. 4 again, the operation of the mirror surface information acquisition apparatus 10 will be described.
Next, the real image mirror image acquisition unit 101 acquires a luminance image (step S003). The real image mirror image acquisition unit 101 acquires information obtained by imaging the amplitude of the light acquired by the light receiving unit 112 for each pixel, like a general TOF sensor.

  Next, the real image mirror image acquisition unit 101 performs feature point detection from the luminance image (step S004). Note that the feature point detection is a process that is performed in order to know the correspondence of some of the pixels captured in common with the real image X and the mirror image X ′. Therefore, the feature point detection method may be any method as long as it is a method capable of taking correspondence between pixels. For example, the feature point detection can be performed by Harris, SIFT, or the like, which are general natural feature points.

  For simplification of description, in this embodiment, it is assumed that the pattern drawn on the calibration board 30 is a checker pattern (checkered pattern), and the feature points are checker pattern grid points. A general detection method may be used for detecting checkered lattice points. In this embodiment, the subject is a calibration board. However, the subject is not limited to this, and the subject may be any other object having a feature point.

The real image mirror image acquisition unit 101 detects all checker pattern lattice points appearing as the real image X and the mirror image X ′ from the luminance image, and the coordinates are the feature point coordinates (u _R , v _R ) and the feature point coordinates (u _M , respectively). , V _M ).

Next, the real image mirror image acquisition unit 101 among the cross-correlation values acquired in all the pixels, the cross-correlation in the pixels of the feature point coordinates (u _R , v _R ) and the feature point coordinates (u _M , v _M ). Only the value is extracted (step S005). The cross-correlation values of the sampling periods in the feature point coordinates (u _R , v _R ) are respectively represented by C ₀ (u _R , v _R ), C ₁ (u _R , v _R ), C ₂ (u _R , v _R ) , And C ₃ (u _R , v _R ), and the cross-correlation value of each sampling period in the feature point coordinates (u _M , v _M ) is C ₀ (u _M , v _M ) C ₁ (u _M , v _{M), C 2 (u M} , v M), and _{C 3 (u M, v M} ) to.

The real image mirror image acquisition unit 101 outputs the feature point coordinates (u _R , v _R ) and the feature point coordinates (u _M , v _M ) to the initial value estimation unit 102 and extracts the extracted cross-correlation value C ₀ (u _M , V _M ), C ₁ (u _M , v _M ), C ₂ (u _M , v _M ), C ₃ (u _M , v _M ), and cross-correlation C ₀ (u _M , v _M ) C ₁ ( u _M , v _M ), C ₂ (u _M , v _M ), and C ₃ (u _M , v _M ) are output to the mirror surface information acquisition unit 105 (step S006).

Next, the initial value estimation unit 102 calculates the normal vector of the mirror 20 from the feature point coordinates (u _R , v _R ), the feature point coordinates (u _M , v _M ), and the three-dimensional shape information of the calibration board 30. n ₀ and the distance d ₀ between the mirror 20 and the light receiving unit 112 are estimated (step S007).

  In addition, when both the three-dimensional point coordinates on the surface of the object having a known shape and the corresponding two-dimensional point coordinates in the image obtained by photographing the three-dimensional point coordinates are known, the posture and position of the known shape object with respect to the camera There are many known methods for estimating the value of, for example, the method of Zhang (Zhang, Z .: A Flexible New Technique for Camera Calibration, IEEE Trans.Pattern Anal. Mach. Intell., Vol. 22, No. 11, pp. 1330 {1334 (2000).) Can be used. By using the Zhang method, the position and orientation of the calibration board 30 as the real image X (checker pattern) and the position and orientation of the calibration board 30 as the mirror image X ′ can be obtained.

Since the real image X and the mirror image X ′ are symmetric with respect to the mirror 20, the position and orientation of the calibration board 30 (checker pattern) as the real image X and the calibration board 30 as the mirror image X ′. The position and orientation of the mirror 20 can be determined based on the position and orientation (of the checker pattern). Further, by the determined position and orientation of the mirror 20, further a normal vector n ₀ of the mirror 20, it is possible to determine the distance d ₀ between the mirror 20 and the light receiving portion 112.

The initial value estimation unit 102 outputs the normal vector n ₀ of the mirror 20 and the distance d ₀ between the mirror 20 and the light receiving unit 112 to the mirror surface information acquisition unit 105.

  Next, the real image acquisition unit 103 acquires the real image X in a state where no interference between the direct light and the reflected light occurs, and acquires a cross-correlation value.

  First, the mirror 20 is shielded by the shielding plate 40 or the like without changing the positions of the mirror 20, the light emitting unit 111, the light receiving unit 112, and the calibration board 30. Since the shielding plate 40 is installed to prevent interference between the reflected light and the direct light, it is desirable that the shielding plate 40 be a color or material that sufficiently absorbs the wavelength component of the light emitted from the light emitting unit 111. In addition, if it is an object which can fully absorb the wavelength component of the light which the light emission part 111 light-emits, it may not be a board, For example, a black cloth etc. may be used instead of the shielding board 40. FIG.

  Next, the real image acquisition unit 103 acquires the cross-correlation value by the light emitting unit 111 and the light receiving unit 112, similarly to the acquisition performed by the real image mirror image acquisition unit 101 (step S008). The signal to be acquired by the real image acquisition unit 103 is the same as the signal to be acquired by the real image mirror image acquisition unit 101. However, since the reflected light is not generated by the shielding plate 40, the cross correlation acquired by the real image mirror image acquisition unit 101 is obtained. A cross-correlation value different from the value is acquired by the real image acquisition unit 103.

Next, the real image acquisition unit 103 extracts the cross-correlation value of each sampling period in the pixel of the feature point coordinates (u _R , v _R ) input from the real image mirror image acquisition unit 101, and extracts them as C ₀ ^′ (u _R , V _R ), C ₁ ^′ (u _R , v _R ), C ₂ ^′ (u _R , v _R ), and C ₃ ^′ (u _R , v _R ) (step S009).

  The real image acquisition unit 103 outputs the extracted cross-correlation value to the position estimation unit 104 as a cross-correlation value at the time of non-interference (step S010).

Hereinafter, the operation of the mirror surface information acquisition apparatus 10 will be described with reference to FIG.
Next, the position estimation unit 104 estimates the position of the calibration board 30 (step S011). In the process performed by the initial value estimation unit 102 described above, the initial value estimation unit 102 obtains the position of the calibration board 30 from the feature point coordinates (u _R , v _R ) and the feature point coordinates (u _M , v _M ). Thus, the position of the mirror 20 is estimated, but the position estimation unit 104 estimates the position of the calibration board 30 with higher accuracy by using the cross-correlation value.

The position estimation unit 104 estimates the three-dimensional position (x, y, z) of the checker pattern lattice point, that is, the three-dimensional point corresponding to the feature point coordinates (u _R , v _R ). That is. Therefore, the three-dimensional position (x, y, z) is calculated from the following equation (4) by obtaining the distance l from the light receiving unit 112 to the checkered lattice point in the feature point coordinates (u _R , v _R ). can do. In Expression (4), F represents the focal length of the light receiving unit 112, and the coordinates (u ₀ , v ₀ ) represent the image center coordinates.

l is obtained from the cross-correlation values C ₀ ^′ (u _R , v _R ), C ₁ ^′ (u _R , v _R ), C ₂ ^′ (u _R , v _R ), C ₃ ^′ (u _R , v _R ). Can be obtained using the following equations (5) and (6) based on the principle of four bucket sampling.

The position estimation unit 104 outputs the three-dimensional coordinates (x, y, z) corresponding to each feature point coordinate (u _R , v _R ) as a true value X to the mirror surface information acquisition unit 105.

The mirror surface information acquisition unit 105 uses the normal vector n _{0 of the} mirror 20 and the distance d ₀ between _the mirror 20 and the light receiving unit 112 as initial values, and uses the three-dimensional coordinates (x , Y, z) and the cross-correlation value at the time of interference, the normal vector n of the mirror 20, the distance d between the mirror 20 and the light receiving unit, and the reflectance ρ are calculated.

Hereinafter, the expression (7) necessary for obtaining the normal vector n of the mirror 20, the distance d between the mirror 20 and the light receiving unit 112, and the reflectance ρ will be described. Equation (7) is a matrix equation obtained by substituting 2πfτ ₀ = θ ₀ , 2πfτ ₁ = θ ₁ , 2πfτ ₂ = θ ₂ , 2πfτ ₃ = θ _{3 with} respect to equation (1), and direct light and reflection This is the formula of Four bucket sampling when light is causing interference. The mirror surface information acquisition unit 105 uses the acquired values C ₀ , C ₁ , C ₂ , C _3, Ψ ₁ , Ψ ₂ , and A ₁ , A ₂ , and B calculated from these values to obtain the normal of the mirror 20. The vector n, the distance d between the mirror 20 and the light receiving unit 112, and the reflectance ρ are calculated.

If the values of A ₁ , A ₂ , and B are obtained as they are from Equation (7), rank drops occur, and therefore an appropriate solution cannot be obtained. Therefore, the mirror surface information acquisition unit 105 first places restrictions on A ₁ and A ₂ (step S012).

First, Equation (8) is given as a constraint on A ₁ and A ₂ . FIG. 7 illustrates the parameters used in equation (8). This restriction condition is that when there is a difference in optical path length as shown in FIG. 7, the optical path length of the optical path A by direct light is the optical path length d ₁ , the optical path length of the optical path B of the reflected light is the optical path length d ₂ , and the irradiated light This is because the intensity of is inversely proportional to the square of the optical path length. Further, light attenuation occurs due to the reflectance ρ of the mirror 20. In addition, there is a difference in the incident angle of near infrared light to the calibration board 30. Based on the above, assuming that the incident angle of the light passing through the optical path A to the calibration board 30 is cos θ ₁ and the incident angle of the light passing through the optical path B to the calibration board 30 is cos θ ₂ , the calibration board 30 is a complete diffusion surface. The constraint condition when it is assumed that is is expression (8).

Returning to FIG. 5 again, the operation of the mirror surface information acquisition apparatus 10 will be described.
Next, the mirror surface information acquisition unit 105 creates Expression (9) as a matrix equation on the real image X side (step S013). Ψ ₁ ^R and ψ ₂ ^R are expressed as ψ ₁ ^R = 4πfd ₁ / c and ψ ₂ ^R = 4πfd ₂ / c using the optical path length d _{1 and the} optical path length d _{2 shown} in FIG. The details are shown in FIG. The optical path length reaching the pixel (u _R , v _R ) of the light receiving unit 112 via the true value X of the calibration board 30 obtained from the light emitting unit 111 by the position estimating unit 104 is the optical path length d ₁ , X. The initial parameters n ₀ and d _{0 of the} mirror 20 are defined as X ′, and the optical path passing through the position X ′, that is, the pixel (u) of the light receiving unit 112 from the light emitting unit 111 through the mirror 20 and the position X in this order. _The optical path length reaching _{R 1} , v _R ) is the optical path length d ₂ .

Next, Formula (10) is created as a matrix equation on the mirror image side (step S014). Ψ ₁ ^M and Ψ ₂ ^M are expressed as Ψ ₁ ^M = 4πfd ₁ / c and Ψ ₂ ^M = 4πfd ₂ / c using the optical path length d _{1 and the} optical path length d ₂ in FIG. The details are shown in FIG. The optical path length reaching the pixel of the feature point coordinates (u _M , v _M ) of the light receiving unit via the pattern position X ′ through the mirror from the light emitting unit is the optical path length d ₁ , and passes from the light emitting unit to X and the mirror in this order. The optical path length reaching the pixel at the feature point coordinates (u _M , v _M ) of the light receiving unit is the optical path length d ₂ .

Next, the mirror surface information acquisition unit 105 calculates the normal vector n of the mirror 20, the distance d between the mirror 20 and the light receiving unit 112, and the reflectance ρ. First, in Expression (9), A1, A2, and B are calculated using Expression (8) as a constraint condition. This is because parameters other than A1, A2, and B are all known, so considering the matrix of equation (11) and assuming that this is a generalized inverse matrix, equation (12) can be obtained, and A1 , A2, B can be solved. However, in reality, since the initial parameters n ₀ and d ₀ of the mirror 20 assumed at the beginning include errors, the left side and the right side of the equation (9) can be obtained by using the obtained A1, A2, and B. Not equal. The same applies to equation (10).

  Therefore, in both formulas (9) and (10), the normal vector n of the mirror 20 and the distance d between the mirror 20 and the light receiving unit 112 such that the error function having the difference between the left side and the right side as an error is minimized. The reflectance ρ is obtained. Since this is a general constrained minimization problem, a detailed description is omitted. The mirror surface information acquisition unit 105 outputs the obtained normal vector n of the mirror 20, the distance d between the mirror 20 and the light receiving unit 112, and the reflectance ρ (step S015).

  As described above, the mirror surface information acquisition apparatus 10 according to the present embodiment can estimate the position and orientation of a mirror surface using an image captured through a reflective object such as a mirror. Furthermore, the mirror surface information acquisition apparatus 10 according to the present embodiment can estimate the reflectance of the mirror together with the estimation of the position and orientation of the mirror.

  You may make it implement | achieve the mirror surface information acquisition apparatus 10 in embodiment mentioned above with a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be a program for realizing a part of the above-described functions, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system. You may implement | achieve using programmable logic devices, such as FPGA (Field Programmable Gate Array).

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10 ... Mirror surface information acquisition apparatus, 11 ... Imaging part, 20 ... Mirror, 30 ... Calibration board, 101 ... Real image mirror image acquisition part, 102 ... Initial value estimation part, 103 ... Real image acquisition part, 104 ... Position estimation part, 105 ... Mirror surface information acquisition unit, 111 ... light emitting unit, 112 ... light receiving unit

Claims (4)

Direct light, which is light that is directly irradiated on the subject from the TOF sensor based on cross-correlation detection and reflected by the subject, and light that is irradiated on the subject via a mirror surface from the TOF sensor based on cross-correlation detection are reflected by the subject. A feature point on the real image that is a feature point of the subject in the real image that is an image based on the direct light and an image based on the reflected light in the case where interference with the reflected light that is reflected light occurs. A feature point on the mirror image that is a feature point of the subject corresponding to the feature point on the real image in a mirror image, and a cross-correlation value at the time of interference indicating a cross-correlation value in light emission and light reception of the TOF sensor based on the cross-correlation detection. A real image mirror image acquisition unit to acquire;
An initial value estimating unit for estimating an initial value of the position of the mirror surface and the direction of the mirror surface based on the feature point on the real image and the feature point on the mirror image;
A real image acquisition unit for acquiring the feature points on the real image and the non-interference cross-correlation value indicating the cross-correlation value when the interference does not occur;
A position estimation unit that estimates the position of the subject based on the non-interference cross-correlation value;
Mirror surface for obtaining mirror surface information indicating the position of the mirror surface and the direction of the mirror surface based on the cross-correlation value at the time of interference, the initial value of the position of the mirror surface and the direction of the mirror surface, and the position of the subject. An information acquisition unit;
A mirror surface information acquisition apparatus.   The mirror surface information acquisition unit acquires the reflectance of the mirror surface based on the cross-correlation value at the time of interference, the initial value of the position of the mirror surface and the direction of the mirror surface, and the position of the subject. The mirror surface information acquisition apparatus according to claim 1. Direct light, which is light that is directly irradiated onto the subject from the TOF sensor based on cross-correlation detection and reflected by the subject, and light that is irradiated on the subject via a mirror surface from the TOF sensor based on cross-correlation detection are reflected by the subject. In the case where interference with reflected light, which is reflected light, occurs, a feature point on the real image that is a feature point of the subject in a real image that is an image based on the direct light, and a mirror image that is an image based on the reflected light A feature point on the mirror image that is a feature point of the subject corresponding to the feature point on the real image and a cross-correlation value at the time of interference indicating a cross-correlation value in light emission and light reception of the TOF sensor based on the cross-correlation detection Real image mirror image acquisition step;
An initial value estimating step of estimating an initial value of the position of the mirror surface and the direction of the mirror surface based on the feature point on the real image and the feature point on the mirror image;
A real image acquisition step of acquiring the feature point on the real image and the cross-correlation value at the time of non-interference indicating the cross-correlation value when the interference does not occur;
A position estimating step for estimating the position of the subject based on the cross-correlation value at the time of non-interference;
Acquisition of acquiring mirror surface information indicating the position of the mirror surface and the direction of the mirror surface based on the cross-correlation value at the time of interference, the initial value of the position of the mirror surface and the direction of the mirror surface, and the position of the subject. Steps,
A method for acquiring mirror surface information.   A computer program for causing a computer to function as the mirror surface information acquisition apparatus according to claim 1.

Images