JP7071633B2 - Distance measuring device, distance measuring method and distance measuring program - Google Patents

Description

The present invention relates to a distance measuring device, a distance measuring method, and a distance measuring program.

There is a TOF (Time Of Flight) method as one of the distance measuring techniques for measuring the distance to an object. The TOF method is a method of irradiating an object with a light beam for distance measurement and measuring the distance to the object based on the time from the irradiation of the light ray to the return of the reflected light.

In addition, as another distance measuring technique, a technique for calculating the degree of polarization based on an image obtained by receiving linear polarization in different polarization directions by a camera and measuring the distance to an object based on the degree of polarization. Has also been proposed.

Furthermore, as another distance measuring technique, both the distance image acquired from the distance (depth) sensor and the polarized image including the degree of polarization obtained from the camera as described above are used, as compared with the case where only the distance sensor is used. Techniques for measuring distance with high accuracy have also been proposed.

Japanese Unexamined Patent Publication No. 2013-044597 U.S. Patent Application Publication No. 2016/0261844

Seungkyu Lee, “Time-of-Flight Depth Camera MotionBlur Detection and Deblurring”, IEEE SIGNAL PROCESSING LETTERS, Vol. 21, No. 6, June 2014 Daisuke Miyazaki, Katsushi Ikeuchi, "Estimation Method of Surface Shape of Transparent Objects by Polarized Ray Tracing Method", IEICE Journal D-II, Vol. J88-DII, No. 8, August 2005, p. 1432-1439

By the way, many TOF sensors measure a distance by using a line scan method. The line scan type TOF sensor emits a light beam for distance measurement as an irradiation pattern extending in a predetermined first direction, and sequentially moves such an irradiation pattern in a second direction orthogonal to the first direction. While receiving the reflected light of the light beam.

In such a line scan type TOF sensor, the time at which the irradiation pattern is emitted to the object is different depending on the position with respect to the first direction. Therefore, when the moving object is distance-measured symmetric, the shape of the image of the object reflected on the TOF sensor may be distorted. When such distortion of the shape of the image occurs, there is a problem that the distance measurement accuracy deteriorates.

In one aspect, it is an object of the present invention to provide a distance measuring device, a distance measuring method, and a distance measuring program capable of measuring a distance to a moving object with high accuracy.

One proposal provides a distance measuring device having the following storage unit and arithmetic unit. The storage unit calculates the degree of polarization for each pixel based on the first distance image acquired from the TOF sensor and a plurality of images acquired from a plurality of cameras that receive linear polarization in different polarization directions. The generated first polarized image is stored. The calculation unit calculates the degree of polarization for each pixel based on the first distance image, and the first distance image is based on the comparison result between the second polarized image and the first polarized image. Is transformed, and the second distance image obtained by the transformation is output.

Further, in one proposal, a distance measuring method is provided in which a computer executes the same processing as the above-mentioned distance measuring device.
Further, in one proposal, a distance measuring program for causing a computer to perform the same processing as the above-mentioned distance measuring device is provided.

On one side, the distance to a moving object can be measured with high accuracy.

It is a figure which shows the configuration example and the processing example of the distance measuring apparatus which concerns on 1st Embodiment. It is a figure which shows the structural example of the shape measurement system which concerns on 2nd Embodiment. It is a figure which shows the hardware configuration example of a shape measuring apparatus and a sensor unit. It is a figure for demonstrating the problem of the depth measurement by a TOF sensor. It is a block diagram which shows the structural example of the processing function which a shape measuring apparatus has. It is a figure which shows an example of the data structure of a depth image. It is a figure which shows the outline of the camera polarization image calculation process. It is a figure which shows the outline of the processing procedure of the detailed shape calculation part. This is an example of a flowchart showing the processing procedure of the shape measuring device.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a diagram showing a configuration example and a processing example of the distance measuring device according to the first embodiment. The distance measuring device 1 shown in FIG. 1 uses the measurement result by the TOF sensor 2 and the images captured by a plurality of cameras to measure the distance between the TOF sensor 2 and an object (not shown) whose distance is to be measured. It is a device that measures each pixel of an image in which an object is captured. The TOF sensor 2 emits a light beam for distance measurement to the subject, and measures the distance (depth) of the subject based on the round-trip time from the light emission to the reception of the reflected light.

The distance measuring device 1 has a storage unit 1a and a calculation unit 1b. The storage unit 1a is realized, for example, as a storage area of a storage device (not shown) included in the distance measuring device 1. The arithmetic unit 1b is realized, for example, as a processor (not shown) included in the distance measuring device 1.

In the storage unit 1a, a distance image 11 (first distance image) acquired from the TOF sensor 2 and a polarized image 12 (first polarization) generated based on a plurality of images acquired from a plurality of cameras, respectively. Image) and is stored. In the present embodiment, as an example, it is assumed that three cameras 3a to 3c are used for generating the polarized image 12, but four or more cameras may be used.

The cameras 3a, 3b, and 3c receive linear polarizations in different polarization directions, and output images 4a, 4b, and 4c, respectively. For example, the camera 3a receives 0 ° linear polarization, the camera 3b receives 45 ° linear polarization, and the camera 3c receives 90 ° linear polarization. The polarized image 12 is generated by calculating the degree of polarization for each pixel based on the images 4a to 4c.

Here, the cameras 3a to 3c capture images 4a to 4c at the same shooting time. The “same shooting time” means that the exposure period is the same for the entire images 4a to 4c. On the other hand, since the TOF sensor 2 has a structure in which the reflected light of the light beam for distance measurement cannot be received at the same time in the entire distance image 11, the distance measurement time is partially different in the distance image 11. For example, when the line scan method is used, the TOF sensor 2 emits light rays for distance measurement as an irradiation pattern extending in a predetermined first direction (for example, in the vertical direction). Then, the TOF sensor 2 receives the reflected light of the light beam while sequentially moving such an irradiation pattern in the second direction (for example, the lateral direction) orthogonal to the first direction. In this case, since the light receiving time of the reflected light differs depending on the position on the distance image 11 with respect to the second direction, the measurement time of the distance also differs depending on the position with respect to the second direction.

As described above, in the distance image 11, the measurement time of the distance differs depending on the position of the pixel. Therefore, when a moving object is used as a distance measuring object, the shape of the image of the object reflected in the distance image 11 may be distorted. When such image shape distortion occurs, the distance measurement accuracy deteriorates. On the other hand, as described above, the shooting times of the images 4a to 4c by the cameras 3a to 3c are the same, and the exposure period is the same for the entire images 4a to 4c. Therefore, the polarized images 12 calculated from the images 4a to 4c are not affected by the shape distortion of the image due to the movement of the object as described above.

Therefore, the calculation unit 1b corrects the distance image 11 by the following processing using the polarized image 12 that is not affected by the shape distortion of the image. The calculation unit 1b generates a polarized image 13 (second polarized image) by calculating the degree of polarization for each pixel based on the distance image 11. Then, the calculation unit 1b compares the polarized image 12 and the polarized image 13, and deforms the distance image 11 based on the comparison result. The calculation unit 1b outputs the distance image 14 (second distance image) obtained by the deformation as a distance image in which the shape distortion of the image is reduced.

By such processing, the calculation unit 1b can correct the distance image 11 so that the shape distortion of the image is reduced with reference to the polarized image 12 which is not affected by the shape distortion of the image. As a result, when the object moves, the influence of the distortion of the shape of the image due to the movement on the distance measurement accuracy can be reduced, and the distance measurement accuracy can be improved.

[Second Embodiment]
FIG. 2 is a diagram showing a configuration example of the shape measurement system according to the second embodiment. The shape measuring system shown in FIG. 2 includes a shape measuring device 100 and a sensor unit 200.

The shape measuring device 100 acquires an image of a subject from the sensor unit 200, and measures the depth (distance) between the subject and the sensor unit 200 captured in the acquired image for each pixel on the image. For example, when the object 300 to be measured is shown in the image, the shape measuring device 100 measures the depth of the image of the object 300 shown in the image for each pixel of the image. As a result, the shape measuring device 100 can measure the three-dimensional shape of the object 300.

The sensor unit 200 includes a TOF sensor 210 and a camera unit 220. The TOF sensor 210 calculates the depth of the subject using the TOF method. Specifically, the TOF sensor 210 emits laser light to the subject and calculates the depth of the subject based on the round-trip time until the reflected light returns from the subject. The TOF sensor 210 outputs a "depth image" in which the depth is plotted for each pixel on the image to the shape measuring device 100.

On the other hand, the camera unit 220 simultaneously captures a plurality of camera images obtained by receiving linearly polarized light components in different polarization directions from the light from the subject, and outputs the camera images to the shape measuring device 100. do. The shape measuring device 100 may be able to acquire a luminance image as each camera image. For this purpose, the camera unit 220 may be configured to capture a luminance image, or the camera unit 220 may be configured to capture a color image and the shape measuring device 100 may be configured to convert the color image into a luminance image.

The shape measuring device 100 acquires a depth image based on the depth output from the TOF sensor 210, and calculates a polarized image based on the depth image. Further, the shape measuring device 100 calculates a polarized image based on the camera image output from the camera unit 220. As will be described later, the shape measuring device 100 has reduced the influence of the shape distortion of the object 300 appearing in the former polarized image by using both the polarized image based on the depth image and the polarized image based on the camera image. , Calculate the highly accurate depth.

FIG. 3 is a diagram showing a hardware configuration example of the shape measuring device and the sensor unit.
First, the shape measuring device 100 is realized as, for example, a computer as shown in FIG. The shape measuring device 100 shown in FIG. 3 includes a processor 101, a RAM (Random Access Memory) 102, an HDD (Hard Disk Drive) 103, a graphic processing device 104, an input interface 105, a reading device 106, a network interface 107, and a communication interface 108. Have.

The processor 101 comprehensively controls the entire shape measuring device 100. The processor 101 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or a PLD (Programmable Logic Device). Further, the processor 101 may be a combination of two or more elements of the CPU, MPU, DSP, ASIC, and PLD.

The RAM 102 is used as the main storage device of the shape measuring device 100. The RAM 102 temporarily stores at least a part of an OS (Operating System) program or an application program to be executed by the processor 101. Further, the RAM 102 stores various data necessary for processing by the processor 101.

The HDD 103 is used as an auxiliary storage device for the shape measuring device 100. The OS program, application program, and various data are stored in the HDD 103. As the auxiliary storage device, another type of non-volatile storage device such as SSD (Solid State Drive) can also be used.

A display device 104a is connected to the graphic processing device 104. The graphic processing device 104 causes the display device 104a to display an image according to an instruction from the processor 101. Display devices include liquid crystal displays and organic EL (Electroluminescence) displays.

An input device 105a is connected to the input interface 105. The input interface 105 transmits a signal output from the input device 105a to the processor 101. The input device 105a includes a keyboard, a pointing device, and the like. Pointing devices include mice, touch panels, tablets, touchpads, trackballs, and the like.

A portable recording medium 106a is attached to and detached from the reading device 106. The reading device 106 reads the data recorded on the portable recording medium 106a and transmits it to the processor 101. Examples of the portable recording medium 106a include an optical disk, a magneto-optical disk, and a semiconductor memory.

The network interface 107 transmits / receives data to / from other devices via the network 107a.
The communication interface 108 transmits / receives data to / from the sensor unit 200.

With the above hardware configuration, the processing function of the shape measuring device 100 can be realized.
On the other hand, the TOF sensor 210 of the sensor unit 200 has an infrared light emitting unit 211 and an infrared light receiving unit 212. The infrared light emitting unit 211 emits an infrared laser to the subject. The infrared light receiving unit 212 receives the reflected light of the infrared laser emitted from the infrared light emitting unit 211. The TOF sensor 210 measures the round-trip time until the reflected light of the infrared laser emitted from the infrared light emitting unit 211 returns from the subject, and calculates the depth of the subject by multiplying the measured round-trip time by the speed of light. do. The TOF sensor 210 outputs a depth image obtained by plotting the depth for each pixel on the image to the shape measuring device 100.

Further, the camera unit 220 of the sensor unit has cameras 221 to 223 and polarizing filters 221a to 223a. Each of the cameras 221 to 223 captures, for example, an RGB (Red / Green / Blue) color image. The polarization filters 221a to 223a are linear polarizing plates that transmit linearly polarized light in a specific polarization direction among the incident light.

A polarizing filter 221a is arranged on the imaging surface of the camera 221 at an angle of rotation of 0 ° about the optical axis of the camera 221, and the camera 221 captures an image via the polarizing filter 221a. Further, a polarizing filter 222a is arranged on the imaging surface of the camera 222 at a rotation angle of 45 ° about the optical axis of the camera 222, and the camera 222 captures an image via the polarizing filter 222a. Further, a polarizing filter 223a is arranged on the imaging surface of the camera 223 at a rotation angle of 90 ° about the optical axis of the camera 223, and the camera 223 captures an image via the polarizing filter 223a. As a result, the camera 221 outputs the camera image obtained by receiving the linear polarization of 0 °, the camera 222 outputs the camera image obtained by receiving the linear polarization of 45 °, and the camera 223 outputs the camera image obtained. , 90 ° linear polarization is received and the obtained camera image is output.

Here, although not shown, the cameras 221 to 223 each have a global shutter. Then, the cameras 221 to 223 can output camera images taken at the same shooting time (that is, exposed in the same exposure period) by opening and closing the global shutters in synchronization with each other. It has become.

Next, FIG. 4 is a diagram for explaining the problem of depth measurement by the TOF sensor. The infrared light emitting unit 211 of the TOF sensor 210 irradiates the infrared laser with a linear irradiation pattern, and gradually shifts the irradiation pattern to irradiate the entire depth measurement range with the infrared laser. In the example shown on the left side of FIG. 4, a linear irradiation pattern is shown by a broken line. In this example, the irradiation pattern has a linear shape extending in the vertical direction, and such an irradiation pattern is gradually shifted from right to left for irradiation. Hereinafter, the shift direction of the irradiation pattern is described as "scan direction".

Since the infrared laser is emitted by such a method, in the depth image output from the TOF sensor 210, the scan time (the time when the reflected light of the infrared laser is received) differs depending on the position with respect to the scan direction. .. This causes the measurement result to be inaccurate, for example, when measuring the three-dimensional shape of the moving object 300.

FIG. 4 illustrates a case where the scanning direction is from right to left and the object 300 moves from top to bottom. In this case, as shown on the right side of FIG. 4, the image 301 of the object 300 shown in the depth image has a shape in which the position is shifted downward toward the left side. In this way, when the object 300 moves, the shape of the image 301 of the object 300 may be distorted. There is a problem that the measured depth becomes inaccurate due to the distortion of the shape of the image 301.

Further, by increasing the scanning speed of the TOF sensor 210, the occurrence of the above-mentioned distortion can be reduced and the depth measurement accuracy can be improved. However, since the scan speed and the number of scans in one image (the number of irradiations of the irradiation pattern with respect to the scan direction) have a trade-off relationship, the number of scans decreases when the scan speed is increased. As a result, there is also a problem that the resolution of the depth image becomes low and the obtained three-dimensional shape becomes a coarse shape.

To solve such a problem, the shape measuring device 100 of the present embodiment corrects a depth image by using a polarized image based on a camera image acquired from the camera unit 220. More specifically, the shape measuring device 100 generates a polarized image based on the depth image, and deforms the depth image so that the error between the polarized image and the polarized image based on the camera image becomes small.

As described above, in the camera unit 220, by using the global shutter, a plurality of camera images obtained by receiving linear polarization having different polarization directions are taken at the same time. Therefore, in the polarized image obtained from such a camera image, the shape of the image 301 of the object 300 is not distorted. Utilizing this property, the shape measuring device 100 corrects the depth image by using the polarized image based on the camera image, thereby reducing the influence of the distortion of the shape of the image appearing in the depth image, and the depth is highly accurate. Is calculated.

FIG. 5 is a block diagram showing a configuration example of a processing function included in the shape measuring device. The shape measuring device 100 includes a storage unit 110, a depth acquisition unit 121, a polarized image calculation unit 122, a camera image acquisition unit 123, a polarized image calculation unit 124, and a detailed shape calculation unit 125.

The storage unit 110 is realized as a storage area of a storage device included in the shape measuring device 100, such as a RAM 102 or an HDD 103. The storage unit 110 stores a depth image 111, a normal image 112, a depth polarized image 113, camera images 114a to 114c, and a camera polarized image 115.

The processing of the depth acquisition unit 121, the polarized image calculation unit 122, the camera image acquisition unit 123, the polarized image calculation unit 124, and the detailed shape calculation unit 125 is realized, for example, by the processor 101 executing a predetermined program.

The depth acquisition unit 121 acquires the depth image 111 from the TOF sensor 210 and stores it in the storage unit 110. The depth image 111 has a structure in which the measured value of the depth is plotted for each pixel on the image. The depth acquisition unit 121 inputs the depth image 111 to the polarized image calculation unit 122. After that, the depth acquisition unit 121 deforms the depth image 111 input to the polarized image calculation unit 122 in response to a request from the detailed shape calculation unit 125, and re-inputs the deformed depth image to the polarized image calculation unit 122. Can be done.

The polarized image calculation unit 122 calculates a polarized image based on the depth image input from the depth acquisition unit 121. In the present specification, a polarized image calculated based on a depth image is referred to as a “depth polarized image”. The polarized image calculation unit 122 first calculates the normal information for each pixel on the input depth image, and generates a normal image 112 in which the normal information is plotted for each pixel on the image. , Stored in the storage unit 110. Then, the polarized image calculation unit 122 calculates the degree of polarization for each pixel on the generated normal image 112, and calculates the depth polarized image 113 in which the information on the degree of polarization is plotted for each pixel on the image. It is stored in the storage unit 110. The degree of polarization is an index indicating how much the light is polarized, and is a value of 0 or more and 1 or less.

The camera image acquisition unit 123 acquires camera images 114a, 114b, 114c taken by the cameras 221, 222, 223 of the camera unit 220, respectively, and stores them in the storage unit 110. These camera images 114a to 114c are images taken at the same time.

The camera images 114a to 114c are stored in the storage unit 110 as luminance images. When a color image is output from the cameras 211,222,223, the camera image acquisition unit 123 converts the color image output from the cameras 221,222,223, respectively, into a luminance image, and the camera images 114a, 114b, 114c, respectively. Is stored in the storage unit 110. Further, the camera images 114a to 114c are stored in the storage unit 110 as images having the same resolution as the depth image 111.

The polarized image calculation unit 124 calculates a polarized image based on the camera images 114a to 114c. In the present specification, the polarized image calculated based on the camera images 114a to 114c is referred to as a “camera polarized image”. The polarized image calculation unit 124 calculates the camera polarized image 115 based on the camera images 114a to 114c and stores it in the storage unit 110. The camera-polarized image 115 has a structure in which the degree of polarization is plotted for each pixel on the image.

The detailed shape calculation unit 125 calculates the squared error between the depth polarized image 113 and the camera polarized image 115, and the depth input from the depth acquisition unit 121 to the polarized image calculation unit 122 so that the squared error is minimized. Transform the image. The detailed shape calculation unit 125 inputs the depth image input from the depth acquisition unit 121 to the polarized image calculation unit 122 when the square error is determined to be the minimum (for example, when the square error becomes equal to or less than a predetermined threshold value). It is output as a depth image with the distortion of the image shape corrected.

Hereinafter, the processing of the shape measuring device 100 will be described in more detail.
<Acquisition of 1st depth image and camera image>
The depth acquisition unit 121 acquires the depth image 111 from the TOF sensor 210 and stores it in the storage unit 110. Further, the camera image acquisition unit 123 acquires the camera images 114a, 114b, 114c taken by the cameras 221, 222, 223 of the camera unit 220, respectively, and stores them in the storage unit 110.

As described above, the cameras 221, 222, and 223 capture and output the camera images 114a, 114b, and 114c at the same time by using the global shutter. Further, the acquisition of the depth image 111 and the acquisition of the camera images 114a to 114c are executed in synchronization with each other. Specifically, the scanning period of the depth image 111 (the period from the scanning time at one end of the scanning direction to the scanning time at the other end) includes the shooting times of the camera images 114a to 114c. Depth images 111 and camera images 114a to 114c are acquired.

FIG. 6 is a diagram showing an example of the data structure of the depth image. The data of the depth image 111 is stored as data having a structure in which a position in the X-axis direction (horizontal direction), a position in the Y-axis direction (vertical direction), and a position in the Z direction (depth direction) are associated with each pixel. It is stored in 110. Of these, the position in the Z direction indicates the depth (distance). The data of the depth image 111 is stored, for example, as a data table 111a as shown in FIG. In the data table 111a shown in FIG. 6, each position in the X direction, the Y direction, and the Z direction is registered in association with the pixel number of each pixel.

<Calculation of camera polarized image>
Next, the calculation process of the camera polarized image 115 by the polarized image calculation unit 124 will be described.

FIG. 7 is a diagram showing an outline of the camera polarization image calculation process. The polarized image calculation unit 124 performs a viewpoint conversion process on each of the camera images 114a to 114c in order to match the viewpoint with the viewpoint of the depth image 111. The following preparations are made for this viewpoint conversion process.

External parameters of the TOF sensor 210 and cameras 221 to 223 are set based on the positions and orientations of the TOF sensor 210 and cameras 221 to 223. In addition, internal parameters of cameras 221 to 223 are set. These parameters are determined by calibration. Then, based on these parameters, a homography matrix for converting the viewpoints of the cameras 221 to 223 into the viewpoints of the TOF sensor 210 is calculated.

The polarized image calculation unit 124 calculates the camera images 114a1, 114b1, 114c1 by converting the camera images 114a, 114b, 114c using a homography matrix, respectively. The camera images 114a1, 114b1, 114c1 show images when the camera images 114a, 114b, 114c are taken from the same position as the TOF sensor 210 in the same direction, respectively.

Next, the polarized image calculation unit 124 calculates the camera polarized image 115 using the camera images 114a1 to 114c1 obtained by the conversion. In this calculation process, the following processing is executed for each pixel of the camera images 114a1 to 114c1.

The polarized image calculation unit 124 approximates the luminance value of the m-th pixel in the camera images 114a1 to 114c1 by the cosine curve represented by the following equation (1).
ym = amcos (Θ + bm) + cm ・ ・ ・ (1)
The polarized image calculation unit 124 obtains the maximum value and the minimum value in the obtained cosine curve, and calculates the degree of polarization of the m-th pixel by the formula "(maximum value / minimum value) / (maximum value + minimum value)". calculate. As a result, the degree of polarization is calculated as a value of 0 or more and 1 or less.

The polarized image calculation unit 124 calculates the degree of polarization for each pixel by the above processing procedure. The polarized image calculation unit 124 generates a camera polarized image 115 in which the degree of polarization is plotted for each pixel and stores it in the storage unit 110.

The degree of polarization can be calculated from camera images obtained by using polarization filters having three or more different rotation angles. In the present embodiment, as an example, the degree of polarization is calculated based on the three camera images 114a to 114c obtained by using the three polarizing filters 221a to 223a, but it is obtained by using four or more polarizing filters. The degree of polarization may be calculated based on the camera image.

<Calculation of depth polarized image>
Next, the calculation process of the depth polarized image 113 by the polarized image calculation unit 122 will be described.

First, the polarized image calculation unit 122 calculates the normal image 112 as intermediate data based on the depth image input from the depth acquisition unit 121 by the following procedure. The normal n at the coordinates (x, y) of the depth image is expressed by the following equation (2). Further, p and q of the equation (2) are calculated by the equation (3) based on the depth H at the coordinates (x, y) of the depth image.

Further, the normal line n is expressed by the equation (4) using the zenith angle θ and the azimuth angle ψ.

From the relationship between the equation (2) and the equation (4), cos θ, which is the z component of the normal n shown in the equation (4), can be expressed by p and q. The normal image 112 is calculated by plotting cos θ, which is the z component of the normal n, for each pixel. The polarization image calculation unit 122 obtains the zenith angle θ for each pixel from the normal image 112, and calculates the degree of polarization ρ for each pixel using the following equation (5). In addition, N of the formula (5) represents a refractive index, and is, for example, 1.5.

The polarized image calculation unit 122 calculates the depth polarized image 113 by plotting the calculated degree of polarization ρ for each pixel.
<Calculation of detailed shape>
Next, the processing of the detailed shape calculation unit 125 will be described.

FIG. 8 is a diagram showing an outline of the processing procedure of the detailed shape calculation unit. The detailed shape calculation unit 125 compares the depth polarized image 113 calculated by the polarized image calculation unit 122 with the camera polarized image 115 calculated by the polarized image calculation unit 124. Here, since the depth polarized image 113 is calculated based on the depth image 111 acquired from the TOF sensor 210, there is a possibility that the image of the object to be imaged is distorted. On the other hand, in the camera polarized images 115 calculated based on the camera images 114a to 114c, such distortion does not occur. Therefore, the polarized image calculation unit 122 transforms the depth image 111, which is the calculation source of the depth polarized image 113, into the depth acquisition unit 121 so that the error between the calculated depth polarized image 113 and the camera polarized image 115 becomes small. Let me.

The detailed shape calculation unit 125 executes the following processing using the least squares method. Let Pdm be the degree of polarization in the mth pixel of the depth polarized image 113, and _Pcm be the degree of polarization in the _mth pixel of the camera polarized image 115. The detailed shape calculation unit 125 deforms the depth image 111 input to the polarized image calculation unit 122 so that the residual sum of squares RSS represented by the following equation (6) is minimized.

As described with reference to FIG. 4, in the depth image 111, the shape of the image is distorted when the object moves in a direction perpendicular to the scanning direction of the irradiation pattern of the infrared laser (vertical direction in the example of FIG. 4). Therefore, the deformation of the depth image 111 is performed so that the pixel trains adjacent to each other in the left-right direction are shifted in the vertical direction with respect to a certain pixel trains parallel to each other in the vertical direction on the image.

For example, assuming that the scanning direction is from right to left as shown in FIG. 4, the image of the object reflected in the depth image 111 has a shape in which the position is shifted upward toward the right side. Therefore, in one modification of the depth image 111, the pixel array adjacent to the right side of a certain pixel array is shifted downward by a predetermined shift amount. The shift amount may be set in units of multiples of pixels, or may be set to an amount smaller than the pixel pitch (for example, 0.5 pixel). In the latter case, the depth value in the transformed depth image is obtained by interpolation calculation.

<Flow chart>
Next, the processing procedure of the shape measuring device 100 will be described with reference to a flowchart.
FIG. 9 is an example of a flowchart showing a processing procedure of the shape measuring device.

[Step S11] The depth acquisition unit 121 acquires the depth image 111 from the TOF sensor 210 and stores it in the storage unit 110. Further, the camera image acquisition unit 123 acquires the camera images 114a, 114b, 114c taken by the cameras 221, 222, 223 of the camera unit 220, respectively, and stores them in the storage unit 110.

[Step S12] The polarized image calculation unit 124 performs a viewpoint conversion process for adjusting the viewpoint to the viewpoint of the depth image 111 for each of the camera images 114a to 114c. Specifically, the polarized image calculation unit 124 calculates the camera images 114a1, 114b1, 114c1 by converting the camera images 114a, 114b, 114c using a homography matrix for viewpoint conversion, respectively.

[Step S13] The polarized image calculation unit 124 calculates the camera polarized image 115 from the camera images 114a1 to 114c1 converted in step S12. Specifically, the polarized image calculation unit 124 approximates the luminance value of the m-th pixel in the camera images 114a1 to 114c1 by the cosine curve represented by the above equation (1). The polarized image calculation unit 124 obtains the maximum value and the minimum value in the obtained cosine curve, and determines the degree of polarization of the m-th pixel by the formula "(maximum value / minimum value) / (maximum value + minimum value)". calculate. The polarized image calculation unit 124 generates a camera polarized image 115 in which the degree of polarization is plotted for each pixel and stores it in the storage unit 110.

[Step S14] The depth acquisition unit 121 inputs the depth image 111 to the polarized image calculation unit 122. The polarized image calculation unit 122 calculates the normal image 112 based on the input depth image 111 and stores it in the storage unit 110. As described above, the polarized image calculation unit 122 calculates the normal image 112 by plotting cos θ, which is the z component of the normal n, for each pixel based on the equations (2) to (4).

[Step S15] The polarized image calculation unit 122 calculates the degree of polarization ρ for each pixel using the above equation (5) based on the normal image 112. The polarized image calculation unit 122 calculates the depth polarized image 113 by plotting the calculated degree of polarization ρ for each pixel, and stores it in the storage unit 110.

[Step S16] The detailed shape calculation unit 125 uses the above equation (6) to calculate the square error between the camera polarized image 115 calculated in step S13 and the depth polarized image 113 calculated in step S15. calculate.

[Step S17] The detailed shape calculation unit 125 determines whether the calculated square error is equal to or less than a predetermined threshold value. Here, as an example, when the square error is equal to or less than the threshold value, it is determined that the square error is minimized. The detailed shape calculation unit 125 executes the process of step S19 when the square error is equal to or less than the threshold value, and executes the process of step S18 when the square error is larger than the threshold value.

[Step S18] The detailed shape calculation unit 125 requests the depth acquisition unit 121 to deform the depth image 111. The depth acquisition unit 121 deforms the depth image 111 input to the polarized image calculation unit 122 in the latest step S14 by one step in response to the deformation request. For example, assuming that the scanning direction is from right to left and the shift amount for one step is S, the depth acquisition unit 121 is on the right side of the pixel rows parallel to each other in the vertical direction in the depth image 111 before deformation. The adjacent pixel strings are shifted downward by the shift amount S. In this case, the pixel sequence of the i-th column (however, i = 1, 2, ...) Is shifted downward by (i-1) × S.

After this, the process returns to step S14. In step S14, the depth acquisition unit 121 inputs the depth image 111 deformed in step S18 to the polarized image calculation unit 122. As a result, the normal image 112 and the depth polarized image 113 are calculated based on the deformed depth image 111.

[Step S19] The detailed shape calculation unit 125 corrects the depth image 111 when the square error becomes equal to or less than the threshold value, that is, the depth image 111 input to the polarized image calculation unit 122 in the latest step S14. Output as the final depth image.

By the above processing, the shape measuring device 100 can calculate a highly accurate depth that reduces the influence of the distortion of the shape of the image appearing in the depth image due to the movement even when the object to be photographed moves.

The processing functions of the devices (for example, the distance measuring device 1 and the shape measuring device 100) shown in each of the above embodiments can be realized by a computer. In that case, a program describing the processing content of the function that each device should have is provided, and the processing function is realized on the computer by executing the program on the computer. The program describing the processing content can be recorded on a computer-readable recording medium. The recording medium that can be read by a computer includes a magnetic storage device, an optical disk, a photomagnetic recording medium, a semiconductor memory, and the like. The magnetic storage device includes a hard disk device (HDD), a magnetic tape, and the like. Optical discs include CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray Discs (BDs, registered trademarks) and the like. The magneto-optical recording medium includes MO (Magneto-Optical disk) and the like.

When a program is distributed, for example, a portable recording medium such as a DVD or a CD on which the program is recorded is sold. It is also possible to store the program in the storage device of the server computer and transfer the program from the server computer to another computer via the network.

The computer that executes the program stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in its own storage device. Then, the computer reads the program from its own storage device and executes the processing according to the program. The computer can also read the program directly from the portable recording medium and execute the processing according to the program. Further, the computer can sequentially execute the processing according to the received program each time the program is transferred from the server computer connected via the network.

1 Distance measuring device 1a Storage unit 1b Calculation unit 2 TOF sensor 3a to 3c Camera 4a to 4c Image 11,14 Distance image 12,13 Polarized image

Claims (7)

To calculate the degree of polarization for each pixel based on the first distance image acquired from the TOF (Time Of Flight) sensor and multiple images acquired from multiple cameras that receive linear polarization in different polarization directions. A storage unit that stores the first polarized image generated in
Based on the comparison result between the second polarized image generated by calculating the degree of polarization for each pixel based on the first distance image and the first polarized image, the first distance image is obtained. A calculation unit that transforms and outputs the second distance image obtained by the transformation,
Distance measuring device with. In the modification, a comparison process of generating the second polarized image based on the first distance image and comparing the first polarized image with the second polarized image is performed on the first distance image. Is executed while deforming, and the second distance image is generated based on the result of the comparison process.
The distance measuring device according to claim 1. In the comparison process, an error between the first polarized image and the second polarized image is calculated.
As the second distance image, a distance image obtained by modifying the first distance image so that the error is minimized is generated.
The distance measuring device according to claim 2. The TOF sensor emits a light beam for distance measurement as an irradiation pattern extending in a predetermined first direction, and reflects the light ray while sequentially moving the irradiation pattern in a second direction orthogonal to the first direction. By receiving light, the first distance image is generated.
The distance measuring device according to any one of claims 1 to 3. The first distance image is deformed by sequentially shifting the pixels adjacent to the second direction in the first direction.
The distance measuring device according to claim 4. The computer
The first distance image is acquired from the TOF sensor, and the degree of polarization is calculated for each pixel based on a plurality of images acquired from a plurality of cameras that receive linear polarization in different polarization directions. Obtain a polarized image of 1 and
Based on the comparison result between the second polarized image generated by calculating the degree of polarization for each pixel based on the first distance image and the first polarized image, the first distance image is obtained. Deform and output the second distance image obtained by the deformation,
Distance measurement method. On the computer
The first distance image is acquired from the TOF sensor, and the degree of polarization is calculated for each pixel based on a plurality of images acquired from a plurality of cameras that receive linear polarization in different polarization directions. Obtain a polarized image of 1 and
Based on the comparison result between the second polarized image generated by calculating the degree of polarization for each pixel based on the first distance image and the first polarized image, the first distance image is obtained. Deform and output the second distance image obtained by the deformation,
A distance measurement program that executes processing.

Images