JP2016136112A - Distance measurement device and distance measurement method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a distance measurement device and distance measurement method that can obtain a range-finding result higher in a reliability level.SOLUTION: A distance measurement device comprises: an imaging unit that chronologically acquires a plurality of shooting images; a characteristic point detection unit 201 that detects a point on a subject in the plurality of shooting images; and a three-dimensional position information calculation unit 203 that calculates a three-dimensional position of the point on the subject, and further comprises; a reliability level calculation unit 204 that calculates a reliability level of the calculated three-dimensional position; and a three-dimensional position determination unit 206 that determines the calculated three-dimensional position. The three-dimensional position determination unit 206 is configured so as to determine the three-dimensional position of an arbitrary point of attention on the subject in the plurality of shooting images on the basis of the three-dimensional position of an arbitrary point of comparison on the subject in the plurality of shooting images existing within an arbitrary range from the point of attention and the reliability level of the three-dimensional position of the arbitrary point of comparison.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a distance measuring apparatus and a distance measuring method capable of calculating a distance to a subject, a distance between two points on a subject, or a distance between subjects by using a plurality of images.

A multi-viewpoint imaging device that can shoot an arbitrary subject from multiple viewpoints is used to generate stereoscopic images, free-viewpoint images, and to generate depth images (depth maps) necessary to realize them. Used.
In such a system, based on the principle of triangulation, by using the parameters of the imaging device and the parallax information (sometimes expressed as disparity and depth) of each captured image, from the imaging device to the subject. And the distance between subjects can also be calculated. The relationship between the parameters of the imaging device and the distance from the imaging device to the subject can be expressed by [Equation 1].

Here, d is the parallax, b is the baseline length between the imaging devices, f is the focal length of the imaging device, and Z is the distance from the imaging device to the subject. [Equation 1] corresponds to a distance calculation formula when the optical axis of the imaging device is arranged in parallel. When the optical axis of the imaging device is arranged at an angle, the camera is calibrated by camera calibration or the like. The same can be applied by calculating the parameters and performing parallelization of the image.
Furthermore, three-dimensional position information can be calculated based on imaging device parameters such as the angle of view and lens distortion. The XY coordinate value can also be calculated using the angle between the optical axis and the projection point, the focal length f, and the distance Z. If three-dimensional position information can be obtained, the distance between any two points in the imaging space can be calculated.

The parallax as described above can be obtained by a stereo matching process between images. In stereo matching, one arbitrarily defined image is a reference image, the other is an evaluation image, a partial image of a region of interest of the reference image (for example, set to 5 × 5) is used as a template, and corresponding points ( This is a method of searching for a corresponding area based on an evaluation function (for example, SAD: Sum of Absolute Difference), and the distance between the attention area and the corresponding area corresponds to parallax.
In this stereo matching, when a characteristic part is not included in the partial image (for example, in the case of a plane having the same brightness and the same color), matching is performed in a plurality of regions, and the true position and The likelihood of matching is not necessarily maximized. That is, the correct parallax may not be obtained.
On the other hand, in the case of a partial image including a characteristic portion such as an edge, disparity information with relatively high accuracy can be obtained except when occlusion occurs or a repetitive pattern is formed.

As a distance measuring system using an edge portion of an image, Patent Document 1 is disclosed.
In Patent Literature 1, two images input from a multi-viewpoint imaging device are used as original images, a resized image in which one of the images is changed is prepared, and an edge (feature end point) is extracted from the resized image. . The corresponding region of the untransformed image (original image) corresponding to the extracted edge is set as a matching pattern of an arbitrary size, and the parallax and the distance are calculated by performing stereo matching with the other image.
With the above configuration, it is possible to improve the parallax detection power of the diagonal edge portion by using the vertical and horizontal edge detection kernel without preparing an edge detection kernel for detecting the diagonal edge for each angle.

JP 2003-98424 A

In the invention described in Patent Document 1, it is possible to perform distance measurement using an edge. However, as an issue for Patent Document 1, the influence of motion blur can be considered.
(1) Influence of motion blur itself When the subject is moving, motion blur (motion blur) occurs near the edge. In particular, when the subject is moving at high speed, the influence of motion blur occurs over a plurality of pixels on the image, and the object cannot be extracted as an edge (feature region), and thus distance measurement cannot be performed. Depending on the setting of the edge detection kernel, there is a possibility that it can be extracted as an edge, but even if it can be extracted as an edge, it is not guaranteed that the true position of the edge when no blur occurs. Therefore, it is considered that the reliability of the distance measurement result is lowered.
(2) Influence of the moving direction of the imaging device and the subject The amount of motion blur is maximized when the optical axis of the imaging device and the moving direction of the subject are orthogonal, and the optical axis of the imaging device and the moving direction of the subject are parallel. To the minimum. Therefore, in order to reduce the influence of motion blur, it is desirable to install the imaging device at a position where the optical axis of the imaging device and the moving direction of the subject are parallel. In other words, assuming that the subject is moving in the front direction of the subject, it is desirable to place the imaging device in front of the subject. However, when the side of the subject is also set as the shooting range, it cannot be always placed in front.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a distance measuring device and a distance measuring method capable of obtaining a more reliable distance measuring result. is there.

In order to solve the above-described problem, the distance measuring device of the present invention includes:
An imaging unit that acquires a plurality of captured images over time;
A subject point detector for detecting points on the subject in the plurality of captured images;
A three-dimensional position calculation unit that calculates a three-dimensional position of a point on the subject,
A reliability calculation unit for calculating the reliability of the calculated three-dimensional position;
A three-dimensional position determining unit for determining the calculated three-dimensional position;
The three-dimensional position determining unit is configured to arbitrarily set a three-dimensional position of an arbitrary point of interest on the subject in the plurality of photographed images within an arbitrary range from the point of interest. The comparison point is determined based on the three-dimensional position of the comparison point and its reliability.

  Therefore, since the distance (distance to the subject, distance between two points on the subject, or distance between subjects) can be calculated based on the determined three-dimensional position, a more reliable ranging result can be obtained. Can be obtained.

Preferably,
The comparison points may be a plurality of points, and at least one point may be configured to be detected from an image different from the captured image in which the point of interest exists.

  With such a configuration, it is possible to obtain a distance measurement result with higher reliability.

Also preferably,
The reliability can be configured to be a reliability based on a theoretical error due to a distance to the subject.

  With this configuration, the distance (the distance to the subject, the distance between two points on the subject, or the distance between the subjects) is determined based on the three-dimensional position determined in consideration of the theoretical error due to the distance to the subject. Thus, it is possible to obtain a distance measurement result with higher reliability.

Also preferably,
A motion blur estimation unit that estimates motion blur by predicting a moving direction of the subject relative to the imaging unit from a plurality of the three-dimensional positions calculated corresponding to each of the plurality of captured images;
Based on the estimation result of the motion blur estimation unit, it is possible to change at least one of the size and shape of the target point detection window used when detecting the target point.

  With this configuration, the attention point can be detected in consideration of the effect of motion blur, so the attention point can be detected more appropriately, and a more reliable ranging result can be obtained. it can.

Also preferably,
It is possible to configure so that the user can designate the attention point.

  By configuring in this way, the attention point that the user is interested in can be known, so only the three-dimensional position of the attention point has to be determined, and the amount of calculation can be reduced.

The distance measuring method of the present invention is
A shooting process of shooting a subject by an imaging unit that acquires a plurality of shot images over time;
A detection step of detecting a point on the subject in the captured image;
A three-dimensional position calculating step for calculating a three-dimensional position of a point on the subject;
A reliability calculation step of calculating the reliability of the calculated three-dimensional position;
Confirming the calculated three-dimensional position, and
In the determination step, a three-dimensional position of an arbitrary point of interest on the subject in the plurality of captured images is an arbitrary comparison point on the subject in the plurality of captured images that exists within an arbitrary range from the point of interest. It is configured to be determined based on the three-dimensional position and its reliability.

  Therefore, since the distance (distance to the subject, distance between two points on the subject, or distance between subjects) can be calculated based on the determined three-dimensional position, a more reliable ranging result can be obtained. Can be obtained.

  According to the present invention, it is possible to obtain a distance measurement result with higher reliability.

It is a figure showing an example of 1 composition of a distance measuring device in a 1st embodiment. It is a figure which shows one structural example of the image process part with which the distance measuring apparatus shown in FIG. 1 is provided. It is a figure which shows an example of an edge detection filter. It is a figure which shows an example of a stereo matching. It is a figure which shows an example of the calculation method of the to-be-photographed object's three-dimensional position. It is a figure which shows an example of the coordinate transformation method from the imaging device coordinate system C to the object coordinate system L. It is a flowchart which shows an example of the operation | movement in a three-dimensional position determination part. It is a figure which shows an example of the selection method of a comparison point. It is a figure which shows one structural example of the image process part with which the distance measuring device in 2nd Embodiment is provided. It is a figure which shows an example of the operation | movement in a motion blur estimation part. It is a figure which shows an example of the operation | movement in a motion blur estimation part. It is a figure which shows an example of the operation | movement in a motion blur estimation part. It is a figure which shows an example of the operation | movement in a motion blur estimation part. It is a figure which shows the example of 1 structure of the image process part with which the distance measuring device in 3rd Embodiment is provided. It is a figure which shows an example of the image for designating the feature point which the user wants to measure displayed on the display part. It is a figure which shows an example of the confirmed feature point information displayed on the display part. It is a figure which shows an example of the measuring method of the thickness of the board part of the pantograph of a vehicle. It is a figure which shows an example of the measuring method of the thickness of the board part of the pantograph of a vehicle. It is a figure which shows an example of the measuring method of the thickness of the board part of the pantograph of a vehicle.

  Embodiments according to the present invention will be described below with reference to the drawings.

<First Embodiment>: Basic Configuration In this embodiment, a basic configuration of the present invention will be described.
Specifically, the subject moving in the shooting space is shot multiple times, and the feature point coordinates and reliability of each of the shot images obtained by the multiple shots are obtained, and based on the feature point coordinates and reliability Then, the distance (distance to the subject, the distance between two points on the subject, or the distance between subjects) and the reliability thereof are measured by correcting the feature point coordinates of the photographed image at a desired photographing timing. A distance measuring apparatus and a distance measuring method that can be used will be described.

First, the overall configuration of the present invention will be described.
FIG. 1 is a configuration example of a distance measuring apparatus according to the present embodiment. The distance measuring device 100 is a distance measuring device according to the present embodiment, and includes an imaging unit 101 for imaging a subject, an image processing unit 102 for processing a captured image, a captured image itself and an image processing result. A display unit 103 for output display, a storage unit 104 for storing captured images themselves, image processing results, and various data used for image processing, and an external input for user input and data transmission to an external device An output unit 105, a control unit 106 for controlling the entire distance measuring apparatus 100, and a data bus 107 for exchanging data between each unit are configured.

The imaging unit 101 is configured to include a plurality of imaging elements such as optical components and CMOS (Complementary Metal Oxide Semiconductor) for capturing the imaging space into the distance measuring device 100, and each captured image data obtained by photoelectric conversion. Is output to the data bus 107. The plurality of optical components and the image pickup device may be arranged in one housing, but are configured as a plurality of image pickup devices having different shooting positions. That is, the imaging unit 101 includes a plurality of imaging devices each including an optical component and an imaging device, and these imaging devices are arranged at positions where captured images having parallax can be captured.
The photographed information may be output to the data bus 107 as raw data, or image data that has been subjected to image processing (luminance imaging, noise removal, etc.) in advance so that the image processing unit 102 can easily process the information. May be output to the data bus 107, or both may be output. Furthermore, it is also possible to configure so that camera parameters such as an aperture value and a focal length at the time of shooting are sent from the imaging unit 101 to the storage unit 104.

  The imaging unit 101 according to the present invention captures a moving subject a plurality of times and transmits each captured image data to each unit. In the present embodiment, a case where the imaging unit 101 includes two imaging devices will be described as an example.

  The image processing unit 102 is configured by an FPGA (Field Programmable Gate Array) or the like, acquires captured image data from the imaging unit 101 or the storage unit 104 storing the captured image data, and displays the results of various image processing as a data bus. It outputs to 107. Details of the image processing unit 102 will be described later.

The display unit 103 is an LCD (Liquid Crystal Display) or an organic EL display (OELD: Organic ElectroLuminescence).
Display), and displays an image processing result output from the image processing unit 102, an image stored in the storage unit 104, and the like. The display unit 103 may be installed outside the distance measuring apparatus 100 via the external input / output unit 105.

  The storage unit 104 includes, for example, a main storage device such as a RAM (Random Access Memory) and an auxiliary storage device such as a hard disk. The main storage device is used to temporarily hold captured image data and image processing results. The auxiliary storage device stores data for long-term storage as storage, such as photographed image data, image processing results, and data used for camera calibration.

The external input / output unit 105 has input / output ports such as USB (Universal Serial Bus) and HDMI (registered trademark) (High Definition Multimedia Interface), and receives external shooting triggers, receives shot image data and image processing results. Send and so on.
The shooting trigger is input from, for example, a passage sensor that detects passage of a subject. The passage sensor may be configured to transmit a shooting trigger immediately upon detecting that the subject has passed through the measurement range of the passage sensor, for example, according to the positional relationship between the passage sensor and the distance measuring device 100. In addition, it is also possible to configure so that a shooting trigger is transmitted after a predetermined time has elapsed since the subject has passed through the measurement range of the passage sensor.

  The control unit 106 is configured by a CPU (Central Processing Unit) or the like, and performs various commands and controls such as imaging processing in the imaging unit 101 and image processing in the image processing unit 102.

  The data bus 107 is a bus for exchanging data between the units.

The above configuration is the entire configuration in the present embodiment.
Next, a detailed configuration example and operation example of the image processing unit 102 in the present invention will be described.
The present invention is characterized in that the same moving subject is photographed a plurality of times, and the measurement result of the distance (three-dimensional position) in the desired photographing is corrected using the preliminary photographing measurement result. Preliminary shooting and desired shooting are, for example, under the condition of shooting twice, the first shooting being the preliminary shooting and the second shooting being the desired shooting. In addition, by storing the first and second shooting results in the storage unit 104, the preliminary shooting and the desired shooting can be switched, and a plurality of shootings other than the desired shooting are set as the preliminary shooting by shooting three or more times. Thus, it can be configured to correct at the time of desired shooting.
Hereinafter, an example will be described in which shooting is performed twice, in which the first shooting is preliminary shooting and the second shooting is desired shooting.

  As shown in FIG. 2, the image processing unit 102 according to the present embodiment includes a feature point detection unit 201 that detects feature points from captured image data received from the imaging unit 101 or the storage unit 104, and a feature point detection unit 201. The parallax calculation unit 202 that calculates the parallax at the detected feature point, and the imaging parameters of the imaging unit 101 such as the parallax calculated by the parallax calculation unit 202 and the focal length stored in the storage unit 104 and the baseline length between the imaging devices Based on (camera parameter) information, a three-dimensional position information calculation unit 203 that calculates a three-dimensional position for each feature point and measurement error (resolution) information at the three-dimensional position, a feature point detection unit 201, and a parallax calculation unit 202 And a reliability calculation unit 204 that calculates the reliability for each feature point based on the processing information from the three-dimensional position information calculation unit 203. A feature point information holding unit 205 that temporarily holds three-dimensional position information and reliability for each feature point, and feature point information (three-dimensional position information and reliability) at the time of preliminary shooting and desired shooting A three-dimensional position determination unit 206 that receives from the point information holding unit 205 and corrects the three-dimensional position information for each feature point at the time of desired photographing can be configured.

The feature point detector 201 receives stereo image data (captured image data captured by one of the two imaging devices and captured image data captured by the other) from the data bus 107, and features of the captured image data Is detected. Here, the feature point is, for example, edge information, and is determined to be a feature point region when the reaction intensity with respect to the Sobel filter (3 × 3) as shown in FIG. Note that the edge detection method is not limited to the Sobel filter, and can be configured to use any one or more of various feature point data called Prewitt filter, Laplacian filter, Canny filter, and SIFT (Scale-Invariant Feature Transform). .
The feature point detection unit 201 transmits position information and / or feature point image data of the detected feature point in the image coordinates, and, if necessary, captured image data to the parallax calculation unit 202. Also, the feature point strength (reaction strength) at each feature point coordinate is sent to the reliability calculation unit 204.

The parallax calculation unit 202 receives position information and / or feature point image data of the feature point image coordinates from the feature point detection unit 201, and performs stereo matching.
Here, an example of stereo matching will be described with reference to FIG. When the imaging devices in the present embodiment are arranged side by side, first, an image in the range of the search window is obtained based on the feature point position of the left image (the captured image captured by the left imaging device). Cut as a template. Here, the range of the search window is, for example, 11 × 11. Next, a matching evaluation value between the search window range of the right image (captured image captured by the right imaging device) corresponding to the feature point position of the left image and the template is obtained. The matching evaluation value is obtained as a SAD (Sum of Absolute Difference) value, for example, as shown in [Formula 2]. Here, L (x, y) is the search window of the left image at the image coordinates (x, y), and R (x, y) is the gradation of each pixel of the search window of the right image at the image coordinates (x, y). Represents a value.

The matching evaluation value is calculated for each position while shifting the position of interest with respect to a preset search range. When calculating by SAD, the position where the matching evaluation value is minimum is the matching position, and the difference between the matching position in the right image and the original attention point (feature point) position in the left image is parallax.
Note that the method for calculating the matching evaluation value may be a method using various evaluation functions used in stereo matching, and may be calculated using SSD (Sum of Square Difference) or the like.
The parallax calculation unit 202 sends the parallax calculation result for each pixel position (each feature point position) to the three-dimensional position information calculation unit 203. The matching evaluation value for each pixel position (each feature point position) is sent to the reliability calculation unit 204.

  The three-dimensional position information calculation unit 203 calculates the distance Z by [Equation 1] based on the parallax information (parallax calculation result) from the parallax calculation unit 202 and various imaging parameters (camera parameters) from the storage unit 104. To do. Here, the parallax calculated by the parallax calculation unit 202 is a parallax in the image coordinate system and is a pixel value. On the other hand, the parallax in [Equation 1] is a parallax (SI unit such as “mm”) in the sensor coordinate system. Therefore, the distance Z can be calculated by performing conversion from the image coordinate system to the sensor coordinate system using the pixel pitch amount P (mm / pixel) from the storage unit 104 as a parameter. The distance Z corresponds to a coordinate value of Z when the optical axis is parallel to the Z axis. Further, as shown in FIG. 5, the three-dimensional position (X, Y, Z) of the subject can be calculated from the distance Z and the coordinate position (x, y) on the image.

  Further, based on the parallax measurement accuracy (parallax calculation accuracy) in the parallax calculation unit 202, a measurement error at the feature point position is calculated. For example, when the parallax measurement accuracy Np in the parallax calculation unit 202 is a unit of pixels, it can be calculated by [Expression 3].

Here, E is the measurement error, P is the pixel pitch amount, Np is the parallax measurement accuracy, f is the focal length, and Z is the distance to the subject. It can be seen that the measurement error E is proportional to the distance Z. Therefore, the measurement error E increases as the distance Z from the imaging device to the subject increases.
The three-dimensional position information calculation unit 203 transmits the distance to the subject (three-dimensional position) information and measurement error information obtained as described above to the reliability calculation unit 204.

  The reliability calculation unit 204 includes the feature point intensity from the feature point detection unit 201, the matching evaluation value from the parallax calculation unit 202, and the three-dimensional position information and measurement error information from the three-dimensional position information calculation unit 203. Each is received and the reliability is calculated. Specifically, the reliability calculation unit 204 calculates, for example, the reliability I_R for the feature point strength, the reliability M_R for the matching evaluation value, and the reliability E_R for the measurement error information. The degree S_R can be calculated. Examples of the respective reliability calculation formulas are shown in [Formula 4] to [Formula 7].

  [Equation 4] is an example of an equation for calculating the feature point reliability I_R. I_B is the feature point intensity, and IS is the coefficient sum of the feature point detection filter. For example, when the image based on the photographed image data is an 8-bit image and the feature point is detected by the Sobel filter (3 × 3), the feature point intensity I_B is 1020 at the maximum (= 255 + 255 × 2 + 255), and the filter coefficient sum IS is 4 (= 1 + 2 + 1). Therefore, the feature point reliability I_R is a value from 0 to 255, and when it is 255, the feature point reliability is maximum.

[Equation 5] is an example of an expression for calculating the matching reliability M_R. M_B is a matching evaluation value, and N is a search window size. As described above, when the parallax calculation unit 202 calculates the parallax by SAD of N = 11, the matching evaluation value M_B is a value from 0 to 30855. By dividing the matching evaluation value M_B by N ² (= 121) which is the total number of pixels of the search window, the averaged matching evaluation value (the second term on the right side of [Formula 5]) is a value between 0 and 255. Become. As the averaged matching evaluation value is closer to 0, the evaluation of matching is higher. Similarly to the feature point reliability I_R, a value obtained by subtracting the averaged matching evaluation value from 255 is set as the matching reliability M_R so that 255 becomes the maximum value.

  [Equation 6] is an example of an equation for calculating the measurement error reliability E_R. E is the measurement error of the feature point position, E_min is the smallest measurement error in the range designated as the measurement range, and E_max is the largest measurement error in the range designated as the measurement range. For example, consider a case where the range that can be measured as specifications is set to Z = 2000 mm to 5000 mm, the focal length f is 75 mm, the pixel pitch amount P is 0.0005 mm / pixel, and the parallax measurement accuracy Np is 1 pixel. From [Equation 3], the minimum measurement error E_min and the maximum measurement error E_max are 0.133 and 0.333, respectively. The measurement error reliability E_R is a value of 255 when the measurement error E is equal to the minimum measurement error E_min, and is a value of 0 when it is equal to the maximum measurement error E_max.

  [Expression 7] is an example of a formula for calculating the integrated reliability S_R. CI, CM, and CE are arbitrary weighted average coefficients. That is, the integrated reliability S_R is a weighted average value of the feature point reliability I_R, the matching reliability M_R, and the measurement error reliability E_R. The weighting value may be input to the user from the external input / output unit 105, and the value may be stored in the storage unit 104. Among the reliability levels, the minimum reliability level is the integrated reliability level S_R. It can also be set as the structure selected as. Note that the weighting in the case of calculating the integrated reliability S_R by [Equation 7] is preferably set so that the CE weight is larger than the CI and CM weights. This is because the feature point reliability I_R and the matching reliability M_R tend to decrease as the subject moves forward, and conversely, the measurement error reliability E_R tends to increase as the subject moves forward. Therefore, it is desirable to set so that “CI + CM” and “CE” are balanced. For example, CI + CM = CE = 0.5 is set.

When setting so that CI + CM = CE = 0.5, the integrated reliability S_R (X, Y, Z) is a value of 0 to 255, and the closer to 255, the higher the reliability. The higher the edge strength, the higher the matching strength, and the smaller the measurement error of the feature point position, the higher the reliability.
The reliability calculation unit 204 stores the coordinate information X, Y, Z of the feature point and the value of the integrated reliability S_R (X, Y, Z) for each coordinate calculated as described above in the feature point information holding unit 205. send.

  The feature point information holding unit 205 temporarily holds the feature point coordinate information X, Y, Z and the integrated reliability S_R (X, Y, Z) from the reliability calculation unit 204 and uses them at the next photographing. To do. In addition, as reliability data for each shooting, information from the reliability calculation unit 204 can be sent to the storage unit 104 and the data can be stored sequentially.

  The three-dimensional position determination unit 206 does not need to perform any particular processing during preliminary shooting.

  As described above, image processing at the time of preliminary shooting is performed. Image processing results at the time of preliminary photographing (results of three-dimensional position measurement, reliability, processed images, etc.) are stored in the storage unit 104, but are sequentially displayed on the display unit 103 or via the external input / output unit 105. Can be transmitted to an external device.

  Next, image processing during desired shooting will be described. In image processing at the time of desired shooting, the feature point detection unit 201, the parallax calculation unit 202, the three-dimensional position information calculation unit 203, the reliability calculation unit 204, and the feature point information holding unit 205 included in the image processing unit 102 are preliminarily shot. By performing the same processing as the image processing at that time, the coordinate information X ′, Y ′, Z ′ of each feature point in the desired photographing and the integrated reliability S_R (X ′, Y ′, Z ′) for each coordinate are obtained. obtain.

  The three-dimensional position determination unit 206 receives coordinate information X, Y, and Z of feature points in preliminary shooting, their integrated reliability S_R (X, Y, Z), and feature points in desired shooting from the feature point information holding unit 205. Coordinate information X ′, Y ′, Z ′ and its integrated reliability S_R (X ′, Y ′, Z ′) are received, and the three-dimensional position information is determined.

  The coordinate information X, Y, Z of the feature points in the preliminary shooting and the coordinate information X ′, Y ′, Z ′ of the feature points in the desired shooting are coordinate data in the image pickup apparatus coordinate system C. Cannot be compared. Therefore, as shown in FIG. 6, coordinate transformation to the subject coordinate system L is performed using [Equation 8] to [Equation 11].

Here, X _L1 , Y _L1 , and Z _L1 are coordinate values in the subject coordinate system L in preliminary shooting, X _L2 , Y _L2 , and Z _L2 are coordinate values in the subject coordinate system L in desired shooting, r ₁₁ , r ₁₂ , r ₁₃ , r ₂₁ , r ₂₂ , r ₂₃ , r ₃₁ , r ₃₂ , r ₃₃ are rotation matrix coefficients to the subject coordinate system L in the preliminary shooting, r ₁₁ ′, r ₁₂ ′, r ₁₃ ′, r ₂₁ ′, r ₂₂ ′, r ₂₃ ′, r ₃₁ ′, r ₃₂ ′, r ₃₃ ′ are rotation matrix coefficients to the subject coordinate system L in the desired shooting, and tx, ty, tz are the subject coordinates in the preliminary shooting. The translation vector coefficients tx ′, ty ′, and tz ′ to the system L are translation vector coefficients to the subject coordinate system L in the desired shooting. RT ₁ and RT ₂ in [Equation 9] and [Equation 11] are transformation matrices from the imaging apparatus coordinate system C to the subject coordinate system L, and vice versa as shown in [Equation 12] and [Equation 13]. By using the matrix, the inverse transformation from the subject coordinate system L to the imaging device coordinate system C can be performed. Therefore, since the three-dimensional position information from the coordinate value in the subject coordinate system L to the subject feature point in the imaging device coordinate system C can be obtained, by determining the coordinate value in the subject coordinate system L, the imaging device coordinate system The three-dimensional position information up to the subject feature point in C is also determined.

  Using each coordinate value in the subject coordinate system L obtained by the above conversion and its integrated reliability S_R, a coordinate value in the desired shooting is determined. Next, the determination method will be described.

FIG. 7 is a flowchart illustrating an example of the operation in the three-dimensional position determination unit 206.
In step S701, a feature point (attention point) in the subject coordinate system is selected. As the attention point, for example, an attention point list in which feature points are arranged in descending order of reliability is created, and the attention point is designated as the attention point from the feature points having high reliability. At this time, the feature points listed in the attention point list may be only the feature points at the time of desired shooting. However, if the feature points at the time of preliminary shooting are also listed, they can be detected at the time of desired shooting. Since it is possible to obtain feature points with high reliability that were not present, it is preferable. Further, by setting a threshold value and not listing feature points with low reliability (feature points less than the threshold value), it is also possible to present only information having a certain reliability level to the user.

  In step S702, a comparison point is selected from the feature points at the time of preliminary shooting and the feature points at the time of desired shooting. Specifically, a feature point within an arbitrary three-dimensional distance range from the feature point (attention point) selected in step S701 is selected as a comparison target (comparison point). For example, as shown in FIG. 8A, a feature point Q included in an arbitrary three-dimensional distance range R at the feature point P (attention point P) is selected as a comparison target. As shown in FIG. 8B, when there is no feature point within the three-dimensional distance range R, “no comparison object (comparison point)” is set. As shown in FIG. 8C, when there are a plurality of feature points within the three-dimensional distance range R, all of the plurality of feature points (in the case of the example shown in FIG. 8C, the feature points Q1 and the feature points). Q2) and both) are selected for comparison. Here, the three-dimensional distance range R is set based on, for example, the measurement accuracy range or allowable error of the distance measuring device 100. For example, when the measurement accuracy is 0.1 mm, it is preferable to set the three-dimensional distance range R to 0.1 mm because highly reliable feature point coordinate data can be obtained in units of measurement accuracy.

In step S703, the reliability of the feature point (attention point) selected in step S701 and the feature point (comparison point) selected in step S702 are compared.
In step S704, the coordinate value of the feature point (attention point) is determined based on the comparison result in step S703.

For example, as shown in FIG. 8A, when one comparison target (feature point Q) is found, the integrated reliability S_RP at the feature point _P and the integrated reliability S_RQ at the feature point _Q are compared, and the reliability A feature point having a high value is defined as a post-determined feature point P ′. Alternatively, as shown in [Equation 14] and [Equation 15], the determined feature point P ′ and its reliability may be obtained using the respective reliability as weights.

Here, Xp ′, Yp ′, and Zp ′ are the coordinate values of the determined feature point P ′, Xp, Yp, and Zp are the coordinate values of the feature point P, Xq, Yq, and Zq are the coordinate values of the feature point Q, and S_R _{P '} Is the post-determining reliability (post-determined integrated reliability) for the post-determining feature point P'
Further, for example, as shown in FIG. 8B, in the case of “no comparison object”, P ′ = P and no particular determination is performed.
For example, as shown in FIG. 8C, when a plurality of comparison targets (feature points Q1 and Q2) are found, the feature point P, the feature point Q1, and the feature point are the same as in FIG. 8A. The feature point with the highest reliability in Q2 may be set as the feature point P ′ after determination, or the feature point P ′ after determination and the reliability thereof may be obtained using the reliability as a weight.

Next, the three-dimensional position determining unit 206 sends the obtained coordinate value of the post-determined feature point P ′ and its reliability S_RP _′ to the storage unit 104, and deletes the feature point selected for comparison from the attention point list. To do.
Then, the three-dimensional position determination unit 206 performs the processing from step S701 to step S704 for all attention points to determine the feature point coordinates.

  As described above, by holding the feature point coordinates in the subject coordinate system L, the distance measuring device 100 calculates the distance between feature points (the distance between two points on the subject, the distance between subjects, etc.). be able to. Here, the distance between two points on the subject (on the subject surface) is the thickness and width of the subject. Further, the distance measuring apparatus 100 is configured to convert the distance from the imaging apparatus by performing coordinate conversion to the imaging apparatus coordinate system C based on [Equation 13] as described above, and send the distance to the storage unit 104. Thus, the distance from the imaging device to the subject can be calculated.

In the above description, the description is given on the assumption that the subject is moving. However, even when the distance measuring device 100 moves while the subject is stationary, the subject is adapted because the movement is relatively generated. Is possible.
In the above description, the case of shooting twice has been described. However, preliminary shooting can be performed a plurality of times, or an image in the current frame can be obtained in a desired image, and an earlier frame can be obtained. It can be easily assumed that a distance can be measured successively in a moving image by processing the image as a captured image obtained by preliminary shooting.

  With the above configuration, the subject moving in the shooting space is shot a plurality of times, and the feature point coordinates and the reliability of each of the shot images obtained by the plurality of shootings are obtained, and based on the feature point coordinates and the reliability. By determining the feature point coordinates of the photographed image at a desired photographing timing, the distance (distance from the imaging device to the subject, the distance between two points on the subject, or the distance between subjects), and its reliability, It is possible to provide a distance measuring device and a distance measuring method capable of measuring the distance.

  According to the distance measuring apparatus 100 of the first embodiment described above, the imaging unit 101 that acquires a plurality of photographed images over time, and a subject point detection unit that detects points on the subject in the plurality of photographed images ( A distance measuring device including a feature point detection unit 201) and a three-dimensional position calculation unit (three-dimensional position information calculation unit 203) that calculates a three-dimensional position of a point on the subject, the calculated three-dimensional A reliability calculation unit 204 that calculates position reliability (integrated reliability S_R), and a three-dimensional position determination unit 206 that determines the calculated three-dimensional position are further provided. Based on the three-dimensional position of an arbitrary comparison point on the subject in a plurality of captured images and the reliability thereof, the three-dimensional position of the arbitrary attention point on the subject in the captured image To confirm It is configured. Then, the distance measuring device 100 calculates a distance (a distance from the imaging device to the subject, a distance between two points on the subject, or a distance between subjects) based on the three-dimensional position determined by the three-dimensional position determining unit 206. It is configured to calculate.

Therefore, since the distance (distance from the imaging device to the subject, distance between two points on the subject, or distance between subjects) can be calculated based on the determined three-dimensional position, measurement with higher reliability can be performed. A distance result can be obtained.
Here, “a plurality of photographed images” acquired over time are a photographed image obtained by preliminary photographing and a photographed image obtained by desired photographing.
Note that the captured image acquired by the imaging unit 101 in one shooting is not limited to the left image and the right image, and may be any captured image having parallax.

  Further, according to the distance measuring apparatus 100 of the first embodiment, there are a plurality of comparison points, and at least one point is detected from an image different from the captured image where the point of interest exists. It can be configured as follows.

  With such a configuration, it is possible to obtain a distance measurement result with higher reliability.

Further, according to the distance measuring apparatus 100 of the first embodiment, the reliability (integrated reliability S_R) can be configured to be a reliability based on a theoretical error due to the distance to the subject.
Specifically, as described above, the feature point reliability I_R and the matching reliability M_R tend to decrease as the subject moves forward (that is, as the distance decreases), and conversely, the measurement error reliability E_R is determined by the subject. There is a tendency to go up as it moves forward (that is, as the distance gets closer). Since the integrated reliability S_R is calculated based on the feature point reliability I_R, the matching reliability M_R, and the measurement error reliability E_R, the integrated reliability S_R can be a reliability based on a theoretical error due to the distance to the subject.

  With this configuration, the distance (the distance from the imaging device to the subject, the distance between two points on the subject, or the distance between subjects is determined based on the three-dimensional position determined in consideration of the theoretical error due to the distance to the subject. Distance) can be calculated, so that a more reliable distance measurement result can be obtained.

  In addition, according to the distance measuring method in the first embodiment described above, a photographing process of photographing a subject by the imaging unit 101 that acquires a plurality of photographed images over time, and a point on the subject in the photographed image is detected. A detecting step for calculating, a three-dimensional position calculating step for calculating a three-dimensional position of a point on the subject, a reliability calculating step for calculating a reliability of the calculated three-dimensional position, and a determined three-dimensional position. And a calculation step for calculating a distance to the subject, a distance between two points on the subject, or a distance between the subjects based on the determined three-dimensional position. The three-dimensional position of an arbitrary point of interest on the subject in the image is determined based on the three-dimensional position of an arbitrary comparison point on the subject in a plurality of photographed images that exists within an arbitrary range from the target point and its reliability. It is configured so that.

  Therefore, since the distance (distance from the imaging device to the subject, distance between two points on the subject, or distance between subjects) can be calculated based on the determined three-dimensional position, measurement with higher reliability can be performed. A distance result can be obtained.

Second Embodiment: Motion Blur Estimation In this embodiment, a feature point detection filter setting method that takes motion blur into account will be described. Specifically, by estimating the subject speed from captured images obtained by multiple shootings, the motion blur in the desired shooting is estimated, and the size and shape of the feature point detection filter are changed according to the motion blur. A method for appropriately detecting feature points will be described.

As shown in FIG. 9, the image processing unit 102 in the present embodiment includes a motion blur estimation unit 901 that estimates motion blur from a plurality of shooting results in addition to the configuration in the first embodiment (FIG. 2). Furthermore, it comprises so that it may comprise.
An example of the operation in the motion blur estimation unit 901 will be described with reference to FIGS.

In step S <b> 1001, the motion blur estimation unit 901 predicts a desired shooting speed from each feature point position obtained by multiple shootings.
Here, as a specific example, a method for predicting the speed based on the measurement result of one preliminary shooting and the provisional measurement result of desired shooting will be described. As shown in FIG. 11, an arbitrary preliminary photographing three-dimensional coordinate (X1, Y1, Z1) is selected, and the image coordinates (u1, v1) corresponding to the preliminary photographing three-dimensional coordinate are set as a search start position. In the same manner as the motion vector detection means, the corresponding coordinates (image coordinates (u2, v2)) in the captured image obtained by the desired imaging are searched. The three-dimensional coordinates (desired photographing three-dimensional coordinates (X2, Y2, Z2)) corresponding to the image coordinates (u2, v2) and the Euclidean distance of the preliminary photographing three-dimensional coordinates (X1, Y1, Z1), and the preliminary photographing and the desired photographing. The velocity vector can be calculated based on the shooting time interval. The conversion from the three-dimensional coordinates to the image coordinates can be performed by using [Equation 1], [Equation 12], [Equation 13], and [Equation 16].

  Here, (u, v) are image coordinates, (X, Y, Z) are three-dimensional coordinates, f is a focal length, xo, yo are images corresponding to image coordinates (0, 0) from the center coordinates of the image element. The element coordinate offsets pw and ph are the width and height of one pixel of the image element.

  In step S1002, the motion blur estimation unit 901 estimates a motion blur vector based on the velocity vector information obtained in step S1001. The motion blur estimation unit 901 receives the shutter speed information of the imaging apparatus from the storage unit 104, and the three-dimensional after the desired shooting three-dimensional coordinates (X2, Y2, Z2) move between the shutter speeds as shown in FIG. The coordinates (X3, Y3, Z3) are calculated. Three-dimensional coordinates (X2, Y2, Z2) and (X3, Y3, Z3) are transformed into corresponding image coordinates (u2, v2) and (u3, v3), respectively, and image coordinates (u2, v2) and A motion blur vector is estimated from the distance (u3, v3).

  In step S1003, the motion blur estimation unit 901 generates filter size change information based on the motion blur information obtained in step S1002, and sends the filter size change information to the feature point detection unit 201. For example, when the size of motion blur (distance between two points) is 3 pixels, the original feature point coordinates (desired shooting three-dimensional coordinates (X2, Y2, Z2)) are the center of the motion blur range. Assuming that the total amount of motion blur is about 6 pixels. Therefore, 8 × 8 pixel information is generated as change information of the filter size as the motion blur range + one surrounding pixel, and is transmitted to the feature point detection unit 201.

  The above method is particularly suitable when the direction component of the motion blur vector is large in the Z-axis direction (the optical axis direction of the imaging device). On the other hand, when the direction component of the motion blur vector is mainly large in the X-axis direction, the filter is deformed in the X-axis direction (for example, 3 × 8), and the direction component of the motion blur vector is mainly large in the Y-axis direction. In this case, the filter may be deformed in the Y axis direction (for example, 8 × 3). For example, when a large motion blur occurs in the X-axis direction, it can be assumed that the influence of the feature points in the Y-axis direction is small. Therefore, the amount of calculation can be reduced by limiting the size deformation (enlargement) direction of the filter in this way.

  In the above, one point is designated as an arbitrary three-dimensional coordinate, and the example at the desired timing from the matching point on the image has been described. For example, the speed is predicted from the movement amount of the center of gravity position of a plurality of feature points of the subject. You may do it. However, since the feature point information includes motion blur, there is a possibility that the feature point position with high accuracy is not necessarily detected, and the motion blur direction is not always estimated. Therefore, if the configuration is such that the size of the motion blur vector is ignored and the filter size is enlarged or deformed at an arbitrary step size (for example, enlargement of one pixel) according to the direction of the motion blur vector, only the determination process is performed. Can be handled, so the processing can be simplified.

  The feature point detection unit 201 receives the filter size change information, and again detects feature points with respect to the captured image obtained by the desired imaging, so that feature point information with higher reliability can be obtained. .

  In the above description, the feature point of the desired shooting is re-detected from the measurement results of the preliminary shooting and the desired shooting. For example, as shown in FIG. Shooting B), calculating a velocity vector in preliminary shooting B, and predicting a three-dimensional position at the time of desired shooting and a three-dimensional position after the shutter based on the speed vector and a shooting time interval until the desired shooting. Thus, when the motion blur is estimated and the filter size and shape at the time of desired shooting are set in advance, the motion blur vector can be calculated when the position of the preliminary shooting B can be assumed to be equal to or the same acceleration motion. It becomes the structure which can be estimated more correctly.

  With the above configuration, by estimating the subject speed from captured images obtained by multiple shootings, motion blur in desired shooting is estimated, and the size and shape of the feature point detection filter are changed according to the motion blur. Thus, it is possible to detect the feature point more appropriately and measure the distance from the imaging device to the subject, the distance between two points on the subject, or the distance between the subjects and the reliability thereof. An apparatus and a distance measurement method can be provided.

  According to the distance measuring apparatus 100 of the second embodiment described above, calculation is performed corresponding to each of a plurality of photographed images (a photographed image obtained by preliminary photographing and a photographed image obtained by desired photographing). A motion blur estimation unit 901 that estimates motion blur by predicting the moving direction of the subject with respect to the imaging unit 101 from a plurality of three-dimensional positions, and based on the estimation result of the motion blur estimation unit 901, attention points (feature points) Is configured to change at least one of the size and shape of the target point detection window (feature point detection filter) used when detecting.

  Therefore, since the attention point can be detected in consideration of the influence of motion blur, the attention point can be detected more appropriately, and a ranging result with higher reliability can be obtained.

<Third embodiment>: Specification by user In the present embodiment, each of the feature point groups designated by the user has a display means and an input means for designating a feature point that the user wants to measure. By configuring to determine only the measurement results of the feature points included in the feature points or the straight lines, curves, or curved surfaces (including planes) formed by those feature points and their error ranges, the amount of calculation is reduced. A method of reducing will be described. In addition, a display method for facilitating designation of feature points by the user by highlighting feature points with high reliability (for example, enclosing them with a circle having a size corresponding to the high reliability level) will be described.

As shown in FIG. 14, the image processing unit 102 according to the present embodiment has a display image generation unit that generates an image for the user to specify a feature point in addition to the configuration according to the first embodiment (FIG. 2). 1401 and a definite feature point selection unit 1402 that selects a feature point group to be corrected based on a user-specified feature point.
In addition to the configuration in the second embodiment (FIG. 9), a display image generation unit 1401 and a deterministic feature point selection unit 1402 can be further provided.

A display image generation unit 1401 generates an image for designating a feature point that the user wants to measure, and sends the generated image to the display unit 103. Thereby, for example, an image as shown in FIG. 15 is displayed on the display unit 103. Here, reference numeral 1501 denotes a captured image area, and reference numeral 1502 denotes a feature point designation setting area. In the photographed image area 1501, a photographed image and a circle surrounding a feature point having a certain reliability or higher are displayed. Here, a circle 1504 surrounding a feature point designated by the user is represented by a solid line, while a circle 1503 surrounding a feature point not designated is represented by a broken line. In addition, the larger the size of the circle, the higher the reliability of the feature point. The reliability threshold value of the feature point to be displayed can be designated by the user, for example, using a GUI (eg, slide bar 1505) displayed in the feature point designation setting area 1502. Further, the user designates an interpolation method using the interpolation method selection unit 1506.
When the user selects the measurement button 1507, the three-dimensional positions of the feature points specified by the user and the feature points in contact with the curved surface or the like formed by these feature points are determined.

The definite feature point selection unit 1402 performs straight line estimation, curve estimation, or curved surface (including plane) estimation based on the feature points specified in the captured image region 1501 and the interpolation method specified by the interpolation method selection unit 1506. For example, in the case of the example shown in FIG. 15, the designated feature points are 3 points (3 out of 4 corners on the upper surface of the cube), and the designated interpolation method is “plane”. Do. The plane can be estimated by calculating the outer product of two vectors formed when one of the three points is the origin.
Subsequently, the definite feature point selection unit 1402 refers to the feature point information held in the feature point information holding unit 205 and selects a feature point group that touches the estimated straight line, curve, or curved surface (including a plane). . For example, when plane estimation is performed, a feature point group in contact with this plane is selected. At this time, feature point candidates within a certain distance in the normal direction from this plane are selected. The constant distance is, for example, a distance estimation error range required for the system, and is set within 0.1 mm, for example. Alternatively, since the estimation error of the distance per pixel of parallax increases as the distance increases, it is preferable that the range be expanded as the distance increases.

Then, only the feature points selected in this way are determined by the three-dimensional position determination unit 206, and the result is stored in the feature point information storage unit 205. The confirmed feature point information is displayed on the display unit 103 as shown in FIG. 16, for example. In FIG. 16, reference numeral 1601 denotes a plane on which the three-dimensional position is calculated. Here, by designating a feature point that the user wants to measure on this plane, the three-dimensional position information is displayed on the image. Further, when a position (point) that is an area (point) on this plane and is not a feature point is designated by the user, for example, information on the closest feature point may be displayed. For example, it is also possible to display the result of interpolation using information on the neighboring feature points in the four directions of the specified position.
With the above configuration, it is possible to provide a display method that makes it easy for the user to specify feature points.

  According to the distance measuring apparatus 100 of the third embodiment described above, the user can designate a point of interest (feature point).

Therefore, since the point of interest that the user is interested in is known, it is only necessary to determine the three-dimensional position of the point of interest, and the amount of calculation can be reduced.
It should be noted that the method of designating the feature points by the user can be selected as appropriate. For example, the feature points can be designated by operating the operation unit (such as a mouse) connected to the external input / output unit 105 by the user. The display unit 103 may be a touch panel type display unit.

<Fourth embodiment>
The distance measuring device 100 described in the above embodiments (first to third embodiments) can be used, for example, for measuring the thickness of a sliding plate portion of a pantograph of a vehicle.

Specifically, an image photographed in response to a photographing trigger from a passage sensor that has detected entry of a pantograph into the photographing range (of course, this image includes a pantograph) is used.
Alternatively, a plurality of images are photographed over time with an imaging device, and an image in which a pantograph is reflected is used among the plurality of images. At this time, for example, it may be detected by pattern matching of a characteristic part, or, for example, as shown in FIG. It can be configured to determine an image in which a pantograph is shown from the variation of the region.

The standard uses a fixed part. A straight line is detected, and a straight line area of a sliding plate is used. A plane is defined, and the thickness from the plane to each edge of the sliding plate is measured. Specifically, first, as shown in FIG. 18A, a straight edge in the longitudinal direction of the sliding plate is extracted. Next, as shown in FIG. 18B, a reference surface is defined. Then, as shown in FIG.18 (c), the distance (thickness) to each position is calculated.
As shown in FIG. 19, the thickness profile in the preliminary shooting and the thickness profile in the desired shooting are matched in the longitudinal direction of the sliding plate (for example, a corresponding position is calculated by obtaining a cross-correlation coefficient or the like). And adjust the position from there.

  In each of the above-described embodiments, the configuration and the like illustrated in the accompanying drawings are merely examples, and are not limited to these, and can be appropriately changed within the scope of the effects of the present invention. It is. In addition, various modifications can be made without departing from the scope of the object of the present invention.

DESCRIPTION OF SYMBOLS 100 Distance measuring device 101 Imaging part 201 Feature point detection part (Subject point detection part)
203 3D position information calculation unit (3D position calculation unit)
204 reliability calculation unit 206 three-dimensional position determination unit 901 motion blur estimation unit

Claims (6)

An imaging unit that acquires a plurality of captured images over time;
A subject point detector for detecting a point on the subject in the captured image;
A three-dimensional position calculation unit that calculates a three-dimensional position of a point on the subject,
A reliability calculation unit for calculating the reliability of the calculated three-dimensional position;
A three-dimensional position determining unit for determining the calculated three-dimensional position;
The three-dimensional position determining unit is configured to arbitrarily set a three-dimensional position of an arbitrary point of interest on the subject in the plurality of photographed images within an arbitrary range from the point of interest. A distance measuring apparatus characterized in that it is determined based on a three-dimensional position of a comparison point and its reliability.   The distance measuring apparatus according to claim 1, wherein the comparison points are a plurality of points, and at least one point is detected from an image different from a captured image in which the point of interest exists.   The distance measuring device according to claim 1, wherein the reliability is a reliability based on a theoretical error depending on a distance to a subject. A motion blur estimation unit that estimates motion blur by predicting a moving direction of the subject relative to the imaging unit from a plurality of the three-dimensional positions calculated corresponding to each of the plurality of captured images;
4. The method according to claim 1, wherein at least one of a size and a shape of an attention point detection window used when detecting the attention point is changed based on an estimation result of the motion blur estimation unit. The distance measuring device according to claim 1.   The distance measuring device according to any one of claims 1 to 4, wherein a user can designate the attention point. A shooting process of shooting a subject by an imaging unit that acquires a plurality of shot images over time;
A detection step of detecting a point on the subject in the captured image;
A three-dimensional position calculating step for calculating a three-dimensional position of a point on the subject;
A reliability calculation step of calculating the reliability of the calculated three-dimensional position;
Confirming the calculated three-dimensional position, and
In the determination step, a three-dimensional position of an arbitrary point of interest on the subject in the plurality of captured images is an arbitrary comparison point on the subject in the plurality of captured images that exists within an arbitrary range from the point of interest. A distance measuring method characterized in that it is determined based on a three-dimensional position and its reliability.