JP6694234B2 - Distance measuring device - Google Patents

Description

The present invention, by utilizing a plurality of images, relates to the distance measuring equipment capable of outputting a distance.

A multi-viewpoint imaging device capable of shooting an arbitrary subject from a plurality of viewpoints is useful for generating a stereoscopic image, a free-viewpoint image, and a depth image (depth map) required to realize them. Used.
In such a system, based on the principle of triangulation, by using the parameters of the image pickup device and the disparity information (also sometimes referred to as disparity or depth) of each imaged image, from the image pickup device to the subject. It is also possible to calculate the distance between and the distance between the subjects. The relationship between the parameters of the imaging device and the distance from the imaging device to the subject can be expressed by [Equation 1].

Here, d is parallax, b is the base length between the image pickup devices, f is the focal length of the image pickup device, and Z is the distance from the image pickup device to the subject. [Equation 1] is equivalent to the distance calculation formula when the optical axes of the image pickup apparatus are arranged in parallel, but when the optical axis of the image pickup apparatus is arranged at an angle, the camera calibration or the like is performed. The same can be applied by calculating the parameters and subjecting the images to parallelization.
Furthermore, three-dimensional position information can be calculated based on parameters of the image pickup device such as the angle of view and lens distortion. XY coordinate values can also be calculated using the angle between the optical axis and the projection point, the focal length f, and the distance Z. If three-dimensional position information can be obtained, the distance between any two points in the shooting space can be calculated.

The parallax as described above can be obtained by stereo matching processing between images. In stereo matching, one image arbitrarily determined is used as a reference image, the other is used as an evaluation image, a partial image of a region of interest of the reference image (for example, set to 5 × 5) is used as a template, and corresponding points of the evaluation image ( This is a method of searching the corresponding area) based on an evaluation function (for example, SAD: Sum of Absolute Difference), and the distance between the attention area and the corresponding area corresponds to the parallax.
In the present stereo matching, if the partial image does not include a characteristic part (for example, in the case of a plane having the same brightness and the same color), matching is performed in a plurality of regions and the true position is not detected. The likelihood of matching is not always maximized. That is, the correct parallax may not be obtained.
On the other hand, in the case of a partial image including a characteristic portion such as an edge, it is possible to obtain parallax information with relatively high accuracy, except when occlusion occurs or when a repetitive pattern is formed.

Patent Document 1 is disclosed as a distance measuring system using an edge portion of an image.
In Patent Document 1, two images input from a multi-viewpoint imaging device are used as original images, a resized image in which the image size of one of the images is changed is prepared, and edges (feature end points) are extracted from the resized image. .. The corresponding region of the image (original image) before deformation corresponding to the extracted edge is set as a matching pattern of arbitrary size, and the parallax and the distance are calculated by performing stereo matching with the other image.
With the above configuration, it is possible to improve parallax detection power of a diagonal edge portion by using the vertical and horizontal edge detection kernels without preparing an edge detection kernel for detecting a diagonal edge for each angle.

JP, 2003-98424, A

In the invention described in Patent Document 1, it is possible to measure the distance by using the edge. However, as a problem with respect to Patent Document 1, the influence of motion blur can be considered.
(1) Effect of motion blur itself When the subject is moving, motion blur occurs near the edge. In particular, when the subject is moving at high speed, motion blurring occurs over a plurality of pixels on the image, and it cannot be extracted as an edge (characteristic region), so that distance measurement cannot be performed. Depending on the setting of the edge detection kernel, it may be possible to extract as an edge, but even if it can be extracted as an edge, it is not guaranteed that it is the true position of the edge when no blur occurs. Therefore, it is considered that the reliability of the distance measurement result decreases.
(2) Influence of the moving direction of the image pickup device and the subject The amount of motion blur becomes maximum when the optical axis of the image pickup device and the moving direction of the subject are orthogonal to each other, and when the optical axis of the image pickup device and the moving direction of the subject are parallel to each other. Is the smallest. Therefore, in order to reduce the influence of motion blur, it is desirable to install the image pickup device at a position where the optical axis of the image pickup device and the moving direction of the subject are parallel to each other. That is, assuming that the subject is moving in the front direction of the subject, it is desirable to place the imaging device in front of the subject. However, when it is desired to set the side surface of the subject as the shooting range, the side surface of the object cannot always be placed in front.

The present invention has been made in view of the above problems, it is an object of the present invention is to provide a distance measuring equipment capable of obtaining a more reliable measurement result.

In order to solve the above problems, the distance measuring device of the present invention,
An imaging unit that acquires a plurality of images taken from a plurality of directions,
A subject point detection unit that detects a point on the subject in the image,
A three-dimensional position calculation unit that calculates a three-dimensional position of a point on the subject by parallax measurement of the plurality of images ,
Reliable based on the measurement error E = P · Np · Z / f (P is the pixel pitch amount, Np is the parallax measurement accuracy, f is the focal length, and Z is the distance to the subject) at the three-dimensional position of the point on the subject. Calculate the degree,
And the three-dimensional position, on the basis of the degrees of reliability, to determine the coordinates of a point on the object, and is configured to output a distance.

Therefore, it is possible to obtain a good Ri reliable distance measurement result.

Alternatively, the distance measuring device of the present invention is
An imaging unit that acquires a plurality of images taken over time,
A subject point detection unit that detects a point on the subject in the image,
A three-dimensional position calculation unit that calculates a three-dimensional position of a point on the subject by parallax measurement of the plurality of images ,
Reliable based on the measurement error E = P · Np · Z / f (P is the pixel pitch amount, Np is the parallax measurement accuracy, f is the focal length, and Z is the distance to the subject) at the three-dimensional position of the point on the subject. Calculate the degree,
And the three-dimensional position, on the basis of the degrees of reliability, to determine the coordinates of a point on the object, and is configured to output a distance.

Therefore , a more reliable distance measurement result can be obtained.

Preferably,
The reliability of the three-dimensional position of the point of interest is calculated by parallax measurement based on the plurality of images during the first shooting, and the parallax measurement is performed based on the plurality of images different from the time during the first shooting during the second shooting. It is possible to calculate the reliability of the three-dimensional position of the comparison point , determine the coordinate value of the point on the subject based on the comparison result of these reliability, and output the distance.

According to such a constitution, it is possible to obtain a high measurement result of good Ri reliability.

Also, preferably,
The distance is a distance between the distance measuring device and a point on the object, a distance between a point on the object and another point on the object, or a point on the object and other points in the image. Can be configured to be the distance between points on the subject .

With this configuration, it is possible to output the distance to the subject, the distance between two points on the subject, or the distance between the subjects .

Also, preferably ,
The reliability of the point on the object, it is possible to configure so as to display as the magnitude of the reliability is found.

With this configuration, the user can know the degree of reliability .

  According to the present invention, it is possible to obtain a highly reliable distance measurement result.

It is a figure which shows one structural example of the distance measuring device in 1st Embodiment. It is a figure which shows one structural example of the image processing part with which the distance measuring device shown in FIG. 1 is equipped. It is a figure which shows an example of an edge detection filter. It is a figure which shows an example of a stereo matching. It is a figure which shows an example of the calculation method of a three-dimensional position of a to-be-photographed object. It is a figure which shows an example of the coordinate conversion method from the imaging device coordinate system C to the to-be-photographed coordinate system L. It is a flow chart which shows an example of operation in a three-dimensional position deciding part. It is a figure which shows an example of the selection method of a comparison point. It is a figure which shows one structural example of the image processing part with which the distance measuring device in 2nd Embodiment is equipped. It is a figure which shows an example of operation | movement in a motion blur estimation part. It is a figure which shows an example of operation | movement in a motion blur estimation part. It is a figure which shows an example of operation | movement in a motion blur estimation part. It is a figure which shows an example of operation | movement in a motion blur estimation part. It is a figure which shows one structural example of the image processing part with which the distance measuring device in 3rd Embodiment is equipped. It is a figure which shows an example of the image for which the user designates the feature point which a user wants to measure displayed on the display part. It is a figure which shows an example of the confirmed characteristic point information displayed on the display part. It is a figure which shows an example of the measuring method of the thickness of the sliding plate part of the pantograph of a vehicle. It is a figure which shows an example of the measuring method of the thickness of the sliding plate part of the pantograph of a vehicle. It is a figure which shows an example of the measuring method of the thickness of the sliding plate part of the pantograph of a vehicle.

  Embodiments according to the present invention will be described below with reference to the drawings.

<First Embodiment>: Basic Configuration In this embodiment, a basic configuration of the present invention will be described.
Specifically, a subject moving in the photographing space is photographed multiple times, the feature point coordinates and reliability of each of the photographed images obtained by the plurality of photographings are obtained, and based on the feature point coordinates and reliability. By correcting the characteristic point coordinates of the captured image at the desired capturing timing, the distance (distance to the subject, distance between two points on the subject, or distance between the subjects) and its reliability are measured. A distance measuring device and a distance measuring method that can be used will be described.

First, the overall configuration of the present invention will be described.
FIG. 1 is a configuration example of a distance measuring device according to the present embodiment. The distance measuring device 100 is a distance measuring device according to the present embodiment, and includes an image capturing unit 101 for capturing an image of an object, an image processing unit 102 for processing a captured image that has been captured, a captured image itself and an image processing result. A display unit 103 for displaying an output, a storage unit 104 for storing a captured image itself, an image processing result, and various data used for image processing, and an external input for user input and data transmission to an external device. The output unit 105, a control unit 106 for controlling the distance measuring device 100 as a whole, and a data bus 107 for exchanging data between the respective units.

The image capturing unit 101 is configured to include a plurality of image capturing elements such as an optical component for capturing a capturing space in the distance measuring device 100 and a CMOS (Complementary Metal Oxide Semiconductor), and each captured image data obtained by photoelectric conversion. Is output to the data bus 107. Although the plurality of optical components and the image pickup device may be arranged in one housing, they are configured as a plurality of image pickup devices having different photographing positions. That is, the image pickup unit 101 has a plurality of image pickup devices each including an optical component and an image pickup element, and these image pickup devices are arranged at positions where picked-up images having a parallax with each other can be picked up.
The captured information may be output to the data bus 107 as raw data as it is, or image data that has been subjected to image processing (luminance imaging, noise removal, etc.) in advance so that it can be easily processed by the image processing unit 102. May be output to the data bus 107 or both of them may be output. Furthermore, the camera parameters such as the aperture value and the focal length at the time of shooting can be configured to be sent from the imaging unit 101 to the storage unit 104.

  The image capturing unit 101 according to the present invention captures a moving subject a plurality of times, and transmits each captured image data to each unit. In the present embodiment, a case where the image pickup unit 101 includes two image pickup devices will be described as an example.

  The image processing unit 102 is configured by an FPGA (Field Programmable Gate Array) or the like, acquires captured image data from the image capturing unit 101 or a storage unit 104 in which captured image data is stored, and outputs the results of various image processing to a data bus. To 107. Detailed description of the image processing unit 102 will be described later.

The display unit 103 is an LCD (Liquid Crystal Display) or an organic EL display (OELD: Organic Electro Luminescence).
Display), and displays the image processing result output from the image processing unit 102, the image saved in the saving unit 104, and the like. The display unit 103 may be installed outside the distance measuring device 100 via the external input / output unit 105.

  The storage unit 104 is composed of, for example, a main storage device such as a RAM (Random Access Memory) and an auxiliary storage device such as a hard disk. The main storage device is used to temporarily hold captured image data and image processing results. The auxiliary storage device stores data for long-term storage as storage, such as captured image data, image processing results, and data used for camera calibration.

The external input / output unit 105 has an input / output port such as a USB (Universal Serial Bus) or HDMI (registered trademark) (High Definition Multimedia Interface), and receives an external shooting trigger and receives captured image data and image processing results. Send, etc.
The shooting trigger is input from, for example, a passage sensor that detects passage of an object. The passage sensor may be configured to transmit an imaging trigger as soon as it detects that the subject has passed the measurement range of the passage sensor, for example, according to the positional relationship between the passage sensor and the distance measuring device 100. However, it is also possible to configure so that the photographing trigger is transmitted after a lapse of a predetermined time after detecting that the subject has passed the measurement range of the passage sensor.

  The control unit 106 is configured by a CPU (Central Processing Unit) or the like, and performs various commands / controls such as imaging processing in the imaging unit 101 and image processing in the image processing unit 102.

  The data bus 107 is a bus for exchanging data between the units.

The above configuration is the overall configuration of the present embodiment.
Next, a detailed configuration example and operation example of the image processing unit 102 according to the present invention will be described.
The present invention is characterized in that the same moving subject is photographed a plurality of times and the measurement result of the preliminary photographing is used to correct the measurement result of the distance (three-dimensional position) in the desired photographing. Preliminary shooting and desired shooting are, for example, when the shooting is performed twice, the first shooting is the preliminary shooting and the second shooting is the desired shooting. By storing the first and second shooting results in the storage unit 104, the preliminary shooting and the desired shooting can be interchanged, or three or more shootings and a plurality of shootings other than the desired shooting are set as the preliminary shootings. By doing so, it is also possible to make a correction at the time of desired photographing.
In the following, an example will be described in which two shots are taken and the first shot is the preliminary shot and the second shot is the desired shot.

  As shown in FIG. 2, the image processing unit 102 according to the present embodiment includes a feature point detection unit 201 that detects a feature point from captured image data received from the imaging unit 101 or the storage unit 104, and a feature point detection unit 201. A parallax calculation unit 202 that calculates the parallax at the detected feature points, and imaging parameters of the imaging unit 101 such as the parallax calculated by the parallax calculation unit 202 and the focal length stored in the storage unit 104 and the baseline length between the imaging devices. A three-dimensional position information calculation unit 203 that calculates a three-dimensional position for each feature point and measurement error (resolution) information at the three-dimensional position based on (camera parameter) information, a feature point detection unit 201, and a parallax calculation unit 202. , And the reliability calculation unit 204 that calculates the reliability for each feature point based on the processing information from the three-dimensional position information calculation unit 203. , A feature point information holding unit 205 that temporarily holds three-dimensional position information and reliability for each feature point, and features each feature point information (three-dimensional position information and reliability) at the time of preliminary shooting and desired shooting A three-dimensional position determination unit 206 that receives from the point information holding unit 205 and corrects the three-dimensional position information for each feature point at the time of desired imaging can be configured.

The feature point detection unit 201 receives stereo image data (captured image data captured by one of the two image capturing devices and captured image data captured by the other of the two image capturing devices) from the data bus 107, and the feature point of the captured image data is received. To detect. Here, the feature point is, for example, edge information, and is determined to be a feature point region when the reaction intensity with respect to the Sobel filter (3 × 3) as shown in FIG. The edge detection method is not limited to the Sobel filter, and may be configured to use any one or more of various feature point data called Prewitt filter, Laplacian filter, Canny filter, and SIFT (Scale-Invariant Feature Transform). ..
The feature point detection unit 201 transmits position information and / or feature point image data at the image coordinates of the detected feature point and, if necessary, captured image data to the parallax calculation unit 202. Further, the feature point strength (reaction strength) at each feature point coordinate is sent to the reliability calculation unit 204.

The parallax calculation unit 202 receives the position information and / or the feature point image data in the image coordinates of the feature point from the feature point detection unit 201 and performs stereo matching.
Here, an example of stereo matching will be described with reference to FIG. When the image pickup devices according to the present embodiment are arranged side by side, first, an image in the range of the search window is set based on the feature point position of the left image (the image captured by the left image pickup device). Cut out as a template. Here, the range of the search window is, for example, 11 × 11. Next, the matching evaluation value between the search window range of the position of the right image (the image captured by the imaging device on the right side) corresponding to the position of the characteristic point of the left image and the template is obtained. The matching evaluation value is obtained as a SAD (Sum of Absolute Difference) value, for example, as shown in [Equation 2]. Here, L (x, y) is the search window of the left image at the image coordinates (x, y), and R (x, y) is the gradation of each pixel of the search window of the right image at the image coordinates (x, y). Represents a value.

The matching evaluation value is calculated for each position while shifting the attention position with respect to a preset search range. When calculated by SAD, the position where the matching evaluation value is the minimum is the matching position, and the difference between the matching position in the right image and the original target point (feature point) position in the left image is the parallax.
The matching evaluation value may be calculated by various evaluation functions used in stereo matching, and may be calculated by SSD (Sum of Square Difference) or the like.
The parallax calculation unit 202 sends the parallax calculation result for each pixel position (each characteristic point position) to the three-dimensional position information calculation unit 203. In addition, the matching evaluation value for each pixel position (each feature point position) is sent to the reliability calculation unit 204.

  The three-dimensional position information calculation unit 203 calculates the distance Z by [Equation 1] based on the parallax information (parallax calculation result) from the parallax calculation unit 202 and various imaging parameters (camera parameters) from the storage unit 104. To do. Here, the parallax calculated by the parallax calculation unit 202 is a parallax in the image coordinate system and is a pixel value. On the other hand, the parallax in [Equation 1] is a parallax (SI unit such as “mm”) in the sensor coordinate system. Therefore, the distance Z can be calculated by converting from the image coordinate system to the sensor coordinate system using the pixel pitch amount P (mm / pixel) from the storage unit 104 as a parameter. The distance Z corresponds to the coordinate value of Z when the optical axis is parallel to the Z axis. Further, as shown in FIG. 5, the three-dimensional position (X, Y, Z) of the subject can be calculated from the distance Z and the coordinate position (x, y) on the image.

  Further, based on the parallax measurement accuracy (parallax calculation accuracy) in the parallax calculation unit 202, the measurement error at the feature point position is calculated. For example, when the parallax measurement accuracy Np in the parallax calculation unit 202 has one pixel as a unit, it can be calculated by [Equation 3].

Here, E is the measurement error, P is the pixel pitch amount, Np is the parallax measurement accuracy, f is the focal length, and Z is the distance to the subject. It can be seen that the measurement error E is proportional to the distance Z. Therefore, the measurement error E has a larger value as the distance Z from the image pickup device to the subject increases.
The three-dimensional position information calculation unit 203 transmits the distance (three-dimensional position) information to the subject and the measurement error information obtained as described above to the reliability calculation unit 204.

  The reliability calculation unit 204 calculates the feature point strength from the feature point detection unit 201, the matching evaluation value from the parallax calculation unit 202, the three-dimensional position information and the measurement error information from the three-dimensional position information calculation unit 203. Each is received and the reliability is calculated. Specifically, the reliability calculation unit 204 calculates, for example, the reliability I_R for the feature point strength, the reliability M_R for the matching evaluation value, and the reliability E_R for the measurement error information. It can be configured to calculate the degree S_R. An example of each reliability calculation formula is shown in [Equation 4] to [Equation 7].

  [Equation 4] is an example of a calculation formula of the feature point reliability I_R. I_B is the strength of the feature point, and IS is the sum of the coefficients of the feature point detection filter. For example, when the image based on the captured image data is an 8-bit image and the feature point is detected by the Sobel filter (3 × 3), the feature point intensity I_B is 1020 (= 255 + 255 × 2 + 255) at the maximum, and the sum of the filter coefficients is IS is 4 (= 1 + 2 + 1). Therefore, the feature point reliability I_R has a value of 0 to 255, and at 255, the feature point reliability becomes maximum.

[Equation 5] is an example of a calculation formula of the matching reliability M_R. M_B is a matching evaluation value, and N is a search window size. As described above, when the parallax calculation unit 202 calculates the parallax by the SAD of N = 11, the matching evaluation value M_B becomes a value of 0 to 30855. By dividing the matching evaluation value M_B by N ² (= 121) which is the total number of pixels in the search window, the averaged matching evaluation value (the second term on the right side of [Equation 5]) becomes a value of 0 to 255. Become. The closer the averaged matching evaluation value is to 0, the higher the matching evaluation. Similar to the feature point reliability I_R, a value obtained by subtracting the averaged matching evaluation value from 255 is set as the matching reliability M_R so that 255 becomes the maximum value.

  [Equation 6] is an example of a calculation formula of the measurement error reliability E_R. E is the measurement error of the feature point position, E_min is the minimum measurement error in the range specified in the measurement range, and E_max is the maximum measurement error in the range specified in the measurement range. For example, let us consider a case where the measurable range is set as Z = 2000 mm to 5000 mm as specifications, the focal length f is 75 mm, the pixel pitch amount P is 0.0005 mm / pixel, and the parallax measurement accuracy Np is 1 pixel. From [Equation 3], the minimum measurement error E_min and the maximum measurement error E_max are 0.133 and 0.333, respectively. The measurement error reliability E_R has a value of 255 when the measurement error E is equal to the minimum measurement error E_min, and has a value of 0 when the measurement error E is equal to the maximum measurement error E_max.

  [Equation 7] is an example of a formula for calculating the integrated reliability S_R. CI, CM, and CE are arbitrary weighted average coefficients. That is, the integrated reliability S_R is a weighted average value of the feature point reliability I_R, the matching reliability M_R, and the measurement error reliability E_R. The weighting value may be input by the user from the external input / output unit 105, and the value may be held in the storage unit 104. Of the reliability values, the minimum reliability value may be the integrated reliability S_R. It is also possible to adopt a configuration such that It is preferable that the weighting in the case of calculating the integrated reliability S_R by [Equation 7] is set so that the weight of CE is larger than the weight of CI and CM. This is because the feature point reliability I_R and the matching reliability M_R tend to decrease as the subject moves forward, and conversely, the measurement error reliability E_R tends to increase as the subject moves forward. Therefore, it is desirable to set the balance between "CI + CM" and "CE". For example, it is set so that CI + CM = CE = 0.5.

When CI + CM = CE = 0.5 is set, the integrated reliability S_R (X, Y, Z) takes a value of 0 to 255, and the closer it is to 255, the higher the reliability is. The higher the edge strength, the higher the matching strength, and the smaller the measurement error of the feature point position, the higher the reliability.
The reliability calculation unit 204 stores the coordinate information X, Y, Z of the feature point and the integrated reliability S_R (X, Y, Z) value for each coordinate calculated as described above in the feature point information holding unit 205. send.

  The feature point information holding unit 205 temporarily holds the coordinate information X, Y, Z of the feature points and the integrated reliability S_R (X, Y, Z) from the reliability calculation unit 204, and uses them at the time of the next shooting. To do. Further, as the reliability data for each photographing, the information from the reliability calculation unit 204 can be sent to the storage unit 104, and the data can be sequentially stored.

  The three-dimensional position determination unit 206 does not need to perform any processing during the preliminary shooting.

  As described above, the image processing during the preliminary shooting is performed. The image processing result (three-dimensional position measurement result, reliability, processed image, etc.) at the time of preliminary shooting is stored in the storage unit 104, but is displayed sequentially on the display unit 103 or via the external input / output unit 105. Then, it can be sent to an external device.

  Next, image processing at the time of desired photographing will be described. At the time of image processing at the time of desired shooting, the feature point detection unit 201, the parallax calculation unit 202, the three-dimensional position information calculation unit 203, the reliability calculation unit 204, and the feature point information holding unit 205 included in the image processing unit 102 perform preliminary shooting. By performing the same processing as the image processing at the time, the coordinate information X ′, Y ′, Z ′ of each feature point in the desired shooting and the integrated reliability S_R (X ′, Y ′, Z ′) for each coordinate are obtained. obtain.

  The three-dimensional position determination unit 206 receives, from the feature point information holding unit 205, coordinate information X, Y, Z of feature points in preliminary shooting, their integrated reliability S_R (X, Y, Z), and feature points in desired shooting. The coordinate information X ′, Y ′, Z ′ and the integrated reliability S_R (X ′, Y ′, Z ′) are received to determine the three-dimensional position information.

  The coordinate information X, Y, Z of the characteristic points in the preliminary photographing and the coordinate information X ', Y', Z'of the characteristic points in the desired photographing are coordinate data in the image pickup apparatus coordinate system C, and the characteristic points are not changed as they are. Can't compare. Therefore, as shown in FIG. 6, coordinate conversion to the subject coordinate system L is performed using [Equation 8] to [Equation 11].

Here, X _L1 , Y _L1 , and Z _L1 are coordinate values in the subject coordinate system L in preliminary shooting, and X _L2 , Y _L2 , and Z _L2 are coordinate values in the subject coordinate system L in desired shooting, r ₁₁ , r ₁₂ , r ₁₃ , r ₂₁ , r ₂₂ , r ₂₃ , r ₃₁ , r ₃₂ , and r ₃₃ are rotation matrix coefficients to the object coordinate system L in preliminary imaging, r ₁₁ ′, r ₁₂ ′, r ₁₃ ′, r ₂₁ ′, r ₂₂ ′, r ₂₃ ′, r ₃₁ ′, r ₃₂ ′, and r ₃₃ ′ are rotation matrix coefficients for the subject coordinate system L in desired shooting, and tx, ty, and tz are subject coordinates in preliminary shooting. The translation vector coefficients tx ′, ty ′, and tz ′ to the system L are translation vector coefficients to the object coordinate system L in desired photographing. RT ₁ and RT ₂ in [Equation 9] and [Equation 11] are conversion matrices from the image pickup apparatus coordinate system C to the subject coordinate system L, and the inverses thereof are obtained as shown in [Equation 12] and [Equation 13]. By using the matrix, it is possible to perform the inverse transformation from the subject coordinate system L to the image pickup apparatus coordinate system C. Therefore, since the three-dimensional position information up to the subject feature point in the image capturing device coordinate system C can be obtained from the coordinate values in the subject coordinate system L, by determining the coordinate values in the subject coordinate system L, the image capturing device coordinate system is determined. The three-dimensional position information up to the subject feature point in C is also determined.

  The coordinate value in the desired photographing is determined using each coordinate value in the subject coordinate system L obtained by the above conversion and its integrated reliability S_R. Next, the confirmation method will be described.

FIG. 7 is a flowchart showing an example of the operation of the three-dimensional position determination unit 206.
In step S701, a feature point (point of interest) in the subject coordinate system is selected. As the attention point, for example, a attention point list in which the feature points are arranged in the order of high reliability is created, and the feature point having the highest reliability is designated as the attention point. At this time, the feature points to be listed in the point of interest list may be only the feature points at the time of desired shooting, but by listing the feature points at the time of preliminary shooting as well, the feature points can be detected at the time of desired shooting. This is preferable because it is possible to obtain feature points with high reliability that were not present. Further, by setting a threshold value and not listing feature points with low reliability (feature points below the threshold value), it is possible to present only information having a certain reliability to the user.

  In step S702, a comparison point is selected from the feature points at the time of preliminary shooting and the feature points at the time of desired shooting. Specifically, feature points within an arbitrary three-dimensional distance range from the feature points (points of interest) selected in step S701 are selected as comparison targets (comparison points). For example, as shown in FIG. 8A, a feature point Q included in an arbitrary three-dimensional distance range R at the feature point P (point of interest P) is selected as a comparison target. As shown in FIG. 8B, when there is no feature point within the three-dimensional distance range R, it is determined that “no comparison target (comparison point)”. As shown in FIG. 8C, when a plurality of feature points exist within the three-dimensional distance range R, all the plurality of feature points (in the case of the example shown in FIG. 8C, the feature point Q1 and the feature point Both Q2) are selected for comparison. Here, the three-dimensional distance range R is set based on, for example, the measurement accuracy range or the tolerance of the distance measuring device 100. For example, when the measurement accuracy is 0.1 mm, it is preferable to set the three-dimensional distance range R to 0.1 mm because feature point coordinate data with high reliability can be obtained in units of measurement accuracy.

In step S703, the reliability of each of the feature points (points of interest) selected in step S701 and the feature points (comparison points) selected in step S702 are compared.
In step S704, the coordinate value of the feature point (point of interest) is determined based on the comparison result in step S703.

For example, as shown in FIG. 8A, when one comparison target (feature point Q) is found, the integrated reliability S_RP at the feature point _P and the integrated reliability S_RQ at the feature point _Q are compared, and the reliability is calculated. A feature point having a high value is defined as a determined feature point P ′. Alternatively, as shown in [Equation 14] and [Equation 15], the determined feature point P ′ and its reliability may be obtained using each reliability as a weight.

Here, Xp ′, Yp ′, Zp ′ are coordinate values of the determined feature point P ′, Xp, Yp, Zp are coordinate values of the feature point P, Xq, Yq, Zq are coordinate values of the feature point Q, and S_R _{P '} Is the post-determination reliability (post-determination integrated reliability) for the post-determination feature point P'.
Further, for example, as shown in FIG. 8B, in the case of “no comparison target”, P ′ = P is set and no particular determination is made.
Further, for example, when a plurality of comparison targets (feature points Q1 and Q2) are found as in FIG. 8C, the feature point P, the feature point Q1, and the feature point are similar to those in FIG. The feature point with the highest reliability of Q2 may be the determined feature point P ′, or the determined feature point P ′ and its reliability may be obtained using the reliability as a weight.

Next, the three-dimensional position determination unit 206 sends the obtained coordinate values of the determined feature point P ′ and its reliability S_RP _′ to the storage unit 104, and deletes the feature point selected as the comparison target from the attention point list. To do.
Then, the three-dimensional position determination unit 206 performs the processing of steps S701 to S704 for all the points of interest to determine the feature point coordinates.

  As described above, by holding the feature point coordinates in the subject coordinate system L, the distance measuring apparatus 100 calculates the distance between the feature points (distance between two points on the subject, distance between subjects, etc.). be able to. Here, the distance between two points on the subject (on the subject surface) is the thickness or width of the subject. Further, the distance measuring device 100 is configured to perform coordinate conversion to the image capturing device coordinate system C based on [Equation 13] as described above to convert the distance to the image capturing device and send the distance to the storage unit 104. Thus, the distance from the imaging device to the subject can be calculated.

In the above description, it is assumed that the subject is moving. However, even when the distance measuring apparatus 100 moves while the subject is stationary, there is a relative movement, which is applicable. It is possible.
Further, in the above description, the case where the image capturing is performed twice has been described, but the preliminary image capturing may be performed a plurality of times, and the image in the current frame may be the captured image obtained by the desired image capturing, and the image in the frame before that. It can be easily assumed that the distance can be sequentially measured in the moving image by treating the image as a captured image obtained in the preliminary photographing.

  With the above configuration, the object moving in the shooting space is photographed multiple times, the feature point coordinates and the reliability of each of the captured images obtained by the multiple shots are obtained, and based on the feature point coordinates and the reliability. By determining the feature point coordinates of the captured image at the desired capturing timing by using the distance (distance from the imaging device to the subject, distance between two points on the subject, or distance between subjects) and its reliability, It is possible to provide a distance measuring device and a distance measuring method capable of measuring the distance.

  According to the distance measuring apparatus 100 of the first embodiment described above, the imaging unit 101 that acquires a plurality of captured images over time, and the subject point detection unit (which detects points on the subject in the plurality of captured images ( A feature point detection unit 201) and a three-dimensional position calculation unit (three-dimensional position information calculation unit 203) that calculates a three-dimensional position of a point on a subject. The three-dimensional position determination unit 206 further includes a reliability calculation unit 204 that calculates the position reliability (integrated reliability S_R) and a three-dimensional position determination unit 206 that determines the calculated three-dimensional position. Based on the three-dimensional position of an arbitrary comparison point on the subject in a plurality of captured images, which exists within an arbitrary range from the attention point, and the reliability thereof. , To confirm It is configured. Then, the distance measuring apparatus 100 determines the distance (distance from the imaging device to the subject, distance between two points on the subject, or distance between the subjects) based on the three-dimensional position determined by the three-dimensional position determining unit 206. Is configured to calculate.

Therefore, the distance (distance from the image pickup device to the subject, distance between two points on the subject, or distance between the subjects) can be calculated based on the determined three-dimensional position, so that the measurement can be performed with higher reliability. Distance results can be obtained.
Here, the "plurality of photographed images" acquired over time are the photographed images obtained by the preliminary photographing and the photographed images obtained by the desired photographing.
Note that the captured images acquired by the image capturing unit 101 in one capture are not limited to the left image and the right image, and may be captured images having parallax with each other.

  Further, according to the distance measuring apparatus 100 of the first embodiment, there are a plurality of comparison points, and at least one point is detected from an image different from the captured image in which the point of interest exists. Can be configured as follows.

  With this configuration, it is possible to obtain a highly reliable distance measurement result.

Further, according to the distance measuring apparatus 100 of the first embodiment, the reliability (integrated reliability S_R) can be configured to be the reliability based on the theoretical error due to the distance to the subject.
Specifically, as described above, the feature point reliability I_R and the matching reliability M_R tend to decrease as the subject moves forward (that is, as the distance decreases), and conversely, the measurement error reliability E_R varies depending on the subject. There is a tendency to go up toward the front (that is, closer to the distance). Since the integrated reliability S_R is calculated based on the feature point reliability I_R, the matching reliability M_R, and the measurement error reliability E_R, the integrated reliability S_R can be reliability based on a theoretical error due to the distance to the subject.

  With this configuration, the distance (distance from the imaging device to the subject, distance between two points on the subject, or distance between the subjects, based on the three-dimensional position determined in consideration of the theoretical error due to the distance to the subject, Since the distance) can be calculated, a more reliable distance measurement result can be obtained.

  Further, according to the distance measuring method in the first embodiment described above, a photographing process of photographing a subject by the image capturing unit 101 that acquires a plurality of photographed images over time, and detecting a point on the subject in the photographed images. Detection step, a three-dimensional position calculation step of calculating a three-dimensional position of a point on the subject, a reliability calculation step of calculating a reliability of the calculated three-dimensional position, and a calculated three-dimensional position are determined. And a calculation step of calculating a distance to the subject, a distance between two points on the subject, or a distance between the subjects based on the determined three-dimensional position. The three-dimensional position of an arbitrary point of interest on the subject in the image is determined based on the three-dimensional position of an arbitrary comparison point on the subject in a plurality of captured images existing within an arbitrary range from the point of interest and its reliability. It is configured so that.

  Therefore, the distance (distance from the image pickup device to the subject, distance between two points on the subject, or distance between the subjects) can be calculated based on the determined three-dimensional position, so that the measurement can be performed with higher reliability. Distance results can be obtained.

<Second Embodiment>: Estimation of Motion Blur In the present embodiment, a method of setting a feature point detection filter in consideration of motion blur will be described. Specifically, by estimating the subject speed from the shot images obtained by multiple shots, the motion blur in the desired shooting is estimated, and the size and shape of the feature point detection filter are changed according to the motion blur. A method of appropriately detecting the feature points will be described.

As shown in FIG. 9, the image processing unit 102 according to the present embodiment includes a motion blur estimation unit 901 that estimates motion blur from a plurality of shooting results, in addition to the configuration according to the first embodiment (FIG. 2). It is configured to further include.
An example of the operation of the motion blur estimation unit 901 will be described with reference to FIGS. 10 to 12.

In step S1001, the motion blur estimation unit 901 predicts the speed at the time of desired shooting from the respective characteristic point positions obtained by a plurality of times of shooting.
Here, as a specific example, a method of predicting the speed based on the measurement result of one preliminary photographing and the provisional measurement result of the desired photographing will be described. As shown in FIG. 11, an arbitrary preliminary photographing three-dimensional coordinate (X1, Y1, Z1) is selected, and the image coordinate (u1, v1) corresponding to the preliminary photographing three-dimensional coordinate is set as a search start position. Similar to the motion vector detecting means, the corresponding coordinates (image coordinates (u2, v2)) in the photographed image obtained by the desired photographing are searched. Euclidean distance between three-dimensional coordinates (desired shooting three-dimensional coordinates (X2, Y2, Z2)) and preliminary shooting three-dimensional coordinates (X1, Y1, Z1) corresponding to image coordinates (u2, v2) and preliminary shooting and desired shooting The velocity vector can be calculated based on the shooting time interval of. The conversion from the three-dimensional coordinates to the image coordinates can be performed by using [Equation 1], [Equation 12], [Equation 13], and [Equation 16].

  Here, (u, v) are image coordinates, (X, Y, Z) are three-dimensional coordinates, f is a focal length, and xo and yo are images corresponding to image coordinates (0, 0) from the center coordinates of the image element. The element coordinate offsets pw and ph are the width and height of one pixel of the image element.

  In step S1002, the motion blur estimation unit 901 estimates a motion blur vector based on the velocity vector information obtained in step S1001. The motion blur estimation unit 901 receives the shutter speed information of the imaging device from the storage unit 104, and as illustrated in FIG. 12, the three-dimensional image after the desired shooting three-dimensional coordinates (X2, Y2, Z2) move between the shutter speeds. The coordinates (X3, Y3, Z3) are calculated. The three-dimensional coordinates (X2, Y2, Z2) and (X3, Y3, Z3) are coordinate-converted into corresponding image coordinates (u2, v2) and (u3, v3), and image coordinates (u2, v2) and The motion blur vector is estimated from the distance of (u3, v3).

  In step S1003, the motion blur estimation unit 901 generates filter size change information based on the motion blur information obtained in step S1002, and sends it to the feature point detection unit 201. For example, when the size of motion blur (distance between two points) is 3 pixels, and the original feature point coordinates (desired photographing three-dimensional coordinates (X2, Y2, Z2)) are the center of the motion blur range. Assuming that the total amount of motion blur is about 6 pixels. Therefore, the information of 8 × 8 pixels is generated as the filter size change information for the motion blur range + the surrounding one pixel, and the information is transmitted to the feature point detection unit 201.

  The above method is particularly suitable when the direction component of the motion blur vector is large in the Z-axis direction (the optical axis direction of the imaging device). On the other hand, when the direction component of the motion blur vector is mainly large in the X axis direction, the filter is deformed in the X axis direction (for example, 3 × 8), and the direction component of the motion blur vector is mainly large in the Y axis direction. In this case, the filter may be deformed in the Y-axis direction (for example, 8 × 3). For example, when a large motion blur occurs in the X-axis direction, it can be assumed that the influence of the feature point in the Y-axis direction is small. Therefore, the amount of calculation can be reduced by thus limiting the size deformation (expansion) direction of the filter.

  In the above description, an example is described in which one point is designated as an arbitrary three-dimensional coordinate and the matching point on the image is obtained at a desired timing. You may do it. However, since the feature point information includes motion blur, the feature point position may not always be detected with high accuracy, and the motion blur direction may not be estimated. Therefore, if the size of the motion blur vector is ignored and the filter size is enlarged or deformed at an arbitrary step size (for example, 1 pixel enlargement) according to the direction of the motion blur vector, only the determination process is performed. Since it can be dealt with, the process can be simplified.

  The feature point detection unit 201 receives the filter size change information, and again detects the feature point for the captured image obtained by the desired shooting, whereby the feature point information with higher reliability can be obtained. ..

  In the above description, the characteristic point of the desired shooting is re-detected from the measurement results of the preliminary shooting and the desired shooting. However, as shown in FIG. 13, for example, the preliminary shooting is performed twice (preliminary shooting A, preliminary shooting). Shooting B), calculating the speed vector in the preliminary shooting B, and predicting the three-dimensional position at the desired shooting and the three-dimensional position after the shutter based on the speed vector and the shooting time interval until the desired shooting. By estimating the motion blur and presetting the filter size and shape at the time of desired photographing, the motion blur vector can be calculated when the position of the preliminary photographing B can be assumed to be the same velocity or the same acceleration motion. The configuration enables more accurate estimation.

  With the above configuration, the motion blur in the desired shooting is estimated by predicting the subject speed from the captured images obtained by multiple shootings, and the size and shape of the feature point detection filter are changed according to the motion blur. In this way, the distance measurement capable of more appropriately detecting the feature points and measuring the distance from the imaging device to the subject, the distance between two points on the subject, or the distance between the subjects and their reliability. An apparatus and a distance measuring method can be provided.

  According to the distance measuring apparatus 100 of the second embodiment described above, it is calculated corresponding to each of a plurality of captured images (captured image obtained by preliminary capture and captured image obtained by desired capture). A motion blur estimation unit 901 that estimates the motion blur by predicting the moving direction of the subject with respect to the imaging unit 101 from a plurality of three-dimensional positions is provided, and the point of interest (feature point) based on the estimation result of the motion blur estimation unit 901. It is configured to change at least one of the size and the shape of the point-of-interest detection window (feature point detection filter) used when detecting the.

  Therefore, the attention point can be detected in consideration of the influence of the motion blur, so that the attention point can be detected more appropriately and the distance measurement result with higher reliability can be obtained.

<Third Embodiment>: Specification by User In the present embodiment, each of the feature point groups designated by the user has a display unit and an input unit for designating a feature point that the user wants to measure. The amount of calculation can be reduced by configuring only the measurement results of the feature points included in the feature points or the straight lines, curves, or curved surfaces (including planes) formed by those feature points and their error ranges. The method of reduction will be described. In addition, a display method for facilitating the user to specify a feature point by highlighting the feature point with high reliability (eg, enclosing it with a circle having a size corresponding to the high reliability) will be described.

As shown in FIG. 14, the image processing unit 102 according to the present embodiment includes a display image generation unit that generates an image for a user to specify a feature point, in addition to the configuration according to the first embodiment (FIG. 2). 1401 and a definite feature point selection unit 1402 that selects a feature point group to be corrected based on a user-specified feature point.
In addition to the configuration (FIG. 9) in the second embodiment, a display image generation unit 1401 and a fixed feature point selection unit 1402 may be further provided.

The display image generation unit 1401 generates an image for the user to specify a feature point to be measured and sends it to the display unit 103. As a result, an image as shown in FIG. 15, for example, is displayed on the display unit 103. Here, 1501 is a photographed image area, and 1502 is a feature point designation setting area. In the captured image area 1501, a captured image and a circle surrounding a feature point having a certain reliability or higher are displayed. Here, a circle 1504 surrounding the feature point designated by the user is represented by a solid line, while a circle 1503 surrounding a feature point not designated is represented by a broken line. In addition, the larger the size of the circle, the higher the reliability of the feature points. The threshold of the reliability of the displayed feature points can be specified by the user, for example, by the GUI (for example, the slide bar 1505) displayed in the feature point specification setting area 1502. Further, the user specifies the interpolation method by the interpolation method selection unit 1506.
Then, when the user selects the measurement button 1507, the three-dimensional positions of the feature points that are in contact with the feature points designated by the user and the curved surface formed by these feature points are determined.

The fixed feature point selection unit 1402 performs straight line estimation, curve estimation, or curved surface (including plane) estimation based on the feature points designated in the captured image area 1501 and the interpolation method designated by the interpolation method selection unit 1506. For example, in the case of the example shown in FIG. 15, the specified feature points are three points (three of the four corners of the upper surface of the cube), and the specified interpolation method is “plane”. To do. The plane can be estimated by calculating the cross product of the two vectors made with one of the three points as the origin.
Subsequently, the definite feature point selection unit 1402 refers to the feature point information stored in the feature point information storage unit 205 and selects a feature point group that is in contact with the estimated straight line, curve, or curved surface (including plane). .. For example, when plane estimation is performed, a feature point group in contact with this plane is selected. At this time, feature point candidates within a certain distance from this plane in the normal direction are selected. This fixed distance is, for example, the range estimation error range required for the system, and is set within 0.1 mm, for example. Alternatively, since the estimation error of the distance per parallax pixel increases as the distance increases, it is preferable to configure the range to increase as the distance increases.

Then, only the feature points selected in this way are determined by the three-dimensional position determination unit 206, and the result is stored in the feature point information storage unit 205. The confirmed feature point information is displayed on the display unit 103 as shown in FIG. 16, for example. Reference numeral 1601 in FIG. 16 indicates a plane on which the three-dimensional position is calculated. Here, by designating a feature point on the plane that the user wants to measure, the three-dimensional position information is displayed on the image. Further, when a position (point) which is an area (point) on this plane and is not a feature point is designated by the user, for example, information on the closest feature point may be displayed. For example, it is possible to display the interpolated result by using the information of each of the neighboring feature points of the designated position in the plane 4 direction.
With the above configuration, it is possible to provide a display method for making it easier for the user to specify the feature points.

  The distance measuring apparatus 100 according to the third embodiment described above is configured so that the user can specify the point of interest (feature point).

Therefore, since the user can recognize the attention point of interest, only the three-dimensional position of the attention point needs to be determined, and the amount of calculation can be reduced.
The method of designating the feature points by the user can be appropriately selected. For example, the user can operate the operation unit (mouse or the like) connected to the external input / output unit 105 to designate the feature points. Alternatively, the display unit 103 may be a touch panel type display unit.

<Fourth Embodiment>
The distance measuring device 100 described in the above embodiments (first to third embodiments) can be used, for example, for measuring the thickness of the sliding plate portion of a vehicle pantograph.

Specifically, an image taken in response to a shooting trigger from a passage sensor that detects the entry of the pantograph into the shooting range (of course, the pantograph is shown in this image) is used.
Alternatively, a plurality of images are taken with an imaging device over time, and the image showing the pantograph is used from among the plurality of images. At this time, for example, it may be detected by pattern matching of a characteristic portion, or, for example, as shown in FIG. 17, by taking a background difference by taking an image of the overhead line area in advance, the overhead line can be used. It can be configured to determine the image in which the pantograph appears from the region variation.

The standard uses the fixed part. Detects a straight line and uses a straight line area in the shape of a strip. Define a plane and measure the thickness from that plane to the edge of each contact plate. Specifically, first, as shown in FIG. 18A, a straight edge in the longitudinal direction of the contact strip is extracted. Next, as shown in FIG. 18B, a reference surface is defined. Then, as shown in FIG. 18C, the distance (thickness) to each position is calculated.
As shown in FIG. 19, the thickness profile in the preliminary shooting and the thickness profile in the desired shooting are matched in the longitudinal direction of the strip (for example, the corresponding position is calculated by obtaining the cross-correlation coefficient or the like). Then, align the positions and start the comparison from there.

  In each of the above-described embodiments, the configurations and the like illustrated in the accompanying drawings are merely examples, and the present invention is not limited to these and can be appropriately changed within the scope of exerting the effects of the present invention. Is. Other than the above, the present invention can be appropriately modified and implemented without departing from the scope of the object of the present invention.

100 distance measuring device 101 imaging unit 201 feature point detection unit (subject point detection unit)
203 3D position information calculation unit (3D position calculation unit)
204 reliability calculation unit 206 three-dimensional position determination unit 901 motion blur estimation unit

Claims (5)

An imaging unit that acquires a plurality of images taken from a plurality of directions,
A subject point detection unit that detects a point on the subject in the image,
A three-dimensional position calculation unit that calculates a three-dimensional position of a point on the subject by parallax measurement of the plurality of images ,
Reliable based on the measurement error E = P · Np · Z / f (P is the pixel pitch amount, Np is the parallax measurement accuracy, f is the focal length, and Z is the distance to the subject) at the three-dimensional position of the point on the subject. Calculate the degree,
A distance measuring device, which determines a coordinate value of a point on the subject based on the three-dimensional position and the reliability and outputs a distance. An imaging unit that acquires a plurality of images taken over time,
A subject point detection unit that detects a point on the subject in the image,
A three-dimensional position calculation unit that calculates a three-dimensional position of a point on the subject by parallax measurement of the plurality of images ,
Reliable based on the measurement error E = P · Np · Z / f (P is the pixel pitch amount, Np is the parallax measurement accuracy, f is the focal length, and Z is the distance to the subject) at the three-dimensional position of the point on the subject. Calculate the degree,
A distance measuring device, which determines a coordinate value of a point on the subject based on the three-dimensional position and the reliability and outputs a distance.   The reliability of the three-dimensional position of the point of interest is calculated by parallax measurement based on the plurality of images during the first shooting, and the parallax measurement is performed based on the plurality of images different from the time during the first shooting during the second shooting. The reliability of a three-dimensional position of a comparison point is calculated, the coordinate value of the point on the subject is determined based on the comparison result of these reliability, and the distance is output. Distance measuring device.   The distance is a distance between the distance measuring device and a point on the object, a distance between a point on the object and another point on the object, or a point on the object and other points in the image. The distance measuring apparatus according to any one of claims 1 to 3, wherein the distance is a distance to a point on the subject.   The distance measuring device according to claim 1, wherein the reliability of a point on the subject is displayed so that the magnitude of the reliability can be seen.