JP4814669B2 - 3D coordinate acquisition device - Google Patents

Description

    The present invention relates to a three-dimensional coordinate acquisition apparatus that acquires three-dimensional coordinates around a vehicle using images obtained by two or more cameras attached to the vehicle. The present invention relates to a three-dimensional coordinate acquisition apparatus having a calibration function to acquire.

    Conventionally, a system has been developed in which a camera is mounted on a vehicle, images of the front and rear of the vehicle are taken, and the surrounding environment is recognized three-dimensionally. As a method of measuring three-dimensional coordinates (distance) with a camera, stereo vision using two or more cameras (hereinafter referred to as compound eye stereo) is generally used. For example, Patent Document 1 discloses a technique for processing an image captured by a stereo camera mounted on a vehicle and measuring a three-dimensional position of an object outside the vehicle.

    In the case of compound-eye stereo, images are taken with multiple cameras placed at spatially separated positions, and object feature points (such as corners, hereinafter referred to as `` feature points '') and texture parallax are used. Thus, the three-dimensional coordinates (distance) can be calculated by the so-called triangulation principle. In that case, relative geometric information between the cameras (translational position shift amount between cameras and rotation amount of the camera optical axis) is required, and these are obtained by calibration in advance.

    Many methods of stereo calibration have been proposed. For example, Patent Document 2 discloses an apparatus that photographs an adjustment pattern with a known distance and adjusts the displacement of the camera mounting position.

Japanese Patent Laid-Open No. 5-114099 JP-A-10-115506

    In compound eye stereo, prior calibration is required, which is generally time-consuming, and for example, as shown in Patent Document 2, a test pattern with a known distance is required.

    Therefore, an object of the present invention is to provide a three-dimensional coordinate acquisition apparatus that can calibrate a stereo camera without requiring addition of hardware or the like or a test pattern. Note that only the relative geometric information between the cameras (translational position deviation and rotation deviation) is calibrated here. Camera internal parameters (focal length and angle of view) need to be known.

    A basic matrix (including relative geometric information between the two cameras) is derived from the correspondence between the feature points in the images obtained by the two cameras constituting the three-dimensional coordinate acquisition device, and the position between the cameras It is possible to estimate the deviation amount and the rotational deviation amount. However, the magnitude of the positional deviation obtained at that time is normalized (for example, 1), and distance measurement by triangulation cannot be performed as it is. Therefore, the magnitude (scale) of this positional deviation amount is estimated using the result of monocular stereo.

Here, monocular stereo (also known as motion stereo, SfM (Structure
from Motion)) is a technology that can realize a stereo view by using a plurality of images taken from different times and different viewpoints with a single camera.

    If the camera installation parameters (height and depression angle) for the road surface (assuming that the road surface is a flat surface) are known, the positional deviation can be achieved by using a feature point belonging to the road surface in the image in a method with a lighter computational load. The magnitude (scale) of the quantity can be estimated. When the relative positional relationship between the two cameras is known, the same method is used by exchanging the compound eye stereo and the monocular stereo, that is, the monocular stereo is calibrated using the result of the compound eye stereo. It is also possible.

    In the present invention, the camera internal parameters (focal length, angle of view, etc.) need to be known.

    It is possible to automatically calibrate a compound eye stereo camera (acquisition of relative positional relationship between two cameras) using the result of monocular stereo without adding hardware. Further, if the camera installation parameters (height and depression angle) with respect to the road surface are known, calibration can be performed by a method with a lighter calculation load by using feature points belonging to the road surface in the image. Contrary to the above, when the relative positional relationship between the two cameras is known, the monocular stereo calibration (one camera) is automatically performed using the result of compound eye stereo without adding hardware. It is also possible to acquire the movement amount of the camera).

    Embodiments of the present invention will be described below with reference to the drawings.

    FIG. 1 is a block diagram illustrating an example of a three-dimensional coordinate acquisition apparatus, FIG. 2 is a block diagram illustrating another example of the three-dimensional coordinate acquisition apparatus, and FIG. 3 is a flowchart of processing in the three-dimensional coordinate acquisition apparatus of FIG. FIG. 4 is a plan view showing an example of mounting a three-dimensional coordinate acquisition device on a vehicle, and FIG. 5 is an example of images acquired by a camera attached to the vehicle and a flow generated therefrom. FIG. 6 is a diagram showing the camera coordinate system, FIG. 7 is a plan view showing another example of mounting the three-dimensional coordinate acquisition device on the vehicle, and FIG. 8 is still another example of the three-dimensional coordinate acquisition device. FIG.

    As shown in FIG. 4, the three-dimensional coordinate acquisition apparatus 1 has two cameras 3A and 3B that are appropriately arranged so that the fields of view of the cameras overlap (the number of cameras is two or more). But at least two are required). It is assumed that the relative positional relationship such as the installation distance between the cameras 3A and 3B is naturally unknown.

    As shown in FIG. 1, a monocular stereo processing unit 5 and a binocular stereo processing unit 6 are connected to the camera 3A, and the monocular stereo processing unit 5 and the binocular stereo processing unit 6 have an inter-camera distance scale estimation. Means 7 are connected.

    The monocular stereo processing unit 3 includes a flow estimation unit 9, a host vehicle momentum estimation unit 10 connected to the camera 3A (or 3B, or both, and will be described later for 3A), and the inter-camera distance scale estimation unit 7 described above. The binocular stereo processing unit 6 includes a corresponding point estimation unit 12, a temporary calibration unit 13, and a temporary three-dimensional coordinate estimation unit connected to the cameras 3A and 3B. 15 and three-dimensional coordinate estimation means 16. The temporary three-dimensional coordinate estimation unit 15 and the three-dimensional coordinate estimation unit 16 are connected to the above-described inter-camera distance scale estimation unit 7.

    Note that the camera 3A. Other than 3B, flow estimation means 9, own vehicle momentum estimation means 10, three-dimensional coordinate estimation means 11, corresponding point estimation means 12, provisional calibration means 13, provisional three-dimensional coordinate estimation means 15, three-dimensional coordinate estimation means 16 and The inter-camera distance scale estimation means 7 schematically displays functions virtually realized by a computer having a CPU executing, for example, a program created based on the process flowchart shown in FIG. However, each means may be configured as hardware using an integrated circuit such as a semiconductor.

    Since the three-dimensional coordinate acquisition apparatus 1 has the above-described configuration, the three-dimensional coordinate acquisition apparatus 1 is mounted when the three-dimensional coordinate acquisition apparatus 1 is mounted on the vehicle 2 or when the arrangement positions of the cameras 3A and 3B are changed. When acquiring the three-dimensional coordinates of the surrounding target while performing the calibration of 1, the camera 3A, 3B attached to the vehicle first captures the situation outside the vehicle at regular time intervals. Images captured by the cameras 3A and 3B are recorded at a predetermined frame rate (the number of images taken per unit time) and stored in a memory (not shown).

    Next, the flow estimation unit 9 of the monocular stereo processing unit 5 provided for calculating the three-dimensional coordinates of the feature points in the image using the time series image acquired by one camera 3A. As shown in step S1 of FIG. 3, the process of tracking the feature points in the continuous image and estimating the flow of the feature points is performed.

    When the flow estimation means 9 compares two images (see FIG. 5) obtained by the camera 3A using different methods with different shooting (acquisition) times and is displayed in common in both images. A point to be judged (a part of the image and a characteristic point in the image, for example, a corner portion, a sharp tip portion, a change point of lightness, saturation, etc.) as a feature point (any point is acceptable) Corresponding point estimation processing is performed, which includes a process of extracting a large number and associating these feature points with each other between different images and estimating corresponding points between the two images. This is generally called optical flow estimation (or simply flow estimation), and the KLT (Kanade Lucas-Tomashi) method is often used. An example of the flow is shown in FIG. As is clear from FIG. 5, the flow is a trajectory in which the feature point has moved between the image one hour before and the image at the current time.

Next, the own vehicle movement amount estimation means 10 estimates the movement amount of the camera 3A as shown in step S2 of FIG. The movement amount of the camera is a translational movement amount in the X, Y, and Z axis directions between the cameras before and after the movement, and a rotation amount around the X, Y, and Z axes (see FIG. 6). Since the camera is fixed to the vehicle, the camera movement amount and the own vehicle movement amount are used interchangeably. As a method for estimating the own vehicle (camera) momentum, there are a method using a sensor such as a wheel speed sensor (see, for example, JP-A-2001-187553) and a method using a flow in an image. As this method, for example, a method for calculating the amount of movement of the host vehicle equipped with the camera using a flow indicating the movement state of the corresponding point belonging to the road surface is known (for example, T. Suzuki (Toyota ), T. Kanade, 'Measurement of Vehicle Motion and
Orientation using Optical Flow ', IEEE conference on ITS, 1999). In addition, as a method using the basic matrix, there are (Yamaguchi (Toyota Chuken) et al., “Detection of obstacles in front vehicle with in-vehicle monocular camera”, Research report of Information Processing Society of Japan, 2005.11). Note that the translation amount directly obtained from the basic matrix is normalized to 1, and it is necessary to estimate its size (scale) separately. Examples of the method for estimating the scale include a method using output from sensors such as a wheel speed sensor and a vehicle speed sensor, and a method using road flow.

    Next, in step S4 of FIG. 3, in the three-dimensional coordinate estimation unit 11 of the monocular stereo processing unit 9, the flow of feature points obtained by the flow estimation unit 9 and the own vehicle obtained by the own vehicle momentum estimation unit 10 are obtained. Using the momentum and camera internal parameters (focal length, angle of view, etc. (these are known)), the three-dimensional coordinates to the feature point can be obtained by a known method (for example, Kenichi Kanaya “Image Understanding-Mathematical Science for Three-Dimensional Recognition- ", The method disclosed in Morikita Publishing, ISBN4-627-82140-9;).

    On the other hand, the three-dimensional coordinate acquisition apparatus 1 includes two cameras 3A and 3B in parallel with the three-dimensional coordinate acquisition processing of the object around the vehicle by the monocular stereo processing unit 5 using the single camera 2A. The two-lens stereo processing unit 6 executes a three-dimensional coordinate acquisition process for an object around the vehicle from the image captured by the above.

    In this process, first, the situation around the vehicle is photographed using the cameras 3A and 3B. This can be performed in parallel with the image acquisition operation via the camera 3A by the monocular stereo processing unit 5 described above. That is, the image data acquired by the camera 3A is output in parallel to the monocular stereo processing unit 5 and the binocular stereo processing unit 6, and the monocular stereo processing unit 5 uses only the image from the camera 3A. The two-dimensional stereo processing unit 6 performs a three-dimensional coordinate acquisition process of the object around the vehicle based on the image from the camera 3B operating simultaneously in addition to the camera 3A. Perform the acquisition process.

    For this purpose, first, in step S5 of FIG. 3, the corresponding point estimation means 12 of the binocular stereo processing unit 6 associates feature points in images captured (acquired) at the same time by two cameras. . For example, there is a method of extracting characteristic points (feature points) in an image using a corner detection filter such as the Harris method, setting a block around the point, and performing a correlation operation such as block matching between the images. is there. Note that the feature point selection method is the same as the flow estimation means 9 by the monocular stereo processing unit 5 described above.

    At this time, at least a part of the feature points selected by the flow estimation unit 9 of the monocular stereo processing unit 5 is made to coincide with the feature points extracted by the corresponding point extraction unit 12. Note that the feature point selection criteria are usually the same in the monocular stereo processing unit 5 and the binocular stereo processing unit 6, so that the feature points selected as a result overlap in both the processing units 5 and 6. Since there are a large number of points, the two processing units 5 and 6 collate them and store the feature point information such as the position on the image for the matching feature points in an appropriate memory. Also good.

    Note that, between the flow estimation unit 9 and the corresponding point extraction unit 12, at least a part of the feature points selected by either one is output to the other estimation unit side as feature points used by the other estimation unit. Feature point output means (built in either of the estimation means 9 and 12) may be provided, and the monocular stereo processing unit 5 and the binocular stereo processing unit 6 may be configured to use a common feature point. Then, the calculation of the common feature point in the monocular stereo processing unit 5 and the binocular stereo processing unit 6 can be performed easily and reliably. Further, since the feature point is extracted by one of the estimation means 9 or 12, and the extracted feature point can be used as it is by the other estimation means 12 or 9, the feature point extraction calculation is greatly increased. It is possible to reduce the calculation load of the computer.

    Next, the provisional calibration means 13 of the binocular stereo processing unit 6 estimates the provisional translational position displacement amount t ′ and the rotational displacement amount r between the two cameras 3A, 3B (both t ′ and r are 3). A vector with two elements (details below). The reason that is provisional here will be described later. In the temporary calibration means 13, first, the basic matrix F is obtained (step S6 in FIG. 3), then the basic matrix E is obtained, and finally the basic matrix E is decomposed into a provisional translational displacement amount t ′ and a rotational displacement amount r. (Step 7 in FIG. 6).

    First, as shown in step S6 of FIG. 3, a fundamental matrix F (Fundamental Matrix) is estimated. The basic matrix F is a 3 × 3 matrix obtained from the translational displacement amount, the rotational displacement amount and the internal parameters of the two cameras 3A and 3B, and the translational displacement amount is expressed as t = [Tx , Ty, Tz] T (T represents transposition), rotation deviation amount is γ = [θ ψ φ] [rad], and camera internal matrix is A (3 × 3 matrix), the basic matrix F is ( It can be calculated by formula (1).

inv(・)は行列・の逆行列を求める演算である。ただし、

inv (·) is an operation for obtaining an inverse matrix of a matrix. However,

［数２］中の×は外積演算を表す。なお、Tx,Ty,TzはそれぞれX,Y,Z軸方向の並進位置ずれ量、θ、ψ、φはぞれぞれ、Y軸周り回転(ヨー回転)角[rad]、X軸周り回転(ピッチ回転)角[rad]、Z軸回り回転(ロール回転)角[rad]である(図６参照)。軸及び回転の向きは仮のものであり、任意の構成が可能である。

X in [Expression 2] represents an outer product operation. Tx, Ty, and Tz are translational displacements in the X, Y, and Z axes, respectively. Θ, ψ, and φ are rotations about the Y axis (yaw rotation), an angle [rad], and rotations about the X axis. (Pitch rotation) angle [rad] and Z axis rotation (roll rotation) angle [rad] (see FIG. 6). The axis and the direction of rotation are temporary, and any configuration is possible.

    Thus, the basic matrix F includes the geometric information between the cameras. If the basic matrix is obtained, t and r can be estimated conversely. However, in reality, the basic matrix cannot be obtained directly using the equation (1) (since t and r are (of course) unknown), so the basic matrix is obtained using another method. As a method for obtaining the basic matrix F, an 8-point algorithm that solves a linear equation using the correspondence of 8 feature points between two images, or a least-squares method using more correspondences of feature points is used. There are ways to do it.

Next, in step S7, the translational displacement amount t and the rotational displacement amount r are derived from the basic matrix F. For details on how to find the fundamental matrix and how to derive t and r from the fundamental matrix, see Kenichi Kanaya, 'Image Understanding-Mathematics of 3D Recognition', Morikita Publishing,
"ISBN4-627-82140-9" and "Xugang, 'Three-dimensional CG from photos", Modern Science, ISBN4-7649-0286-9, etc. However, the amount of translational displacement (obtained as t ′) obtained by the above method is normalized (to 1). This is the reason why I said <provisional> earlier. That is, it is necessary to separately obtain the magnitude (scale) of the translational position shift amount. In the present invention, this scale (Sc) is estimated using the result of monocular stereo vision.

    A method for obtaining the scale Sc will be described below. Using the temporary calibration value obtained by the temporary calibration means 13, that is, the correspondence between the temporary translational position deviation amount t ′ and the rotational deviation quantity r and the feature points obtained by the corresponding point estimation means 12 (disparity information). Then, as shown in step S8 of FIG. 3, provisional three-dimensional coordinates up to each feature point are calculated. For the calculation of the provisional three-dimensional coordinates, a method (known) similar to the three-dimensional coordinate estimation means 11 of the monocular stereo processing unit 9 is used.

    Next, as shown in step S9 of FIG. 3, the inter-camera distance scale estimation unit 7 includes the feature points extracted by the monocular stereo processing unit 5 and the feature points extracted by the binocular stereo processing unit 6. For the same feature points that coincide with each other, the three-dimensional coordinates of the feature points obtained by the three-dimensional coordinate estimation means 11 are compared with the temporary three-dimensional coordinates of the feature points obtained by the provisional three-dimensional coordinate estimation means 15. The scale Sc is estimated.

    That is, the distance from the camera of a certain feature point obtained by the monocular stereo processing unit 5 is defined as Ds. On the other hand, it is assumed that the temporary distance obtained by the binocular stereo processing unit 6 for the feature point is Dm. In fact, since Dm and Ds should be equal, the actual scale between cameras is Sc = Ds / Dm. That is, the accurate translational displacement between the cameras 3A and 3B is t = Sc × t ′. When there are multiple feature points, the scale is first calculated using all feature points, and the average value, median value, or reliability calculated separately (the greater the parallax, the greater the reliability, the closer the distance, the more reliable The final scale is the weighted average value, etc., weighted by (eg, large). Thus, an accurate relative geometric relationship (t and r) between the two cameras 3A and 3B is obtained, and the calibration is completed.

    Next, in step S10 of FIG. 3, the three-dimensional coordinate estimation unit 16 uses the obtained inter-camera distance scale Sc to temporarily store each feature point in the image obtained by the temporary three-dimensional coordinate estimation unit 15. The actual three-dimensional coordinates are calculated by correcting the three-dimensional coordinates.

    FIG. 2 shows another example of the three-dimensional coordinate acquisition apparatus 1. The same parts as those shown in FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted.

    In the three-dimensional coordinate acquisition apparatus 1 of FIG. 2, the flow estimation means 9 and the own vehicle momentum estimation means 10 replace the road surface feature point extraction means 17 as compared with the apparatus of FIG. 1. The road surface feature point extraction means 17 extracts feature points belonging to the road surface from the image obtained by the camera 3A. If the installation parameters (camera height, camera depression angle, yaw angle, etc.) with respect to the road surface (assuming that the road surface is a plane) of the camera 3A that has captured the feature points belonging to the road surface are known, the feature points can be obtained even from one still image. 3D coordinates (distance) can be calculated immediately, so that monocular stereo processing (processing for estimating the three-dimensional coordinates of feature points using the flow estimation means 9 and the own vehicle momentum estimation means 10) is unnecessary. The calculation load is reduced.

    The processing on the binocular stereo processing unit 6 side is the same as in the case of FIG. That is, the provisional three-dimensional coordinates of the same feature point as the feature point on the road surface are obtained on the binocular stereo processing unit 6 side, and the inter-camera distance scale Sc is estimated by comparing the two.

    Note that the scale estimation processing by the inter-camera distance scale estimation means 7, that is, the processing from step S 1 to step S 9 in FIG. 3 is performed a plurality of times over time, and the results are integrated to obtain a more accurate calibration. Is possible.

    The calibration of the stereo cameras 3A and 3B need not be performed every time. Therefore, once the calibration is completed, the calibration can be prevented from being performed thereafter. In addition, if the epipolar constraint obtained from the basic matrix F obtained in the course of processing is used, the amount of computation for estimating the correspondence between feature points between images in compound eye stereo can be greatly reduced.

    Also, the opposite of the previous embodiment is possible. In other words, when the relative positional relationship between the two cameras is known (that is, the compound-eye stereo camera is already calibrated), the monocular stereo calibration is performed using the three-dimensional coordinate value calculated by the compound-eye stereo (the own vehicle). Equivalent to the movement amount estimation) (FIG. 8). That is, the calibration processing referred to in this specification is not applied only when obtaining the positional relationship between two cameras arranged side by side in the processing with compound eye stereo. This method is also used when estimating the actual size (scale) of the temporary translation of the camera normalized to 1 (obtained directly from the basic matrix) during 3D coordinate estimation processing using monocular stereo. It can be applied. In this case, there is an advantage that it is not necessary to prepare special hardware such as a wheel speed sensor only for scale estimation.

    For example, as shown in FIG. 7, the optical axes 3c and 3d of the cameras of the two cameras 3A and 3B are facing each other, and the fields of view of these cameras partially overlap in front of the optical axes 3c and 3d. Suppose that the cameras 3A and 3B are installed such that the vehicle traveling direction is a compound eye stereo view (compound eye stereo region WS), and the other case is a monocular stereo view (monocular stereo region SS). In this case, in the front area WS where compound eye stereo is possible, the compound eye stereo processing unit 6 using the cameras 3A and 3B whose relative installation position relationship (geometric position relationship) is known is extracted by the corresponding point estimation unit 12. The corresponding points, that is, the actual three-dimensional coordinates of the target are estimated by the three-dimensional coordinate estimation means 16 from the geometrical positional relationship between the two cameras and the camera internal parameters, and at the same time, the monocular stereo processing unit 5 The flow estimation means 9 and the provisional vehicle momentum estimation means 19 use the basic matrix to calculate the provisional momentum of the camera 3A (or camera 3B), that is, the provisional translational displacement amount and the rotational displacement amount (wheel speed). Without using a sensor). Based on the provisional momentum, provisional three-dimensional coordinate estimation means 20 estimates a provisional three-dimensional coordinate value of the object (feature point).

    Next, the camera movement amount scale estimation means 21 performs provisional three-dimensional coordinate calculation estimation for the same feature points that coincide with each other among the feature points of the flow estimation means 9 and the corresponding point estimation means 12. The provisional three-dimensional coordinates calculated by the means 20 and the three-dimensional coordinates calculated by the three-dimensional coordinate estimation means 16 of the binocular stereo processing unit 6 are compared, and the camera movement amount scale (actual movement amount of the camera) is compared. The size is estimated.

    Next, in the three-dimensional coordinate estimation means 11 of the monocular stereo processing unit 5, the provisional translational movement amount and rotation normalized to 1 of the camera 3A (and the camera 3B) obtained by the provisional three-dimensional coordinate estimation means 20 are obtained. Temporary three-dimensional coordinates of feature points based on the quantities are corrected to actual three-dimensional coordinate values. Thus, the three-dimensional coordinates of the object other than the front of the vehicle in FIG. 7 (that is, the monocular stereo area SS capable of monocular stereo only) are obtained. As a result, it is possible to acquire the three-dimensional coordinates of the target in a wider range by stereo vision.

    INDUSTRIAL APPLICABILITY The present invention can be used in a three-dimensional coordinate acquisition device that is mounted on a vehicle and acquires three-dimensional coordinates of a vehicle peripheral object using two or more cameras.

FIG. 1 is a block diagram illustrating an example of a three-dimensional coordinate acquisition apparatus. FIG. 2 is a block diagram illustrating another example of a three-dimensional coordinate acquisition apparatus. FIG. 3 is a flowchart showing a flow of processing in the three-dimensional coordinate acquisition apparatus of FIG. FIG. 4 is a plan view showing an example of mounting the three-dimensional coordinate acquisition apparatus on a vehicle. FIG. 5 is a diagram illustrating an example of images acquired by a camera attached to a vehicle and a flow generated therefrom. FIG. 6 is a diagram illustrating a relationship in coordinates between the translational movement amount and the rotational movement amount of the own vehicle. FIG. 7 is a plan view showing another example of mounting the three-dimensional coordinate acquisition apparatus on a vehicle. FIG. 8 is a block diagram showing still another example of the three-dimensional coordinate acquisition apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Three-dimensional coordinate acquisition apparatus 2 ... Vehicle 3A, 3B ... Camera 9 ... Flow estimation means 10 ... Own vehicle momentum estimation means 11 ... Three-dimensional coordinate estimation means 12 ... Corresponding point estimation means 13 ... Temporary calibration means 15, 20... Temporary three-dimensional coordinate estimation means 16... Three-dimensional coordinate estimation means 17.

Claims (6)

3 of the target around the vehicle, using at least two cameras mounted on the vehicle and arranged so that the fields of view of the cameras overlap, and using a plurality of images acquired by the at least two cameras. In a three-dimensional coordinate acquisition apparatus that estimates and acquires dimensional coordinates,
Feature point extraction means for extracting a plurality of first feature points from an image acquired by one of the at least two cameras;
First three-dimensional coordinate calculation means for calculating and estimating the three-dimensional coordinates of the extracted first feature point;
The first three-dimensional coordinate calculation means extracts the first feature points that are displayed in common in two images acquired by the one camera over time, and the extracted first features Calculating and estimating the three-dimensional coordinates of the first feature point based on the process of estimating the corresponding point between the images by associating the feature point with each other between the images and the process of estimating the momentum of the own vehicle;
A plurality of second feature points from two images acquired simultaneously by two of the at least two cameras so that the first feature points coincide with at least some of the feature points. Corresponding point estimation means for extracting and associating these second feature points in the two images;
Temporary calibration is performed by calculating a basic matrix from the correspondence between the second feature points, and estimating and calculating a temporary translational position shift amount and a rotational shift amount between the two cameras from the basic matrix as temporary calibration values. Means,
A provisional three-dimensional coordinate estimation means for estimating and calculating a provisional three-dimensional coordinate of the second feature point based on the provisional calibration value;
Three-dimensional coordinates calculated by the first three-dimensional coordinate calculating means and the temporary three-dimensional coordinate estimating means for the same feature points that coincide with each other among the first feature points and the second feature points. And a distance scale estimation means between cameras for estimating a distance scale between the two cameras,
Based on the estimated distance scale between the two cameras and the provisional three-dimensional coordinates of the second feature point calculated by the provisional three-dimensional coordinate estimation means, the second feature point Second 3D coordinate calculation means for calculating and estimating 3D coordinates;
A three-dimensional coordinate acquisition apparatus. The first three-dimensional coordinate calculation means includes flow estimation means, and the flow estimation means associates the extracted first feature points with each other between the images and estimates the corresponding points between the two images. Is performed by estimating a flow between the images of the first feature points,
The three-dimensional coordinate acquisition apparatus according to claim 1, wherein the three-dimensional coordinates of the first feature point are calculated and estimated based on the estimated flow. 3 of the target around the vehicle, using at least two cameras mounted on the vehicle and arranged so that the fields of view of the cameras overlap, and using a plurality of images acquired by the at least two cameras. In a three-dimensional coordinate acquisition apparatus that estimates and acquires dimensional coordinates,
Road surface feature point extraction means for extracting a plurality of feature points belonging to the road surface as first feature points from an image acquired by one of the at least two cameras;
First three-dimensional coordinate calculation means for calculating and estimating the three-dimensional coordinates of the extracted first feature point;
The first three-dimensional coordinate calculation means calculates the three-dimensional coordinates of the first feature points from the extracted first feature points and installation parameters of the camera that has captured the first feature points. Having road surface feature point coordinate calculation means,
A plurality of second feature points from two images acquired simultaneously by two of the at least two cameras so that the first feature points coincide with at least some of the feature points. Corresponding point estimation means for extracting and associating these second feature points in the two images;
The corresponding point estimation unit includes a road surface corresponding point estimation unit that extracts a first feature point belonging to the road surface as a second feature point and associates the second feature point in the two images. ,
Temporary calibration is performed by calculating a basic matrix from the correspondence between the second feature points, and estimating and calculating a temporary translational position shift amount and a rotational shift amount between the two cameras from the basic matrix as temporary calibration values. Means,
A provisional three-dimensional coordinate estimation means for estimating and calculating a provisional three-dimensional coordinate of the second feature point based on the provisional calibration value;
Three-dimensional coordinates calculated by the first three-dimensional coordinate calculating means and the temporary three-dimensional coordinate estimating means for the same feature points that coincide with each other among the first feature points and the second feature points. And a distance scale estimation means between cameras for estimating a distance scale between the two cameras,
Based on the estimated distance scale between the two cameras and the provisional three-dimensional coordinates of the second feature point calculated by the provisional three-dimensional coordinate estimation means, the second feature point Second 3D coordinate calculation means for calculating and estimating 3D coordinates;
A three-dimensional coordinate acquisition apparatus.     Feature point output means for outputting the feature point extracted by one of the estimation means to the other estimation means as the feature point used by the other estimation means between the flow estimation means and the corresponding point estimation means The three-dimensional coordinate acquisition apparatus according to claim 2, comprising: A plurality of cameras acquired by the at least two cameras, which are mounted on a vehicle and have at least two cameras arranged so as to maintain a predetermined relative geometric position so that the fields of view of the cameras overlap each other; In the three-dimensional coordinate acquisition apparatus that estimates and acquires the three-dimensional coordinates of the target around the vehicle using the image,
Feature point extraction means for extracting a plurality of first feature points from an image acquired by one of the at least two cameras;
The first feature that is commonly displayed in two images acquired over time when the one camera moves with a translational displacement amount and a rotational displacement amount as the vehicle moves. Flow estimation means for extracting points and estimating a flow between the images of the first feature points;
Temporary three-dimensional coordinate estimation means for computing and estimating temporary three-dimensional coordinates of the extracted first feature points based on the estimated flow. Two of the at least two cameras are simultaneously A plurality of second feature points are extracted from the two acquired images so that the first feature points coincide with at least some of the feature points, and the second feature points are extracted from the two images. Corresponding point estimation means to associate in
First three-dimensional coordinate calculation means for estimating and calculating three-dimensional coordinates of the second feature point based on the predetermined relative geometric position between the two cameras;
Among the first feature point and the second feature point, for the same feature point that coincides with each other, the temporary three-dimensional coordinates calculated by the temporary three-dimensional coordinate calculation estimation means and the first three A scale estimation means for estimating a moving amount scale of the camera by comparing with the three-dimensional coordinates calculated by the dimension coordinate calculation means;
Based on the estimated movement amount scale of the camera and the temporary three-dimensional coordinates of the first feature point calculated by the temporary three-dimensional coordinate estimation means, the three-dimensional coordinates of the first feature point A second three-dimensional coordinate calculation means for calculating and estimating
A three-dimensional coordinate acquisition apparatus. The at least two cameras are arranged such that their optical axes are directed outward from each other, and the fields of view of these cameras are disposed in front of the optical axis and partially overlap each other.
The three-dimensional coordinate acquisition apparatus according to any one of claims 1 to 5.

Images