JP4814669B2  3D coordinate acquisition device  Google Patents
3D coordinate acquisition device Download PDFInfo
 Publication number
 JP4814669B2 JP4814669B2 JP2006087728A JP2006087728A JP4814669B2 JP 4814669 B2 JP4814669 B2 JP 4814669B2 JP 2006087728 A JP2006087728 A JP 2006087728A JP 2006087728 A JP2006087728 A JP 2006087728A JP 4814669 B2 JP4814669 B2 JP 4814669B2
 Authority
 JP
 Japan
 Prior art keywords
 cameras
 dimensional coordinate
 feature points
 estimating
 means
 Prior art date
 Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
 Active
Links
Images
Description
The present invention relates to a threedimensional coordinate acquisition apparatus that acquires threedimensional coordinates around a vehicle using images obtained by two or more cameras attached to the vehicle. The present invention relates to a threedimensional coordinate acquisition apparatus having a calibration function to acquire.
Conventionally, a system has been developed in which a camera is mounted on a vehicle, images of the front and rear of the vehicle are taken, and the surrounding environment is recognized threedimensionally. As a method of measuring threedimensional coordinates (distance) with a camera, stereo vision using two or more cameras (hereinafter referred to as compound eye stereo) is generally used. For example, Patent Document 1 discloses a technique for processing an image captured by a stereo camera mounted on a vehicle and measuring a threedimensional position of an object outside the vehicle.
In the case of compoundeye stereo, images are taken with multiple cameras placed at spatially separated positions, and object feature points (such as corners, hereinafter referred to as `` feature points '') and texture parallax are used. Thus, the threedimensional coordinates (distance) can be calculated by the socalled triangulation principle. In that case, relative geometric information between the cameras (translational position shift amount between cameras and rotation amount of the camera optical axis) is required, and these are obtained by calibration in advance.
Many methods of stereo calibration have been proposed. For example, Patent Document 2 discloses an apparatus that photographs an adjustment pattern with a known distance and adjusts the displacement of the camera mounting position.
In compound eye stereo, prior calibration is required, which is generally timeconsuming, and for example, as shown in Patent Document 2, a test pattern with a known distance is required.
Therefore, an object of the present invention is to provide a threedimensional coordinate acquisition apparatus that can calibrate a stereo camera without requiring addition of hardware or the like or a test pattern. Note that only the relative geometric information between the cameras (translational position deviation and rotation deviation) is calibrated here. Camera internal parameters (focal length and angle of view) need to be known.
A basic matrix (including relative geometric information between the two cameras) is derived from the correspondence between the feature points in the images obtained by the two cameras constituting the threedimensional coordinate acquisition device, and the position between the cameras It is possible to estimate the deviation amount and the rotational deviation amount. However, the magnitude of the positional deviation obtained at that time is normalized (for example, 1), and distance measurement by triangulation cannot be performed as it is. Therefore, the magnitude (scale) of this positional deviation amount is estimated using the result of monocular stereo.
Here, monocular stereo (also known as motion stereo, SfM (Structure
from Motion)) is a technology that can realize a stereo view by using a plurality of images taken from different times and different viewpoints with a single camera.
If the camera installation parameters (height and depression angle) for the road surface (assuming that the road surface is a flat surface) are known, the positional deviation can be achieved by using a feature point belonging to the road surface in the image in a method with a lighter computational load. The magnitude (scale) of the quantity can be estimated. When the relative positional relationship between the two cameras is known, the same method is used by exchanging the compound eye stereo and the monocular stereo, that is, the monocular stereo is calibrated using the result of the compound eye stereo. It is also possible.
In the present invention, the camera internal parameters (focal length, angle of view, etc.) need to be known.
It is possible to automatically calibrate a compound eye stereo camera (acquisition of relative positional relationship between two cameras) using the result of monocular stereo without adding hardware. Further, if the camera installation parameters (height and depression angle) with respect to the road surface are known, calibration can be performed by a method with a lighter calculation load by using feature points belonging to the road surface in the image. Contrary to the above, when the relative positional relationship between the two cameras is known, the monocular stereo calibration (one camera) is automatically performed using the result of compound eye stereo without adding hardware. It is also possible to acquire the movement amount of the camera).
Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a block diagram illustrating an example of a threedimensional coordinate acquisition apparatus, FIG. 2 is a block diagram illustrating another example of the threedimensional coordinate acquisition apparatus, and FIG. 3 is a flowchart of processing in the threedimensional coordinate acquisition apparatus of FIG. FIG. 4 is a plan view showing an example of mounting a threedimensional coordinate acquisition device on a vehicle, and FIG. 5 is an example of images acquired by a camera attached to the vehicle and a flow generated therefrom. FIG. 6 is a diagram showing the camera coordinate system, FIG. 7 is a plan view showing another example of mounting the threedimensional coordinate acquisition device on the vehicle, and FIG. 8 is still another example of the threedimensional coordinate acquisition device. FIG.
As shown in FIG. 4, the threedimensional coordinate acquisition apparatus 1 has two cameras 3A and 3B that are appropriately arranged so that the fields of view of the cameras overlap (the number of cameras is two or more). But at least two are required). It is assumed that the relative positional relationship such as the installation distance between the cameras 3A and 3B is naturally unknown.
As shown in FIG. 1, a monocular stereo processing unit 5 and a binocular stereo processing unit 6 are connected to the camera 3A, and the monocular stereo processing unit 5 and the binocular stereo processing unit 6 have an intercamera distance scale estimation. Means 7 are connected.
The monocular stereo processing unit 3 includes a flow estimation unit 9, a host vehicle momentum estimation unit 10 connected to the camera 3A (or 3B, or both, and will be described later for 3A), and the intercamera distance scale estimation unit 7 described above. The binocular stereo processing unit 6 includes a corresponding point estimation unit 12, a temporary calibration unit 13, and a temporary threedimensional coordinate estimation unit connected to the cameras 3A and 3B. 15 and threedimensional coordinate estimation means 16. The temporary threedimensional coordinate estimation unit 15 and the threedimensional coordinate estimation unit 16 are connected to the abovedescribed intercamera distance scale estimation unit 7.
Note that the camera 3A. Other than 3B, flow estimation means 9, own vehicle momentum estimation means 10, threedimensional coordinate estimation means 11, corresponding point estimation means 12, provisional calibration means 13, provisional threedimensional coordinate estimation means 15, threedimensional coordinate estimation means 16 and The intercamera distance scale estimation means 7 schematically displays functions virtually realized by a computer having a CPU executing, for example, a program created based on the process flowchart shown in FIG. However, each means may be configured as hardware using an integrated circuit such as a semiconductor.
Since the threedimensional coordinate acquisition apparatus 1 has the abovedescribed configuration, the threedimensional coordinate acquisition apparatus 1 is mounted when the threedimensional coordinate acquisition apparatus 1 is mounted on the vehicle 2 or when the arrangement positions of the cameras 3A and 3B are changed. When acquiring the threedimensional coordinates of the surrounding target while performing the calibration of 1, the camera 3A, 3B attached to the vehicle first captures the situation outside the vehicle at regular time intervals. Images captured by the cameras 3A and 3B are recorded at a predetermined frame rate (the number of images taken per unit time) and stored in a memory (not shown).
Next, the flow estimation unit 9 of the monocular stereo processing unit 5 provided for calculating the threedimensional coordinates of the feature points in the image using the time series image acquired by one camera 3A. As shown in step S1 of FIG. 3, the process of tracking the feature points in the continuous image and estimating the flow of the feature points is performed.
When the flow estimation means 9 compares two images (see FIG. 5) obtained by the camera 3A using different methods with different shooting (acquisition) times and is displayed in common in both images. A point to be judged (a part of the image and a characteristic point in the image, for example, a corner portion, a sharp tip portion, a change point of lightness, saturation, etc.) as a feature point (any point is acceptable) Corresponding point estimation processing is performed, which includes a process of extracting a large number and associating these feature points with each other between different images and estimating corresponding points between the two images. This is generally called optical flow estimation (or simply flow estimation), and the KLT (Kanade LucasTomashi) method is often used. An example of the flow is shown in FIG. As is clear from FIG. 5, the flow is a trajectory in which the feature point has moved between the image one hour before and the image at the current time.
Next, the own vehicle movement amount estimation means 10 estimates the movement amount of the camera 3A as shown in step S2 of FIG. The movement amount of the camera is a translational movement amount in the X, Y, and Z axis directions between the cameras before and after the movement, and a rotation amount around the X, Y, and Z axes (see FIG. 6). Since the camera is fixed to the vehicle, the camera movement amount and the own vehicle movement amount are used interchangeably. As a method for estimating the own vehicle (camera) momentum, there are a method using a sensor such as a wheel speed sensor (see, for example, JPA2001187553) and a method using a flow in an image. As this method, for example, a method for calculating the amount of movement of the host vehicle equipped with the camera using a flow indicating the movement state of the corresponding point belonging to the road surface is known (for example, T. Suzuki (Toyota ), T. Kanade, 'Measurement of Vehicle Motion and
Orientation using Optical Flow ', IEEE conference on ITS, 1999). In addition, as a method using the basic matrix, there are (Yamaguchi (Toyota Chuken) et al., “Detection of obstacles in front vehicle with invehicle monocular camera”, Research report of Information Processing Society of Japan, 2005.11). Note that the translation amount directly obtained from the basic matrix is normalized to 1, and it is necessary to estimate its size (scale) separately. Examples of the method for estimating the scale include a method using output from sensors such as a wheel speed sensor and a vehicle speed sensor, and a method using road flow.
Next, in step S4 of FIG. 3, in the threedimensional coordinate estimation unit 11 of the monocular stereo processing unit 9, the flow of feature points obtained by the flow estimation unit 9 and the own vehicle obtained by the own vehicle momentum estimation unit 10 are obtained. Using the momentum and camera internal parameters (focal length, angle of view, etc. (these are known)), the threedimensional coordinates to the feature point can be obtained by a known method (for example, Kenichi Kanaya “Image UnderstandingMathematical Science for ThreeDimensional Recognition ", The method disclosed in Morikita Publishing, ISBN4627821409;).
On the other hand, the threedimensional coordinate acquisition apparatus 1 includes two cameras 3A and 3B in parallel with the threedimensional coordinate acquisition processing of the object around the vehicle by the monocular stereo processing unit 5 using the single camera 2A. The twolens stereo processing unit 6 executes a threedimensional coordinate acquisition process for an object around the vehicle from the image captured by the above.
In this process, first, the situation around the vehicle is photographed using the cameras 3A and 3B. This can be performed in parallel with the image acquisition operation via the camera 3A by the monocular stereo processing unit 5 described above. That is, the image data acquired by the camera 3A is output in parallel to the monocular stereo processing unit 5 and the binocular stereo processing unit 6, and the monocular stereo processing unit 5 uses only the image from the camera 3A. The twodimensional stereo processing unit 6 performs a threedimensional coordinate acquisition process of the object around the vehicle based on the image from the camera 3B operating simultaneously in addition to the camera 3A. Perform the acquisition process.
For this purpose, first, in step S5 of FIG. 3, the corresponding point estimation means 12 of the binocular stereo processing unit 6 associates feature points in images captured (acquired) at the same time by two cameras. . For example, there is a method of extracting characteristic points (feature points) in an image using a corner detection filter such as the Harris method, setting a block around the point, and performing a correlation operation such as block matching between the images. is there. Note that the feature point selection method is the same as the flow estimation means 9 by the monocular stereo processing unit 5 described above.
At this time, at least a part of the feature points selected by the flow estimation unit 9 of the monocular stereo processing unit 5 is made to coincide with the feature points extracted by the corresponding point extraction unit 12. Note that the feature point selection criteria are usually the same in the monocular stereo processing unit 5 and the binocular stereo processing unit 6, so that the feature points selected as a result overlap in both the processing units 5 and 6. Since there are a large number of points, the two processing units 5 and 6 collate them and store the feature point information such as the position on the image for the matching feature points in an appropriate memory. Also good.
Note that, between the flow estimation unit 9 and the corresponding point extraction unit 12, at least a part of the feature points selected by either one is output to the other estimation unit side as feature points used by the other estimation unit. Feature point output means (built in either of the estimation means 9 and 12) may be provided, and the monocular stereo processing unit 5 and the binocular stereo processing unit 6 may be configured to use a common feature point. Then, the calculation of the common feature point in the monocular stereo processing unit 5 and the binocular stereo processing unit 6 can be performed easily and reliably. Further, since the feature point is extracted by one of the estimation means 9 or 12, and the extracted feature point can be used as it is by the other estimation means 12 or 9, the feature point extraction calculation is greatly increased. It is possible to reduce the calculation load of the computer.
Next, the provisional calibration means 13 of the binocular stereo processing unit 6 estimates the provisional translational position displacement amount t ′ and the rotational displacement amount r between the two cameras 3A, 3B (both t ′ and r are 3). A vector with two elements (details below). The reason that is provisional here will be described later. In the temporary calibration means 13, first, the basic matrix F is obtained (step S6 in FIG. 3), then the basic matrix E is obtained, and finally the basic matrix E is decomposed into a provisional translational displacement amount t ′ and a rotational displacement amount r. (Step 7 in FIG. 6).
First, as shown in step S6 of FIG. 3, a fundamental matrix F (Fundamental Matrix) is estimated. The basic matrix F is a 3 × 3 matrix obtained from the translational displacement amount, the rotational displacement amount and the internal parameters of the two cameras 3A and 3B, and the translational displacement amount is expressed as t = [Tx , Ty, Tz] T (T represents transposition), rotation deviation amount is γ = [θ ψ φ] [rad], and camera internal matrix is A (3 × 3 matrix), the basic matrix F is ( It can be calculated by formula (1).
inv (·) is an operation for obtaining an inverse matrix of a matrix. However,
X in [Expression 2] represents an outer product operation. Tx, Ty, and Tz are translational displacements in the X, Y, and Z axes, respectively. Θ, ψ, and φ are rotations about the Y axis (yaw rotation), an angle [rad], and rotations about the X axis. (Pitch rotation) angle [rad] and Z axis rotation (roll rotation) angle [rad] (see FIG. 6). The axis and the direction of rotation are temporary, and any configuration is possible.
Thus, the basic matrix F includes the geometric information between the cameras. If the basic matrix is obtained, t and r can be estimated conversely. However, in reality, the basic matrix cannot be obtained directly using the equation (1) (since t and r are (of course) unknown), so the basic matrix is obtained using another method. As a method for obtaining the basic matrix F, an 8point algorithm that solves a linear equation using the correspondence of 8 feature points between two images, or a leastsquares method using more correspondences of feature points is used. There are ways to do it.
Next, in step S7, the translational displacement amount t and the rotational displacement amount r are derived from the basic matrix F. For details on how to find the fundamental matrix and how to derive t and r from the fundamental matrix, see Kenichi Kanaya, 'Image UnderstandingMathematics of 3D Recognition', Morikita Publishing,
"ISBN4627821409" and "Xugang, 'Threedimensional CG from photos", Modern Science, ISBN4764902869, etc. However, the amount of translational displacement (obtained as t ′) obtained by the above method is normalized (to 1). This is the reason why I said <provisional> earlier. That is, it is necessary to separately obtain the magnitude (scale) of the translational position shift amount. In the present invention, this scale (Sc) is estimated using the result of monocular stereo vision.
A method for obtaining the scale Sc will be described below. Using the temporary calibration value obtained by the temporary calibration means 13, that is, the correspondence between the temporary translational position deviation amount t ′ and the rotational deviation quantity r and the feature points obtained by the corresponding point estimation means 12 (disparity information). Then, as shown in step S8 of FIG. 3, provisional threedimensional coordinates up to each feature point are calculated. For the calculation of the provisional threedimensional coordinates, a method (known) similar to the threedimensional coordinate estimation means 11 of the monocular stereo processing unit 9 is used.
Next, as shown in step S9 of FIG. 3, the intercamera distance scale estimation unit 7 includes the feature points extracted by the monocular stereo processing unit 5 and the feature points extracted by the binocular stereo processing unit 6. For the same feature points that coincide with each other, the threedimensional coordinates of the feature points obtained by the threedimensional coordinate estimation means 11 are compared with the temporary threedimensional coordinates of the feature points obtained by the provisional threedimensional coordinate estimation means 15. The scale Sc is estimated.
That is, the distance from the camera of a certain feature point obtained by the monocular stereo processing unit 5 is defined as Ds. On the other hand, it is assumed that the temporary distance obtained by the binocular stereo processing unit 6 for the feature point is Dm. In fact, since Dm and Ds should be equal, the actual scale between cameras is Sc = Ds / Dm. That is, the accurate translational displacement between the cameras 3A and 3B is t = Sc × t ′. When there are multiple feature points, the scale is first calculated using all feature points, and the average value, median value, or reliability calculated separately (the greater the parallax, the greater the reliability, the closer the distance, the more reliable The final scale is the weighted average value, etc., weighted by (eg, large). Thus, an accurate relative geometric relationship (t and r) between the two cameras 3A and 3B is obtained, and the calibration is completed.
Next, in step S10 of FIG. 3, the threedimensional coordinate estimation unit 16 uses the obtained intercamera distance scale Sc to temporarily store each feature point in the image obtained by the temporary threedimensional coordinate estimation unit 15. The actual threedimensional coordinates are calculated by correcting the threedimensional coordinates.
FIG. 2 shows another example of the threedimensional coordinate acquisition apparatus 1. The same parts as those shown in FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted.
In the threedimensional coordinate acquisition apparatus 1 of FIG. 2, the flow estimation means 9 and the own vehicle momentum estimation means 10 replace the road surface feature point extraction means 17 as compared with the apparatus of FIG. 1. The road surface feature point extraction means 17 extracts feature points belonging to the road surface from the image obtained by the camera 3A. If the installation parameters (camera height, camera depression angle, yaw angle, etc.) with respect to the road surface (assuming that the road surface is a plane) of the camera 3A that has captured the feature points belonging to the road surface are known, the feature points can be obtained even from one still image. 3D coordinates (distance) can be calculated immediately, so that monocular stereo processing (processing for estimating the threedimensional coordinates of feature points using the flow estimation means 9 and the own vehicle momentum estimation means 10) is unnecessary. The calculation load is reduced.
The processing on the binocular stereo processing unit 6 side is the same as in the case of FIG. That is, the provisional threedimensional coordinates of the same feature point as the feature point on the road surface are obtained on the binocular stereo processing unit 6 side, and the intercamera distance scale Sc is estimated by comparing the two.
Note that the scale estimation processing by the intercamera distance scale estimation means 7, that is, the processing from step S 1 to step S 9 in FIG. 3 is performed a plurality of times over time, and the results are integrated to obtain a more accurate calibration. Is possible.
The calibration of the stereo cameras 3A and 3B need not be performed every time. Therefore, once the calibration is completed, the calibration can be prevented from being performed thereafter. In addition, if the epipolar constraint obtained from the basic matrix F obtained in the course of processing is used, the amount of computation for estimating the correspondence between feature points between images in compound eye stereo can be greatly reduced.
Also, the opposite of the previous embodiment is possible. In other words, when the relative positional relationship between the two cameras is known (that is, the compoundeye stereo camera is already calibrated), the monocular stereo calibration is performed using the threedimensional coordinate value calculated by the compoundeye stereo (the own vehicle). Equivalent to the movement amount estimation) (FIG. 8). That is, the calibration processing referred to in this specification is not applied only when obtaining the positional relationship between two cameras arranged side by side in the processing with compound eye stereo. This method is also used when estimating the actual size (scale) of the temporary translation of the camera normalized to 1 (obtained directly from the basic matrix) during 3D coordinate estimation processing using monocular stereo. It can be applied. In this case, there is an advantage that it is not necessary to prepare special hardware such as a wheel speed sensor only for scale estimation.
For example, as shown in FIG. 7, the optical axes 3c and 3d of the cameras of the two cameras 3A and 3B are facing each other, and the fields of view of these cameras partially overlap in front of the optical axes 3c and 3d. Suppose that the cameras 3A and 3B are installed such that the vehicle traveling direction is a compound eye stereo view (compound eye stereo region WS), and the other case is a monocular stereo view (monocular stereo region SS). In this case, in the front area WS where compound eye stereo is possible, the compound eye stereo processing unit 6 using the cameras 3A and 3B whose relative installation position relationship (geometric position relationship) is known is extracted by the corresponding point estimation unit 12. The corresponding points, that is, the actual threedimensional coordinates of the target are estimated by the threedimensional coordinate estimation means 16 from the geometrical positional relationship between the two cameras and the camera internal parameters, and at the same time, the monocular stereo processing unit 5 The flow estimation means 9 and the provisional vehicle momentum estimation means 19 use the basic matrix to calculate the provisional momentum of the camera 3A (or camera 3B), that is, the provisional translational displacement amount and the rotational displacement amount (wheel speed). Without using a sensor). Based on the provisional momentum, provisional threedimensional coordinate estimation means 20 estimates a provisional threedimensional coordinate value of the object (feature point).
Next, the camera movement amount scale estimation means 21 performs provisional threedimensional coordinate calculation estimation for the same feature points that coincide with each other among the feature points of the flow estimation means 9 and the corresponding point estimation means 12. The provisional threedimensional coordinates calculated by the means 20 and the threedimensional coordinates calculated by the threedimensional coordinate estimation means 16 of the binocular stereo processing unit 6 are compared, and the camera movement amount scale (actual movement amount of the camera) is compared. The size is estimated.
Next, in the threedimensional coordinate estimation means 11 of the monocular stereo processing unit 5, the provisional translational movement amount and rotation normalized to 1 of the camera 3A (and the camera 3B) obtained by the provisional threedimensional coordinate estimation means 20 are obtained. Temporary threedimensional coordinates of feature points based on the quantities are corrected to actual threedimensional coordinate values. Thus, the threedimensional coordinates of the object other than the front of the vehicle in FIG. 7 (that is, the monocular stereo area SS capable of monocular stereo only) are obtained. As a result, it is possible to acquire the threedimensional coordinates of the target in a wider range by stereo vision.
INDUSTRIAL APPLICABILITY The present invention can be used in a threedimensional coordinate acquisition device that is mounted on a vehicle and acquires threedimensional coordinates of a vehicle peripheral object using two or more cameras.
DESCRIPTION OF SYMBOLS 1 ... Threedimensional coordinate acquisition apparatus 2 ... Vehicle 3A, 3B ... Camera 9 ... Flow estimation means 10 ... Own vehicle momentum estimation means 11 ... Threedimensional coordinate estimation means 12 ... Corresponding point estimation means 13 ... Temporary calibration means 15, 20... Temporary threedimensional coordinate estimation means 16... Threedimensional coordinate estimation means 17.
Claims (6)
 3 of the target around the vehicle, using at least two cameras mounted on the vehicle and arranged so that the fields of view of the cameras overlap, and using a plurality of images acquired by the at least two cameras. In a threedimensional coordinate acquisition apparatus that estimates and acquires dimensional coordinates,
Feature point extraction means for extracting a plurality of first feature points from an image acquired by one of the at least two cameras;
First threedimensional coordinate calculation means for calculating and estimating the threedimensional coordinates of the extracted first feature point;
The first threedimensional coordinate calculation means extracts the first feature points that are displayed in common in two images acquired by the one camera over time, and the extracted first features Calculating and estimating the threedimensional coordinates of the first feature point based on the process of estimating the corresponding point between the images by associating the feature point with each other between the images and the process of estimating the momentum of the own vehicle;
A plurality of second feature points from two images acquired simultaneously by two of the at least two cameras so that the first feature points coincide with at least some of the feature points. Corresponding point estimation means for extracting and associating these second feature points in the two images;
Temporary calibration is performed by calculating a basic matrix from the correspondence between the second feature points, and estimating and calculating a temporary translational position shift amount and a rotational shift amount between the two cameras from the basic matrix as temporary calibration values. Means,
A provisional threedimensional coordinate estimation means for estimating and calculating a provisional threedimensional coordinate of the second feature point based on the provisional calibration value;
Threedimensional coordinates calculated by the first threedimensional coordinate calculating means and the temporary threedimensional coordinate estimating means for the same feature points that coincide with each other among the first feature points and the second feature points. And a distance scale estimation means between cameras for estimating a distance scale between the two cameras,
Based on the estimated distance scale between the two cameras and the provisional threedimensional coordinates of the second feature point calculated by the provisional threedimensional coordinate estimation means, the second feature point Second 3D coordinate calculation means for calculating and estimating 3D coordinates;
A threedimensional coordinate acquisition apparatus.  The first threedimensional coordinate calculation means includes flow estimation means, and the flow estimation means associates the extracted first feature points with each other between the images and estimates the corresponding points between the two images. Is performed by estimating a flow between the images of the first feature points,
The threedimensional coordinate acquisition apparatus according to claim 1, wherein the threedimensional coordinates of the first feature point are calculated and estimated based on the estimated flow.  3 of the target around the vehicle, using at least two cameras mounted on the vehicle and arranged so that the fields of view of the cameras overlap, and using a plurality of images acquired by the at least two cameras. In a threedimensional coordinate acquisition apparatus that estimates and acquires dimensional coordinates,
Road surface feature point extraction means for extracting a plurality of feature points belonging to the road surface as first feature points from an image acquired by one of the at least two cameras;
First threedimensional coordinate calculation means for calculating and estimating the threedimensional coordinates of the extracted first feature point;
The first threedimensional coordinate calculation means calculates the threedimensional coordinates of the first feature points from the extracted first feature points and installation parameters of the camera that has captured the first feature points. Having road surface feature point coordinate calculation means,
A plurality of second feature points from two images acquired simultaneously by two of the at least two cameras so that the first feature points coincide with at least some of the feature points. Corresponding point estimation means for extracting and associating these second feature points in the two images;
The corresponding point estimation unit includes a road surface corresponding point estimation unit that extracts a first feature point belonging to the road surface as a second feature point and associates the second feature point in the two images. ,
Temporary calibration is performed by calculating a basic matrix from the correspondence between the second feature points, and estimating and calculating a temporary translational position shift amount and a rotational shift amount between the two cameras from the basic matrix as temporary calibration values. Means,
A provisional threedimensional coordinate estimation means for estimating and calculating a provisional threedimensional coordinate of the second feature point based on the provisional calibration value;
Threedimensional coordinates calculated by the first threedimensional coordinate calculating means and the temporary threedimensional coordinate estimating means for the same feature points that coincide with each other among the first feature points and the second feature points. And a distance scale estimation means between cameras for estimating a distance scale between the two cameras,
Based on the estimated distance scale between the two cameras and the provisional threedimensional coordinates of the second feature point calculated by the provisional threedimensional coordinate estimation means, the second feature point Second 3D coordinate calculation means for calculating and estimating 3D coordinates;
A threedimensional coordinate acquisition apparatus.  Feature point output means for outputting the feature point extracted by one of the estimation means to the other estimation means as the feature point used by the other estimation means between the flow estimation means and the corresponding point estimation means The threedimensional coordinate acquisition apparatus according to claim 2, comprising:
 A plurality of cameras acquired by the at least two cameras, which are mounted on a vehicle and have at least two cameras arranged so as to maintain a predetermined relative geometric position so that the fields of view of the cameras overlap each other; In the threedimensional coordinate acquisition apparatus that estimates and acquires the threedimensional coordinates of the target around the vehicle using the image,
Feature point extraction means for extracting a plurality of first feature points from an image acquired by one of the at least two cameras;
The first feature that is commonly displayed in two images acquired over time when the one camera moves with a translational displacement amount and a rotational displacement amount as the vehicle moves. Flow estimation means for extracting points and estimating a flow between the images of the first feature points;
Temporary threedimensional coordinate estimation means for computing and estimating temporary threedimensional coordinates of the extracted first feature points based on the estimated flow. Two of the at least two cameras are simultaneously A plurality of second feature points are extracted from the two acquired images so that the first feature points coincide with at least some of the feature points, and the second feature points are extracted from the two images. Corresponding point estimation means to associate in
First threedimensional coordinate calculation means for estimating and calculating threedimensional coordinates of the second feature point based on the predetermined relative geometric position between the two cameras;
Among the first feature point and the second feature point, for the same feature point that coincides with each other, the temporary threedimensional coordinates calculated by the temporary threedimensional coordinate calculation estimation means and the first three A scale estimation means for estimating a moving amount scale of the camera by comparing with the threedimensional coordinates calculated by the dimension coordinate calculation means;
Based on the estimated movement amount scale of the camera and the temporary threedimensional coordinates of the first feature point calculated by the temporary threedimensional coordinate estimation means, the threedimensional coordinates of the first feature point A second threedimensional coordinate calculation means for calculating and estimating
A threedimensional coordinate acquisition apparatus.  The at least two cameras are arranged such that their optical axes are directed outward from each other, and the fields of view of these cameras are disposed in front of the optical axis and partially overlap each other.
The threedimensional coordinate acquisition apparatus according to any one of claims 1 to 5.
Priority Applications (1)
Application Number  Priority Date  Filing Date  Title 

JP2006087728A JP4814669B2 (en)  20060328  20060328  3D coordinate acquisition device 
Applications Claiming Priority (1)
Application Number  Priority Date  Filing Date  Title 

JP2006087728A JP4814669B2 (en)  20060328  20060328  3D coordinate acquisition device 
Publications (2)
Publication Number  Publication Date 

JP2007263669A JP2007263669A (en)  20071011 
JP4814669B2 true JP4814669B2 (en)  20111116 
Family
ID=38636812
Family Applications (1)
Application Number  Title  Priority Date  Filing Date 

JP2006087728A Active JP4814669B2 (en)  20060328  20060328  3D coordinate acquisition device 
Country Status (1)
Country  Link 

JP (1)  JP4814669B2 (en) 
Families Citing this family (28)
Publication number  Priority date  Publication date  Assignee  Title 

JP5171196B2 (en)  20071009  20130327  株式会社エヌ・ティ・ティ・ドコモ  Wireless communication system, wireless communication method, and base station 
JP2009186353A (en) *  20080207  20090820  Fujitsu Ten Ltd  Object detecting device and object detecting method 
JP5228614B2 (en) *  20080515  20130703  株式会社豊田中央研究所  Parameter calculation apparatus, parameter calculation system and program 
JP2009294109A (en) *  20080605  20091217  Fujitsu Ltd  Calibration apparatus 
JP4702569B2 (en) *  20080930  20110615  マツダ株式会社  Image processing apparatus for vehicle 
DE102009031650A1 (en) *  20090703  20110105  Volkswagen Ag  Method for enhancing camera system of vehicle assistance system for vehicle, involves arranging camera and evaluation unit in common housing, where camera system is extended by another camera 
JP5445069B2 (en) *  20091126  20140319  富士通株式会社  3D position measuring apparatus and 3D position measuring method 
JP2011163900A (en) *  20100209  20110825  Casio Computer Co Ltd  Image processor, image processing program, and image processing method 
JP4865065B1 (en)  20100714  20120201  Ｂｉ２−Ｖｉｓｉｏｎ株式会社  Stereo imaging device control system 
KR101733131B1 (en)  20101214  20170510  한국전자통신연구원  3D motion recognition method and apparatus 
EP2551819A1 (en)  20110729  20130130  BI2Vision Co.  Control system for stereo imaging device 
JP5515018B2 (en)  20111206  20140611  本田技研工業株式会社  Environment recognition device 
JP5865726B2 (en) *  20120220  20160217  日立マクセル株式会社  Video display device 
WO2014002725A1 (en)  20120629  20140103  富士フイルム株式会社  3d measurement method, device, and system, and image processing device 
JP5745178B2 (en)  20120629  20150708  富士フイルム株式会社  Threedimensional measurement method, apparatus and system, and image processing apparatus 
JP6003673B2 (en) *  20130123  20161005  株式会社デンソー  3D position estimation apparatus, vehicle control apparatus, and 3D position estimation method 
WO2014181581A1 (en) *  20130507  20141113  日立オートモティブシステムズ株式会社  Calibration device, calibration system, and imaging device 
WO2014192061A1 (en) *  20130527  20141204  パイオニア株式会社  Image processing device, image processing method, and image processing program 
KR101551026B1 (en)  20131224  20150907  현대자동차주식회사  Method of tracking vehicle 
DE102014201271A1 (en) *  20140124  20150730  Robert Bosch Gmbh  A method and controller for detecting a change in a relative yaw angle within a stereo video system for a vehicle 
JP6247569B2 (en) *  20140313  20171213  ヤマハ発動機株式会社  Distance estimating device and vehicle equipped with the same 
DE112014006493T5 (en)  20140527  20170119  Application Solutions (Electronics and Vision) Ltd.  Determine a scale of threedimensional information 
JP6282377B2 (en) *  20150224  20180221  株式会社日立製作所  Threedimensional shape measurement system and measurement method thereof 
CN108307654A (en) *  20150617  20180720  索尼公司  Image processing apparatus and image processing method 
JP2017139612A (en) *  20160203  20170810  パナソニックＩｐマネジメント株式会社  Onvehicle camera calibration system 
JP6404525B2 (en) *  20160517  20181010  株式会社オプティム  Spherical camera captured image display system, omnidirectional camera captured image display method and program 
WO2018235256A1 (en) *  20170623  20181227  株式会社日立製作所  Stereo measurement device and system 
CN108437898B (en) *  20180512  20190430  吉林大学  A kind of environment sensing device based on eyetracking 
Family Cites Families (5)
Publication number  Priority date  Publication date  Assignee  Title 

JP2993995B2 (en) *  19900323  19991227  マツダ株式会社  Image processing apparatus of the moving car 
JPH0843055A (en) *  19940729  19960216  Canon Inc  Method and apparatus for recognizing shape of three dimensional object 
JP3436074B2 (en) *  19970610  20030811  トヨタ自動車株式会社  Car stereo camera 
JP2001012946A (en) *  19990630  20010119  Toshiba Corp  Dynamic image processor and processing method 
JP2005024463A (en) *  20030704  20050127  Fuji Heavy Ind Ltd  Stereo wide visual field image processing apparatus 

2006
 20060328 JP JP2006087728A patent/JP4814669B2/en active Active
Also Published As
Publication number  Publication date 

JP2007263669A (en)  20071011 
Similar Documents
Publication  Publication Date  Title 

EP2733457B1 (en)  Position and orientation calibration method and apparatus  
KR101188588B1 (en)  Monocular Motion StereoBased Free Parking Space Detection Apparatus and Method  
US20100246901A1 (en)  Operation Support System, Vehicle, And Method For Estimating ThreeDimensional Object Area  
JP2010072807A (en)  Device for detecting and determining road boundary  
US9589326B2 (en)  Depth image processing apparatus and method based on camera pose conversion  
JP3719095B2 (en)  Behavior detection apparatus and gradient detection method  
JP4480083B2 (en)  Object recognition device  
US20090015675A1 (en)  Driving Support System And Vehicle  
US9578310B2 (en)  Automatic scene calibration  
EP2959315B1 (en)  Generation of 3d models of an environment  
JP4919036B2 (en)  Moving object recognition device  
US20100204964A1 (en)  Lidarassisted multiimage matching for 3d model and sensor pose refinement  
JP2009041972A (en)  Image processing device and method therefor  
JP2007129560A (en)  Object detector  
JP3976900B2 (en)  Method for determining viewpoint position and orientation and camera device  
US9509979B2 (en)  Stereo autocalibration from structurefrommotion  
JP4406381B2 (en)  Obstacle detection apparatus and method  
WO2017163596A1 (en)  Autonomous navigation using visual odometry  
KR20120084635A (en)  Apparatus and method for estimating camera motion using depth information, augmented reality system  
US8872920B2 (en)  Camera calibration apparatus  
JP2013232195A5 (en)  
EP2169589A2 (en)  Image processing device for vehicle and corresponding method and computer program product  
CN103020952A (en)  Information processing apparatus and information processing method  
JP6107081B2 (en)  Image processing apparatus, image processing method, and program  
EP2380136B1 (en)  A method of estimating a motion of a multiple camera system, a multiple camera system and a computer program product 
Legal Events
Date  Code  Title  Description 

A621  Written request for application examination 
Free format text: JAPANESE INTERMEDIATE CODE: A621 Effective date: 20080807 

A977  Report on retrieval 
Free format text: JAPANESE INTERMEDIATE CODE: A971007 Effective date: 20110224 

A131  Notification of reasons for refusal 
Free format text: JAPANESE INTERMEDIATE CODE: A131 Effective date: 20110308 

A521  Written amendment 
Free format text: JAPANESE INTERMEDIATE CODE: A523 Effective date: 20110406 

TRDD  Decision of grant or rejection written  
A01  Written decision to grant a patent or to grant a registration (utility model) 
Free format text: JAPANESE INTERMEDIATE CODE: A01 Effective date: 20110802 

A01  Written decision to grant a patent or to grant a registration (utility model) 
Free format text: JAPANESE INTERMEDIATE CODE: A01 

A61  First payment of annual fees (during grant procedure) 
Free format text: JAPANESE INTERMEDIATE CODE: A61 Effective date: 20110826 

R150  Certificate of patent or registration of utility model 
Ref document number: 4814669 Country of ref document: JP Free format text: JAPANESE INTERMEDIATE CODE: R150 Free format text: JAPANESE INTERMEDIATE CODE: R150 

FPAY  Renewal fee payment (event date is renewal date of database) 
Free format text: PAYMENT UNTIL: 20140902 Year of fee payment: 3 

FPAY  Renewal fee payment (event date is renewal date of database) 
Free format text: PAYMENT UNTIL: 20140902 Year of fee payment: 3 

R250  Receipt of annual fees 
Free format text: JAPANESE INTERMEDIATE CODE: R250 

R250  Receipt of annual fees 
Free format text: JAPANESE INTERMEDIATE CODE: R250 

R250  Receipt of annual fees 
Free format text: JAPANESE INTERMEDIATE CODE: R250 

R250  Receipt of annual fees 
Free format text: JAPANESE INTERMEDIATE CODE: R250 

R250  Receipt of annual fees 
Free format text: JAPANESE INTERMEDIATE CODE: R250 

R250  Receipt of annual fees 
Free format text: JAPANESE INTERMEDIATE CODE: R250 