JP2013104660A - Momentum estimation method based on stereo vision using monocular camera image and momentum estimation device using the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method of calculating momentum and three-dimensional coordinates with high accuracy, while reducing an arithmetic load, upon performing momentum estimation based on monocular stereo vision.SOLUTION: Upon performing momentum estimation based on stereo vision using an image captured by a monocular camera, in consideration of each optical flow direction, feature points that form optical flows different in direction are efficiently sampled to be used for calculating a basic matrix. Concretely, in sampling the feature points that form the optical flows to be used for calculating the basic matrix, the sampled feature points are sorted into a predetermined number of groups on the basis of the direction of the optical flow. The feature points for the group including a plurality of feature points is converged. As for the group including no feature point, feature points are selected again. The above processing is repeated so that each of all the groups has one feature point. On the basis of the feature points, a basic matrix is calculated.

Description

  The present invention relates to a momentum estimation method based on stereo vision using an image captured by a camera attached to a moving body such as a vehicle, and a momentum estimation apparatus using the method. More specifically, the present invention relates to a method for improving the estimation accuracy of a method for estimating momentum based on stereo vision using an image captured by a monocular camera attached to a moving body such as a vehicle, and momentum estimation using the method. Relates to the device.

  2. Description of the Related Art Conventionally, a system has been developed in which an image of a peripheral object is captured by a camera mounted on a moving body such as a vehicle and the surrounding environment is three-dimensionally recognized. As a method for measuring the three-dimensional coordinates (distance) from a camera image, stereo vision using a plurality of cameras (hereinafter, also referred to as “compound eye stereo vision”) is common, but even a single camera is different. It is known that stereo vision can be realized by using a plurality of images taken from viewpoints (different positions and postures) (hereinafter also referred to as “monocular stereo vision”).

  As is well known in the technical field, as a method for estimating the momentum based on monocular stereo vision, for example, there is a method using optical flow (optical flow) of feature points in a plurality of images taken from different viewpoints. Can be mentioned. In this method, first, two images are taken from different viewpoints, and the taken images are taken into an image processing apparatus. A plurality of feature points are calculated from one of the two images thus captured. Points corresponding to the plurality of feature points are searched from the other image, and an optical flow indicating the movement state of the feature points is calculated. A predetermined number of feature points are selected from a plurality of feature points, a basic matrix is calculated based on the selected feature points, and an epipolar error is calculated for the selected feature points based on the calculated basic matrix. The selection of the feature points, the calculation of the basic matrix, and the calculation of the epipolar error are repeated, and the optimal basic matrix showing the smallest epipolar error is selected. The momentum of the camera (a moving body such as a vehicle on which the camera is mounted) is calculated based on the optimal basic matrix thus selected, and the three-dimensional coordinates of the feature points are calculated based on the calculated momentum.

  As described above, when estimating the momentum based on monocular stereo vision using the optical flow, every time a predetermined number of feature points used for calculation of each basic matrix is selected, an epipolar about the selected feature points is selected. Since it is necessary to obtain an error, the calculation load required to select an optimal base matrix is very large. Accordingly, in the technical field, various methods have been proposed for the purpose of reducing the calculation load.

  For example, instead of obtaining an epipolar error for a predetermined number of feature points that are selected each time an individual basic matrix is calculated, a camera (equipped with information obtained from momentum detection means such as a wheel speed sensor) is installed. Calculating or estimating a temporary momentum of a moving body such as a vehicle, etc., estimating a temporary basic matrix based on the temporary momentum obtained in this way, and A method has been proposed in which an epipolar error for a feature point is calculated in advance, a feature point having a large epipolar error is removed, and an optimal basic matrix is calculated based on the remaining feature points (see, for example, Patent Document 1). ).

  According to the above method, it is not necessary to obtain an epipolar error for a selected feature point every time a predetermined number of feature points are selected for calculation of each basic matrix. The required calculation load can be reduced. However, since optical flow has an error in the first place, when a predetermined number of feature points used for calculation of each basic matrix are selected randomly and unconditionally as in the above method, it is calculated due to the error of optical flow. The underlying matrix may not always be correct. Therefore, in order to increase the calculation accuracy of the basic matrix, it is necessary to increase the number of feature points used for calculation of each basic matrix to reduce the influence of errors of the optical flow. As a result, even in the method as described above, it is difficult to sufficiently reduce the calculation load required to calculate the optimal basic matrix if the accuracy of calculation of momentum and three-dimensional coordinates is to be increased. .

  Therefore, in this technical field, for the purpose of reducing the influence of the error of the optical flow, the feature points used for calculation of each basic matrix are not randomly selected unconditionally (sampling), but individually. In consideration of the size of the optical flow, it has been proposed to select a predetermined number of feature points forming optical flows having different sizes and use them in the calculation of the basic matrix (see, for example, Patent Document 2). .

  However, in the above method, when selecting a predetermined number of feature points used for calculation of each basic matrix, the size of the optical flow is considered, but the direction of the optical flow is not considered, so It is difficult to sufficiently increase the calculation accuracy of momentum and three-dimensional coordinates.

  As described above, in this technical field, there is a continuous demand for a technology that can realize a momentum estimation based on stereo vision using a monocular camera image with high accuracy while reducing a calculation load.

JP 2007-256029 A JP 2010-085240 A

    Multiple View Geometry in Computer Vision, Second Edition, 2003, Richard Hartley & Andrew Zisserman, Cambridge University Press

  As described above, in this technical field, there is a continuous demand for a technique that can realize a momentum estimation based on stereo vision using a monocular camera image with high accuracy while reducing a calculation load. The present invention has been made to meet such a demand. More specifically, one object of the present invention is to provide a method for calculating the momentum and the three-dimensional coordinates with high accuracy while reducing the calculation load in performing the momentum estimation based on monocular stereo vision. is there.

One object of the present invention is to
An imaging means mounted on the moving body for imaging a predetermined area;
Feature point extracting means for extracting a plurality of feature points from a first image picked up when the image pickup means is in a first position and a first posture;
A point corresponding to each of the feature points extracted from the first image by the feature point extraction unit is a second position and a second point where the imaging unit is different from the first position and the first posture. An optical flow calculating unit that searches the second image captured when the posture is in the position and calculates an optical flow that represents a moving state of the feature point;
A feature point selection step of selecting a predetermined number n of feature points from the plurality of feature points extracted by the feature point extraction means; and a base matrix calculation step of calculating a base matrix based on the selected n feature points; An error calculation step of calculating epipolar errors for all feature points extracted by the feature point extraction means based on the calculated basic matrix, and repeating the predetermined N times, N A basic matrix calculation means for selecting a basic matrix indicating the smallest epipolar error among the basic matrices of as an optimal basic matrix;
By calculating the momentum of the imaging means based on the points corresponding to the feature points obtained by the optical flow from the optimal basic matrix calculated by the basic matrix calculation means and the internal parameters of the imaging means, A momentum calculating means for calculating the momentum of the moving body;
Three-dimensional coordinate calculation means for calculating three-dimensional coordinates of individual feature points based on the momentum of the imaging means calculated by the momentum calculation means;
In the momentum estimation device comprising:
In the feature point selection step, after selecting a predetermined number n of feature points from the plurality of feature points extracted by the feature point extraction means,
A grouping step for dividing the selected n feature points into predetermined n groups based on the direction of the optical flow;
Among the n groups, with respect to a group having two or more feature points belonging to the group, only one of the two or more feature points belonging to the group is left and the other feature points are deleted. A feature point convergence step;
A feature point reselecting step of selecting again the same number of feature points as the number of the groups of the n groups for which no feature points belong to the group,
Is repeated until one feature point belongs to each of the n groups.
This is achieved by the momentum estimation method.

  As described above, in the momentum estimation method according to the present invention, when performing momentum estimation based on stereo vision using an image captured by a monocular camera, the feature points that form optical flows having different directions are efficiently sampled. Therefore, since it is used for calculation of the basic matrix, the momentum of the camera and the three-dimensional coordinates of the feature points can be calculated with high accuracy while reducing the calculation load.

It is a flowchart showing the outline | summary of the flow of the various processes performed in the momentum estimation method which concerns on one embodiment of this invention. It is a flowchart showing the flow of the various processes performed in the base matrix calculation step (S400) included in the flowchart shown in FIG. It is a flowchart showing the flow of the various processes performed in the feature point selection step (S410) included in the flowchart shown in FIG.

  As described above, one object of the present invention is to provide a method for calculating the momentum and the three-dimensional coordinates with high accuracy while reducing the calculation load when performing the momentum estimation based on monocular stereo vision. .

  As a result of diligent research to achieve the above object, the present inventor, when performing momentum estimation based on stereo vision using an image captured by a monocular camera, uses an optical flow used to calculate individual basic matrices as in the prior art. Rather than randomly and unconditionally selecting feature points (corresponding points) that form an image, the feature points that form optical flows having different orientations are efficiently sampled in consideration of the direction of each optical flow. The present invention has been conceived by finding that it is possible to calculate the momentum of the camera and the three-dimensional coordinates of the feature points with high accuracy while reducing the calculation load by using the calculation of the basic matrix.

That is, the first embodiment of the present invention is:
An imaging means mounted on the moving body for imaging a predetermined area;
Feature point extracting means for extracting a plurality of feature points from a first image picked up when the image pickup means is in a first position and a first posture;
A point corresponding to each of the feature points extracted from the first image by the feature point extraction unit is a second position and a second point where the imaging unit is different from the first position and the first posture. An optical flow calculating unit that searches the second image captured when the posture is in the position and calculates an optical flow that represents a moving state of the feature point;
A feature point selection step of selecting a predetermined number n of feature points from the plurality of feature points extracted by the feature point extraction means; and a base matrix calculation step of calculating a base matrix based on the selected n feature points; An error calculation step of calculating epipolar errors for all feature points extracted by the feature point extraction means based on the calculated basic matrix, and repeating the predetermined N times, N A basic matrix calculation means for selecting a basic matrix indicating the smallest epipolar error among the basic matrices of as an optimal basic matrix;
By calculating the momentum of the imaging means based on the points corresponding to the feature points obtained by the optical flow from the optimal basic matrix calculated by the basic matrix calculation means and the internal parameters of the imaging means, A momentum calculating means for calculating the momentum of the moving body;
Three-dimensional coordinate calculation means for calculating three-dimensional coordinates of individual feature points based on the momentum of the imaging means calculated by the momentum calculation means;
In the momentum estimation device comprising:
In the feature point selection step, after selecting a predetermined number n of feature points from the plurality of feature points extracted by the feature point extraction means,
A grouping step for dividing the selected n feature points into predetermined n groups based on the direction of the optical flow;
Among the n groups, with respect to a group having two or more feature points belonging to the group, only one of the two or more feature points belonging to the group is left and the other feature points are deleted. A feature point convergence step;
A feature point reselecting step of selecting again the same number of feature points as the number of the groups of the n groups for which no feature points belong to the group,
Is repeated until one feature point belongs to each of the n groups.
It is a momentum estimation method.

  The imaging means is an imaging device such as a CCD (Charge Coupled Device) camera, for example, and is mounted on a moving body such as a vehicle to image a predetermined area such as the front of the moving body. The captured image can be stored in a data storage unit such as an image memory, for example, as an A / D (analog / digital) converted image signal.

  The feature point extracting unit extracts a plurality of feature points from the first image captured when the imaging unit is in the first position and the first posture. As is well known to those skilled in the art, a feature point is a point required to define the shape of a boundary in an image, and corresponds to, for example, a corner or an edge. For the extraction of such feature points, various methods widely known in the technical field can be used. Specifically, for example, a Harris operator or a SUSAN operator can be used to extract feature points.

  The optical flow calculating means determines a point corresponding to each of the feature points extracted from the first image by the feature point extracting means by using a second position different from the first position and the first posture by the imaging means. A search is made from the second image captured when the camera is in the second posture, and an optical flow representing the movement state of the feature point is calculated. As is well known to those skilled in the art, the optical flow is a trajectory in which feature points have moved between the first image and the second image. In other words, the optical flow is obtained by obtaining a velocity field of an image using a set of time-series images and expressing the velocity field as a vector set.

  For the calculation of the optical flow, various methods widely known in the art can be used. In general, optical flow calculation methods can be broadly classified into gradient methods and block matching methods. In the gradient method, an optical flow is obtained by adding a constraint condition to a constraint equation of luminance temporal / spatial differentiation (luminance gradient) called “optical flow constraint equation”. Typical gradient methods include, for example, the Lucas-Kanade method and the KLT (Kanade-Lucas-Tomasi) tracker. On the other hand, in the block matching method, a specific block in one image is used as a template, and an optical flow is obtained by searching for a corresponding portion in the other image.

  The basic matrix calculation means calculates a basic matrix based on the feature point selection step of selecting a predetermined number n of feature points from the plurality of feature points extracted by the feature point extraction means, and the selected n feature points. A basic matrix calculation step, and an error calculation step of calculating epipolar errors for all feature points extracted by the feature point extraction means based on the calculated basic matrix, are repeated a predetermined number of times N. A basic matrix indicating the smallest epipolar error among the obtained N basic matrices is selected as the optimal basic matrix.

  The basic matrix calculation means selects a predetermined number n of feature points from the plurality of feature points extracted by the feature point extraction means in the feature point selection step. The momentum estimation method according to the present embodiment is characterized by the feature point selection step, which will be described later in detail. The predetermined number n, which is the number of feature points selected in the feature point selection step, can be set to an arbitrary natural number according to, for example, an algorithm used for calculation of a basic matrix described later. As the predetermined number n increases, the calculation accuracy of each basic matrix increases, but the calculation load required for calculating the basic matrix also increases. Conversely, the smaller the predetermined number n, the lower the calculation load required for calculating each basic matrix, but the lower the calculation accuracy of the basic matrix.

  Next, the basic matrix calculation means calculates a basic matrix based on the selected n feature points in the basic matrix calculation step. As is well known to those skilled in the art, a basic matrix is a basic matrix that mathematically describes point correspondence between images in stereo vision based on epipolar geometry. In other words, the basic matrix is a matrix that represents epipolar constraints between images. For the calculation of the basic matrix, various methods widely known in the technical field can be used. Specifically, for example, a known Hartley eight-point algorithm, a seven-point algorithm, a five-point algorithm, or the like can be used to calculate the basic matrix.

  Next, the basic matrix calculation means calculates an epipolar error for all the feature points extracted by the feature point extraction means based on the calculated basic matrix in the error calculation step. As is well known to those skilled in the art, the epipolar error is the distance between the epipolar line and the feature point based on the calculated basic matrix.

  The basic matrix calculation means repeats the above feature point selection step, basic matrix calculation step, and error calculation step a predetermined number of times N, and the basic matrix indicating the smallest epipolar error among the N basic matrices thus obtained. Select a matrix as the optimal basis matrix. As a result, for example, the optical flow calculation means sets the points (corresponding points) corresponding to each of the feature points extracted from the first image by the feature point extraction means, and the imaging means sets the first position and the first posture. When there is a large error in some of the corresponding points, such as when an incorrect point is detected when searching from the second image captured when the second position and the second posture are different from In addition, the influence of such outliers (outliers) can be eliminated, and the calculation accuracy of the basic matrix can be improved. In other words, in the calculation of the optimal basic matrix by the basic matrix calculation means, it is possible to improve the calculation accuracy of the basic matrix by using a robust method called RANSAC (RANdam Sample Sample Consensus) that uses the epipolar error as an evaluation amount. it can.

  Note that the predetermined number N, which is the number of times to repeatedly execute the feature point selection step, the basic matrix calculation step, and the error calculation step, is, for example, Multiple View Geometry Measurement, Second Edition, 2003, Richard Hartley & Andrew Zis. It can be set to an arbitrary natural number according to a statistical technique described in Cambridge University Press (Non-patent Document 1). As the predetermined number N is increased, the calculation accuracy of the optimum basic matrix to be finally selected increases, but the calculation load required for calculating the optimal basic matrix also increases. Conversely, the smaller the predetermined number N, the lower the calculation load required for calculating the optimum basic matrix, but the lower the calculation accuracy of the optimal basic matrix.

  The momentum calculating means calculates the momentum of the imaging means based on the points corresponding to the feature points obtained by the optical flow and the internal parameters of the imaging means, from the optimum basic matrix calculated by the basic matrix calculating means. Thus, the momentum of the moving body is calculated. As is well known to those skilled in the art, the internal parameters of the imaging means refer to, for example, the focal length, angle of view, optical axis point of the lens provided in the imaging means.

  The three-dimensional coordinate calculating means calculates three-dimensional coordinates of individual feature points based on the momentum of the imaging means calculated by the momentum calculating means.

  It should be noted that the feature point extracting means, optical flow calculating means, basic matrix calculating means, momentum calculating means, and three-dimensional coordinate calculating means provided in the momentum estimating apparatus using the momentum estimating method according to the present embodiment are not necessarily individual. It need not be configured as a component. For example, these various means may be implemented as a function realized in an electronic control unit (ECU: Electronic Control Unit) provided in a moving body such as a vehicle on which the momentum estimation device is mounted. These various means may be implemented in an ECU dedicated to the momentum estimation device, or may be implemented in an ECU for other purposes provided in a moving body such as a vehicle on which the momentum estimation device is mounted. Good. Further, these various means may be implemented in a single ECU, or may be implemented in a distributed manner in a plurality of ECUs.

  In addition, as a configuration of the ECU, for example, a central processing unit (CPU: Central Processing Unit) (for example, a microcomputer) for executing arithmetic processing corresponding to various processes performed by the various means, for example, A data storage device (for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a HDD (Hard Disk) for storing a program for causing the CPU to execute the arithmetic processing and the like, an image signal from the imaging means, and the like. Drive) etc.), for example, a configuration including a data input / output port or the like for receiving an image signal or the like from the imaging means, or sending the result of the arithmetic processing to another device (for example, a display device or the like) Can be mentioned. However, the configuration for realizing various means included in the momentum estimation apparatus using the momentum estimation method according to the present embodiment is not limited by the above description, and any configuration as long as the function of each means can be realized. It may be.

  As described above, the momentum estimation method according to the present embodiment includes an imaging unit, a feature point extraction unit, an optical flow calculation unit, a basic matrix calculation unit, a momentum calculation unit, and a three-dimensional coordinate calculation unit. In the momentum estimation device provided, when performing momentum estimation based on stereo vision, the feature points (corresponding points) that form the optical flow used for calculation of each basic matrix are selected randomly and unconditionally as in the prior art. In addition, taking into account the direction of individual optical flows, the feature points that form optical flows with different orientations are efficiently sampled and used to calculate the basic matrix, thereby reducing the computational load and reducing the momentum of the camera. And the three-dimensional coordinates of the feature points are calculated with high accuracy.

Specifically, according to the momentum estimation method according to the present embodiment, after selecting a predetermined number n of feature points from a plurality of feature points extracted by the feature point extraction unit in the feature point selection step,
A grouping step for dividing the selected n feature points into predetermined n groups based on the direction of the optical flow;
Among the n groups, with respect to a group having two or more feature points belonging to the group, only one of the two or more feature points belonging to the group is left and the other feature points are deleted. A feature point convergence step;
A feature point reselecting step of selecting again the same number of feature points as the number of the groups of the n groups for which no feature points belong to the group,
Is repeated until one feature point belongs to all of the n groups.

  In the grouping step, the selected n feature points are sorted into n groups determined in advance based on the direction of the optical flow. The n groups predetermined based on the direction of the optical flow are, for example, n groups (that is, selected according to the range of the angle of the direction of the optical flow formed by individual feature points). It may be a group of the same number as the number of feature points). More specifically, the n groups predetermined based on the direction of the optical flow are, for example, X− where the horizontal direction in the image captured by the imaging unit is the X axis direction and the vertical direction is the Y axis direction. When the angle of the positive direction on the X axis in the Y coordinate plane is 0 ° (zero degree) and the counterclockwise direction is defined as the direction in which the angle increases, 360 ° is defined as n angular regions. The feature points can be grouped according to which angle region the angle of the direction of each optical flow (hereinafter sometimes simply referred to as “θ”) corresponds. In this case, the n angle regions may be equally divided or may be unevenly divided. Further, the angle (θ) of the direction of the optical flow formed by the individual feature points can be geometrically calculated from the coordinates of the individual feature points in the first image and the coordinates in the second image. .

  In the feature point convergence step, as a result of the n feature points being assigned to the n groups in the grouping step, among the n groups, there are groups in which two or more feature points belonging to the group exist. If it exists, only one of the two or more feature points belonging to the group remains, and the other feature points are deleted, so that the feature points belonging to the group converge to one. At this time, one feature point remaining in a group to which two or more feature points belong (hereinafter may be referred to as “representative feature points”) can be selected randomly and unconditionally, for example.

  Or the angle of the direction of the optical flow formed by the individual feature points belonging to the group to which two or more feature points are distributed and the direction of the optical flow formed by the representative feature points of two groups adjacent to the group The angle difference (d1) between the representative feature point of one of the two groups adjacent to the group and the representative feature point of the other group (angle difference) is calculated. A feature point that is more equal to d2) (a difference between d1 and d2 is smaller) can be selected as a representative feature point of the group.

  The “adjacent group” in the above description refers to a group in which the angular areas of the optical flow directions defining the respective groups are adjacent. For example, 360 ° is equally divided into 45 ° angular regions, and eight groups corresponding to the individual angular regions are divided into group 1, 45 ° to 90 ° angular regions corresponding to 0 ° to 45 ° angular regions. When the group 8 corresponding to the corresponding group 2,..., 315 to 360 ° angle region is set, the two groups adjacent to the group 2 are the group 1 and the group 3.

  In the feature point reselection step, as a result of the n feature points being assigned to the n groups in the grouping step, among the n groups, the feature points belonging to the group do not exist. The same number of feature points as the number of the group is selected again. As a result of the feature point convergence step, there is one feature point in every group where the feature points belonging to the group exist, so the number of groups where no feature points belonging to the group exist in the feature point reselection step As a result, the predetermined number n of feature points are newly selected from the plurality of feature points extracted by the feature point extracting means.

  As described above, according to the momentum estimation method according to the present embodiment, in the feature point selection step, the grouping step, the feature point convergence step, and the feature point reselection step include feature points in all n groups. Is repeated until one of them becomes a state. Thereby, it is possible to efficiently sample feature points forming optical flows having various different directions and use them for calculation of the basic matrix. As a result, the calculation accuracy of each basic matrix increases, and the calculation accuracy of the three-dimensional coordinates of the momentum of the moving object and the feature points also increases.

  Further, according to the momentum estimation method according to the present embodiment, when selecting (sampling) the feature points used for calculation of the basic matrix, the optical flows having various different directions are considered in consideration of the directions of the individual optical flows. It is possible to positively and efficiently sample the feature points forming. As a result, for example, it is possible to reduce the possibility of calculating a basic matrix indicating a larger error because, for example, the feature points forming the optical flow having the direction corresponding to the specific angle region are sampled biased. As a result, it is possible to reduce the calculation load required for calculating the optimum basic matrix.

  That is, according to the momentum estimation method according to the present embodiment, it provides a method for calculating the momentum and the three-dimensional coordinates with high accuracy while reducing the calculation load when performing the momentum estimation based on monocular stereo vision. One object of the present invention can be achieved.

  By the way, as described above, the basic matrix calculation means selects a predetermined number n of feature points from the plurality of feature points extracted by the feature point extraction means in the feature point selection step. The predetermined number n can be set to an arbitrary natural number according to, for example, an algorithm used for calculating the basic matrix. On the other hand, for calculation of the basic matrix, various methods widely known in the technical field such as the known Hartley 8-point algorithm, 7-point algorithm, and 5-point algorithm can be used.

  As described above, various algorithms can be used to calculate the basic matrix. For example, when using an algorithm that uses eight feature points in the calculation of the basic matrix, such as the Hartley 8-point algorithm described above, the predetermined number n is 8. Such an embodiment is also included in the scope of the present invention as one embodiment of the present invention.

Accordingly, the second embodiment of the present invention provides:
A momentum estimation method according to the first embodiment of the present invention, comprising:
The predetermined number n is 8.
It is a momentum estimation method.

  As described above, in the momentum estimation method according to the present embodiment, the predetermined number n is 8. That is, according to the momentum estimation method according to the present embodiment, eight feature points are selected from the plurality of feature points extracted by the feature point extraction means in the feature point selection step described above. Thereafter, in the grouping step, the selected eight feature points are divided into eight groups predetermined based on the direction of the optical flow, and in the feature point convergence step, among the eight groups, For a group having two or more feature points belonging to a group, only one of the two or more feature points belonging to the group is left, and other feature points are deleted. Among the groups, for a group in which no feature point belonging to the group exists, the same number of feature points as the number of the group is selected again. These steps are repeatedly executed, and eventually one feature point belongs to all eight groups.

  When eight feature points are selected (sampled) as described above, a basic matrix is calculated based on the selected eight feature points in the basic matrix calculation step described above. At this time, for the calculation of the basic matrix, for example, an algorithm that uses eight feature points for calculation of the basic matrix, such as the aforementioned Hartley 8-point algorithm, is used.

  By the way, as described above, the present invention also relates to a momentum estimation apparatus that uses various momentum estimation methods including the various embodiments described above and modifications thereof. Therefore, an embodiment of the present invention as such a momentum estimation apparatus will be described below. However, since it overlaps with the description of the above-described embodiment of the present invention as a momentum estimation method, the embodiment of the present invention as a momentum estimation apparatus will be described. Detailed explanation is omitted.

That is, the third embodiment of the present invention
An imaging means mounted on the moving body for imaging a predetermined area;
Feature point extracting means for extracting a plurality of feature points from a first image picked up when the image pickup means is in a first position and a first posture;
A point corresponding to each of the feature points extracted from the first image by the feature point extraction unit is a second position and a second point where the imaging unit is different from the first position and the first posture. An optical flow calculating unit that searches the second image captured when the posture is in the position and calculates an optical flow that represents a moving state of the feature point;
A feature point selection step of selecting a predetermined number n of feature points from the plurality of feature points extracted by the feature point extraction means; and a base matrix calculation step of calculating a base matrix based on the selected n feature points; An error calculation step of calculating epipolar errors for all feature points extracted by the feature point extraction means based on the calculated basic matrix, and repeating the predetermined N times, N A basic matrix calculation means for selecting a basic matrix indicating the smallest epipolar error among the basic matrices of as an optimal basic matrix;
By calculating the momentum of the imaging means based on the points corresponding to the feature points obtained by the optical flow from the optimal basic matrix calculated by the basic matrix calculation means and the internal parameters of the imaging means, A momentum calculating means for calculating the momentum of the moving body;
Three-dimensional coordinate calculation means for calculating three-dimensional coordinates of individual feature points based on the momentum of the imaging means calculated by the momentum calculation means;
In the momentum estimation device comprising:
In the feature point selection step, after selecting a predetermined number n of feature points from the plurality of feature points extracted by the feature point extraction means,
A grouping step for dividing the selected n feature points into predetermined n groups based on the direction of the optical flow;
Among the n groups, with respect to a group having two or more feature points belonging to the group, only one of the two or more feature points belonging to the group is left and the other feature points are deleted. A feature point convergence step;
A feature point reselecting step of selecting again the same number of feature points as the number of the groups of the n groups for which no feature points belong to the group,
Is repeated until one feature point belongs to each of the n groups.
It is a momentum estimation device.

The fourth embodiment of the present invention is
A momentum estimation apparatus according to the third embodiment of the present invention,
The predetermined number n is 8.
It is a momentum estimation device.

  The configuration and the like of the momentum estimation method according to some embodiments of the present invention will be described below with reference to the accompanying drawings. However, the following description is for illustrative purposes only, and the scope of the present invention should not be construed as being limited to the following description.

1. Flow of various processes executed in the momentum estimation method (outline)
FIG. 1 is a flowchart showing an overview of the flow of various processes executed in the method of estimating the momentum according to one embodiment of the present invention as described above. As shown in FIG. 1, the momentum estimation method according to the present embodiment is
An image capturing step (S100) for capturing a predetermined area by an imaging means (for example, a camera) mounted on a moving body such as a vehicle and capturing the captured image;
A feature point extracting step (S200) for extracting a plurality of feature points from a first image captured when the imaging means is in the first position and the first posture;
A point corresponding to each of the feature points extracted from the first image in the feature point extraction step (S200) is a second position where the imaging means is different from the first position and the first posture. And an optical flow calculating step (S300) for searching from the second image captured when in the second posture and calculating an optical flow representing the moving state of the feature point;
A basic matrix calculation step (S400) in which a predetermined number n of feature points are selected from the plurality of feature points extracted in the feature point extraction step (S200), and a basic matrix is calculated based on the selected n feature points. When,
From the basic matrix calculated in the basic matrix calculation step (S400), the momentum of the imaging means is calculated based on corresponding points (points corresponding to feature points) obtained by optical flow and internal parameters of the imaging means. From the momentum calculation step (S500) for calculating the momentum of the moving body,
A three-dimensional coordinate calculating step (S600) for calculating three-dimensional coordinates of individual feature points based on the momentum of the imaging means calculated in the momentum calculating step (S500);
including.

  In the image capturing step (S100), as described above, for example, an imaging unit such as a CCD (Charge Coupled Device) camera is mounted on a moving body such as a vehicle, and a predetermined area such as the front of the moving body, for example. A plurality of images are taken. The captured image can be stored in a data storage unit such as an image memory, for example, as an A / D (analog / digital) converted image signal. In addition, the said imaging means image | photographs the condition of the moving body periphery which changes every moment, for example. Therefore, the imaging unit may be of a type that captures an image at a predetermined frame rate (the number of images captured per unit time) (for example, a moving image capturing unit).

  In the feature point extraction step (S200), from among the images captured in the image capture step (S100), from the first image captured when in the first position and the first posture, for example, A plurality of feature points such as corners and edges are extracted. As described above, for example, a Harris operator or a SUSAN operator can be used to extract feature points.

  In the optical flow calculation step (S300), the optical flow representing the movement state of the feature points extracted in the feature point extraction step (S200) is selected from the images captured in the image capture step (S100). It calculates using the 2nd image imaged when it exists in a 2nd position and a 2nd attitude | position. As described above, a gradient method such as the Lucas-Kanade method or a block matching method can be used for calculating the optical flow.

  In the basic matrix calculation step (S400), a predetermined number n of feature points are selected from the plurality of feature points extracted in the feature point extraction step (S200), and the basis is based on the selected n feature points. Calculate the matrix. The momentum estimation method according to the present embodiment has a feature in the feature point selection method, and details will be described later. As described above, for example, the known Hartley 8-point algorithm, 7-point algorithm, and 5-point algorithm can be used to calculate the basic matrix. Furthermore, as described above, in the momentum estimation method according to the present embodiment, for the purpose of improving the calculation accuracy of the basic matrix, for example, in the calculation of the basic matrix using the RANSAC method with the epipolar error as the evaluation amount. It is possible to reduce the influence of outliers (outliers) and improve the calculation accuracy of the basic matrix. The improvement of the calculation accuracy of the basic matrix in the momentum estimation method according to the present embodiment will also be described in detail later.

  In the momentum calculating step (S500), corresponding points (points corresponding to feature points) obtained by optical flow from the basic matrix calculated in the basic matrix calculating step (S400) and internal parameters of the imaging means (for example, The momentum of the imaging means is calculated based on the focal length, angle of view, optical axis point, etc. of the lens provided in the imaging means. As described above, since the imaging unit is mounted on the moving body, the momentum of the moving body can be calculated by calculating the momentum of the imaging unit.

  In the three-dimensional coordinate calculation step (S600), the three-dimensional coordinates of the individual feature points are calculated based on the momentum of the imaging means calculated in the momentum calculation step (S500). Thus, according to the momentum estimation method according to the present embodiment, the moving body is obtained by capturing an image around the moving body with an imaging unit (for example, a monocular camera) mounted on the moving body such as a vehicle. The surrounding environment can be recognized in three dimensions.

  In the momentum estimation method according to the present embodiment, as described above, the RANSAC method is used to improve the calculation accuracy of the basic matrix, and the feature point sampling method used for calculating the basic matrix is devised to improve the basic matrix calculation. The calculation load required for the calculation is reduced. Therefore, these features in the method for estimating momentum according to the present embodiment will be described below.

2. 2. Improvement of calculation accuracy of basic matrix by RANSAC method using epipolar error as evaluation quantity FIG. 2 is a flow of various processes executed in the basic matrix calculation step (S400) included in the flowchart shown in FIG. It is a flowchart showing. As shown in FIG. 2, in the basic matrix calculation step (S400) of the momentum estimation method according to the present embodiment, the above-mentioned feature point selection step (S410), basic matrix calculation step (S420), and error calculation step (S430). ) Is repeated a predetermined number of times N, and then, in the optimal basic matrix selection step (S440), the basic matrix indicating the smallest epipolar error among the N basic matrices thus obtained is selected as the optimal basic matrix. .

  As described above, for example, in the above-described optical flow calculation step (S300), the imaging unit sets the points corresponding to each of the feature points extracted from the first image by the feature point extraction unit (corresponding points) at the first position. And when searching from the second image captured when the second position and the second posture are different from the first posture, it is large for some corresponding points, such as when an erroneous point is detected. When there is an error, the influence of such outliers (outliers) can be eliminated, and the calculation accuracy of the basic matrix can be improved. That is, a series of processes by the feature point selection step (S410), the basic matrix calculation step (S420), the error calculation step (S430), and the optimum basic matrix selection step (S440) executed in the basic matrix calculation step (S400). This flow corresponds to the RANSAC method using an epipolar error as an evaluation amount, and can improve the calculation accuracy of the optimum basic matrix.

  Next, in the momentum estimation method according to this embodiment, the feature point sampling method used for calculation of the basic matrix is devised to reduce the calculation load required for calculation of the basic matrix. This will be described below.

3. 3. Improvement of basic matrix calculation accuracy by RANSAC method using epipolar error as evaluation quantity FIG. 3 shows the flow of various processes executed in the feature point selection step (S410) included in the flowchart shown in FIG. It is a flowchart showing. As shown in FIG. 3, in the feature point selection step (S410) included in the basic matrix calculation step (S400) of the momentum estimation method according to the present embodiment, in the above-described feature point extraction step (S200) in step S411. A predetermined number n of feature points are selected from the extracted feature points. As described above, the predetermined number n can be set to an arbitrary natural number according to, for example, an algorithm used for calculating the basic matrix. In the present embodiment, description will be made assuming that the predetermined number n is 8.

  Next, in the grouping step (S412), the selected eight (n) feature points are divided into eight (n) groups predetermined based on the direction of the optical flow. Here, the eight (n) groups predetermined based on the direction of the optical flow are divided according to the range of the angle of the direction of the optical flow formed by the individual feature points as described above. N groups (eight in this embodiment). In the present embodiment, these eight groups are positive orientations on the X-axis in the XY coordinate plane in which the horizontal direction in the image captured by the imaging unit is the X-axis direction and the vertical direction is the Y-axis direction. When the angle is defined as 0 ° (zero degree) and the counterclockwise direction is defined as the direction in which the angle increases, 360 ° is set by dividing it into eight angle regions.

  That is, in this embodiment, 360 ° is equally divided into 45 ° angular regions, and eight groups corresponding to the individual angular regions are set. As a result, in this embodiment, the angle region of group 1 is 0 to 45 °, the angle region of group 2 is 45 to 90 °,..., And the angle region of group 8 is 315 to 360 °. Divided the group. However, in the present embodiment, the eight groups are divided so as to have an equal angle region, but as described above, the individual groups may be divided unevenly.

  Further, as described above, the angle (θ) of the direction of the optical flow formed by the individual feature points is calculated from the coordinates of the first feature image and the second image of the feature points. Can be calculated. More specifically, the coordinates of the feature point p1 in the first image captured when the imaging unit is in the first position and the first posture are (u1, v1), and the imaging unit is in the first position and the first position. Let (u2, v2) be the coordinates of the point p2 corresponding to the feature point p1 in the second image captured at the second position and the second posture different from the first posture. In this case, the angle (θ) of the direction of the optical flow formed by the feature point p1 can be calculated by the following equation, for example.

  In the grouping step (S412), each of the eight feature points selected in step S411 is based on the optical flow direction angle (θ) calculated as described above. Are classified into groups 1 to 8. As a result, eight feature points may be allocated to each of the eight groups, but a plurality of (ie, two or more) feature points are present in any of the eight groups. It is possible that an empty group will be allocated, resulting in no feature points being allocated. However, according to the present invention, as described above, when selecting feature points (corresponding points) forming an optical flow used for calculation of each basic matrix, the optical used for calculation of each basic matrix as in the prior art is selected. Rather than randomly and unconditionally selecting the feature points forming the flow, the feature points forming the optical flows having different orientations are efficiently sampled in consideration of the direction of the individual optical flows, and the base matrix By using it for calculation, it is intended to calculate the momentum of the camera and the three-dimensional coordinates of the feature points with high accuracy while reducing the calculation load.

  Therefore, in the momentum estimation method according to the present embodiment, it is confirmed whether or not a group to which a plurality of feature points belongs is present among the eight groups (step S413). When there is a group to which a plurality of feature points belong (step S413: Yes), in the feature point convergence step (S415), only one of the plurality of feature points belonging to the group is left and the other feature points are deleted. . At this time, one feature point (representative feature point) remaining in a group to which two or more feature points belong is selected, for example, randomly and unconditionally, or 2 adjacent to the group. It can be determined by selecting a feature point having a more uniform angular difference from the representative feature points of two groups.

  On the other hand, in the momentum estimation method according to the present embodiment, it is also checked whether or not there is an empty group to which no feature point belongs (step S414). If there is an empty group to which no feature point belongs (step S415: Yes), the same number of feature points as the number of empty groups is selected again in the feature point reselecting step (S416). As a result, since eight feature points are selected again, the process returns to the grouping step (S412) and the above steps are repeated.

  On the other hand, when there is no empty group to which no feature point belongs (step S415: No), it means that one feature point belongs to all eight groups. Therefore, in the momentum estimation method according to the present embodiment, the process proceeds to the next basic matrix calculation step (S420).

  As described above, according to the momentum estimation method according to the present embodiment, in the feature point selection step (S410), the grouping step (S412), the feature point convergence step (S415), and the feature point reselection step (S416). ) Is repeated until one feature point belongs to all eight groups. Thereby, it is possible to efficiently sample feature points forming optical flows having various different directions and use them for calculation of the basic matrix. As a result, according to the momentum estimation method according to the present embodiment, the calculation accuracy of each basic matrix is increased, and the momentum of the moving body and the calculation accuracy of the three-dimensional coordinates of the feature points are also increased.

  Although several embodiments having specific configurations have been described above for the purpose of illustrating the present invention, the scope of the present invention is not limited to these exemplary embodiments, and patents Needless to say, modifications can be made as appropriate within the scope of the claims and the description of the specification.

Claims (4)

An imaging means mounted on the moving body for imaging a predetermined area;
Feature point extracting means for extracting a plurality of feature points from a first image picked up when the image pickup means is in a first position and a first posture;
A point corresponding to each of the feature points extracted from the first image by the feature point extraction unit is a second position and a second point where the imaging unit is different from the first position and the first posture. An optical flow calculating unit that searches the second image captured when the posture is in the position and calculates an optical flow that represents a moving state of the feature point;
A feature point selection step of selecting a predetermined number n of feature points from the plurality of feature points extracted by the feature point extraction means; and a base matrix calculation step of calculating a base matrix based on the selected n feature points; An error calculation step of calculating epipolar errors for all feature points extracted by the feature point extraction means based on the calculated basic matrix, and repeating the predetermined N times, N A basic matrix calculation means for selecting a basic matrix indicating the smallest epipolar error among the basic matrices of as an optimal basic matrix;
By calculating the momentum of the imaging means based on the points corresponding to the feature points obtained by the optical flow from the optimal basic matrix calculated by the basic matrix calculation means and the internal parameters of the imaging means, A momentum calculating means for calculating the momentum of the moving body;
Three-dimensional coordinate calculation means for calculating three-dimensional coordinates of individual feature points based on the momentum of the imaging means calculated by the momentum calculation means;
In the momentum estimation device comprising:
In the feature point selection step, after selecting a predetermined number n of feature points from the plurality of feature points extracted by the feature point extraction means,
A grouping step for dividing the selected n feature points into predetermined n groups based on the direction of the optical flow;
Among the n groups, with respect to a group having two or more feature points belonging to the group, only one of the two or more feature points belonging to the group is left and the other feature points are deleted. A feature point convergence step;
A feature point reselecting step of selecting again the same number of feature points as the number of the groups of the n groups for which no feature points belong to the group,
Is repeated until one feature point belongs to each of the n groups.
The momentum estimation method. The momentum estimation method according to claim 1,
The predetermined number n is 8.
The momentum estimation method. An imaging means mounted on the moving body for imaging a predetermined area;
Feature point extracting means for extracting a plurality of feature points from a first image picked up when the image pickup means is in a first position and a first posture;
A point corresponding to each of the feature points extracted from the first image by the feature point extraction unit is a second position and a second point where the imaging unit is different from the first position and the first posture. An optical flow calculating unit that searches the second image captured when the posture is in the position and calculates an optical flow that represents a moving state of the feature point;
A feature point selection step of selecting a predetermined number n of feature points from the plurality of feature points extracted by the feature point extraction means; and a base matrix calculation step of calculating a base matrix based on the selected n feature points; An error calculation step of calculating epipolar errors for all feature points extracted by the feature point extraction means based on the calculated basic matrix, and repeating the predetermined N times, N A basic matrix calculation means for selecting a basic matrix indicating the smallest epipolar error among the basic matrices of as an optimal basic matrix;
By calculating the momentum of the imaging means based on the points corresponding to the feature points obtained by the optical flow from the optimal basic matrix calculated by the basic matrix calculation means and the internal parameters of the imaging means, A momentum calculating means for calculating the momentum of the moving body;
Three-dimensional coordinate calculation means for calculating three-dimensional coordinates of individual feature points based on the momentum of the imaging means calculated by the momentum calculation means;
In the momentum estimation device comprising:
In the feature point selection step, after selecting a predetermined number n of feature points from the plurality of feature points extracted by the feature point extraction means,
A grouping step for dividing the selected n feature points into predetermined n groups based on the direction of the optical flow;
Among the n groups, with respect to a group having two or more feature points belonging to the group, only one of the two or more feature points belonging to the group is left and the other feature points are deleted. A feature point convergence step;
A feature point reselecting step of selecting again the same number of feature points as the number of the groups of the n groups for which no feature points belong to the group,
Is repeated until one feature point belongs to each of the n groups.
Momentum estimation device. The momentum estimation apparatus according to claim 3,
The predetermined number n is 8.
Momentum estimation device.

Images