JP2009048516A - Information processor, information processing method and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To input images photographed by two cameras arranged at different positions almost in parallel and calculate a parallax offset based on the photographed images. <P>SOLUTION: Feature point pairs included in images photographed by two cameras are detected and a parallax offset in the photographed images of the two cameras is calculated by analyzing the feature point pairs. At first, a feature point pair determined to be highly reliable is selected by filtering the feature point pairs, a histogram of parallax offset data corresponding to the selected feature point pair is generated, a constraint in parallax offset values is set by using the histogram, and a parallax offset optimum value is calculated on the basis of the set constraint. In this configuration, even when a part of parameters of one camera is unknown, an optimum parallax offset can be calculated on the basis of the photographed images of respective cameras and a three-dimensional map can be generated. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an information processing apparatus, an information processing method, and a computer program. More specifically, the present invention relates to an information processing apparatus, an information processing method, and a computer program that perform a parallax offset calculation using images captured by two cameras and a three-dimensional map creation process.

  There is a technique in which images are taken by two cameras from different directions and a three-dimensional map is created based on analysis of the taken images. This method is called a stereo method, and a camera used for photographing is called a stereo camera.

  In order to create a three-dimensional map by the stereo method, a process for obtaining a correspondence relationship between each pixel (pixel) of a photographed image of a camera and a three-dimensional position in a real space is necessary. This camera calibration is necessary. For this, a calculation applying the parameters of each stereo camera is required.

  As camera parameters used for camera calibration, for example, the lens focal length of each stereo camera, distortion parameters caused by the lens, camera internal parameters such as the image center position, and external parameters such as camera position and orientation information There is. A camera calibration is performed by calculation using these parameters, and a three-dimensional map is created. For example, Patent Document 1 (US Patent No. 6009359), Patent Document 2 (US Patent Publication No. 2005/0237385), Patent Document 3 (US Patent No. 6606404), which are disclosed as techniques for creating a three-dimensional map based on the stereo method. There exists patent document 4 (US Published Patent 2003/0156751) etc.

If the internal parameters and external parameters of the camera to be used are known, camera calibration using these parameters and creation of a three-dimensional map are possible. For that purpose, the two cameras used for the stereo camera are Therefore, it is essential to use a stereo camera in which all parameters are known, and not only the change of the stereo camera is allowed, but also the positions and orientations of the two cameras must be completely fixed. When the camera is changed or when the camera setting position or posture is deviated, the known parameters cannot be used.
US Patent 6009359 US Published Patent 2005/0237385 US Pat. No. 6,606,404 US published patent 2003/0156751

  The present invention has been made in view of the above-described problems. For example, even when at least a part of parameters of a camera that captures an image used to create a three-dimensional map based on the stereo method is unknown. An object of the present invention is to provide an information processing apparatus, an information processing method, and a computer program capable of efficiently calculating parameters from a photographed image of a camera and performing camera calibration to create a three-dimensional map And

The first aspect of the present invention is:
An information processing apparatus that inputs images captured by two cameras installed substantially in parallel at different positions, and calculates a parallax offset based on the captured images.
A feature point extraction unit for extracting feature points included in the captured images of the two cameras;
A feature point matching processing unit that detects a feature point pair composed of a combination of similar feature points by comparison processing of feature points extracted from the captured images of the two cameras;
A parallax offset calculating unit that calculates a parallax offset in the captured images of the two cameras by analyzing the feature point pair;
The parallax offset calculation unit includes:
A feature point pair determined to have high reliability by filtering the feature point pair is selected, a histogram is generated as occurrence frequency distribution information of parallax offset data corresponding to the selected feature point pair, and the generated histogram is used. The information processing apparatus is characterized in that a constraint condition for an allowable value of parallax offset is set and an optimum value of the parallax offset is calculated based on the set constraint condition.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the parallax offset calculation unit includes a parallax offset corresponding to a horizontal direction (X) and a vertical direction (Y) as a parallax offset corresponding to the selected feature point pair ( dx, dy) [dx] histogram and [dy] histogram are generated, and the constraint condition of the parallax offset allowable value is set using the generated histogram, and the parallax offset of the parallax offset is set based on the set constraint condition It is the structure which calculates an optimal value, It is characterized by the above-mentioned.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the parallax offset calculation unit corresponds to a peak position of a [dy] histogram that defines a parallax offset in the vertical direction (Y) in the vertical direction (Y). A process for determining the optimum value of the parallax offset (dy) is executed.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the parallax offset calculation unit may select [dx] according to a preset appearance frequency rate in a histogram of [dx] that defines the parallax offset in the horizontal direction (X). ] Is set as a constraint condition for the parallax offset tolerance value, and based on the set constraint condition, the parallax offset of the horizontal direction (X) is set. It is the structure which calculates an optimal value, It is characterized by the above-mentioned.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the parallax offset calculation unit includes a plurality of [dx] set between the minimum allowable value [dx, min] and the maximum allowable value [dx, max]. The error and free (Outliers) data number of the feature point when the value is applied is verified, and the value of [dx] that minimizes the number of error and free data is the optimal parallax offset in the horizontal direction (X) A process for determining a value is executed.

Furthermore, in one embodiment of the information processing apparatus of the present invention, the parallax offset calculation unit is configured as a parallax offset corresponding to the feature point pair as a filtering process for selecting a feature point pair determined to have high reliability. About the value of the parallax offset (dx, dy) corresponding to the horizontal direction (X) and the vertical direction (Y),
(A) Epipolar constraint on condition that dy is in a preset range
(B) Disparity constraint on condition that dx is in a preset range,
(C) When there are a plurality of pairs (one-to-many) satisfying both of the above (a) and (b), these pairs are determined to be feature points pairs with high ambiguity and eliminated. Restraint,
The present invention is characterized in that it is configured to execute a filtering process that executes determination of whether or not at least one of the constraint conditions (a) to (c) is satisfied, and leaves only those that satisfy the constraint condition. To do.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the information processing apparatus further executes a camera calibration and a three-dimensional map generation process to which the optimum value of the parallax offset calculated by the parallax offset calculation unit is applied. And a three-dimensional map generating unit.

Furthermore, the second aspect of the present invention provides
In the information processing apparatus, an information processing method for inputting images captured by two cameras installed substantially in parallel at different positions and calculating a parallax offset based on the captured image,
A feature point extracting unit for extracting a feature point included in the captured images of the two cameras;
A feature point matching processing step in which a feature point matching processing unit detects a feature point pair consisting of a combination of similar feature points by a comparison process of feature points extracted from the captured images of the two cameras;
A parallax offset calculating unit includes a parallax offset calculating step of calculating a parallax offset in the captured images of the two cameras by analyzing the feature point pair;
The parallax offset calculating step includes:
A feature point pair determined to have high reliability by filtering the feature point pair is selected, a histogram is generated as occurrence frequency distribution information of parallax offset data corresponding to the selected feature point pair, and the generated histogram is used. In this information processing method, a constraint condition for the allowable value of the parallax offset is set, and an optimum value of the parallax offset is calculated based on the set constraint condition.

  Furthermore, in one embodiment of the information processing method of the present invention, the parallax offset calculating step includes a parallax offset (corresponding to a horizontal direction (X) and a vertical direction (Y) as a parallax offset corresponding to the selected feature point pair ( dx, dy) [dx] histogram and [dy] histogram are generated, and the constraint condition of the parallax offset allowable value is set using the generated histogram, and the parallax offset of the parallax offset is set based on the set constraint condition It is a step of calculating an optimum value.

  Furthermore, in an embodiment of the information processing method of the present invention, the parallax offset calculating step corresponds to a peak position of a [dy] histogram defining the parallax offset in the vertical direction (Y) in the vertical direction (Y). It is a step that executes a process for determining the optimum value of the parallax offset (dy).

  Furthermore, in an embodiment of the information processing method of the present invention, the parallax offset calculating step may perform [dx] according to a preset appearance frequency rate in a histogram of [dx] that defines the parallax offset in the horizontal direction (X). ] Is set as a constraint condition for the parallax offset tolerance value, and based on the set constraint condition, the parallax offset of the horizontal direction (X) is set. It is a step of calculating an optimum value.

  Furthermore, in an embodiment of the information processing method of the present invention, the parallax offset calculation step includes a plurality of [dx] set between the minimum allowable value [dx, min] and the maximum allowable value [dx, max]. The error and free (Outliers) data number of the feature point when the value is applied is verified, and the value of [dx] that minimizes the number of error and free data is the optimal parallax offset in the horizontal direction (X) It is a step which performs the process determined as a value, It is characterized by the above-mentioned.

Furthermore, in an embodiment of the information processing method of the present invention, the parallax offset calculating step includes a filtering process for selecting a feature point pair determined to be highly reliable, as a parallax offset corresponding to the feature point pair, About the value of the parallax offset (dx, dy) corresponding to the horizontal direction (X) and the vertical direction (Y),
(A) Epipolar constraint on condition that dy is in a preset range
(B) Disparity constraint on condition that dx is in a preset range,
(C) When there are a plurality of pairs (one-to-many) satisfying both of the above (a) and (b), these pairs are determined to be feature points pairs with high ambiguity and eliminated. Restraint,
It is a step of executing a filtering process that executes determination of whether or not at least one of the constraint conditions (a) to (c) is satisfied, and leaves only those that satisfy the constraint condition. To do.

  Furthermore, in one embodiment of the information processing method of the present invention, the information processing method further includes camera calibration in which a three-dimensional map generation unit applies the optimum value of the parallax offset calculated in the parallax offset calculating step, and 3 A three-dimensional map generation step for executing a generation process of the three-dimensional map is provided.

Furthermore, the third aspect of the present invention provides
In the information processing apparatus, a computer program that inputs images taken by two cameras installed substantially in parallel at different positions and calculates a parallax offset based on the taken images,
A feature point extracting step for causing the feature point extracting unit to extract feature points included in the captured images of the two cameras;
A feature point matching processing step for causing the feature point matching processing unit to detect a feature point pair composed of a combination of similar feature points by comparison processing of feature points extracted from the captured images of the two cameras;
A parallax offset calculating step for causing the parallax offset calculating unit to calculate a parallax offset in the captured images of the two cameras by analyzing the feature point pair;
The parallax offset calculating step includes:
A feature point pair determined to have high reliability by filtering the feature point pair is selected, a histogram is generated as occurrence frequency distribution information of parallax offset data corresponding to the selected feature point pair, and the generated histogram is used. The computer program is characterized in that a constraint condition for an allowable value of parallax offset is set and an optimum value of the parallax offset is calculated based on the set constraint condition.

  The computer program of the present invention is, for example, a computer program that can be provided by a storage medium or a communication medium provided in a computer-readable format to a general-purpose computer system that can execute various program codes. . By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer system.

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

  According to the configuration of one embodiment of the present invention, in the configuration in which images taken by two cameras installed at different positions are input and calculation of the optimum parallax offset based on the taken image is performed, the two cameras A feature point pair is detected from the captured image, and a parallax offset between the captured images of the two cameras is calculated by analyzing the feature point pair. In the parallax offset calculation process, a feature point pair that is determined to have high reliability by filtering the feature point pair is selected, and a histogram is generated as occurrence frequency distribution information of the parallax offset data corresponding to the selected feature point pair. Then, a constraint condition for the allowable value of the parallax offset is defined using the generated histogram, and an optimum value of the parallax offset is calculated based on the defined constraint condition. According to this configuration, even when some of the parameters for one camera are unknown, the optimum parallax offset (dx, dy) can be calculated based on the captured image of each camera, and all parameters are known. A three-dimensional map can be generated without using a camera.

  Details of an information processing apparatus, an information processing method, and a computer program according to embodiments of the present invention will be described below with reference to the drawings.

  An information processing apparatus according to an embodiment of the present invention performs a process of creating a three-dimensional map (3Dmap) based on a captured image. Specifically, a parallax offset is calculated based on the captured image, a parameter is acquired based on the calculated parallax offset, and camera calibration is performed using the acquired parameter to create a three-dimensional map (3Dmap). Details of this processing will be described below. FIG. 1 is a diagram showing a configuration of an embodiment of an information processing apparatus according to the present invention.

First, an outline of processing executed by the information processing apparatus of the present invention will be described with reference to FIG. Details of the processing of each component will be described later. Two cameras 101a and 101b are set at different positions, and images are taken from different directions. Note that at least one of the parameters of the cameras 101a and 101b has already been acquired and some of the parameters are unknown. Specifically, for example,
(A) Internal parameters Image size, pixel size, focal length, distortion,
(B) External parameters Stereo baseline These parameters are known in both the two cameras 101a, 101b,
(A) Internal parameters Image point (principal point)
(B) External parameters Camera position and orientation (homography)
These parameters are known in one of the two cameras 101a, 101 and unknown in the other.

  The feature point extraction unit 102 analyzes the captured images of the two cameras 101a and 101b, and performs feature point extraction processing. The feature point matching processing unit 103 executes a matching process for each feature point of the captured images of the two cameras 101a and 101b extracted by the feature point extraction unit 102. The parallax offset calculation unit 104 receives the feature point matching information generated by the feature point matching processing unit 103 and calculates the parallax offsets (Disparity Offset) of the two cameras 101a and 101b.

  The calibration & 3D map generation unit 105 applies the parallax offsets (Disparity Offset) of the two cameras 101a and 101b calculated by the parallax offset calculation unit 104 to execute camera calibration, and performs a three-dimensional map (3Dmap) based on the captured image. ).

  The flowchart shown in FIG. 2 is an overall sequence of a three-dimensional map (3Dmap) creation process executed in the information processing apparatus according to an embodiment of the present invention. In the information processing apparatus of the present invention, a three-dimensional map is created by processing according to the flowchart shown in FIG.

  First, in step S101, captured image acquisition processing by two cameras is executed. Images are taken by cameras 101a and 101b which are set at different positions shown in FIG. 1 and take images from different directions.

  Next, in step S102, feature point extraction from images captured by the two cameras is executed. This is executed in the feature point extraction unit 102 shown in FIG. 1, and the feature point extraction unit 102 analyzes the captured images of the two cameras 101a and 101b, and performs feature point extraction processing.

  As an example of the feature point extraction processing, a processing example using a Harris corner detector will be described with reference to FIG. The data processing unit of the information processing apparatus generates a plurality of Harris corner images 210 to 212 and Laplacian images 220 to 222 from an image 200 photographed by a camera as shown in FIG.

  A Harris corner image is image data generated by applying a Harris Corner Detector to an acquired image. From these Harris corner images 210 to 212, for example, a pixel point (maxima point) 215 having a higher value than the surrounding eight pixels is extracted as a detection point. Further, a LoG (Laplacian of Gaussian) filter is applied to the acquired image 200 to generate multi-level (resolution) Laplacian images 220 to 222. The LoG (Laplacian of Gaussian) filter is a kind of second-order differential filter used for image edge enhancement, and is performed until information from the retina is relayed by the outer knee in the human visual system. It is used as an approximate model of the processing.

  The feature point extraction processing corresponds to the position of the detection point obtained from the Harris corner images 210 to 212 whether there is a change in position due to a change in resolution within a predetermined level range of the Laplacian images 220 to 222 that are LoG filter output images. A point is examined, and a point having no change is defined as a feature point. Thereby, it is possible to realize matching between feature points that is robust to an enlargement / reduction operation of an image. Details of these feature point extraction processes are described in, for example, Japanese Patent Application Laid-Open No. 2004-326693 (Japanese Patent Application No. 2003-124225), and this feature point extraction process in the present invention in the feature point extraction unit 102 is described in this document. The method can be applied.

  This feature point detection is executed for each of the images taken by the cameras 101a and 101b shown in FIG. 1 that take images from different directions, and the detected feature point information is the feature point matching processing unit shown in FIG. 2, the process of step S103 of the flow shown in FIG. 2, that is, the feature point matching process of the captured image of each camera is executed.

  A detailed sequence of this feature point matching process will be described with reference to the flowchart shown in FIG. In step S201, feature points of the first camera are extracted, and in step S202, feature points of the second camera are extracted. These feature points are feature points extracted by the feature point extraction unit 102.

  Next, a feature point pair is detected in step S203. The feature point pair detection is an association process between the feature points of the first camera and the feature points of the second camera. Specifically, for example, the technique described in Japanese Patent Application Laid-Open No. 2004-326693 (Japanese Patent Application No. 2003-124225) introduced earlier can be applied.

  An outline of the feature point pair detection method described in Japanese Patent Application Laid-Open No. 2004-326693 (Japanese Patent Application No. 2003-124225) will be described. First, feature quantities are extracted and stored for each extracted feature point. As the feature amount, density gradient information (gradient strength and gradient direction) at each point in the vicinity region of the feature point is used. As the feature point vicinity region for calculating the feature amount, it is preferable to select a region that is symmetrical with respect to the feature point so that the structure does not change with respect to the rotation change. Thereby, the robustness with respect to a rotation change is realizable. For example, (i) a method in which a region within a radius r pixel from a feature point is used as a feature point neighboring region, or (ii) a method of applying a two-dimensional Gaussian weight symmetric about a feature point having a width σ around the feature point to the density gradient. Can be used. Further, a histogram (direction histogram) regarding the gradient direction in the vicinity of the feature point is also acquired as a feature amount.

  Next, feature amounts corresponding to feature points acquired from two cameras are compared, and a feature point pair (candidate corresponding feature point pair) having similar feature amounts is selected. For example, K feature points having a small difference from the feature amount (direction histogram) of one feature point selected from the photographed image of the first camera are selected from the photographed image of the second camera, and k feature points are selected. A pair candidate is selected, and a feature point pair candidate having an error equal to or greater than a predetermined threshold value is removed to leave only a probable feature point pair.

  The feature point matching processing unit 103 illustrated in FIG. 1 performs feature point matching processing of the captured image of each camera through such processing. The details of the feature point matching processing are described in Japanese Patent Application Laid-Open No. 2004-326693 (Japanese Patent Application No. 2003-124225) as described above.

  When the above feature point matching is completed, the process of step S104 in the flow shown in FIG. 2 is executed. The process of step S104 is a first stage process for calculating a parallax offset, and specifically, a process for calculating a disparity constraint. Thereafter, in step S105, the parallax offset calculation process is performed by applying the disparity constraint calculated in step S104. These processes are executed in the parallax offset calculation unit 104 shown in FIG.

  First, the parallax offset will be described with reference to FIGS. The parallax offset is a value corresponding to a pixel shift including a physical positional relationship between two cameras, that is, the camera 101a illustrated in FIG. 1 and corresponding feature points of the camera 101b and a measurement error.

  The parallelized image will be described with reference to FIG. For example, images taken by two cameras arranged in an arbitrary posture are image data from different directions as images 211 and 212 shown in FIG. Even if this captured image is used as it is, the epipolar line set on the captured image does not become a parallel line. For example, when the corresponding point search is performed using the epipolar line as a search line, the amount of calculation becomes enormous. . Therefore, the images 211 and 212 shown in FIG. 5A are converted to generate images similar to those taken by the cameras in parallel arrangement, that is, the parallelized images 215 and 216, and the parallelized images 215 are displayed on the parallelized images 215. Search for corresponding points in the set epiphora line.

  An example of the parallelized image data is data as shown in FIG. 5B, and image data from different directions are converted into parallel arrangements like images 211 and 212 shown in FIG. 5B. It is possible to easily extract corresponding points by generating the same images taken by a certain camera, that is, parallelized images 215 and 216 and searching for corresponding points on the epiphora line set on the parallelized image 215. Become.

A transformation matrix applied to this parallelization process is called an interplane transformation matrix. By performing the image conversion according to the plane between the transformation matrix H ₁ relative to the captured image 211 shown in FIG. 5 (b), collimating the image 215 is acquired with respect to the captured image 212 shown in FIG. 5 (b) by performing the image conversion according to the plane between the transformation matrix H _2, collimated image 216 is acquired. The epipolar lines set in these parallel images 215 and 216 are set in parallel, and the corresponding point (feature point pair) search can be easily executed. Image conversion processing using an inter-plane conversion matrix is called stereo image parallelization processing.

  FIG. 6 shows two images 215 and 216 generated by two cameras whose postures are set substantially parallel to each other, and a captured image center 231 of the parallelized image is displayed on each of the images 215 and 216, respectively. 232 is shown. Since the position and orientation of the camera are not strictly collimated, the captured image centers (primary points) 231 and 232 of the collimated image are shifted from each other.

The captured image center (e _x1 , e _y1 ) of the parallelized image of the first image 215 is indicated as a coordinate from the center of the first image 215 (image center). The horizontal direction is indicated by x and the vertical direction is indicated by y. Similarly, the captured image center (e _x2 , e _y2 ) of the parallelized image of the second image 216 is indicated as a coordinate from the center of the second image 216 (image center).

Based on two images 215 and 216 captured by two cameras having such captured image centers (parallel points) 231 and 232 and arranged so that their postures are substantially parallel to each other. Thus, the calculation of the disparity offset (Disparity Offset) is performed. The parallax offset is calculated for each of the corresponding pixels of the two images, and can be expressed as set data (dx, dy) of a shift (dx) in the x direction and a shift (dy) in the y direction of each corresponding pixel. Using the precondition that the postures of each other are installed almost in parallel, the parallax offset (dx, dy) is
Interplane transformation matrix of the first image: H ₁ = (t _x1 , t _y1 )
Interplane transformation matrix of the second image: H ₂ = (t _x2 , t _y2 )
Image center of first image: (e _x1 , e _y1 )
Image center of second image: (e _x2 , e _y2 )
And can be approximated with
Parallax offset _{(dx, dy) = (e} x1 + t x1 -e x2 -t x2, e y1 + t y1 -e y2 -t y2)
Can be expressed as

If this parallax offset can be calculated, camera calibration is possible. That is, for example, one camera (first camera) is fixed, and the image center of the parallelized image of the first image: (e _x1 , e _y1 ) is the image center of the parallelized image (image center). If the configuration is set to (0, 0), the other camera (second camera)
(A) Internal parameters Image point (principal point)
(B) External parameters Camera position and orientation (homography)
Even when these parameters are unknown, the correspondence between the captured images of the respective cameras is obtained using the image center data of the first image that is a parallelized image based on the captured image of the first camera: (e _x1 , e _y1 ). With respect to points (feature point pairs), parallax offsets can be calculated, and camera calibration can be performed.

  A detailed sequence of the process of step S104 in the flow shown in FIG. 2, that is, a calculation process such as disparity constraint performed as the first stage process for calculating the parallax offset will be described with reference to the flowchart of FIG. To do. This process is executed in the parallax offset calculation unit 104 shown in FIG.

  The disparity constraint calculation process performed as the first stage process for calculating the parallax offset is a process of setting a constraint condition of the parallax offset, and is performed in the x direction of each of the corresponding pixels of the two parallelized images. This is a process of setting an allowable range of a parallax offset (dx, dy) that is a set data of a deviation (dx) and a deviation in the y direction (dy). Specifically, calculation of the optimum value (dy, best) in the y direction of the parallax offset (dx, dy), and calculation of the minimum allowable value (dx, min) and the maximum allowable value (dx, max) in the x direction. Is executed.

  First, in step S301 of the flowchart shown in FIG. 7, the initial setting of the parallax constraint is performed as the parallax offset constraint condition. This initial value can be determined based on approximate camera setting position data or a captured image.

For example, as an initial setting example of the parallax constraint indicating the allowable range of the parallax offset (dx, dy), the allowable range for each of the dx and dy of the parallax offset (dx, dy) is set to the minimum allowable value: d ′ _{x, min} , Using d ′ _{y, min} and the maximum allowable values: d ′ _{x, max} , d ′ _{y, max} ,
d ′ _{x, min} <d _x <d ′ _{x, max} ,
d ′ _{y, min} <d _y <d ′ _{y, max} ,
When expressed as above,
d ′ _{x, min} = −50,
d ′ _{x, max} = 50,
d ′ _{y, min} = −50,
d ′ _{y, max} = 50,
Such an initial setting is used.
Note that the unit of −50 and 50 is, for example, the number of pixels. The unit is not limited to the number of pixels, and an actual distance on the image, a distance in real space, or the like may be used.

In step S301, as an initial setting example of the parallax constraint indicating the allowable range of the parallax offset (dx, dy), for each of dx and dy of the parallax offset (dx, dy), for example,
−50 <d _x <50,
−50 <d _y <50,
Such initial setting is performed.

  Next, in step S302, a feature point pair that does not satisfy the parallax constraint or the like set in step S301 by applying a stereo filter to images captured by two cameras whose postures are set substantially parallel to each other (step S302). (Corresponding point) is deleted.

  This process is performed for each of the feature point pairs extracted in steps S101 and S102 of the flow shown in FIG. 2, that is, the feature point pairs extracted by the feature point pair extraction process described above with reference to the flow of FIG. This is a process of deleting feature point pairs that do not satisfy the parallax constraint set in step S301.

A filter for deleting a feature point pair that does not satisfy constraint conditions such as parallax constraint is referred to as a stereo filter here. The stereo filter performs the following processing.
(input)
(1) Feature point pairs: (f ₁ , f _r ) = (x ₁ , y ₁ ), (x _r , y _r ),
(2) Minimum allowable values for each of dx and dy of the parallax offset (dx, dy): d ′ _{x, min} , d ′ _{y, min} , and maximum allowable values: d ′ _{x, max} , d ′ _{y, max} ,
And Note that (f ₁ , f _r ) = (x ₁ , y ₁ ), (x _r , y _r ) are obtained from the captured images of the two cameras extracted in steps S101 and S102 of the flow shown in FIG. The coordinate position in the parallelized image of the feature point pair is shown.
In the feature point pair, the coordinate position of the feature point of the left camera (camera L) is f ₁ = (x ₁ , y ₁ ), and the coordinate position of the feature point of the right camera (camera R) is f _r = (x _r , y _r ).

Based on these input information, it is determined whether each feature point pair satisfies the following conditions.
(Determination process)
(F ₁ , f _r ) = (x ₁ , y ₁ ), (x _r , y _r )
But,
(A) Epipolar constraint
_{d y, min -δ y <y} l -y r <d y, max + δ y,
Here, δ _y is a preset epipolar line setting error.
(B) Disparity constraint
d _{x, min} <x ₁ −x _r ,
(C) Unique match constraint
When there are a plurality of pairs (one-to-many) satisfying both of the above (a) and (b), these pairs are determined to be feature points pairs with high ambiguity, and are excluded.

(output)
Only feature point pairs that satisfy all of the constraint conditions (a) to (c) are selected and output.

  In step S302, a stereo filter is applied to the parallelized image, and the above processing is executed to delete a feature point pair that does not satisfy the parallax constraint or the like and to leave only a highly reliable feature point pair. I do.

Specific examples of the filtering processes (a) to (c) will be described with reference to FIGS.
FIG. 8 (A)
(A) Epipolar constraint
_{d y, min -δ y <y} l -y r <d y, max + δ y,
It is a figure which shows the example of a deletion and selection process of the feature point pair to which this constraint condition is applied.

In FIG. 8A, two feature points corresponding to the feature point [f ₁₁ = (x ₁₁ , y ₁₁ )] selected from the first camera image (L) are _extracted from the second camera image (R). When the feature points [f _r1a = (x _r1a , y _r1a )] and [f _r1b = (x _r1b , y _r1b )] are selected,
First feature point pair: [f _l1 = (x _l1 , y _l1 )] and [f _r1a = (x _r1a , y _r1a )]
The y coordinate data of the first feature point pair is
Epipolar _{constraint: d y, min -δ y <} y l -y r <d y, max + δ y,
Not satisfied
Second feature point pair: [f _l1 = (x _l1 , y _l1 )] and [f _r1b = (x _r1b , y _r1b )]
The y coordinate data of this second feature point pair is
Epipolar _{constraint: d y, min -δ y <} y l -y r <d y, max + δ y,
Satisfied.
in this case,
The first feature point pair: [f _l1 = (x _l1 , y _l1 )] and [f _r1a = (x _r1a , y _r1a )] are deleted,
Second feature point pair: [f _l1 = (x _l1 , y _l1 )] and [f _r1b = (x _r1b , y _r1b )]
This feature point pair is selected as a feature point pair that satisfies the epipolar constraint.

FIG. 8 (B)
(B) Disparity constraint
d _{x, min} <x ₁ −x _r ,
It is a figure which shows the example of a deletion and selection process of the feature point pair to which this constraint condition is applied.

In FIG. 8B, a feature point pair corresponding to one feature point [f _r2a = (x _r2a , y _r2a )] selected from the second camera image (R) is selected from the first camera image (L). _Selected feature points [f _l2a = (x _l2a , y _l2a )] and [f _l2b = (x _l2b , y _l2b )] are selected,
First feature point pair: [f _l2a = (x _l2a , y _l2a )] and [f _r2a = (x _r2a , y _r2a )]
The x-coordinate data of this first feature point pair is
Parallax constraint: d _{x, min} <x ₁ −x _r ,
Not satisfied
Second feature point pair: [f _l2b = (x _l2b , y _l2b )] and [f _r2a = (x _r2a , y _r2a )]
The x-coordinate data of this second feature point pair is
Parallax constraint: d _{x, min} <x ₁ −x _r ,
Satisfied.
in this case,
The first feature point pair: [f _l2a = (x _l2a , y _l2a )] and [f _r2a = (x _r2a , y _r2a )] are deleted,
Second feature point pair: [f _l2b = (x _l2b , y _l2b )] and [f _r2a = (x _r2a , y _r2a )] This feature point pair satisfies the disparity constraint (Disparity constraint) Selected as.

  FIG. 9 illustrates a non-conforming example and a conforming example of (c) unique match constraint. When there are a plurality of pairs (one-to-many) satisfying both the above-described epipolar constraint and parallax constraint, the unique correspondence constraint is determined as a feature point pair with high ambiguity and is excluded. It is a constraint condition.

FIG. 9A shows a non-conforming example of the inherent correspondence constraint. Feature points [f _l1a = (x _l1a , y _l1a )] selected from the first camera image (L), and feature points [f _r1a = (x _r1a , y _r1a )] of the second camera image (R) Is selected as a feature point pair, and this feature point pair satisfies both epipolar constraint and parallax constraint.
However, the feature point [f _l1a = (x _l1a , y _l1a )] selected from the first camera image (L) is another feature point [f _r1b = (x _r1b , y _r1b )] is selected as a feature point pair, and this feature point pair also satisfies both epipolar constraint and parallax constraint.

Further, another feature point [f _l1b = (x _l1b , y _l1b )] selected from the first camera image (L) shown in FIG. Both feature points [f _r1a = (x _r1a , y _r1a )] and [f _r1b = (x _r1b , y _r1b )] are selected as feature point pairs that satisfy both epipolar constraint and parallax constraint .

  As described above, when there are a plurality of pairs (one-to-many) satisfying both the epipolar constraint and the parallax constraint, these pairs are determined to be feature point pairs with high ambiguity and are excluded.

FIG. 9B shows an example of adaptation of the inherent correspondence constraint. The first camera image (L) (collimated image) is selected from the feature points _{[f l1a = (x l1a,} y l1a)] and, feature points of the second camera image (R) (collimated image) _{[f r1a} = (X _r1a , y _r1a )] is selected as a feature point pair, and this feature point pair satisfies both epipolar constraint and parallax constraint.

The first selected feature points from the camera image _{(L) [f l1a = (} x l1a, y l1a)] , in addition to feature points of the second camera image _{(R) [f r1b = (} x r1b, y r1b )] Is selected as a feature point pair, but this feature point pair does not satisfy both epipolar constraint and parallax constraint.

In this case, the feature point [f _l1a = (x _l1a , y _l1a )] of the first camera image (L) and the feature point [f _r1a = (x _r1a , y _r1a )] of the second camera image (R) The feature point pair consisting of is selected as a feature point pair that satisfies the matching condition of the unique correspondence constraint.

Similarly, and FIG. 9 (B) selected from the first camera image (L) shown in the characteristic point _{[f l1b = (x l1b,} y l1b)] and, feature points of the second camera image (R) _{[f r1b} = (X _r1b , y _r1b )] is selected as a feature point pair, and this feature point pair satisfies both epipolar constraint and parallax constraint.

The feature point [f _l1b = (x _l1b , y _l1b )] selected from the first camera image (L) is the other feature point [f _r1a = (x _r1a , y _r1a ) of the second camera image (R). )] Is selected as a feature point pair, but this feature point pair does not satisfy both epipolar constraint and parallax constraint.

In this case, the feature point of the first camera image _{(L) [f l1b = (} x l1b, y l1b)] and, feature points of the second camera image _{(R) [f r1b = (} x r1b, y r1b)] The feature point pair consisting of is selected as a feature point pair that satisfies the matching condition of the unique correspondence constraint.

As described above, in step S302 of the flow shown in FIG. 7, a stereo filter is applied to images taken by two cameras installed so that their postures are substantially parallel to each other, and the setting is made in step S301. A feature point pair (corresponding point) that does not satisfy the parallax constraint or the like is deleted. Specifically, as described above,
(A) Epipolar constraint
(B) Disparity constraint
(C) Unique match constraint
A feature point pair that does not satisfy these constraint conditions is determined to be a feature point pair with low reliability and is deleted.

  FIG. 10 shows an example of reducing feature point pairs by eliminating feature point pairs with low reliability in this way. FIG. 10A shows an example of extraction of feature point pairs extracted by the processing of steps S102 and S103 in the flow of FIG. 2 and the feature point pair extraction processing according to the flow shown in FIG. A black rectangular frame shown in the first camera image (L) and the second camera image (R) is an area selected as a feature point pair.

A stereo filter is applied to the data shown in FIG.
(A) Epipolar constraint
(B) Disparity constraint
(C) Unique match constraint
By determining that feature point pairs that do not satisfy these constraint conditions are feature point pairs with low reliability and deleting them, only feature point pairs with high reliability are selected as shown in FIG. The

  The black rectangular frames shown in the first camera image (L) and the second camera image (R) shown in FIG. 10B are feature point pairs remaining after applying the stereo filter, and are feature points pairs with high reliability. The selected area.

Next, the process proceeds to step S303 in the flow shown in FIG. 7 to generate a parallax offset histogram based on the remaining feature point pairs. That is, apply a stereo filter,
(A) Epipolar constraint
(B) Disparity constraint
(C) Unique match constraint
A parallax offset histogram is generated based on a feature point pair satisfying these constraint conditions, specifically, a highly reliable feature point pair as shown in FIG.

As described above, the parallax offset is calculated for each of the corresponding pixels that are the feature point pairs of the two camera images arranged in parallel with each other, and the horizontal shift (x direction) of each corresponding pixel (dx) And set data (dx, dy) of the deviation (dy) in the vertical direction (y direction), that is,
Parallax offset _{(dx, dy) = (e} x1 + t x1 -e x2 -t x2, e y1 + t y1 -e y2 -t y2)
Can be expressed as However,
Interplane transformation matrix of the first image: H ₁ = (t _x1 , t _y1 )
Interplane transformation matrix of the second image: H ₂ = (t _x2 , t _y2 )
Image center of first image: (e _x1 , e _y1 )
Image center of second image: (e _x2 , e _y2 )
And approximate.

  In step S303, a histogram (frequency distribution) of each value [dx] and [dy] of the parallax offset (dx, dy) for the highly reliable feature point pair left by the application of the stereo filter is generated. That is, a [dx] histogram and a [dy] histogram corresponding to the horizontal direction (X) and the vertical direction (Y) are generated.

  For example, the histograms shown in FIGS. 11 and 12 are generated. FIG. 11 is a histogram (frequency distribution) of values [dy] of parallax offsets (dx, dy) for highly reliable feature point pairs left by applying the stereo filter, and FIG. 12 is a graph of [dx]. It is a histogram (frequency distribution).

Next, in step S304, a new disparity constraint is set based on the disparity offset histogram. This new parallax constraint setting is
For each value [dx], [dy] of the parallax offset (dx, dy),
dx: Allowable minimum value [d _{x, min} ] and allowable maximum value [d _{x, max} ]
dy: optimal value [dy _{, best} ]
This process is performed to determine these values.

First, from the histogram (frequency distribution) of the value [dy] of the parallax offset (dx, dy) for the highly reliable feature point pair left by the application of the stereo filter shown in FIG. 11, the optimum value [dy _{, The} process for determining “ _best ” will be described.

dy represents a shift in the Y-axis direction of the epipolar line in the left and right images. Since dy is a parameter common to all matching pairs, the optimal value [dy _{, best} ] of dy is set to the value of dy that is the peak position of the dy histogram. As shown in FIG. 11, the value of dy that is the peak position of the dy histogram is determined as the optimum value of dy [dy _{, best} ].

Next, from the histogram (frequency distribution) of the value [dx] of the parallax offset (dx, dy) for the highly reliable feature point pair left by the application of the stereo filter shown in FIG. _The process for determining _{x, min} ] and the allowable maximum value [d _{x, max} ] will be described.

The allowable minimum value [d _{x, min} ] and the allowable maximum value [d _{x, max} ] of dx are determined based on preset threshold values [thout] and [thinf].
The threshold value [thout] is a threshold value that determines the allowable minimum value [d _{x, min} ] of dx. For example, the parallax offset [dx = xl−xr] at which the appearance frequency reaches 1% starting from the minimum value of dx. Set to the point.
This is based on the assumption that data having an appearance frequency of 1% or less is likely to be an error.

On the other hand, the threshold value [thin] is a threshold value that determines the allowable maximum value [d _{x, max} ] of dx. For example, the parallax offset [dx in the x direction that starts from the minimum value of dx and has an appearance frequency of 10%. = Xl-xr].
This is based on the assumption that the maximum frequency of appearance of the assumed free point (Outlier) is 10%. This free point includes an infinite point where parallax is zero.
Many portions of the pixels included in the captured image are captured images of an object (Object) located far enough compared to the parallax between the two cameras. In this case, these pixels are assumed to be infinity points. be able to. Such a feature point pair (corresponding point) at an infinite point is data that cannot be an element for calculating correct parallax data, and the threshold [thin] is a threshold for excluding these data.

A sharp rise of the histogram occurs at a point d _{x, max} shown in the histogram (frequency distribution) of the value [dx] of the parallax offset (dx, dy) shown in FIG. Indicates that data corresponding to a subject at infinity is counted. In order to eliminate such data, an allowable maximum value [d _{x, max} ] of dx is determined based on the threshold value [thin].

Through the above processing, in step S304 of the flow shown in FIG. 7, the setting of a new disparity constraint (Disparity constraint) based on the disparity offset histogram, that is,
For each value [dx], [dy] of the parallax offset (dx, dy),
dx: Allowable minimum value [d _{x, min} ] and allowable maximum value [d _{x, max} ]
dy: optimal value [dy _{, best} ]
These values are determined.

  With these processes, the process of step S104 in the flow shown in FIG. 2, that is, the process of calculating the disparity constraint performed as the first stage process for calculating the parallax offset is completed, and then, in step S105, The parallax offset calculation process is executed by applying the disparity constraint calculated in step S104. This process is executed in the parallax offset calculation unit 104 shown in FIG.

  A detailed sequence of the parallax offset calculation processing in step S105 will be described with reference to the flowchart shown in FIG.

First, in step S351, the parallax offset [dx] between the allowable minimum value [d _{x, min} ] and the allowable maximum value [d _{x, max} ] of the parallax offset dx in the x direction set as a result of the processing flow of FIG. Select. In the flow illustrated in FIG. 13, for example, a fixed number (n) of parallax offset values at equal intervals between the allowable minimum value [d _{x, min} ] and the allowable maximum value [d _{x, max} ] of the parallax offset dx in the x direction. dx1 to dxn are selected, and Steps S351 to S353 are executed.

  In step S352, the occurrence state of error and free data (Outlier) when the parallax offset [dx] in the x direction of the feature point pair selected in step S351 is applied is verified. A detailed sequence of this process will be described with reference to the flow of FIG.

First, in step S381, a stereo filter is applied to delete feature point pairs that do not satisfy the parallax constraint or the like. This process is the same filtering process as the stereo filter applied in step S302 of the flow of FIG.
(A) Epipolar constraint
_{d y, min -δ y <y} l -y r <d y, max + δ y,
δ _y is a preset epipolar line setting error.
(B) Disparity constraint
d _{x, min} <x ₁ −x _r ,
(C) Unique match constraint
When there are a plurality of pairs (one-to-many) satisfying both of the above (a) and (b), these pairs are determined to be feature points pairs with high ambiguity, and are excluded.
This is a process in which at least one of these determination processes is executed and only those satisfying each constraint condition remain.

However, in the process of step S352,
The minimum allowable value and the maximum allowable value for dy of the parallax offset (dx, dy) applied to the determination condition: d ′ _{y, min} , d ′ _{y, max} calculated in the process of step S104 in FIG. 2 dy: optimal Let it be the value [d _{y, best} ]. That is,
d' _{y, min} = d' _{y, min} = _{dy, best}
And
Further, regarding the minimum allowable value and the maximum allowable value for dx of the parallax offset (dx, dy) applied to the determination condition: d ′ _{x, min} , d ′ _{x, max} ,
The maximum permissible value d ′ _{x, max} is the maximum value [d _{x, max} ] calculated in the process of step S104 of FIG. 2, while the minimum permissible value d ′ _{x, min} is determined in step S351 of the flow of FIG. The selected dx. That is,
d ′ _{x, min} = dx
And

Under this condition,
(input)
(1) Feature point pairs: (f ₁ , f _r ) = (x ₁ , y ₁ ), (x _r , y _r ),
(2) Minimum allowable values for each of dx and dy of the parallax offset (dx, dy): d ′ _{x, min} , d ′ _{y, min} , and maximum allowable values: d ′ _{x, max} , d ′ _{y, max} ,
(Determination process)
(F ₁ , f _r ) = (x ₁ , y ₁ ), (x _r , y _r )
But,
(A) Epipolar constraint
_{d y, min -δ y <y} l -y r <d y, max + δ y,
Here, δ _y is a preset epipolar line setting error.
(B) Disparity constraint
d _{x, min} <x ₁ −x _r ,
(C) Unique match constraint
When there are a plurality of pairs (one-to-many) satisfying both of the above (a) and (b), these pairs are determined to be feature points pairs with high ambiguity, and are excluded.
(output)
Only feature point pairs that satisfy all of the constraint conditions (a) to (c) are selected and output.
These stereo filtering processes are executed.

The process of the next step S382 is executed only for the feature point pair selected here. In step S382, verification processing of the verification image to be processed and model image data is executed to verify the number of errors and free data from feature point pairs included in the verification image. The model image is a stereo image having highly reliable feature point pair information, and includes, for example, a stereo image one frame before the verification target image.
The process of step S382 will be described with reference to FIG.

  FIG. 15B shows a first camera image (L) and a second camera image (R) as verification target images. The black rectangular frames shown in these images are feature point pairs with high reliability selected in the stereo filtering process of step S381.

  FIG. 15A shows a model image. For example, a preceding image one frame before the verification target image is used. This model image also shows highly reliable feature point pairs selected in the stereo filtering process in the images of the first camera image (L) and the second camera image (R), as in the verification image.

  The procedure of the process in step S382 will be described. First, feature point pairs that are not included in the verification image and the model image data are deleted. For example, the first camera image (L) as the verification target image is compared with the first camera image (L) of the model image, and feature point pair data not included in both is deleted from the verification image.

  Next, using the feature point pair information included in both the verification target image and the model image, the number of errors and free data included in the feature point pair of the verification target image is verified. This process will be described.

Processing is performed using one of the model image and the verification image, for example, the first camera image (L).
The set data of the three-dimensional positions of the feature points of the first camera image (L) of the model image is
P = {p1, p2,... Pn}
A set data set of three-dimensional positions of feature points of the first camera image (L) of the verification image,
Q = {q1, q2,... Qn}
And Each of p1 to pn and q1 to qn has position information in three-dimensional data.

In addition, the position and orientation change at the time of shooting the model image and verification image of the camera that shot the first camera image (L),
ξ = (R, T)
And
Where R is camera posture change information, Euler angles corresponding to the x, y and z axes,
R = rot (α, z) · rot (β, y) · rot (γ, x)
It is.
T represents a translation vector that is a three-dimensional position change of the camera,
T = (tx, ty, yz)
It is.

Here, the feature point set of the first camera image (L) of the model image,
P = {p1, p2,... Pn}
The feature point set of the first camera image (L) of the verification image is
Q = {q1, q2,... Qn}
Applying these data, Euclidean distance (Euclidean distance),
Is calculated.

  When the feature point sets [P] and [Q] extracted from the model image and the verification image include incorrect data, for example, free points and errors, the Euclidean distance becomes a large value, and there are few free points and errors. A small value is calculated.

In addition, change of camera position and orientation
ξ = (R, T)
However, if the camera posture change [R] is small,
It is possible to assume that the rotation angles with respect to the x, y and z axes: α, β, γ≈0,
cos (α) ≈1,
sin (α) ≈α,
sin (α) sin (β) ≈0
And can be approximated. By using the above approximate expression, the camera posture change [R] can be approximated as follows.

  The purpose of the processing here is to detect the number of error data or free point (Outlier) data included in the feature point pair of the verification target image. First, the number of free points (Outlier) included in the feature point pair of the test subject image is obtained as follows.

A set of feature points of the first camera image (L) of the model image;
P = {p1, p2,... Pn}
The feature point set of the first camera image (L) of the verification image is
Q = {q1, q2,... Qn}
The free points (Outliers) included in the feature point pairs of the test target image are obtained by histogram filtering using the Hough transform using the feature points of the verification target image and the corresponding feature point data in the model image. Is detected.

The Hough space is six dimensions (α, β, γ, tx, ty, tz) constituting ξ = (R, T) that defines the position and orientation change of the camera.
A set of feature points of the first camera image (L) of the model image;
P = {p1, p2,... Pn}
The feature point set of the first camera image (L) of the verification image is
Q = {q1, q2,... Qn}
A four-dimensional curve in the Hough domain is defined by the data of each pair of these corresponding feature point pairs, and the above-described camera position and orientation change is defined by applying this four-dimensional curve ξ = (R, T 6-dimensional data (α, β, γ, tx, ty, tz) that constitutes) can be obtained.

  Here, an example of experimental data in which β = 0, γ = 0, tz = 0 and a three-dimensional Hough space (α, tx, ty) will be described with reference to FIGS. 16 and 17.

  FIG. 16 is an example of a histogram of the rotation angle [α] generated by the corresponding points of the feature point set [P] of the model image of the first camera (L) and the feature point set [Q] of the verification image. FIG. 17 is a diagram illustrating an example of a histogram of translation vectors [tx, ty] [tx, ty].

  FIG. 16A is a model image of the first camera (L), and is an image including a feature point set: P = {p1, p2,... Pn}. FIG. 16B is a verification image of the first camera (L), which is an image including a feature point set: Q = {q1, q2,... Qn}.

  FIG. 16C shows n pieces calculated based on the corresponding feature points (p1, q1) to (pn, qn) of the model image of FIG. 16A and the verification image of FIG. Is a histogram as a frequency distribution of rotation angle (α) data. In this histogram, an area within a preset threshold centered on the peak position is defined as a support area, and a feature point obtained by calculating a rotation angle (α) included in the support area is defined as a correct feature point. A feature point for which a rotation angle other than the region is calculated is defined as a free point (outlier).

  FIG. 17 is a diagram illustrating an example of a histogram of the translation vector [tx, ty]. FIG. 17A is a model image of the first camera (L), and is an image including a feature point set: P = {p1, p2,... Pn}. FIG. 17B is a verification image of the first camera (L), and is an image including a feature point set: Q = {q1, q2,... Qn}.

  FIG. 17 (c) shows n pieces calculated based on the corresponding feature points (p1, q1) to (pn, qn) of the model image of FIG. 17 (a) and the verification image of FIG. 17 (b). Is a histogram as a frequency distribution of the translation vector [tx, ty] data. In this histogram, a region within a preset threshold centered on the peak position is defined as a support region, and a feature point obtained by calculating a translation vector [tx, ty] included in the support region is defined as a correct feature point. The feature point for which the translation vector [tx, ty] other than the support region is calculated is set as a free point (outlier).

  In this manner, free points (outliers) are discriminated from the feature points [Q = {q1, q2,... Qn}] included in the verification image, and the number thereof is counted. This count number is the number of free points.

Further, the calculation formula for the Euclidean distance is used to calculate the number of error data included in the feature point pair of the verification target image.
A set of feature points of the first camera image (L) of the model image;
P = {p1, p2,... Pn}
The feature point set of the first camera image (L) of the verification image is
Q = {q1, q2,... Qn}
Applying these data, Euclidean distance (Euclidean distance),
Is calculated.

  As described above, when the feature point sets [P] and [Q] extracted from the model image and the verification image include incorrect data, for example, free points and errors, the Euclidean distance becomes a large value, and free points and When there are few errors, a small value is calculated. Therefore, the error index value is the calculated value of the above formula.

  With this process, the process of step S382 in the flowchart shown in FIG. In step S383, the error index value of the feature point obtained in step S382 and the number of free point (Outlier) data are the error index values corresponding to the parallax offset [dx] in the x direction selected in step S351 of the flow shown in FIG. And set as the number of free point (Outlier) data.

  After this processing, the process proceeds to step S353 in the flow of FIG. 13, and the error index value calculated in step S352 and the number of free points (Outlier) are set to the parallax offset in the x direction selected in step S351 of the flow shown in FIG. An error index value corresponding to [dx] and the number of free points (Outlier) are recorded.

Next, the process proceeds to step S354, and it is determined whether there is unprocessed [dx] data. As described above in the description of the processing in step S351, the processing in steps S351 to S353 is performed with the allowable minimum value [d _{x, min} ] of the parallax offset dx in the x direction set as a result of the processing flow in FIG. A predetermined number (n) of parallax offset values dx1 to dxn are set as verification target data, for example, at regular intervals from the maximum value [d _{x, max} ], and are repeatedly executed.

  In step S354, it is determined whether or not the verification for all the verification target data dx1 to dxn has been completed. If not completed, the processes of steps S351 to S353 are executed for the unprocessed data. When the processes in steps S351 to S354 are completed for all the verification target data, the process proceeds to step S355.

In step S355, the parallax offset [dx] that minimizes the number of errors and free data (Outlier) is determined as the optimum parallax offset [dx, best]. At the stage where the determination in step S354 is Yes, the error index values and the number of free points (Outliers) corresponding to all the verification target data dx1 to dxn are calculated. That is, for example,
dx1: Error index value = Xa, number of free data (Outlier) = Ya dx2: error index value = Xb, number of free data (Outlier) = Yb dx3: error index value = Xc, number of free data (Outlier) = Yc Pieces
:
dxn: Error index value = Xp, number of free data (Outlier) = Yp These data have been calculated.

  In step S355, the parallax offset [dx] that minimizes the number of errors and free data (Outlier) is determined as the optimum parallax offset [dx, best] from these calculated data. With this process, the process of step S355 ends, and the optimum parallax offset [dx, best] is determined.

  By this process, the process of step S105 of the flow shown in FIG. As a result, the optimum values [dx, best] and [dy, best] of the parallax offset (dx, dy) in the x and y directions are calculated.

  In step S106, the optimal values [dx, best], [dy, best] of the parallax offset are assumed to be the parallax offsets of the two cameras 101a and 101b shown in FIG. To do.

As described above with reference to FIG. 6, if the parallax offset can be calculated using images taken by two cameras installed almost in parallel at different positions, camera calibration becomes possible. A three-dimensional map can be generated. That is, as described above with reference to FIG. 6, the parallax offset is calculated for each of the corresponding pixels of the two parallelized images, and the shift in the x direction (dx) and the shift in the y direction (dy ) And data that can be expressed as pair data (dx, dy), and the parallax offset (dx, dy) is
Interplane transformation matrix of the first image: H ₁ = (t _x1 , t _y1 )
Interplane transformation matrix of the second image: H ₂ = (t _x2 , t _y2 )
Image center of first image: (e _x1 , e _y1 )
Image center of second image: (e _x2 , e _y2 )
And approximating
Parallax offset _{(dx, dy) = (e} x1 + t x1 -e x2 -t x2, e y1 + t y1 -e y2 -t y2)
Can be expressed as

If this parallax offset (dx, dy) can be calculated, for example, the image center of the first image: (e _x1 , e _y1 ) Camera calibration can be performed using this data. That is, for example, one camera (first camera) is fixed, and the image center of the first image: (e _x1 , e _y1 ) is set to the center of the parallelized image (image center (Image Center)) (0, 0). ), The other camera (second camera)
(A) Internal parameters Image point (principal point)
(B) External parameters Camera position and orientation (homography)
Even when these parameters are unknown, the correspondence between the captured images of the respective cameras is obtained using the image center data of the first image that is a parallelized image based on the captured image of the first camera: (e _x1 , e _y1 ). With respect to points (feature point pairs), parallax offsets can be calculated, and camera calibration can be performed.

  In the configuration of the present invention, the parallax offset optimum value [dy, best] in the y direction calculated in step S104 of the flow shown in FIG. 2 and the parallax offset optimum value [dx, best] in x direction calculated in step S105 are applied. Then, camera calibration is executed to generate a three-dimensional map.

  According to the above-described processing, it is possible to calculate the optimum parallax offset (dx, dy) based on the captured image of each camera even when some of the parameters for one camera are unknown. A dimensional map can be generated.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

  The series of processing described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run. For example, the program can be recorded in advance on a recording medium. In addition to being installed on a computer from a recording medium, the program can be received via a network such as a LAN (Local Area Network) or the Internet, and installed on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

  As described above, according to the configuration of an embodiment of the present invention, images taken by two cameras installed substantially in parallel at different positions are input and a parallax offset is calculated based on the taken images. In this configuration, a pair is detected from the images captured by the two cameras, and a parallax offset between the images captured by the two cameras is calculated by analyzing the feature point pair. In the parallax offset calculation process, a feature point pair that is determined to have high reliability by filtering the feature point pair is selected, and a histogram is generated as occurrence frequency distribution information of the parallax offset data corresponding to the selected feature point pair. Then, a constraint condition for the allowable value of the parallax offset is defined using the generated histogram, and an optimum value of the parallax offset is calculated based on the defined constraint condition. According to this configuration, even when some of the parameters for one camera are unknown, the optimum parallax offset (dx, dy) can be calculated based on the captured image of each camera, and all parameters are known. A three-dimensional map can be generated without using a camera.

It is a figure which shows the information processing apparatus structural example of one Example of this invention. It is a figure which shows the flowchart explaining the process sequence performed in the information processing apparatus of this invention. It is a figure explaining the example of a feature point extraction process using a Harris corner (Harris Corner) image. It is a figure which shows the flowchart explaining the sequence of a feature point matching process. It is a figure explaining parallax offset. It is a figure explaining parallax offset. It is a figure which shows the flowchart explaining the detailed sequence of the calculation process of a parallax constraint (Disparity constraint). It is a figure explaining the specific process example of the filtering process to which a stereo filter is applied. It is a figure explaining the specific process example of the filtering process to which a stereo filter is applied. It is a figure explaining the specific process example of the filtering process to which a stereo filter is applied. It is a figure which shows the histogram (frequency distribution) of the value [dy] of parallax offset (dx, dy) about the reliable feature point pair left by application of a stereo filter. It is a figure which shows the histogram (frequency distribution) of the value [dx] of parallax offset (dx, dy) about the reliable feature point pair left by application of a stereo filter. It is a figure which shows the flowchart explaining the detailed sequence of the calculation process of parallax offset. It is a figure which shows the flowchart explaining the process sequence which verifies the generation | occurrence | production condition of the error and free data (Outlier) at the time of applying the parallax offset [dx] of the x direction of a feature point pair. It is a figure explaining the process which counts the data number judged to be an error and free data (Outlier) from the feature point pair contained in a verification image. It is a figure which shows the example of the histogram of rotation angle [(alpha)] produced | generated by the feature point set [P] of the model image of a 1st camera (L), and the feature point set [Q] of a verification image. Example of translation vector [tx, ty] [tx, ty] generated by corresponding points of feature point set [P] of model image of first camera (L) and feature point set [Q] of verification image FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Camera 102 Feature point extraction part 103 Feature point matching process part 104 Parallax offset calculation part 105 Calibration & 3D map generation part 200 Image 210-212 Harris corner image 220-222 Laplacian image

Claims (15)

An information processing apparatus that inputs images captured by two cameras installed substantially in parallel at different positions, and calculates a parallax offset based on the captured images.
A feature point extraction unit for extracting feature points included in the captured images of the two cameras;
A feature point matching processing unit that detects a feature point pair composed of a combination of similar feature points by comparison processing of feature points extracted from the captured images of the two cameras;
A parallax offset calculating unit that calculates a parallax offset in the captured images of the two cameras by analyzing the feature point pair;
The parallax offset calculation unit includes:
A feature point pair determined to have high reliability by filtering the feature point pair is selected, a histogram is generated as occurrence frequency distribution information of parallax offset data corresponding to the selected feature point pair, and the generated histogram is used. An information processing apparatus having a configuration in which a constraint condition for an allowable value of parallax offset is set and an optimum value of the parallax offset is calculated based on the set constraint condition. The parallax offset calculation unit includes:
Generating a [dx] histogram and a [dy] histogram of parallax offsets (dx, dy) corresponding to the horizontal direction (X) and the vertical direction (Y) as the parallax offset corresponding to the selected feature point pair; 2. The information processing according to claim 1, wherein a constraint condition for an allowable value of parallax offset is set using the generated histogram, and an optimum value of the parallax offset is calculated based on the set constraint condition. apparatus. The parallax offset calculation unit includes:
A process of determining a peak position of a [dy] histogram that defines a parallax offset in the vertical direction (Y) as an optimum value of the parallax offset (dy) corresponding to the vertical direction (Y) is executed. Item 3. The information processing device according to Item 2. The parallax offset calculation unit includes:
In the [dx] histogram that defines the parallax offset in the horizontal direction (X), the minimum allowable value [dx, min] and the maximum allowable value [dx, max] of [dx] are set according to a preset appearance frequency rate. 3. The information processing according to claim 2, wherein the information processing unit is configured to set as a constraint condition of an allowable value of the parallax offset, and to calculate an optimal value of the parallax offset in the horizontal direction (X) based on the set constraint condition. apparatus. The parallax offset calculation unit includes:
The error of the feature point and the number of independent (Outliers) data are verified when a plurality of [dx] values set between the minimum allowable value [dx, min] and the maximum allowable value [dx, max] are applied. 5. The information processing according to claim 4, wherein a process of determining a value of [dx] that minimizes the number of errors and free data as an optimum value of the parallax offset in the horizontal direction (X) is performed. apparatus. The parallax offset calculation unit includes:
As a filtering process to select feature point pairs judged to be highly reliable,
As the parallax offset corresponding to the feature point pair, the values of the parallax offset (dx, dy) corresponding to the horizontal direction (X) and the vertical direction (Y),
(A) Epipolar constraint on condition that dy is in a preset range
(B) Disparity constraint on condition that dx is in a preset range,
(C) When there are a plurality of pairs (one-to-many) satisfying both of the above (a) and (b), these pairs are determined to be feature points pairs with high ambiguity and eliminated. Restraint,
The present invention is characterized in that it is configured to execute a filtering process that executes determination of whether or not at least one of the constraint conditions (a) to (c) is satisfied, and leaves only those that satisfy the constraint condition. The information processing apparatus according to claim 1. The information processing apparatus further includes:
The apparatus according to any one of claims 1 to 6, further comprising a three-dimensional map generation unit that executes camera calibration and a three-dimensional map generation process to which an optimum value of the parallax offset calculated by the parallax offset calculation unit is applied. Information processing device. In the information processing apparatus, an information processing method for inputting images captured by two cameras installed substantially in parallel at different positions and calculating a parallax offset based on the captured image,
A feature point extracting unit for extracting a feature point included in the captured images of the two cameras;
A feature point matching processing step in which a feature point matching processing unit detects a feature point pair consisting of a combination of similar feature points by a comparison process of feature points extracted from the captured images of the two cameras;
A parallax offset calculating unit includes a parallax offset calculating step of calculating a parallax offset in the captured images of the two cameras by analyzing the feature point pair;
The parallax offset calculating step includes:
A feature point pair determined to have high reliability by filtering the feature point pair is selected, a histogram is generated as occurrence frequency distribution information of parallax offset data corresponding to the selected feature point pair, and the generated histogram is used. An information processing method comprising: setting a constraint condition for an allowable value of parallax offset and calculating an optimum value of the parallax offset based on the set constraint condition. The parallax offset calculating step includes:
Generating a [dx] histogram and a [dy] histogram of parallax offsets (dx, dy) corresponding to the horizontal direction (X) and the vertical direction (Y) as the parallax offset corresponding to the selected feature point pair; The information processing according to claim 8, wherein a constraint condition of an allowable value of parallax offset is set using the generated histogram, and an optimal value of the parallax offset is calculated based on the set constraint condition. Method. The parallax offset calculating step includes:
A step of executing a process of determining a peak position of a histogram of [dy] that defines a parallax offset in the vertical direction (Y) as an optimum value of the parallax offset (dy) corresponding to the vertical direction (Y). The information processing method according to claim 9. The parallax offset calculating step includes:
In the [dx] histogram that defines the parallax offset in the horizontal direction (X), the minimum allowable value [dx, min] and the maximum allowable value [dx, max] of [dx] are set according to a preset appearance frequency rate. 10. The information processing according to claim 9, wherein the information processing step is a step of setting as an allowable value of the parallax offset allowable value and calculating an optimal value of the horizontal direction (X) parallax offset based on the set limiting condition. Method. The parallax offset calculating step includes:
The error of the feature point and the number of independent (Outliers) data are verified when a plurality of [dx] values set between the minimum allowable value [dx, min] and the maximum allowable value [dx, max] are applied. 12. The step of executing a process of determining a value of [dx] that minimizes the number of errors and outliers as an optimal value of a parallax offset in a horizontal direction (X). Information processing method. The parallax offset calculating step includes:
As a filtering process to select feature point pairs judged to be highly reliable,
As the parallax offset corresponding to the feature point pair, the values of the parallax offset (dx, dy) corresponding to the horizontal direction (X) and the vertical direction (Y),
(A) Epipolar constraint on condition that dy is in a preset range
(B) Disparity constraint on condition that dx is in a preset range,
(C) When there are a plurality of pairs (one-to-many) satisfying both of the above (a) and (b), these pairs are determined to be feature points pairs with high ambiguity and eliminated. Restraint,
It is a step of executing a filtering process that executes determination of whether or not at least one of the constraint conditions (a) to (c) is satisfied, and leaves only those that satisfy the constraint condition. The information processing method according to claim 8. The information processing method further includes:
The three-dimensional map generation unit includes a three-dimensional map generation step for executing camera calibration and a three-dimensional map generation process to which an optimum value of the parallax offset calculated in the parallax offset calculation step is applied. The information processing method in any one of 8-13. In the information processing apparatus, a computer program that inputs images taken by two cameras installed substantially in parallel at different positions and calculates a parallax offset based on the taken images,
A feature point extracting step for causing the feature point extracting unit to extract feature points included in the captured images of the two cameras;
A feature point matching processing step for causing the feature point matching processing unit to detect a feature point pair composed of a combination of similar feature points by comparison processing of feature points extracted from the captured images of the two cameras;
A parallax offset calculating step for causing the parallax offset calculating unit to calculate a parallax offset in the captured images of the two cameras by analyzing the feature point pair;
The parallax offset calculating step includes:
A feature point pair determined to have high reliability by filtering the feature point pair is selected, a histogram is generated as occurrence frequency distribution information of parallax offset data corresponding to the selected feature point pair, and the generated histogram is used. A computer program characterized in that it sets a constraint condition for an allowable value of parallax offset and calculates an optimum value of the parallax offset based on the set constraint condition.