WO2007066002A1

WO2007066002A1 - Method of bringing steoreoscopic images into correspondence

Info

Publication number: WO2007066002A1
Application number: PCT/FR2006/002659
Authority: WO
Inventors: Lucile Martin; Christophe Leroux
Original assignee: Commissariat A L'energie Atomique
Priority date: 2005-12-06
Filing date: 2006-12-06
Publication date: 2007-06-14
Also published as: FR2894355B1; FR2894355A1; EP1958157A1

Abstract

The present invention relates to a method of bringing a first image and a second image into correspondence, the said images coming from acquisition devices, the optical axes of which are approximately parallel, and forming two stereoscopic images, which method comprises the steps of: selecting, from said first and second images, first and second lists of analysis points; calculating the relative angles between the straight lines joining the points of a first set of pairs of analysis points to the straight lines joining the points of a second set of pairs of analysis points, said pairs belonging to at least part of the set of pairs of analysis points, the first element of said pairs of points of said set belonging to said first list, and the second element of said pairs of points of said set belonging to said second list; and selecting, as pairs of analogous points, from said at least part of said set, the pairs corresponding to the angles having a predominant distribution.

Description

Image matching method

stereoscopic

The present invention relates to the field of analysis and processing of stereoscopic images.

More specifically, the subject of the invention is a method of matching analogous points of a first image and a second image, the first and second images forming two stereoscopic images.

This type of method for matching analogous points is known per se and consists, for example, from two images from two separate cameras, of determining the positions in each camera image, of points corresponding to the projection of the same physical point in three-dimensional space.

It can also be used to determine the displacement of a camera having acquired two successive images. In this case, if we can determine that a point of the first image corresponds to the same physical point in three dimensions as another point of the second image, we can determine the movement of the camera from the position of the colon in the field.

In general, two points of two stereoscopic images corresponding to the same physical point in three dimensions will be called analogous points.

The publication "IEEE Transactions on pattern analysis and machine intelligence" (BROWN, BURSCKA, HAGER, VOL. 25, N ° 8, August 2003) offers a review of the various known methods of matching two stereoscopic images. These methods include for example the matching of image blocks (or “block matching” in English), the optimization of gradients (or “gradient-based optimization” in English), or the matching of elements (or “feature matching” in English).

However, the reliability of these mapping methods is sometimes insufficient and the occurrences of erroneous mapping distort subsequent three-dimensional reconstructions.

One of the objects of the present invention is therefore to find an alternative to the methods for matching known stereoscopic images.

There are also mapping methods based on the distances between the mapped points to locate a mobile relative to a target object.

The disadvantage of these distance-based methods is that the points positioned in the real scene at different depths do not move at the same apparent speed in the images of said scene. In fact the distances between the homologous points corresponding to the same 3D point in the images of the real scene are different depending on the depth of this point. We cannot therefore judge the reliability of the pairings on homologous points corresponding to 3D points of different depths based solely on the distance separating them. Another object of the present invention is therefore to provide a method which takes into account the depth of the objects of a reference scene. In addition, several methods of matching require calculation times that are too long to be able to be used quickly, for example in the context of a robot provided with two cameras, and seeking to orient itself in space.

Another object of the present invention is therefore to provide a fast matching method.

Also known in the prior art, the publication of Adam et al., XP 002395780, "Rejection of Outliers by rotations in Stereo Matching", IEEE 2000. In the method described in this article, any analysis points of two stereoscopic images are joined on the same common coordinate system so as to form a set of segments. Random rotations are then applied to these segments so as to determine which rotation provides a distribution of the segments in substantially the same direction.

In this document, it is therefore necessary to test a priori an infinity of rotations to determine which rotation will provide a uniform set of segments. In addition, since the analogous points are not known before having performed rotations, it is not possible to know how to choose points of the two images so as to form segments of analogous points. The object of the above-mentioned publication is therefore a method for eliminating clearly non-analogous points in two stereoscopic images, but not a method for searching for analogous points in two stereoscopic images.

The invention aims, on the contrary, to enable such analogous points to be determined. To this end, the present invention relates to a method of matching analogous points of a first image and a second image, the first and second images forming two stereoscopic images coming from acquisition devices including the optical axes. are substantially parallel and it is characterized in that it comprises steps consisting in: that it comprises steps consisting in:

Select in said first image, a first list of analysis points ([Ai, A ₂ , ... A _n ]);

- Select in said second image, a second list of analysis points ([Bi, B ₂ , --- Bp]);

Calculate the relative angles (θjk; Θ _s tuv) between the lines joining the points of a first set of pairs of analysis points and the lines joining the points of a second set of pairs of analysis points, said pairs belonging to at least one part (C ₁ C) of the set (C) of the pairs of analysis points ([(Aj, B _k )], j = 1, n, k = 1, p), the first element of said pairs of points of said set (C) belonging to said first list and the second element of said pairs of points of said set (C) belonging to said second list;

Select as pairs of analogous points from said at least part (C, C) of said set (C), the pairs ([(Aj ₁ B _k )]) corresponding to the angles (θ _m ) having a majority distribution.

Thanks to the measurement of the relative angle between pairs of points, and to the selection of the majority distribution relative angles, it is therefore possible, according to the invention, to determine pairs of analogous points in two stereoscopic images.

The fact of making measurements on relative angles makes it possible in particular to process the two images without passing through a common reference frame for the two images. The relative angle can therefore be represented in an arbitrary coordinate system.

Preferably, said at least part (C, C) of the set (C) of the pairs of analysis points is chosen as a function of displacement constraints relating to the pairs of analysis points of said set (C). These constraints are for example chosen so as to ensure the uniqueness of said pairs of points.

They can depend on at least one parameter chosen from distance, brightness, order and pyramid stability, that is to say multi-scale stability at several energy resolutions.

Thanks to this process and to the purely statistical orientation criterion used, two images are matched, for example in the context of a 3D spatial reconstruction. The invention also relates to a computer program comprising a set of instructions for carrying out the method as described above, as well as a recording medium for recording such instructions.

The invention is particularly suitable when the devices for acquiring the first image and the second image have substantially parallel optical axes. Other aspects, goals and benefits of. the invention will appear on reading the detailed description of the invention.

The invention will also be better understood using the drawings in which:

- Figure 1 is a general view of an application environment of the method according to the present invention;

- Figure 2 illustrates an example of implementation of the method according to the invention;

- Figure 3 illustrates an example of a histogram for matching according to the present invention.

- Figure 4 illustrates an example of definition of the pairs of points defining lines for obtaining a histogram of the orientations according to the present invention.

FIG. 5 represents the histogram of the orientations associated with FIG. 4.

- Figure 6 illustrates an example of application of the constraints within the framework of the method according to the present invention.

In the different figures, the same references designate identical or similar elements. The extraction of positioning and / or orientation information from an object is for example applicable in the context of the location of a mobile vehicle or a robot arm equipped with cameras with respect to its environment.

FIG. 1 represents such an environment in which the invention can be implemented.

In this figure, a mobile robot 1 is equipped with cameras 2A ₁ 2B and must locate itself in its environment here containing a cylindrical object 3. The camera system 2A,

2B of the robot is connected by wire or wireless connection to a computer

5 carrying out the acquisition and processing of the images. The camera system 2 allows the acquisition of a couple of images of the environment in front of it. These 4D, 4G images of the environment notably include an image of the object 3. As part of the analysis of stereoscopic images, the two images 4D and 4G have at least one common part, here the object 3.

According to the invention, one begins by selecting a certain number of points of interest on one of the 4D or 4G images, for example here 4D. Known techniques for selecting points of interest are the Moravec detectors, the Harris and Stephens detectors, and the SUSAN filter. These point selection methods often use a 4D image analysis to distinguish their singularities, such as the limits of the elements they represent, from sudden variations in brightness from one point from one image to another. . It should be noted that the selection of points does not depend on the operator but on the program alone and that it remains unknown to the operator; however, the operator can limit the selection of the points of the 4D image to the regions comprising the image of a particular object 3 and its surroundings, by specifying for example a frame outside of which the selected points are separated.

We will then either select a certain number of points of interest on the other image, here 4G, using the same technique as for the 4D image; either deduce the points detected in the 4D image from potentially homologous points in the 4G image by so-called tracking or pyramidal techniques which seek the points of 4G having invariance characteristics similar to those of 4D successively stable and robust in several under parts of 4G. The points finally selected in the 4D image are noted by the letter A followed by a numerical index and those selected in the 4G image are noted by the letter B followed by a numerical index. The analysis points selected are therefore the points (A1, A2, ..., An) in the 4D image, and the points (B1, B2, ..., Bp) in the 4G image.

For the purposes of the explanation, the 4D and 4G images of the object 3 are superimposed in the same frame. After this step of selecting the points, according to the invention, the orientations of the lines are identified (Aj Bk) .

As illustrated in FIG. 2, these orientations can correspond to the angle between a reference line D, for example horizontal, and the line passing through the analysis points (Aj BR). This angle will then be noted θj _k . They can also correspond to the angle between the lines passing through a first pair of points (A _s , B _t ) and a second pair of points (A ₁₁ , B _v ), this second pair of points

(Au ₁ Bv) being distinct from the first pair of points (A _s , Bt) - In this case, the angle will be noted Θ _s tuv.

It is understood that the relative orientations between the lines passing through different pairs of points can be measured as well with respect to a reference line and with the variable%, as with respect to the angles Θ _s tuv-

It will be observed in this respect that the first case is only a special case of the second in which one of the lines is always the same and is supposed to pass through any two points (Ao ₇ B ₀ ) of the images 4D and 4G respectively .

The relative orientation between pairs of points of the two images can also be calculated with respect to the straight line passing through the calculated or assumed projections of the optical centers of the cameras in the images.

In the rest of the description, we will only use the variables θj _k , defined with respect to a reference line. In general, the angle of a pair of points thus corresponds to the angle between the line defined by said pair of points, and a reference line.

According to a first embodiment of the invention, all the points A and all the points B are thus matched by an orientation. For analysis points (Ai, A ₂ , ..., A _n ) in the 4D image and (Bi, B2, • .., B _p ) in the 4G image, np couples are therefore formed in a set of couple C.

The% orientations between all points A and all points B within the calculated exhaustiveness limits are then listed and those whose values are almost similar or almost similar to the orientation of the straight line passing through the optical centers of the two 4D images and 4G are grouped together. We then look for the most frequent non-zero orientation value θ _m f _m ax for the% in a tolerance X. This is for example achievable by statistical analysis of a histogram as illustrated in figure 3. As illustrated in this figure , we select by all the angles in the interval θ _m - X / 2, θ _m + χ / 2. This most frequent orientation θ _m then represents that of the displacement of the image of the object 5 between the two images 4D and 4G.

Thus, if an angle θj _k corresponding to the majority angular distribution and is therefore between θ _m - X / 2, θ _m + X / 2, we select the pair (Aj ₁ Bk) as a pair of similar points in the two stereoscopic images.

Similarly, if an absolute angle Θ _s tuv corresponding to the angle between the lines passing through the pairs (As ₁ Bt) and (Au, Bv) corresponding to the majority angular distribution and is therefore between θ _m - X / 2, θ _m + X / 2, we select the pairs (A ₃₁ Bt) and (A _U , B _V ) as pairs of analogous points in the two stereoscopic images.

According to the invention, the most frequent non-zero orientation criterion (or the maximum of substantially identical orientations) is therefore used as a purely statistical criterion for determining the displacement of the image of the object. This method is particularly applicable in the case of a monocular system where the displacement of the camera between the 4D and 4G images is a horizontal and / or vertical translation, and, in the case of a system of several cameras in which the axes optics of these cameras are parallel and the cameras are rigidly fixed. Under these conditions, the amplitude of the displacements (or the distances) of the images of the objects of the environment vary according to the distance of said objects relative to the camera system, but the orientation and the direction of the displacement remain the same .

According to a variant of the invention, once np couples formed in the set C, the pairs of points are eliminated according to predefined constraints making it possible to guarantee the uniqueness of pairing of each homologous point. We only retain, for example, the orientations compatible with a displacement constraint D _rs associated with a couple (Ar ₁ Bs) reflecting for example the displacement between the successive acquisition of two pairs of images. This constraint can be estimated using an annex sensor of the inertial central type. It is also possible to select the pairs of points of FIGS. 4G and 4D according to a constraint of resemblance of the points of the two images which depends on the relative luminosities of the points A _j and B _k in their respective neighborhoods. In this case, it is sought to match at a point A only the points B having specific luminosities and similar luminosities in all directions. Following the elimination of a certain number of pairs of points, the method of statistical analysis on the orientations is then implemented as described previously. In the case of constraints on displacements D _rs , the use of polar coordinates to estimate the orientations of the lines passing by the pairs of points and displacements thus makes it possible to make a selection according to the angles θj _k of displacements Dj _k on the images, then another selection according to the values of Dj _k .

This type of elimination of the pairs of points corresponds to a preliminary filtering before the stage of statistical analysis. This filtering has the advantage of speeding up the statistical processing time as a function of the orientations on the remaining pairs and of guaranteeing the uniqueness of pairing of each homologous point.

This filtering step can either be carried out following the calculation of the set C corresponding to all the possible pairs associated with lists A and B, or else during the formation of these pairs, by testing, for each candidate pair, whether the constraints are respected, and retaining only the pairs respecting the constraints. In the latter case, the set C of the pairs obtained corresponds to a subset of the set C and takes direct account of the constraints on the pairs of points of the 4D and 4G images.

Illustrated in FIGS. 4 and 5, an example of obtaining a histogram of the orientations according to the present invention is provided from two stereographic images. In this figure, object 3 corresponds to a line segment. On the one hand, the analysis points are points B1, B2, B3, points B1 and

B3 delimiting the line segment, and point B2 being in the middle of the segment, and points A1, A2, A3, points A1 and A3 delimiting the line segment, and point A2 being in the middle of the segment. It seems obvious that the pairs of analogous points are the pairs (A1. B1), (A2, B2), (A3, B3). The method according to the invention confirms this conclusion.

Indeed, we build the 9 lines passing on the one hand by the set of points (B1, B2, B3) and on the other hand by the set of points (A1, A2, A3). In FIG. 4, all the lines of the same relative orientation then bear the same name. There are then the lines of the same orientation and therefore of the same name on the histogram in FIG. 5. There are 3 lines D ₀ , 2 lines Di, 2 lines Di, 1 line D ₂ , and a line D-2. According to the invention, the pairs of points corresponding to the lines D ₀ are therefore analogous points. According to the invention, the pairs (A1, B1), (A2, B2) and (A3, B3) are therefore selected as pairs of analogous points, which confirms the intuitive analysis carried out previously. FIG. 6 illustrates an example of application of displacement constraints within the framework of the present invention.

In this figure, the lines (B1, A1), (B ¹ I ₁ A1), (B1, A'1) and (B'1, A'1), define an identical angle, for example with respect to a line horizontal reference.

According to the invention, the distances are then compared

B1A1, B'1A1, B1A'1, and B'1A'1. The maximum distance distribution corresponds to the pairs (B1, A1) and (B'1, A'1), and it is therefore these couples which will be retained as pairs of analogous points. The method according to the present invention is applicable in the context of locating a mobile vehicle or a robot arm equipped with cameras with respect to its environment. For this, two cameras mounted on the mobile vehicle are used for example, which take images of their environment and possibly of an object to be reached during the movement of the vehicle. It is also possible to use a single camera which successively takes images shifted laterally. A computer is used to locate the cameras in relation to their environment from the stereoscopic images taken by the cameras. This location information can be recalculated and enriched as the cameras move and new acquisitions are considered. Obtaining the localization parameters from stereoscopic images goes through the following stages:

a first step of calibrating the cameras and / or images, which may be known per se;

a matching step according to the present invention;

a step of 3D spatial reconstruction can be known per se, knowing the triangulation parameters of the stereo images and the magnification of the cameras.

Claims

1. Method for matching analogous points of a first image and a second image, the first and second images forming two stereoscopic images coming from acquisition devices whose optical axes are substantially parallel, characterized in that it includes steps of:

Select in said first image, a first list of analysis points ([Ai, A2, ... A _n ]);

Select in said second image, a second list of analysis points ([Bi, B ₂ , ... B _p ]);

Calculate the relative angles (θj _k ; Θ _s tuv) between the lines joining the points of a first set of pairs of analysis points and the lines joining the points of a second set of pairs of analysis points, said lines pairs belonging to at least one part (C ₁ C) of the set (C) of the pairs of analysis points ([(Aj ₁ Bk)], j = 1, n, k = 1, p), the first element of said pairs of points of said set (C) belonging to said first list and the second element of said pairs of points of said set (C) belonging to said second list;

Select as pairs of analogous points, from said at least one part (C, C) of said set (C), the pairs ([(Aj ₁ Bk)]) corresponding to the angles (θ _m ) having a majority distribution.

2. Method for matching a first image and a second image according to claim 1, wherein said at least part (C, C) of the set (C) of the pairs of analysis points is chosen as a function of displacement constraints relating to the pairs of analysis points of said set (C).

3. A method of matching a first image and a second image according to claim 1 or 2, wherein said at least a part (C, C) of the set (C) of the pairs of points of analysis is chosen as a function of resemblance constraints of the analysis points of said set (C), which depends on the relative luminosities of the points Aj and Bk in their respective neighborhoods.

4. Method for matching a first image and a second image according to claim 2, in which said constraints depend on at least one parameter chosen from distance, brightness, order and multi-scale stability. to several energy resolutions.

5. Method for matching a first image and a second image according to claim 1, in which said relative angles are calculated with respect to a reference line.

6. Computer program comprising an instruction set for implementing the method according to any one of claims 1 to 5.

7. A computer-readable digital data recording medium on which instructions for the implementation of the method according to any one of claims 1 to 5 are recorded.