JP4812782B2 - Feature point correspondence method between images, feature point correspondence device between images, feature point correspondence program between images, and a recording medium storing the program. - Google Patents

Description

  According to the present invention, the position, angle, and internal parameters (lens distortion, intersection position between a lens optical axis and an image sensor, etc.) of a plurality of photographing devices (for example, a digital camera and a digital video camera) are obtained only by image processing. In addition, the present invention relates to processing (so-called labeling processing) of how feature points of images captured by each imaging device correspond.

  In the field of robot John and machine vision using a camera as a visual sensor, when the camera input image is analyzed and semantic information (information with a specific meaning) is extracted from the real space, the camera mounting position is accurate. Need to be measured. This is generally called camera calibration or camera calibration.

  For this calibration, a flat board 1 printed with white and black block patterns as shown in FIG. 4 is often used. The plane board is photographed with a plurality of cameras, and the vertexes of each quadrilateral of the plane board reflected on each image are detected as feature points, and the process of associating the cameras (so-called labeling process) is performed. .

  The processing steps (S501 to S506) for associating the images A and B obtained from the two cameras will be described below based on the processing flow of FIG. Note that when two or more cameras are used, the same processing steps (S501 to S506) are repeatedly performed two by two.

  S501: A corner portion (vertex) is extracted as a feature point by image processing. Here, for the entire image, significant points in the light and shade region, that is, corner portions (vertices) are detected, and the coordinate values of these points are recorded. The same process is performed for all images.

S502: First, four corner points (vertices) are manually associated. On the image A, four points that are not on the same straight line are selected, and these are set as “P _i , i = 1, 2, 3, 4”. The respective coordinate values are assumed to be “(x _iA , y _iA ), i = 1, 2, 3, 4”. In the image B, the coordinates of the corresponding points of each P _i are obtained manually, and these are set as “(x _iB , y _iB ), i = 1, 2, 3, 4”.

  S503: An inter-plane transformation matrix is obtained from four pairs of corresponding points. When the corresponding points are on the same plane, the following interplane transformation matrix holds.

However, “f _A , f _B ” are focal lengths of the respective cameras, and are given as arbitrary constants other than “0” when the corresponding points are assigned. “H is a 3 × 3 matrix” and is referred to as an interplane transformation matrix or homography. Substituting the above four pairs of corresponding point coordinates “(x _iA , y _iA ), (x _iB , y _iB ), i = 1, 2, 3, 4” into the above [Equation 1], If an equation is obtained and solved, an interplane transformation matrix is obtained. The “H” obtained here theoretically satisfies the above [Equation 1] between corresponding points on the same plane.

  S504: The coordinates of the corresponding position on the image A of the feature point on the image B are calculated using the transformation matrix. Here, in the image B, if one feature point on the plane board is selected and the conversion matrix H is used, the corresponding coordinate value on the image A can be calculated from the [Equation 1].

  S505: Search for feature points in the vicinity of the calculated coordinates in the image A. That is, in the image A, the feature point closest to the corresponding coordinate value calculated in S504 is searched from the feature points detected on the image.

  S506: The searched feature points are recorded as corresponding points, and a pair of corresponding points is finished. The search result is recorded as a corresponding point on the image A, and the pairing is finished.

Then, the entire images are associated with each other by repeating steps S504 to S506 for all feature points. This conventional method is described in Non-Patent Document 1.2.
Z. Zhang. A flexible new technology for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22 (11): 1330-1334, 2000. Kenichi Kanaya, Image comprehension-Mathematics of three-dimensional recognition-, 4.2-4.4, p. 82-98, Morikita Publishing Co., Ltd., ISBN: 4-627-82140-9, 1990.

  The prior art strongly depends on the theoretical plane constraint property, that is, the property that all feature points exist on the same plane, and the conversion example H is obtained only once in the initial stage of the corresponding point assignment. The transformation matrix H calculated in (1) is used over the entire image.

However, in reality, the association cannot be easily performed due to the following causes, and the accuracy and efficiency of calibration may be reduced.
(1) Influence of Lens Distortion As shown in FIG. 6A, since the image of the flat board is photographed as a curve due to the influence of lens distortion, only the transformation matrix H calculated from the initial four corresponding points is used. In this case, the corresponding position to be calculated may be greatly deviated from the actual feature point position, and the correlation may fail. FIG. 7 shows an example of the failure. Here, four points in the upper left block in the figure are used as the initial four corresponding points. A region surrounded by a line in the lower right on the image indicates a portion where the association has failed.
(2) Influence of shooting direction As shown in FIG. 6B, when the plane board has to be shot from an oblique direction due to the camera arrangement, the blocks on the image are distorted and projected. The size varies greatly depending on the location. This creates a region where feature points on the image are dense, making it difficult to associate them.

  Therefore, the present invention solves the problem of improving the accuracy and efficiency of calibration by suppressing the effects of lens distortion and the shooting direction (such as the positional relationship between the shooting device and the flat board) to achieve stable matching. It is an issue.

  The present invention is a technical idea created in order to solve the above-mentioned problem, and first performs a labeling process between feature points with high accuracy in a certain local specific region, and then uses the coordinates of the feature points. Find the inter-plane transformation matrix, then label the feature points in the vicinity of the specific region, and use the labeled feature points again to obtain the inter-plane transformation matrix. Labeling process is performed.

Specifically, a method for associating feature points between images according to claims 1 to 3, a device for associating feature points between images according to claims 4 to 6, and a feature between images according to claim 7. The program comprises a point correspondence program and a recording medium recording the program according to claim 8.
<Invention of method>
The invention according to claim 1 is a method of photographing a plane object with a plurality of photographing devices, and associating feature points between images obtained from the photographing devices using a computer. A first step of extracting feature points of the plane object from the entirety of the image, and a second step in which the computer selects a part of each image as an initial region and associates the feature points of the initial region between the images. And the third step in which the computer obtains the initial plane transformation matrix from the corresponding point coordinates of the initial block, and the corresponding pointing is performed immediately before the region in which the computer has already assigned the characteristic points in the specific image. It was selected neighboring region adjacent to the region, by using a planar transformation matrix calculated immediately before, and a fourth step of determining a corresponding position in the other image feature points of said neighboring area, computer Sixth step but the searching feature points closest other images at corresponding positions, to calculate a fifth step of recording the search result as the corresponding points, a transformation matrix computer from the corresponding point coordinate of the search results again And the cycle of the fourth step to the sixth step is repeated until the matching of the feature points of the entire planar object on the image is completed.

  In the invention according to claim 2, in the fourth step, in the first cycle, a neighboring area adjacent to the initial area is selected, and the corresponding position is obtained using an initial plane transformation matrix. In the cycle, a neighboring region of the block associated in the previous cycle is selected, and the corresponding position is obtained using the plane transformation matrix calculated in the previous cycle.

According to a third aspect of the present invention, a planar plate on which a plurality of polygonal block patterns are depicted is used as the planar object, and each of the regions is constituted by one block of the block pattern reflected on the image. It is characterized by being.
<Invention of apparatus>
The invention according to claim 4 is an apparatus that captures a plane object with a plurality of imaging devices, and associates feature points between images obtained from the imaging apparatuses, wherein the plane object is detected from the entire image. First means for extracting feature points; second means for selecting a part of each image as an initial region and associating feature points of the initial region between images; and corresponding point coordinates of the initial block The third means for obtaining the initial plane transformation matrix from the above, and selecting a neighboring region adjacent to the region for which the corresponding points are assigned immediately before the corresponding points for the feature points in the specific image, Using the calculated plane transformation matrix, the fourth means for obtaining the corresponding position in the other image of the feature point of the neighboring area, and searching for the feature point of the other image closest to the corresponding position, and corresponding the search result Fifth means for recording as points, and the search result Sixth means for recalculating the transformation matrix from the corresponding point coordinates, and the fourth means to the sixth means repeat the processing cycle until the matching of the feature points of the entire planar object on the image is completed. It is characterized by executing.

  According to a fifth aspect of the present invention, in the first processing cycle, the fourth means selects a neighboring area adjacent to the initial area and obtains the corresponding position using an initial plane transformation matrix. In the processing cycle, the neighboring region of the block associated in the previous processing cycle is selected, and the corresponding position is obtained using the plane transformation matrix calculated in the previous processing cycle.

The invention according to claim 6 uses, as the plane object, a plane plate on which a plurality of polygonal block patterns are depicted, and each of the areas is constituted by one block of the block pattern reflected on the image. It is characterized by being.
<Invention of program / recording medium>
The invention described in claim 7 causes a computer to execute the first step to the sixth step in the method for associating feature points between images according to any one of claims 1 to 3, and the fourth step to the It is characterized in that the sixth step cycle is repeated until the computer completes the matching of the feature points of the entire planar object on the image.

  The invention described in claim 8 relates to a recording medium storing the feature point correspondence program between images described in claim 7.

  According to the first to eighth aspects of the present invention, when the feature point labeling process is performed, it is possible to always use the inter-plane conversion matrix suitable for the situation of the feature point, thereby improving the accuracy of the labeling process.

  Therefore, high-accuracy and high-efficiency calibration can be realized without being affected by lens distortion and shooting direction (such as the positional relationship between the shooting device and the flat board).

  For camera calibration, feature points to be photographed in real space are extracted on the image and matching between the images is indispensable. However, for high-precision calibration, the number of feature points to be matched In many cases, it is necessary to increase the amount of work.

  Therefore, the present invention proposes a method for automatically associating feature points, thereby increasing the number of feature points to be efficiently associated, and realizing highly accurate and highly efficient calibration. . One embodiment of the technique of the present invention will be described with reference to FIGS.

  FIG. 1 shows the configuration of an embodiment of the present invention. Here, a flat board 1 on which a white and black block pattern is printed, two photographing devices C1 and C2 (for example, a digital camera and a digital video camera) for photographing the flat board 1, and both photographing devices C1 and C2 And a processing device 2 connected via a network.

  The flat board 1 will be described as an example using a square block pattern in which the same black and white pattern as the conventional one is repeated uniformly. However, other pattern formats, for example, a hexagon that is equally repeated depending on the usage scene. The pattern may be changed to

  As shown in FIGS. 1 and 2, the processing device 2 includes a communication interface unit 3 that receives image data from the photographing devices C1 and C2 via a network, and a storage unit 4 that can store the image data and the like (for example, A hard disk drive device), storage means 5 (for example, a memory) capable of temporarily storing the coordinate values and transformation matrixes of the feature points of the image data, and a processing unit 6 for controlling and calculating each component. [A CPU (Central Processing Unit), a GPU (Graphic Processing Unit), an OS (Operating System)], an input device 7 such as a keyboard and a mouse, and a display device D such as a monitor, and the like. Yes.

  The processing unit 6 includes feature point detection means 8, initial block selection means 9, vertex correspondence means 10, transformation matrix calculation means 11, neighboring block selection means 12, correspondence position calculation means 13, correspondence point search means 14, and correspondence. Vertex of each square drawn on the plane board 1 based on the image data A transmitted from the imaging device C1 and the image data B transmitted from the imaging device C2 functioning as the scoring means 15 and the transformation matrix updating means 16 (Feature points) are associated (so-called labeling processing).

More specifically, the processing unit 6 receives the image data A.1. After associating the feature points of the local specific region of B, the feature points are sequentially associated with each adjacent region. Hereinafter, an outline of the processing steps (S301 to S309) of the processing unit 6 will be described along the processing flow of FIG.
<Processing Step of Processing Unit 6>
S301: Corner portions (vertices) are extracted as feature points from the image processing of the feature point detection means 8. Here, as in the conventional S501, the image data A.I. From the block pattern of the flat board 1 shown in B, a remarkable point in the light and shade area, that is, a corner (vertex) is detected, and these coordinate values are recorded in the storage means 4.

  S302: The initial block selection means 9 receives the image data A.S. One rectangular block is selected from the block patterns shown on B.

  S303: The vertex correspondence means 10 uses the automatic processing such as template matching for the four vertices of the initial block, and the image data A.S. Association between B is performed.

  S304: The transformation matrix calculation means 11 obtains an initial interplane transformation matrix using the corresponding point coordinates of the four vertices of the initial block. Thereafter, the processing cycles of S305 to S309 are repeated for all other blocks.

  S305: The neighboring block selecting means 12 selects one neighboring block of the immediately preceding block that has already been scored in the image data B. As this neighboring block, the block adjacent to the initial block is selected in the first processing cycle of S305 to S309, while in the second and subsequent processing cycles, the corresponding block is marked in the previous processing cycle. Adjacent blocks are selected.

  S306: The corresponding position calculation means 13 calculates the coordinates of the corresponding position on the image data A using the plane transformation matrix calculated immediately before for the four vertices of the selected neighboring block. At this time, the initial plane conversion matrix is used in the first processing cycle of S305 to S309, while the plane conversion matrix calculated in the previous processing cycle is used in the second and subsequent processing cycles.

  S307: Corresponding point search means 14 searches the image data A for feature points in the vicinity of the coordinates calculated for the four vertices.

  S308: The corresponding point assigning means 15 records the feature points searched for the four vertices as corresponding points, and finishes the association of the four vertices of the neighboring blocks.

S309: The transformation matrix update means 16 recalculates the inter-plane transformation matrix using the four vertex coordinates of the block pair immediately before the association. Thereafter, the process returns to S305 and the processing cycles of S305 to S309 are executed for all the blocks that have not been associated.
<Specific contents of each processing step>
Hereinafter, each process step of S301-S309 is demonstrated concretely. Here, since the processing step of S301 is the same as that of S501 of the conventional example, the description thereof will be omitted and the processing step of S302 will be described.

S302: One rectangular block is selected from a plurality of captured blocks on the image of the block pattern plane board. This block is represented by “B ₀ ”.

S303: Corresponding points are assigned to the four vertices of block B ₀ by automatic processing such as template matching. As a result, four pairs of corresponding point coordinates not on the same straight line are obtained.

S304: The corresponding point coordinates of the four vertices of the block B ₀ are substituted into the [Equation 1] to obtain the inter-plane transformation matrix “H ₀ ”. In addition, a parameter “c” for counting the corresponding blocks is provided. In this step, “c = 0” is set.

S305: In the image data B, one block adjacent to the block B _c is selected, and this block is set to “B _{c + 1} ”. As a method of selecting this neighboring block, for example, it can be performed by an 8-direction neighborhood search of the immediately preceding block. Further, for example, a template matching method can be performed according to the pattern used.

S306: Using the transformation matrix H _c , the coordinates of the corresponding position on the image data A are calculated by substituting the coordinates on the image data B for the four vertices of the block B _{c + 1} into the [Equation 1].

S307: A feature point search is performed in the vicinity of the corresponding position on the image data A for the four vertices of the block B _{c + 1} .

S308: For the four vertices of the block B _{c + 1} , the feature points searched on the image data A are recorded as corresponding points, and the corresponding points of the block B _{c + 1} are finished. Here, one associated block is counted, and the value of “c” is incremented by one. This is expressed as follows.

S309: Since the corresponding points of the four vertices of the block B _c are obtained, the coordinate values of these four pairs of points are substituted into the [Equation 1] to obtain the inter-plane transformation matrix “H _c ”. Then, as long as there is a block that has not been assigned a corresponding score, the processing steps of S305 to S309 are repeated.

  In this case, the coordinate values and conversion matrix obtained in the processing steps of S302 to S309 may be appropriately stored in the storage unit 5. In addition, it is efficient to display the progress of the processing steps of S301 to S309 on the display device D as appropriate.

  As described above, according to the processing method according to the present embodiment, the photographing devices C1. When the camera calibration of C2 is performed, the image data A.B. Corresponding points between B can be efficiently and stably performed. In the conventional method, the entire area of the image data depends on the precondition that the influence of the distortion of the optical system and the distortion due to the shooting direction is small, but here it does not depend on the precondition. Focusing on the fact that the effects of distortion and other factors are minimal when viewed locally in the image data, local processing is performed, and at the same time the global feature point adjacency is used to determine the overall feature point of the image data. The association is performed by iterative processing.

  In other words, using a single transformation matrix for the entire area of the plane board on the image data and using the information on the neighboring block vertices associated with each block on the plane board, compared to the conventional method of associating all feature points The stability of the corresponding points is ensured by updating the transformation matrix.

  As a result, it is possible to stably assign corresponding points between camera images having a large lens distortion, which is difficult with the conventional method, and corresponding points of flat board images photographed in an oblique line-of-sight direction. As a result, efficient camera calibration can be realized in the field of robot vision and machine vision, and costs such as system setup and maintenance can be suppressed.

  In addition, this invention is not limited to the said embodiment, A various deformation | transformation is possible in the range described in each claim. For example, it can be constructed as a program that causes a computer to function as the processing device 2.

  In this case, it can also be realized by supplying a recording medium storing the program code to a computer, and reading and executing the program code stored in the recording medium by a processing unit (CPU or the like) of the computer.

  Then, the program code itself read from the recording medium causes the computer to execute the processing steps S301 to S309, and repeatedly executes the processing steps S305 to S309 until the feature points are associated with each other.

  This program code is stored in, for example, a CD-ROM, DVD-ROM, CD-R, CD-RW, DVD-R, DVD-RW, MO, HDD, or the like. Further, the program may be downloaded from an Internet site to realize the function of the measuring apparatus 100.

Schematic which shows the structure of the method which concerns on embodiment of this invention. Schematic of the processing apparatus. FIG. An example figure of the plane board for camera calibration. The processing flow figure of a conventional method. (A) is an example of an image of a flat board affected by lens distortion, and (b) is an example of an image taken from an oblique viewing direction. An example of an image in which matching with the conventional method has failed.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Plane board 2 ... Processing apparatus 3 ... Communication interface 4 ... Storage means 5 ... Storage means 6 ... Processing part 8 ... Feature point detection means 9 ... Initial block selection means 10 ... Vertex correspondence means 11 ... Transformation matrix calculation means 12 ... Neighborhood Block selecting means 13 ... corresponding position calculating means 14 ... corresponding point searching means 15 ... corresponding point assigning means 16 ... transform matrix updating means C1. C2 ... Imaging device D ... Display device S301 ... Feature point detection processing S302 ... Initial block selection processing S303 ... Initial correspondence marking processing S304 ... Initial interplane transformation matrix calculation processing S305 ... Neighboring block selection processing S306- S308: Processing for determining corresponding points S309: Processing for updating inter-plane transformation matrix S501: Processing for feature point detection of conventional method S502: Processing for initial matching of conventional method S503: Calculation of initial inter-plane conversion matrix of conventional method Processing S504 to S506: Processing for determining corresponding points of the conventional method

Claims (8)

A method of photographing a plane object with a plurality of photographing devices and associating feature points between images obtained from the respective photographing devices using a computer,
A first step in which a computer extracts feature points of a plane object from the entirety of each image;
A second step in which a computer selects a part of each image as an initial region and associates feature points of the initial region between images; and
A third step in which a computer obtains an initial plane transformation matrix from corresponding point coordinates of the initial block;
The computer selects a neighboring area adjacent to the area where the corresponding points have been assigned just before the corresponding points of the feature points in the specific image, and uses the plane transformation matrix calculated immediately before, A fourth step of obtaining corresponding positions in other images of the feature points of the neighboring area;
A fifth step in which a computer searches for a feature point of another image closest to the corresponding position and records the search result as a corresponding point;
A sixth step in which the computer recalculates a transformation matrix from the corresponding point coordinates of the search result,
The method of associating feature points between images, wherein the cycle of the fourth step to the sixth step is repeated until the feature points of the entire plane target on the image are completely associated. The fourth step includes
In the first cycle, a neighboring area adjacent to the initial area is selected, and the corresponding position is obtained using an initial plane transformation matrix,
The neighborhood of the block associated in the previous cycle is selected in the second and subsequent cycles, and the corresponding position is obtained using the plane transformation matrix calculated in the previous cycle. A method for associating feature points between images according to 1. As the plane object, using a plane plate on which a plurality of polygonal block patterns are depicted,
The method for associating feature points between images according to claim 1, wherein each of the regions is constituted by one block of a block pattern reflected on the image. A device for photographing a plane object with a plurality of photographing devices, and associating feature points between images obtained from the photographing devices,
First means for extracting feature points of a plane object from the whole of each image;
A second means for selecting a part of each image as an initial region and associating feature points of the initial region between images;
A third means for obtaining an initial plane transformation matrix from the corresponding point coordinates of the initial block;
Among the areas where the corresponding points of the feature points in the specific image have been assigned, select the neighboring area adjacent to the area where the corresponding points have been assigned immediately before, and use the plane transformation matrix calculated immediately before, the neighboring area A fourth means for obtaining a corresponding position in another image of the feature point;
Searching for a feature point of another image closest to the corresponding position and recording the search result as a corresponding point;
A sixth means for recalculating the transformation matrix from the corresponding point coordinates of the search result,
The apparatus for associating feature points between images, wherein the fourth means to the sixth means repeatedly execute the processing cycle until the feature points of the entire planar object on the image are completely matched. The fourth means includes
At the time of the first processing cycle, a neighboring region adjacent to the initial region is selected, and the corresponding position is obtained using an initial plane transformation matrix,
In the second and subsequent processing cycles, the neighboring area of the block associated in the previous processing cycle is selected, and the corresponding position is obtained using the plane transformation matrix calculated in the previous processing cycle. The apparatus for associating feature points between images according to claim 4. As the plane object, using a plane plate on which a plurality of polygonal block patterns are depicted,
The device for associating feature points between images according to any one of claims 4 and 5, wherein each of the regions is constituted by one block of a block pattern reflected on the image. A computer executes the first step to the sixth step in the method for associating feature points between images according to any one of claims 1 to 3,
A program for associating feature points between images, which causes the computer to repeat the cycle of the fourth step to the sixth step until the feature points of the entire plane object on the image are completely associated.   A recording medium storing the feature point correspondence program between images according to claim 7.