JP2007272668A - Image processing apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus and method to detect a dense plane area even in an image sequence obtained with a monocular camera. <P>SOLUTION: A feature point detection means 2 detects feature points from a plurality of images input to an image input means 1, and a correspondence relation detecting means 3 associates the feature points of the images with one another. A conversion matrix calculating means 4 calculates a conversion matrix H for converting the pixel positions of at least three selected feature points in the first image to the positions of the corresponding points in the second image. An image conversion means 5 provides an image obtained by converting a predetermined area of the first image H using the conversion matrix H. A determining means 7 determines whether or not the area is a plane according to the similarity between the converted image and the second image that is calculated by a similarity calculating means 6. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and method, and more particularly to an image processing apparatus and method for estimating a plane portion in an image.

As a technique for detecting and estimating a planar area in an image, a technique disclosed in Patent Document 1 is known. In this technique, an image including a target region is acquired from a plurality of viewpoints using a plurality of cameras, the three-dimensional coordinates of each point in the image are obtained by stereo restoration, and the points sampled at random from those point groups Plane estimation is performed by performing Hough transform using.
JP 2003-269937 A

  However, the above technique has the following problems. First, a plurality of cameras whose positional relationships are known are required, and the hardware configuration is complicated. Secondly, the only point where the three-dimensional coordinates can be obtained by stereo restoration is a characteristic point in the image, and the restoration result becomes rough. Third, even if a flat surface is applied to the result thus obtained, sufficient accuracy cannot be obtained.

  Therefore, an object of the present invention is to provide an image processing apparatus and method capable of detecting a dense plane region even in an image sequence acquired by a monocular camera.

  In order to solve the above problems, an image processing apparatus according to the present invention includes (1) an image acquisition unit that acquires a plurality of images, and (2) a feature point that is associated between the images acquired by the image acquisition unit. Corresponding point detecting means for detecting a certain corresponding point; and (3) a conversion matrix for converting at least three points on the same plane from a pixel position on one image to a pixel position on the other image among the corresponding points. (4) image conversion means for generating a new image from one image by pixel position conversion based on the obtained conversion matrix, and (5) predetermined between the generated image and the other image. (6) on the plane formed by the corresponding points used when the predetermined matrix calculates the transformation matrix by the transformation matrix calculation unit based on the obtained similarity Whether or not it is a point Characterized by comprising determination means for disconnection, the.

  On the other hand, the image processing method according to the present invention includes (1) acquiring a plurality of images, (2) detecting corresponding points that are feature points associated with the acquired images, and (3) , Obtaining a transformation matrix for converting at least three points on the same plane from a pixel position on one image to a pixel position on the other image, and (4) converting the pixel position from one image based on the obtained transformation matrix To generate a new image, (5) calculate the similarity for a predetermined area between the generated image and the other image, and (6) convert the conversion matrix to the predetermined area based on the obtained similarity. Each step of determining whether or not the point is on a plane formed by the corresponding point used in the determination is provided.

  According to the present invention, feature points are associated between two different images. At least three points located on the same plane are extracted from the corresponding points, and a conversion matrix for performing conversion from a pixel position on one image to a pixel position on the other image is obtained using these three points as a reference. When each pixel position on one image is converted by this conversion matrix, the pixel located on the same plane as these three points is converted to the same position as the pixel position on the other image. Pixels located on the plane are converted to different positions. Therefore, the area located on the same plane as these three points is located at the same position in the image transformed from this image by this transformation matrix and in the other image. Both images will be similar. Based on this relationship, the plane in the screen can be determined.

  Corresponding points used for obtaining the transformation matrix may be selected from points whose distance in the image or in the three-dimensional space is a predetermined value or less. This is because a closer point is more likely to be an end point on the same plane than a distance in the image or in a three-dimensional space.

  The image conversion is preferably performed by obtaining the luminance value at the pixel position after conversion by interpolation. An image to be processed usually has a configuration in which pixels are arranged in a tile shape. In this case, the pixel position after conversion by the conversion matrix and the pixel position of the other image may be shifted by a distance smaller than the pixel interval. In order to correct this shift and match the pixel position, the luminance value may be obtained by interpolation.

  The similarity calculation may be performed by comparing a difference in luminance value around the same coordinate on the image before and after conversion with a predetermined threshold value. That is, when the difference in luminance value is large, it is determined that they are not similar, and when the difference is small, it is determined that they are similar.

  The similarity calculation may be performed by comparing the even power of the luminance value around the same coordinate on the image before and after conversion or the integrated value of the absolute value with a predetermined threshold value. In this case, it is easier to determine that they are not similar when there is a pixel with a large difference in luminance value, compared to the case where a simple difference in luminance value is used.

  The predetermined area for calculating the similarity may be limited to the closed area formed by the corresponding points used when the transformation matrix is used. The closed area is a triangle formed by three points when there are three corresponding points, and a rectangle formed by four points when there are four corresponding points. In the case of 5 points or more, it corresponds to a polygonal region surrounded by corresponding points. This is because the feature points serving as the corresponding points are obtained by edge extraction or the like, and therefore usually end points of planes or curved surfaces are often formed.

  Alternatively, the image processing apparatus according to the present invention includes (1) an image acquisition unit, (2) a unit that detects a region in an image captured by the image acquisition unit, and (3) whether the detected region is a plane. And (4) means for further determining whether or not a small area in the area is a plane when it is determined that the area in the image is not a plane. To do.

  In another image processing method according to the present invention, (1) an image is acquired, (2) an area in the acquired image is detected, (3) it is determined whether the detected area is a plane, 4) The method includes the steps of further determining whether or not the small area in the area is a plane when it is determined that the area in the image is not a plane.

  Even if the first detected area is not a plane, it may be an area including a small area of the plane. Therefore, the plane is searched by further dividing the first area and verifying the area again.

  According to the present invention, between at least two images acquired from a plurality of images, for example, images having different positional relationships with a continuously acquired object, from one image to the other image on a specific plane. An image sequence obtained by a monocular camera is obtained by obtaining a transformation matrix that converts the position of the point (pixel) of the image, and when one image is transformed by this transformation matrix, an area similar to the other image is determined as a planar area. It is possible to detect a planar area in step (b). In particular, when there is a non-planar surface (for example, a curved surface) between these corresponding points, erroneous determination of this as a planar region can be reduced.

  By making the corresponding points used when obtaining the transformation matrix close in space or in the image, it is possible to verify each small region that is highly likely to be a plane. Further, by interpolating the luminance value at the time of image conversion, the pixel positions of the areas to be compared can be matched, and similarity determination is facilitated.

  If the similarity is calculated based on the difference in luminance values, the determination criterion becomes clear. Furthermore, by using an even power or an integrated value of absolute values, it is easy to determine that the pixels are not similar when pixels with a large difference in luminance values exist nearby.

  Corresponding points are highly likely to be points that constitute end points of the surface area, so that it is possible to prevent a plurality of continuous surfaces from being simultaneously determined by performing plane determination for the inside of the occlusion area by the corresponding points. .

  Even if the first detected area is not a plane, it is possible to determine a plane even for a plane whose edge is unclear by further subdividing the area and performing plane determination.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to the present invention. This image processing apparatus includes an image input means 1 for inputting a plurality of images obtained by photographing a recognition target, a feature point detection means 2 for detecting a feature point from each image, and correspondence between feature points between the images. Correspondence detection unit 3 for detecting the relationship, conversion matrix calculation unit 4 for converting the position of the selected feature point in one image into a position in the other image, and an image based on the obtained conversion matrix Image conversion means 5 that performs conversion, similarity calculation means 6 that calculates similarity between predetermined areas between predetermined images, determination means 7 that performs plane determination from the calculated similarity, and results of determination means 7 And a processing area designating unit 8 for redesignating the processing area if necessary.

  The image input unit 1 is a device that captures moving images such as a television camera and captures them as an image sequence. In addition, a digital camera device, an image scanner, etc., as long as it is a video capture device that captures an image sequence from moving image data held in a video recorder or the like, or a device that captures moving images or still images from different viewpoints Good. Alternatively, an image already stored in a storage device such as a computer may be targeted.

  The feature point detection unit 2, the correspondence relationship detection unit 3, the transformation matrix calculation unit 4, the image conversion unit 5, the similarity calculation unit 6, the determination unit 7, and the processing area designation unit 8 are all configured by a CPU, a ROM, a RAM, and the like. Is done. Each of these means may be configured as independent and separate hardware, but some or all of the hardware may be shared and each means may be realized by software. In this case, the software constituting each means may be independent, but a part of the software may be shared, or a plurality of means may be incorporated in one software. .

  Next, the image processing method according to the present invention, which is the operation of the present embodiment, will be specifically described with reference to the flowchart of FIG. First, a plurality of images are input to the image input unit 1 (step S1). Here, a case where two images having different viewpoint positions relative to the object are input will be described as an example. The input two images may be stored in a temporary storage unit (not shown) (for example, a memory such as a RAM or a magnetic disk).

  Next, the feature point detection means 2 detects feature point candidates in the first image I (step S2). Specifically, a technique for detecting a point having a characteristic luminance / luminance pattern may be used. For example, a Harris operator (C. Harris, M. Stephens, “A Combined Corner and Edge Detector,” Proc. Alley Vision Conf. , Pp. 147 to 151, 1988)). The position of the detected feature point on the image is stored in the temporary storage means described above.

The correspondence relationship detection means 3 obtains the correspondence relationship between the detected feature points in both images by template matching (step S3). Specifically, the pixel position in the second image T corresponding to the feature point in the first image I is calculated. By the expression (1), the similarity between the feature point (x ₁ , y ₁ ) of the first image I and the position (x, y) of the second image T is obtained, and the pixel position having a good similarity is defined as the corresponding position. What is necessary is just to judge. U and V represent sets in which ranges for determining similarity are set in the horizontal and vertical directions, respectively.

  When the correspondence relationship between the feature points is detected, the transformation matrix calculation unit 4 extracts predetermined three points from the feature points for which the correspondence relationship has been obtained (step S4). These three points are not the three points that are located on the same straight line, but are the three points that are expected to be located on one plane projected in the image. For example, the distance within the image is within a predetermined distance. Or three points close to each other within a predetermined distance in a three-dimensional space. Other selection criteria are: (1) No other feature points exist in the area surrounded by the three points, or even if they exist, the number is equal to or less than the predetermined number. (2) Two of the three points are on the same contour line. The other one point is located on another contour line that is the same as one of the two points.

Next, a conversion matrix H for converting the selected three pixel positions is obtained. This H can be expressed as follows, assuming that the pixel position P of the feature point on the first image I is (u ₁ , v ₁ ) and the pixel position P ′ of the feature point on the second image T is (u ₂ , v ₂ ). Satisfy (2).

A satisfying H satisfying this relationship with respect to three sets of feature points P ₁ -P ₁ ′, P ₂ -P ₂ ′, and P ₃ -P ₃ ′ as shown in FIG. 3 is obtained by the method of least squares or the like.

Next, an image conversion process (steps S6 to S9) is executed. First, pixels in a region formed by three points on the converted image are extracted. For this extraction, among the pixel positions P ₁ ′, P ₂ ′, and P ₃ ′ of the feature points on the second image T, for example, P ₁ ′ is the origin, and the remaining two feature points P ₂ ′ from P ₁ ′. , P ₃ ′ is t and s, the point P _n ′ in the region (blocking region) surrounded by the three points satisfies the following expression (3). Here, α + β is a scalar value from 0 to 1.

Therefore, a pixel position satisfying the expression (3), that is, a position where the coordinate value takes an integer in the coordinates in the original image (grid position as shown in FIG. 4) may be extracted.

Obtain the pixel position P _n at the next determined conversion prior image corresponding to the pixel position of the P _{n '(first} image) (step S7). This can be obtained by equation (4) by using the inverse matrix H ⁻¹ of the transformation matrix H obtained by equation (2).

Next, a luminance value corresponding to the pixel position _Pn is calculated (step S8). Here, the coordinate values of P _n is different from P n _', not necessarily take integer values (lattice position). Therefore, the following method is used. First, if the coordinate values are both integer values and a pixel exists at the corresponding position of the first pixel I, the value of that pixel is used, and if it exists between them, the closest pixel is used. And a second method is to calculate the luminance value by interpolation using the coordinate values and luminance values of surrounding pixels (4 to 16 pixels). As such an interpolation method for obtaining the luminance value by interpolation, a bilinear method, a bicubic method, or the like can be used. When the luminance value of the pixel position P _n is obtained in this way, the luminance value is set as the luminance value of the pixel position Pn ′ of the converted image (step S9). Thereby, the image T ′ after conversion is obtained.

Next, the similarity calculating means 6 calculates the similarity between the other image T and the converted image T ′, and determines whether the judging means 7 has a similar relationship (step S10). Here, the similarity is calculated in an area corresponding to the pixel position P _n ′ of the image T ′. As an evaluation method based on the degree of similarity, for example, it may be evaluated that they are similar when the difference in luminance value between corresponding positions of the images T and T ′ is within a predetermined value. Note that the absolute value of the difference between individual luminance values may be within a predetermined value, but in order to eliminate the influence of noise or the like in the acquired image, pixels that are within the predetermined value have a predetermined ratio or more. The evaluation may be performed based on whether the maximum difference is equal to or less than a predetermined value. Furthermore, evaluation may be performed by an integrated value of absolute values of differences in luminance values, a cumulative sum of squares, an average value, a mean square, or the like. Weighting may be performed in this integration or accumulation. Alternatively, a method may be used in which it is determined that the magnitude relationship between the luminance values of the pixel and the surrounding pixels is similar when the converted image T ′ and the other image T are the same.

  When it is determined that they are similar according to such a criterion, the region is determined to be a plane (step S11), and when it is determined that they are not similar, it is determined to be a non-plane (step S12). ). After completion of the determination, the process proceeds to step S13 to determine whether or not the combination of three points has been completed. If the combination has not been completed, the process proceeds to step S14 to select the next three points, and step S5 By going back to, all three combinations are verified.

  The transformation matrix H is a transformation into a position occupied on the second image T by points on the plane formed by the three selected points on the first image I. Therefore, it can be estimated that a point located at the same position in the image T ′ converted from the first image I by the conversion matrix H and the second image T is a point on the plane. In this embodiment, this relationship is used, and similarity determination can be performed for each pixel in the region surrounded by the three points. Therefore, it is possible to perform determination for a wide region with high accuracy.

  In particular, in order to obtain the transformation matrix, it is sufficient that the target plane looks different between the two images. Therefore, the images before and after the movement when the camera or the object moves are not acquired by a plurality of cameras. May be used. Therefore, the plane can be determined also from the image sequence (moving image) acquired by the monocular camera.

  When the feature points are spaced apart in the image or in the three-dimensional space, the region surrounded by the selected three points may be constituted by a plurality of planes or curved surfaces including one plane. In such a case, since the entire area does not exist on the same plane as the plane formed by the three selected points, it is determined that the area is not a plane. Therefore, the processing area designating means 8 subdivides the first three selected areas, and performs the determination processing in steps S5 to S12 for each of the subdivided areas. A plane can be determined from the configured area. As a subdivision method, a method of detecting feature points based on another feature point detection method in the region, a method of dividing a region by equally dividing the region, or the like may be used.

  In the above description, an example in which three points are selected has been described, but four or more points may be selected. If four or more points are selected, the accuracy of the transformation matrix H increases. However, unlike the case of three points, the four points are not necessarily on the same plane, so the four or more points are end points on the same plane. It is preferable to use it when it is expected that there is a high possibility.

1 is a block diagram showing a schematic configuration of an image processing apparatus according to the present invention. It is a flowchart which shows the specific operation | movement process of the image processing method concerning this invention. It is a figure explaining calculation of a conversion matrix. It is a figure which shows the mode of a pixel.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Image input means, 2 ... Feature point detection means, 3 ... Correspondence relationship detection means, 4 ... Conversion matrix calculation means, 5 ... Image conversion means, 6 ... Similarity calculation means, 7 ... Determination means.

Claims (14)

Image acquisition means for acquiring a plurality of images;
Corresponding point detection means for detecting corresponding points that are feature points associated between images acquired by the image acquisition means;
Conversion matrix calculation means for obtaining a conversion matrix for converting at least three points on the same plane from a pixel position on one image to a pixel position on the other image among the corresponding points;
Image conversion means for generating a new image by pixel position conversion from one image based on the obtained conversion matrix;
Similarity calculation means for calculating the similarity for a predetermined area between the generated image and the other image;
Judging means for judging whether or not the predetermined area is a point on a plane formed by corresponding points used when the transformation matrix calculating means obtains the transformation matrix based on the obtained similarity;
An image processing apparatus comprising:   2. The image processing apparatus according to claim 1, wherein the corresponding points used when the transformation matrix calculation means obtains the transformation matrix are selected from points whose distance in the image or three-dimensional space is a predetermined value or less. .   2. The image processing apparatus according to claim 1, wherein the image conversion means performs image conversion by obtaining a luminance value at the pixel position after conversion by interpolation.   The image processing apparatus according to claim 1, wherein the similarity calculation unit is configured to compare a difference between luminance values around the same coordinate on the image before and after conversion with a predetermined threshold value.   2. The similarity calculation means is performed by comparing an even power of luminance values around the same coordinates on an image before and after conversion or an integrated value of absolute values with a predetermined threshold value. The image processing apparatus described.   6. The predetermined area calculated by the similarity calculation means is limited to a closed area formed by corresponding points used when the conversion matrix is used by the conversion matrix calculation means. The image processing apparatus described. Image acquisition means;
Means for detecting a region in the image acquired by the image acquisition means;
Means for determining whether or not the detected region is a plane;
If it is determined that the region in the image is not a plane, means for further determining whether or not the small region in the region is a plane;
An image processing apparatus comprising: Acquire multiple images,
Detect corresponding points, which are feature points associated between acquired images,
Among the corresponding points, for at least three points on the same plane, a conversion matrix for converting from a pixel position on one image to a pixel position on the other image is obtained.
Generate a new image from one image by pixel position conversion based on the obtained conversion matrix,
Calculate the similarity for a given area between the generated image and the other image,
An image processing method comprising: determining whether or not the predetermined region is a point on a plane formed by corresponding points used when obtaining a transformation matrix based on the obtained similarity .   9. The image processing method according to claim 8, wherein the corresponding points used when obtaining the transformation matrix are selected from points whose distance in the image or three-dimensional space is a predetermined value or less.   9. The image processing method according to claim 8, wherein the image conversion is performed by obtaining a luminance value at a pixel position after conversion by interpolation.   9. The image processing method according to claim 8, wherein similarity calculation is performed by comparing a difference in luminance value around the same coordinate on the image before and after conversion with a predetermined threshold value.   9. The image according to claim 8, wherein the similarity is calculated by comparing an even power of luminance values around the same coordinates on the image before and after conversion, or an absolute value integrated value with a predetermined threshold value. Processing method.   The image processing method according to claim 11 or 12, wherein the predetermined area for calculating the similarity is limited to a closed area formed by a corresponding point used when the transformation matrix is used. Get an image,
Detect areas in the acquired image,
Determine whether the detected area is a plane,
An image processing method comprising the steps of further determining whether or not a small area in the area is a plane when it is determined that the area in the image is not a plane.

Images