JP2002197472A - Method for recognizing object - Google Patents

Abstract

PROBLEM TO BE SOLVED: To recognize a three-dimensional object appearing in a two-dimensional image and the posture estimation of the three-dimensional object under a perspective projection camera model causing no distortion, and also to perform processing with a small amount of calculation. SOLUTION: In this object recognizing method, an input image is collated with an object model, by collating an image edge in the input image with the model edge of the object model by one at a time, for each posture of the object model to a camera. In such a case, the object posture is divided into a rotating component and a parallel moving component, it is first checked whether a projected image to the image plane of the model edge and the image edge exist on the same straight line with respect to the discrete value of the turning component, and a candidate for rotating component value and edge correspondence is narrowed down. Next, a distribution of parallel moving component values fitting each obtained candidate is calculated, and the parallel moving component value of the highest frequency is calculated. Then that parallel moving component value, and the turning component value and the edge correspondence at that time are defined as being the solutions.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an object recognition method using a computer, and more particularly to a method for recognizing an object in an input image by using an object model registered in advance.
The present invention relates to a method of estimating a posture of the object with respect to a camera.

[0002]

2. Description of the Related Art An alignment method is known as a method for recognizing an object in a two-dimensional image using a three-dimensional object model (Reference: DP Huttenlocher an.
dS. Ullman, "Recognizing S
old Objects by Alignment
with an Image ", Internati
onal Journal of Computer
Vision, Vol. 5, No. 2, pp. 195-
212, 1990). In the alignment method, a camera model is approximated by weak perspective projection, and a transformation matrix for projecting a three-dimensional object onto a two-dimensional image plane is defined. Then, object recognition is formulated as a problem for obtaining the transformation matrix from the feature point set of the object model and the feature point set on the image. Specifically, a transformation matrix is obtained when three feature points on the object model are associated with three feature points on the image, and the result of projecting other model feature points on the image plane using the transformation matrix is different. Find out if the image feature points match well. Perform this process for each combination of model feature points and image feature points,
A transformation matrix having a high degree of coincidence is used as a solution. As the feature points, corners, intersections, inflection points, and the like of edges extracted from the image are used.

[0003]

Normally, perspective projection is used for accurate modeling of a camera. In perspective projection, an object appears small on an image in inverse proportion to depth. But,
In the alignment method, since the projection is approximated by weak perspective projection, when the depth of the object is long, the distortion of the projected image becomes large, and there is a problem that the image cannot be correctly recognized.

In the alignment method, if the number of model feature points is M and the number of image feature points is N, the amount of calculation is M
There is a problem that the calculation time becomes enormous as the number of feature points increases in proportion to ³ N ² logN.

[0005] In addition to the alignment method, three-dimensional object recognition has a problem that a search space is large and a calculation amount is enormous. This is because the problem of object recognition needs to simultaneously solve the problem of associating model features with image features and the problem of finding an object posture.

[0006]

In order to solve the above problems, in the object recognition method of the present invention, a two-dimensional edge (referred to as an image edge) in an input image and a three-dimensional edge of an object model (a model edge and a model edge) are used. ) Are collated one by one to collate the input image with the object model. This collation is performed for each posture of the object, and the object posture is divided into a rotation component and a translation component in a stepwise manner.

According to a first aspect of the present invention, a rotation component of the posture of the object model in a coordinate system of a camera which has taken the input image is discretized, and each discrete value of the rotation component constitutes the object model. It is checked whether the projected image of the model edge on the image plane and each image edge of the image edge set can be on the same straight line, and a set of a model edge and an image edge pair that can be on the same straight line is determined as an edge pair set. When the number of model edges included in the edge pair set exceeds a predetermined threshold, for each edge pair included in the edge pair set, the image plane of the model edge included in the edge pair is transferred to the image plane. A translation component value of the posture of the object model in which both end points of the projected image of the object model coincide with both end points of the image edge included in the edge pair, and the distribution of the translation component value with respect to the edge pair set A representative value of the region Oite concentration degree is greater as the estimated value of the translation component, the estimated value of the rotation component value and the parallel movement component and a candidate object position.

According to a second aspect of the present invention, for each of the object posture candidates, a degree of coincidence of the model edge is calculated from a distance between the image edge and a projection image of each model edge on the image plane in the object posture candidate, The image edge that best matches is selected as the image edge corresponding to the model edge, and the candidate object posture with the best sum of the degrees of coincidence of all model edges is selected as a solution for posture estimation.

[0009]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The object recognition method of the present invention uses a three-dimensional object model composed of a polyhedron to recognize an object shown in one two-dimensional image and to estimate an object posture with respect to a camera. It is. Matching the input image with the object model
This is performed by checking the image edge and the model edge one by one. Since the object model is a polyhedron, all model edges are straight lines. The target two-dimensional image is a digital image captured by a computer, and an image edge is obtained by an edge extraction process widely performed in the field of image processing. It is also assumed that all image edges are divided into straight lines using high curvature points and intersections as division points. FIG.
The example of the image edge of a desk is shown below. However, the actual image includes many image edges other than the target object. FIG.
The following shows an example of a desk object model. The object model may represent only a characteristic part of the object, and in this example,
Only the front part of the desk that can be seen from a normal viewpoint is modeled.

Next, the principle of the object recognition method of the present invention will be described. First, a camera model based on perspective projection is formulated as follows. FIG. 3 shows an example of the posture relationship between the object model and the camera. It is assumed that the shape of the object model is defined in the local coordinate system of the object model. The coordinate transformation parameters from the object model coordinate system to the camera coordinate system are τ = <τ _r , τ _t
>. τ _r = (θ, φ, ψ) is a rotation component, and τ _t =
(X ^t , y ^t , z ^t ) ^T is the translation vector (T
Represents transposition). At this time, the value ^Pc of the point P on the object model in the camera coordinate system is as shown in Expression 1. Where R
(Τ _r ) is a rotation matrix based on τ _r . Moreover, the point of the camera coordinate system ^{P c = (x c, y} c, z c) projected point ^{P s} to the screen coordinate system of ^T is as Equation 2.

[0011]

## EQU1 ## P ^c = R (τ _r ) P + τ _t

[0012]

(Equation 2)

[0013] represents a set of image edge L, and a set of model edge E, the model edge e∈E by the attitude tau (referred to as the projection edge) projected edges in a two-dimensional image and a e ^s or e ^{s (tau)} . The problem of recognizing an object in an input image is represented by a posture τ satisfying Expression 3, and a mapping m from E to L =
It is formulated as a problem to obtain {(e, l) | e {E, l {L}}. In Equation 3, D is a distance between two line segments. For example, D is defined as a sum of Euclidean distances between both end points of the two line segments. If so, another distance scale may be used. Also,
There may be model edges that do not have corresponding image edges.

[0014]

(Equation 3)

In S, the entire object posture τ must be searched, and the amount of calculation becomes enormous. Therefore, the search space of S is divided and the solution is obtained in a stepwise manner, thereby reducing the amount of calculation. Specifically, first, the corresponding edges are narrowed down according to whether or not the straight line equations of the edges can match according to Equation 4, and then the corresponding edges are determined according to Equation 5 by matching the end points. This is the absolute value of the difference between the inclinations of the two edges when the point Q coincides, and is defined by Equation 6. The slope (x) in Equation 6 is the slope of the edge x. Whether linear equations of the projected edge and the image edge may coincide with D ₁ is determined.

[0016]

(Equation 4)

[0017]

(Equation 5)

[0018]

(Equation 6)

[0019] S ₁ is the set of pairs of model edge e and image edges l linearly equation can match. S _2, of the edge correspondence candidates obtained in S _1, the set of those end points of the two edges are matched, S 2 _{= S} holds. Thus, in order to determine the S, and first obtains the _{S 1,} then _{S 2}
Should be obtained.

Next, a specific calculation method of S ₁ and S ₂ and a reduction in the search space at that time will be described. First, S ₁ depends only on the rotation component τ _r and not on τ _t . This It is only necessary to guide that it is immutable. This proof is shown below.

[0021] The inclination of the projection edge e ^s in attitude τ is as Equation 8. Here, from Equation 1, It does not depend. U and v are determined by the image. Therefore, the slope (e ^s ) in Equation 8 is invariant to τ _t .
(Proof end) Incidentally, S ₁ does not change the Q for any point on the image edge l.

[0022]

(Equation 7)

[0023]

(Equation 8)

By [0024] As described above, since it is not necessary to consider the S ₁ τ _t, the search space is reduced become the only τ _r. Therefore, in order to obtain the _{S 1,} tau _r discretized with an appropriate compartments, for each of its discrete values _{Σ (e, l) ∈m D} 1 (e s
An edge correspondence m that satisfies (τ), l) = 0 is obtained. τ _r
Can be covered by a finite number of sections because each angle is within 0 to 360 degrees.

S ₂ is, for each τ _r obtained in S ₁ ,
A translation component τ with a high degree of coincidence between the projected edge and the image edge
_{It is obtained} by calculating _t . The translation component τ _t at which both end points of the image edge and the projection edge coincide with each other is expressed by Expression 9 and Expression 1
0 can be calculated. Here, P ₁ and P ₂ are end points of the model edge, and Q ₁ = (u ₁ , v ₁ ) ^T , Q ₂ = (u ₂ , v ₂ )
^T is the end point of the image edge.

[0026]

Τ _t = FR (τ _r ) (P ₂ −P ₁ ) −R (τ _r ) P ₁

[0027]

(Equation 10)

The translation component τ _t having a high degree of coincidence between the projected edge and the image edge is obtained as follows. First, for each τ _r obtained in S ₁ , for each edge pair (e, l) included in the edge correspondence m at that τ _r , τ _t
To obtain the distribution of τ _t . Then, the most frequent _t in the distribution is selected. Actually, τ _t is not concentrated on one point due to an error or the like, but is dispersed over a certain range, so that the most frequent τ _t is obtained by a method such as voting or clustering. This means that the selected edge pair largest number tau _t of satisfying D = 0 to approximately as a solution of S _2. Finally, τ _t and τ _r and m at that time are paired and set as a solution candidate.

The above method is based on the premise that the end points of the image edges have been accurately extracted. If the image edge can be completely extracted, the end point may be adopted as it is and collated with the end point of the model edge. However, in an actual image, edges may not be extracted properly due to lighting conditions or contrast, or edges may be hidden due to the overlapping of other objects, and the end points of the image edges may not be completely extracted. In this case, the above method is applied by using the intersection of two image edges as the end point candidates of each image edge. The intersection of the image edges referred to here is the intersection of a straight line extending the image edge. Since the intersection of the image edges can be obtained if a straight line portion can be extracted to some extent, even if the image edge is interrupted, the above-mentioned method can stably obtain the end point candidates of the image edge.

An embodiment of the object recognition method according to the present invention will be described below with reference to the drawings. FIG. 1 is a flowchart showing one embodiment of the present invention. Steps S1 to S7
Is the scope of claim 1, and steps S1 to S11 are the scope of claim 2. First, in step S1, the rotation component τ _r of the object posture is discretized in appropriate sections. As a discretization method, for example, there is a method in which τ _r is represented by an Euler angle, and each of the three angles is discretized. Alternatively, surround the object model with a sphere, divide the sphere into polygons,
There is also a method using a set of an azimuth angle of the vector from the center of the sphere to the center of the polygon (which becomes two discretized angles) and a discrete value of a rotation angle around the vector. Note that the range of τ _r may be limited to the range of postures that the object can take with respect to the camera.

Next, the steps S2 and S3, for each discrete value of tau _r, repeats steps S4 to S7. In step S4, at the current τ _r , for each model edge e, the projected edge e ^s (τ) exists on the same straight line as one of the image edges l. Find out if it will. th ₁ is a threshold, and D ₁ (e
^{s (τ),} l) <if ^{_{th 1, D 1 (e s}} (τ),
l) = 0 is assumed to hold. This is because D ₁ ( ^es (τ), 1) is rarely actually 0 due to various errors. As shown in paragraph 0021, D
Τ _t is not required for the calculation of ₁ . Then, a pair (edge pair) of a model edge and an image edge that can exist on the same straight line is registered in an edge pair set M.

Next, in step S5, it is checked whether or not the number of model edges included in the edge pair set M has exceeded a predetermined threshold value. If it has, the process proceeds to step S6. If not, the process returns to step S2. The following τ _r is calculated.

In step S6, each pair of the model edge and the image edge included in the edge pair set M obtained in step S4 is translated by using equations (9) and (10) based on their both end points. Calculate the component τ _t .

Next, in step S7, at the current τ _r , an area where τ _t is concentrated is determined based on the distribution of the translation component τ _{t with} respect to all the edge pairs of the edge pair set M, and a high degree of concentration is obtained. The representative value of the region is set as a solution candidate. As a method of obtaining the area to be concentrated, for example, a voting (votin
g) and clustering. In the voting method, the range of τ _t is appropriately discretized to obtain τ
Create a histogram of _t , τ _t with large frequency (number of votes)
Is a solution candidate. In the process according to clustering by examining the number of other tau _t in the neighborhood of the tau _t, the number of large tau _t and candidate solutions. The obtained [tau] _t and the current [tau] _r are paired and set as a candidate for the object posture.

Next, in steps S8 and S9, step S10 is repeated for each object posture candidate obtained in step S7. In step S10,
Current posture candidate <τ _r, τ _t> each edge pair in the original, to calculate the degree of matching of the model edge. Then, the sum of the degrees of coincidence is set as the degree of coincidence of the object model in the posture candidate.

The coincidence of the model edges is calculated, for example, as follows. That is, in the image edge forming an edge pair with the model edge e, the D
(E ^{s, l)} and is an image edges corresponding to l that minimizes the e, its time of D (e ^{s, l)} the value of the matching degree e. In this case, the smaller the degree of coincidence, the better the coincidence. At this time, when there is no image edge corresponding to the model edge e, by giving an appropriate penalty point to e, processing is added to prevent an object posture having no model edge corresponding to the image edge from being selected as a solution. Is also good.

Finally, in step S11, the object posture <τ _r , τ _t > with the highest degree of coincidence between the object models is selected as a solution for posture estimation. In that position, D
(E ^{s, l)} a solution of object recognition the set of edge pair that minimizes the.

FIG. 2 is a block diagram showing an example of a system configuration for executing the object recognition method of the present invention. The range surrounded by the dotted line in FIG. 2 is a portion for executing the object recognition method of the present invention. First, the edge extracting unit 1 extracts an image edge from an input image, and passes the result to the edge dividing unit 2. The extraction of the image edge is performed by image processing such as, for example, differentiating the image and tracking its extreme value. The edge dividing unit 2 divides an image edge into straight lines at a high curvature point, an intersection, or the like, and divides the result into an end point detecting unit 3, a posture calculating unit 4,
The information is passed to the edge matching unit 5. The end point detection unit 3 obtains the end points of the image edges divided into straight lines, and passes the result to the posture calculation unit 4 and the edge comparison unit 5. As described in paragraph 0029, the end point of the image edge includes a method of using the end point of the straight line as it is and a method of obtaining the end point of the image edge as an intersection of the image edge. The end point detection unit 3 implements either one or both, and allows the user to select one. Next, the posture calculation unit 4 refers to the object model in the object model storage unit 6 by the processing shown in steps S1 to S7 in FIG.
The model edge and the image edge are collated to obtain candidates for the object posture, and the candidates are passed to the edge collation unit 5. The edge matching unit 5 performs step S for each of the obtained candidates for the object posture.
Through the processing shown in steps S8 to S11, the object posture in which the model edge and the image edge best match with each other is determined with reference to the object model in the object model storage unit 6, and the edge correspondence at that time is determined.

FIG. 6 is an explanatory diagram showing an example of the configuration of the object model. The object model is a polyhedron, and includes vertex information, edge information, and edge connection information. The vertex information is a set of coordinate values in a three-dimensional space of the vertices forming the polyhedron. As the coordinate system, it is convenient to set a local coordinate system for each object and to set the coordinate values within the coordinate system. For example, in FIG. 6, the coordinate value of the vertex p1 is (100, 200, 0). The side information is a set of sets of vertices that are end points of the sides of the polyhedron. The side corresponds to the model edge. For example, FIG.
Where the side e1 is defined as a line segment connecting the vertices p1 and p2.

The side connection information specifies another side for obtaining an end point of the side as an intersection. For example, in FIG.
Is an intersection with the side e5 and the side e6, and the other end is an intersection with the side e2 and the side e4. The edge connection information is used as follows when calculating an end point of an image edge as an intersection with another image edge in the calculation of the translation component τ _t or the edge matching. Now, it is assumed that the connection information of the side e1 includes the side e2. Then, on the model, one end point of the side e1 is an intersection with the side e2, so that the end point of the image edge corresponding to the side e1 on the image is the intersection with any one of the image edges forming an edge pair with the side e2. Should be. Therefore, instead of calculating the tau _t for all image edges, to calculate the tau _t Search in image edge forming the edge e2 and edge pairs.

Next, the calculation amount of the object recognition method of the present invention will be described. First, in step S4, since the calculation is performed once for every pair of the model edge and the image edge, the calculation amount is proportional to MN. Here, M is the number of model edges, and N is the number of image edges. Computational steps S6~S7 varies depending Determination of the end point of the image edges in the calculation of the parallel movement component tau _t. When the end point of the image edge is used as it is, one pair is set for all pairs of the model edge and the image edge having the projection edge which can exist on the same straight line.
Since the calculation is performed each time, the calculation amount is proportional to MN '. N 'is the model You. Next, when the intersection of the image edges is used as an end point, each of the N ′ image edges is averagely combined with another image edge to obtain both end points of (N′−1) ^2. since there is only, the amount of calculation is proportional to the MN ^'3. The calculation amount in step S10 is also determined in steps S6 to S7.
Is the same as the calculation amount of

As described above, the calculation amount of the present invention is M (k ₁ N + k ₂ N ′ + k ₃ N ′) when the end point of the image edge is used as it is in the calculation of the translation component τ _t . k ₁ ,
k ₂ and k ₃ are proportional constants, and include the number of discretized sections of the rotation component τ _r . When an intersection of image edges is used as an end point, M (k ₁ N + k ₂ N ′ ³ + k ₃ N ′ ³ )
Becomes

[0043]

As described above, according to the object recognition method of the present invention, since the camera is modeled by perspective projection, the object is compared with the method of approximation by weak perspective projection described in the prior art. There is an effect that correct recognition can be performed even when the depth is long.

As described above, the calculation amount of the object recognition method of the present invention is M (k ₁ N + k ₂ N ′ + Each of the computational amounts is the computational amount M ³ N ² logN of the prior art.
Smaller. Therefore, there is an effect that the calculation time is shorter than in the conventional technique. This becomes remarkable when M and N are large.

[Brief description of the drawings]

FIG. 1 is a flowchart showing a processing procedure of an object recognition method of the present invention.

FIG. 2 is a block diagram showing the configuration of a system that executes the object recognition method of the present invention.

FIG. 3 is an explanatory diagram showing a posture relationship between a camera and an object model.

FIG. 4 is an explanatory diagram illustrating an example of an edge image of an object in an image.

FIG. 5 is an explanatory diagram showing an example of an object model.

FIG. 6 is an explanatory diagram showing an example of data representation of an object model.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Edge extraction part, 2 ... Edge division part, 3 ... Endpoint detection part, 4 ... Attitude calculation part, 5 ... Edge collation part, 6 ... Object model storage part, 7 ... Camera, 8 ... Camera coordinate system, 9 ... Image Plane, 10: screen coordinate system, 11: object model, 12
... object coordinate system. S1 to S11 are steps in the processing procedure.

Claims (2)

[Claims] 1. A method for recognizing an object appearing in an input image by comparing a set of image edges extracted from the input image with an object model registered in advance, the coordinate system of a camera which has taken the input image , The rotation component of the posture of the object model is discretized, and for each discrete value of the rotation component, a projected image of each model edge constituting the object model on an image plane and each image edge of the image edge set are It is checked whether or not the model edge can exist on the same straight line, and a set of a pair of the model edge and the image edge which can exist on the same straight line is obtained as an edge pair set. Then, for each edge pair included in the edge pair set, both end points of the projected image of the model edge included in the edge pair onto the image plane and the image edge included in the edge pair. Determine the translation component value of the orientation of the object model that matches the two end points of the edge, the representative value of the region where the degree of concentration is large in the distribution of the translation component value for the edge pair set as the estimated value of the translation component, An object recognition method, wherein the rotation component value and the estimated value of the translation component are set as candidates for an object posture. 2. For each of the object posture candidates, a degree of coincidence of the model edge is calculated from a distance between the image edge and a projection image of each model edge on the image plane in the object posture candidate. 2. The object recognition method according to claim 1, wherein an edge is selected as an image edge corresponding to the model edge, and an object posture candidate having the best sum of matching degrees of all model edges is selected as a solution for posture estimation.