JP4379459B2 - Object collation method, object collation apparatus, and recording medium recording the program - Google Patents

Description

  The present invention relates to an object collation method, an object collation apparatus, and a recording medium storing the program for collating an object with an image, and in particular, the three-dimensional shape of an object to be collated and the reflectance and color information of the object surface. The present invention relates to an object collation method, an object collation apparatus, and a recording medium on which the program is recorded that is robust against fluctuations in photographing conditions such as the position and orientation of an object on an image and illumination conditions by being registered in advance. Hereinafter, an object registered in advance is referred to as a registered object, and an object that is a target for collation with the registered object is referred to as a target object.

  As shown in FIG. 6, the object matching technique registers what a target object is appropriately arranged in a three-dimensional space using an input image (group) acquired by an image capturing device such as a camera. This is a technology that matches objects. The object collation process is composed of two processes: a registration process in which registered objects are registered in advance, and a collation process in which what is captured in the input image (group) of the target object is compared with the registered object. Has been. In each process, the captured image is used as it is as a two-dimensional image having a two-dimensional extent, or a technique described in Reference 1 (“three-dimensional image measurement”, Iguchi, Sato, Shokodo). For example, it is converted into a three-dimensional shape and used. Conventional object matching techniques are described below with reference to the literature.

・ Prior art 1
As an example of an object matching technique in which a two-dimensional image is registered and the two-dimensional image is used as an input, there is a technique disclosed in Document 2 (Japanese Patent No. 2872776 “Face Image Matching Device”). This technique assumes a human face as an object to be collated, and has a configuration as shown in FIG. At the time of registration, a two-dimensional image taken by the camera 11 is stored in the storage unit 12. At the time of collation, the camera 13 captures a two-dimensional face image as an input image, and the normalizing means 14 performs image processing on face feature points that serve as reference for posture and size such as eye and nose positions from the input image. The image is extracted by technology, and a normalized image obtained by normalizing the two-dimensional position and size on the image with reference to the coordinate position of the feature point is output. Finally, the image comparison unit 15 compares the registered image read from the storage unit 12 with the normalized image using a pattern recognition technique, and outputs a collation result.

・ Prior art 2
Further, as an example of a conventional object matching technique using a three-dimensional shape, there is a technique disclosed in Document 3 (Japanese Patent Laid-Open No. 9-259271 “Person matching device”). This technique takes a configuration as shown in FIG. At the time of registration, the three-dimensional shape color information measuring unit 21 measures the three-dimensional shape and color information of the registered object as face data and stores them in the storage unit 22. Also at the time of collation, the three-dimensional shape color information measuring unit 23 measures the three-dimensional shape and color information of the target object as input data, and the parallel movement / rotation unit 24 uses the input data so that the center of gravity coincides with the face data. A rotational face data group that is translated and added with a slight rotation is generated, the minimum error calculation means 25 obtains the minimum value of the error to correct the three-dimensional position and orientation, and collation is performed based on the minimum error. Do.

・ Prior art 3
Reference 4 (Japanese Unexamined Patent Publication No. Hei 6-168317 “Personal Identification Device”) discloses a verification technique for capturing a two-dimensional image both during registration and during verification. This technique takes a configuration as shown in FIG. At the time of registration, a two-dimensional image of a registered object is photographed by the camera 31, a pixel position having a large luminance variation is detected by the feature extraction unit 32, and a feature point position is output and stored in the storage unit 33. At the time of collation, the camera 34 captures a two-dimensional image of the target object as an input image, the feature extraction unit 35 detects a pixel position with a large luminance fluctuation, outputs a feature point position, and the registration unit 37 registers the position. The comparison is performed by comparing the feature point position with the feature point position of the input image. At this time, in order to absorb fluctuations in the position and orientation of the target object, the position and orientation normalization means 36 prepares a standard three-dimensional shape model of the object in advance and uses the standard three-dimensional shape model. Normalize the position and orientation.

・ Conventional technology 4
Reference 5 (“Visual Learning and Recognition of 3” is a technique that uses only ordinary two-dimensional images in both the registration process and the verification process in order to correct not only the position and orientation fluctuations but also the fluctuations according to the illumination conditions. -D Objects from Appearance ”, Hiroshi Murase and Shree K. Nayer, Int. J. Computer Vision, vol. 14, pp. 5-24, 1995). This technique takes a configuration as shown in FIG. At the time of registration, a two-dimensional image group covering all possible postures and illumination conditions in the input image of the target object for the registered object is photographed by the photographing means 41, and the change of the two-dimensional image group can be sufficiently expressed by the manifold calculation means 42. Such a basis vector group is obtained by principal component analysis, a feature space characterized by correlation with the basis vector group is generated, a trajectory in the feature space of the two-dimensional image group is obtained as a manifold, and stored in the storage means 43. Keep it. At the time of collation, a two-dimensional image of the target object is taken as an input image by the camera 44, the distance calculation means 45 calculates the distance in the feature space between the input image and the manifold, and collation is performed using the distance as a scale. Thereby, it is possible to collate input images taken under various positions and orientations and illumination conditions.

・ Prior art 5
For the change of the two-dimensional image depending on the illumination condition when the position and orientation of the object are fixed, see Reference 6 (“What Is the Set of Images of an Object Under All Possible Illumination Conditions?”, Peter N. Belhumeur and David J. Kriegman, Int. J. Computer Vision, vol. 28, pp. 245-260, 1998). If the position and orientation of the object are fixed, an image under an arbitrary illumination condition can be decomposed into a sum of images under one point light source. Therefore, an image under an arbitrary number of light sources can be represented by a linear sum of images under the one light source, with the intensity of each light source as a coefficient.

  Based on the above analysis, Document 6 proposes a method called the Illumination Subspace Method (hereinafter referred to as Method 1) having a configuration as shown in FIG. In the photographing means 51, three or more different illumination conditions are set so that there are as few shadow pixels as possible, and a two-dimensional image group of the registered object is photographed. In the normal calculation means 52, a vector group corresponding to the product of the reflectance and normal vector of the object surface corresponding to each pixel of the image is obtained from the two-dimensional image group by principal component analysis. Subsequently, the image generation unit 53 generates an image group called extreme ray, which is an image when there is illumination in the direction represented by the outer product of any two vectors of the vector group, and stores it in the storage unit 54. To do. At the time of collation, the camera 55 captures a two-dimensional image of the target object as an input image. When the reflection characteristic of the object surface is completely scattered and the shape is convex, an image under an arbitrary illumination condition can be expressed as a linear sum in which the coefficient of the extreme ray group is positive. Can be calculated using the least squares method under the condition that is not negative. The illumination correction means 56 performs the least square calculation, and generates a comparison image of the target object under the same illumination conditions as the input image by the linear sum of the extreme ray group using the obtained coefficient group. In the image comparison means 57, collation processing is performed by calculating the similarity between the comparison image and the input image.

・ Prior art 6
Reference 7 ("Illumination Cones for Recognition Under Variable Lighting: Faces", AS Georghiades, Proc. IEEE Int. Conf. CVPR, pp.52-58, 1998), ray tracing when calculating extreme ray in Method 1. A method of calculating which pixel becomes a shadow from a three-dimensional shape of an object using a computer graphics technique such as the above and performing a process of adding a shadow (hereinafter referred to as method 2) is shown. As a result, the method 1 can be applied to an object having a non-convex shape.

・ Prior art 7
The document 7 also proposes a method having a configuration as shown in FIG. 27 as a sampling method (hereinafter referred to as method 3). Since it is time-consuming to calculate all the extreme rays as in the method 1, at the time of registration, for example, in the photographing means 61, it is appropriate that the angles of θ and φ in FIG. A number of illumination directions are set to photograph a two-dimensional image group, and the two-dimensional image group is used as extreme ray. Thereafter, similarly to the method 1, the non-negative least square method is applied to perform illumination correction and perform object recognition.

・ Prior art 8
Document 8 (Japanese Patent Application No. 2000-105399 “Image collation apparatus, image collation method, and recording medium on which the program is recorded”) registers the three-dimensional shape of the registered object, and the position and orientation of the input image of the target object. Even if the position and orientation are changed, a group of illumination fluctuation textures according to the position and orientation is generated by computer graphics, and an image space including the image group is obtained and illumination correction processing is performed to obtain the position and orientation and Even when the lighting conditions change together, highly accurate verification is possible.
Japanese Patent No. 2872776 Japanese Patent Laid-Open No. 9-259271 JP-A-6-168317 Japanese Patent Application 2000-105399 3D image measurement ", Iguchi, Sato, Shokodo Visual Learning and Recognition of 3-D Objects from Appearance ”, Hiroshi Murase and Shree K. Nayer, Int. J. Computer Vision, vol.14, pp.5-24,1995 What Is the Set of Images of an Object Under All Possible Illumination Conditions? '', Peter N. Belhumeur and David J. Kriegman, Int. J. Computer Vision, vol.28, pp.245-260,1998 Illumination Cones for Recognition Under Variable Lighting: Faces '', AS Georghiades, Proc.IEEE Int. Conf.CVPR, pp.52-58,1998

  However, an object to be collated is generally accompanied by a three-dimensional parallel movement, a rotational movement, etc. in front of an image capturing device such as a camera unless fixed or adjusted. In addition, as apparent from the fact that the illumination conditions change every moment, such as outdoors, on the two-dimensional image input as the object to be collated, there appears to be a very large change. The prior art has a problem in that the application range is very limited because the variation of the position and orientation and the illumination conditions cannot be sufficiently corrected. Hereinafter, specific problems in the technology described in each document will be described in detail.

  In a technique for simply comparing two-dimensional images as described in Document 2, an apparent appearance on a two-dimensional image due to three-dimensional rotation variation of an object to be collated or illumination condition variation at the time of image capturing. It cannot cope with fluctuations, and its application range is extremely limited.

  The collation technique of Document 3 requires a three-dimensional shape not only at the time of registration but also at the time of collation. Therefore, a three-dimensional shape measuring device as shown in Document 1 is indispensable for the collation device, which is expensive. There was a problem. This is a particular problem when a location different from that at the time of registration or when it is desired to capture and collate input images at a plurality of locations. In addition, in order to measure the shape, there is a problem that the object to be collated must be stationary until the measurement is completed, or accurate shape data cannot be obtained unless it is in a dark room or dim environment. After all, the application range is limited.

  The method of detecting a pixel position with a large luminance variation as shown in Reference 4 is suitable for a building block with a very large three-dimensional curvature, or a black marker on a white plate with a very large reflectance variation. Although it is effective, it is known that it is not suitable for a human face as mentioned in the document 4, so that stable coordinate position detection is generally difficult. In Reference 4, the posture is corrected by the standard three-dimensional shape of the object group to be collated. However, this cannot be applied when the degree of similarity between the objects in the object group is not high. was there.

  In the technique of Reference 5, when various illumination conditions such as a plurality of light sources and extended light sources are considered as illumination conditions for the input image of the target object, a large amount of sample images of registered objects that cover these are required. . In addition, since there is no assumption about the shape of the manifold in the feature space, a search for the parameters of the shooting conditions is required when obtaining the distance from the input image. Therefore, there is a problem that a large amount of calculation is required.

  In the technique of method 1, the procedure for calculating extreme ray according to the complexity of the shape requires a large amount of calculation. According to Document 6, when there are m linearly independent normal vectors on the object surface, the maximum number of extreme rays is m (m−1). Therefore, unless the object shape is as simple as a building block, a huge number of images must be calculated, so calculating all extreme rays for a general object with a complex shape is a calculation There is a problem in terms of quantity. Further, when the object shape is not convex and there is a shadow caused by the other part blocking the light source, it cannot be applied as it is.

  In both methods 1 and 2, when the position and orientation of an object change, it is necessary to take an image of the object in accordance with the position and orientation and recalculate all extreme rays. In particular, in the technique of method 2, a calculation process for shadowing an object image is performed at the time of extreme ray calculation, but this process requires a large amount of calculation such as ray tracing. The problem is that it takes time.

  In the technique of the method 3, it is necessary to take an image in which an object to be collated is illuminated from many directions, and a special illumination device is required at the time of registration.

  In addition, when the position or orientation of the object to be collated changes, it is necessary to re-capture an image of the object in accordance with the position and orientation under a number of illumination conditions. Therefore, since it is necessary to capture images at every possible position and orientation in the input image, it takes time for registration processing, and images captured at positions and orientations that are not registered in advance cannot be collated. There are problems such as.

  In the technique of Literature 8, even if the position and orientation of the target object in the input image change, it is possible to generate an image in which the position and orientation are matched by the three-dimensional shape of the registered object. You don't have to shoot the group at registration. However, the process of generating an image under a large number of illumination conditions that match the position and orientation of the input image requires a very large amount of calculation such as a shadowing process. there were.

  The present invention has been made in view of the above problems, and does not require a three-dimensional shape as input data used for collation, and can perform collation using a two-dimensional image captured by a normal camera. It is an object of the present invention to provide a device and a recording medium recording the program.

  In addition, the present invention can correct a change in the three-dimensional position and orientation of a target object in an input image, can easily measure necessary data of a registered object at the time of registration, and is photographed under various illumination conditions. An object of the present invention is to provide an object collation method, an object collation apparatus, and a recording medium on which the program is recorded, which can realize correction of illumination conditions for the input image by high-speed processing.

  In order to achieve the above object, the object matching method of the present invention is based on a basis for expressing a texture space spanned by a texture group representing luminance and color information at each position on the surface of a registered object under various illumination conditions. A step of deforming a vector according to the position and orientation of the target object in the input image using the three-dimensional shape of the registered object, the closest image in the illumination variation space represented by the deformed basis vector, and the input Determining whether the target object is a registered object based on a distance from the image.

  In order to achieve the above object, the object matching apparatus according to the present invention provides a basis for expressing a texture space spanned by a texture group representing luminance and color information at each position on the surface of a registered object under various illumination conditions. Means for deforming a vector according to the position and orientation of the target object in the input image using the three-dimensional shape of the registered object, the closest image in the illumination variation space represented by the deformed basis vector, and the input Means for determining whether the target object is a registered object based on a distance from the image.

  In order to achieve the above object, the object matching program of the present invention provides a computer with a texture space spanned by texture groups representing luminance and color information of each position on the surface of a registered object under various lighting conditions. A process of transforming a base vector to be represented according to the position and orientation of the target object in the input image using the three-dimensional shape of the registered object, and the closest image in the illumination variation space represented by the deformed base vector And a process of determining whether the target object is a registered object based on the distance between the input image and the input image.

  As is clear from the above description, according to the present invention, only the registration unit needs to measure the three-dimensional shape of the registered object and the reflectance of the object surface or an image under appropriate illumination conditions. As the photographing means, it is sufficient to have an image pickup device such as a video camera for picking up a normal two-dimensional image, and a practical device can be configured without the need for a three-dimensional shape measuring device at the collation stage.

  Further, since the three-dimensional shape of the registered object is registered, it is possible to completely correct the three-dimensional position / posture variation of the target object in the input image. Furthermore, sufficient correction can be performed for variations in illumination conditions.

  Furthermore, the illumination correction process of the present invention generates the illumination variation space by converting the illumination variation texture space that has been calculated in advance in the registration process according to the position and orientation of the target object in the input image. However, it is not necessary to regenerate the illumination variation texture group, and the collation determination can be performed by a process faster than the conventional methods 1 and 3.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  (1) First Embodiment The object collation method according to the first embodiment of the present invention is executed by the object collation apparatus of the first embodiment shown in FIG. The process of registering the image in the database stores the three-dimensional shape of the object and the texture space, which is an image space spanned by texture groups representing the brightness and color information of each position on the object surface under various lighting conditions A matching process for matching an input image with registered data, a step of photographing a two-dimensional image as an input image, and a position and orientation of a target object in the input image Then, a step of generating an illumination variation space, which is an image space spanned by an image group under various illumination conditions, from the three-dimensional shape and the texture space, and a distance between the illumination variation space and the input image. And a process for performing processing such as checking whether the object is a target to be collated, searching for which of registered objects is similar, searching for similar objects of registered objects, and the like. It is.

  In the object collation method of the invention according to the first embodiment, the texture representing the brightness or color of each position on the object surface to be collated changes depending on the illumination condition, but various texture groups generated by these illumination fluctuations By registering the space (texture space) spanned by the object and the three-dimensional shape of the registered object, the texture space is generated by the fluctuation of illumination conditions when the target object is in the required position and orientation. It is possible to convert to an illumination variation space, which can generate an image of the target object in the position and orientation under various illumination conditions, and perform high-precision collation by performing illumination correction processing Is used.

  Furthermore, according to the object collation method of the invention according to the first embodiment, the illumination variation space at the target position and orientation of the target object can be generated only by simple coordinate transformation of the texture space of the registered object. Therefore, the amount of calculation in the matching process can be greatly reduced, and high-speed object matching processing can be performed.

  First, the registration process will be described.

  In the registered object measuring step, the three-dimensional shape of the registered object and the reflectance of the object surface are measured. For this, various measuring devices and methods can be used. As an example, for example, a three-dimensional shape measuring apparatus described in Document 9 (Japanese Patent Application No. 11-123687) can be used. In addition, various devices and means as described in Document 1 can be used.

  Next, in the texture group generation step, a texture space coordinate system (texture coordinate system) is set on the object surface, and the color or luminance information of each point on the object surface is represented as an image using the texture coordinate system, The image is textured. Various methods can be used as the texture coordinate system setting method. As an example, as shown in FIG. 3, a sphere covering the object with the center of gravity of the object as a center is considered, and each point P on the object surface is set. Is projected onto the sphere surface around the center of gravity, and the longitude and latitude (s, t) of the projected point Q can be used as texture coordinates. In addition, various devices and means can be used.

  By using computer graphics techniques or the like, brightness and color changes due to shadows and shadows at each position on the object surface are calculated, and texture groups under various lighting conditions are generated. Changes in luminance and color due to changes in the shade of each pixel of the texture can be calculated from the reflectance of the object surface corresponding to the pixel, the normal direction that can be calculated from the three-dimensional shape, and the direction of illumination. Regarding the generation of shadows, it is possible to determine whether or not light is exposed under the illumination conditions set for the pixels by performing processing such as ray tracing using the three-dimensional shape.

  A texture group can be generated by setting various illumination conditions and performing the texture space generation process.

  Next, in the texture space generation step, a texture space is generated as a space spanned by the texture group.

  Assuming a perfect scattering surface as the reflectance characteristic of the object surface, and assuming that the object shape is convex and there is no shadow caused by shielding the light source by other parts, and the light source is at infinity, each pixel (s, The luminance value I (s, t) of t) is modeled by Equation 1 according to the reflectance B (s, t) of the object surface corresponding to the pixel, the normal direction, the intensity li and the direction of each illumination. it can.

  Here, if the effect of max () is ignored, an arbitrary illumination condition including a case where there are a plurality of illuminations can be represented by one illumination vector as shown in Equation 2.

  Therefore, the degree of freedom of the texture of the object generated by the illumination variation is at most three dimensions of the illumination vector, but in reality, the light source is shielded by the effect of max () and other parts of the object. Therefore, there is an effect by the fact that shadows can be made and the reflection characteristics are not a perfect scattering surface. However, since most can be expressed in a three-dimensional subspace, actual texture fluctuations can be sufficiently approximated as a low-dimensional subspace. Hereinafter, this low-dimensional subspace is called an object texture space.

  Principal component analysis is used to obtain texture space basis vectors. A large number of textures of objects under various lighting conditions are prepared, and an entire set of textures that change due to variations in lighting conditions is approximated. Each texture is a texture under a single point light source at infinity, and many light sources are set at appropriate intervals so as to include all possible directions for lighting conditions when shooting the input image. Prepare the texture. Since the texture under multiple lighting is the sum of the textures of the single lighting, only the texture under the single lighting is sufficient. The texture group is generated using the registered three-dimensional shape and surface reflectance. As an example of the generation means, there is a method of using a basic function of computer graphics. The functions of computer graphics are described in detail in Reference 10 ("OpenGL Programming Guide", Mason Woo, Jackie Neider, Tom Davis, Addison Wesley Publishers Japan). As a standard function of a computer, there are many functions that generate only a shadow by using a reflection characteristic of an object surface as a complete scattering model, but in the present invention, not only a shadow but also a ray tracing technique is used to detect a shadow. It is possible to generate an image as close to reality as possible.

  As described above, the use of a computer graphics function for image generation is an example, and it is of course possible to generate a texture by calculating luminance values for pixels necessary for collation by numerical calculation. .

  Hereinafter, a texture is represented by a vector in which luminance values of pixels in an area used for collation in the entire texture are vertically arranged. When there are N textures in the texture group, V is expressed by Equation 3 when each texture is expressed by a vector.

  Next, up to M eigenvalues σi and eigenvectors of V are calculated in descending order of eigenvalues. Then, the texture space of the object j is approximated by an M-dimensional linear space Ψj having a vector as a base. Here, the dimension M of the texture space can be determined in consideration of the accuracy required for the illumination correction process. When M eigenvectors are used, the cumulative contribution ratio of the eigenvalues can be calculated by Equation 4.

  The cumulative contribution rate is a numerical value representing how accurately each texture can be expressed by the texture space when the error of the texture is evaluated using the difference in luminance value. If a threshold value is defined for this value, M can be automatically determined as the number of dimensions required to exceed the threshold value.

  Next, in the storing step, the three-dimensional shape and the eigenvector of the texture space are stored as registered data.

  Next, the verification process will be described.

  In the input image photographing step, a two-dimensional image of the target object is photographed as an input image using an imaging device such as a video camera.

  In the position / orientation estimation step, the position / orientation between the target object and the imaging apparatus, which are imaging conditions when the input image is captured, and internal parameters of the imaging apparatus are estimated. This position and orientation estimation step can use various methods that are performed manually or automatically.

  For example, as an example of a manual method, when an image generated by computer graphics using the three-dimensional shape and texture of a registered object (hereinafter referred to as a CG image) and an input image of a target object are superimposed, You can use a method of adjusting and inputting parameters of position and orientation using an interactive interface so that the two are as close as possible.

  As an example of a method for automatically estimating the position and orientation, a CG image of a registered object at various positions and orientations is generated, and each CG image is compared with an input image of the target object to obtain the most similar image. The method of determining the position and orientation and the parameters of the imaging device can be used.

  Also, instead of comparing images, the position of a characteristic region or point (hereinafter referred to as a feature point) such as a portion where the luminance value greatly changes on the image of the object is input to the CG image of the registered object and the target object. A method of calculating the position and orientation of the target object and the parameters of the imaging apparatus by obtaining a CG image that is detected from the image and has the closest feature point position can also be used.

  There is also a method for detecting the position of the target object from the input image, and obtaining the position and orientation of the target object using information on the positional relationship between the feature points. Reference 11 (“Ananalytic solution for the pose determination of human faces from a monocular image”, Shinn-Ying Ho, Hui-Ling Huang, Pattern Recognition Letters, Vol. 19, 1045-1054, 1998) When using a human face as an object, use feature points such as the corners of the eyes and mouth, and use a positional relationship such as the straight line connecting the feature points of both eyes and the straight line connecting the feature points of the left and right mouths. A method for determining the posture is described.

In addition, a camera calibration method can be used by registering the positions of the feature points of the registered object. There are many ways to do this. For example, see Reference 12 (“An Efficient and Accurate Camera Calibration Technique”
for 3D Machine Vision ", Roger Y.
Tsai, Proc. CVPR '86, pp. 364-374, 1986) can be used.

  Next, the illumination variation space generation step and the distance calculation step will be described.

  The collation of the object can be performed using the distance between the input image and the illumination fluctuation space Ψ′j of the object j as a scale. Here, the illumination variation space is an image space including image variation due to illumination variation when the object j is in the position and orientation of the input image. As an example of the distance calculation method, for example, a method of calculating a distance between an input image and an image closest to the input image in the illumination variation space Ψ′j can be used. Various measures of distance can be used, but here, an example in which the square error of the luminance value is directly used will be described as an example. These methods are only examples, and various other distance definitions and calculation methods can be used.

  First, in the illumination variation space generation step, an illumination variation space is generated using the three-dimensional shape of the registered object, the illumination variation texture space, and the estimated position and orientation. In the conventional techniques such as method 1 and method 3, it is necessary to generate a large number of illumination variation images in this illumination variation space generation step, and a large amount of calculation is required for calculation of shadows and shadows in that step. There was a problem that processing took time.

  In the present embodiment, the illumination variation space can be easily generated by converting the texture space Ψj of the registered object in accordance with the estimated position and orientation. Various methods can be used for this conversion. For example, the following method can be used to generate the illumination fluctuation space by obtaining the transformation from the coordinate system of the texture space to the coordinate system of the input image. is there. The value of each element of each basis vector of the registered texture space Ψj is set as the luminance value of the object surface to which the element corresponds, and an image at the estimated position and orientation is generated using the three-dimensional shape of the registered object. Since this process can be performed only by a standard function of computer graphics, high-speed calculation is possible. If each generated image is a base vector, an illumination variation space Ψ′j can be generated as a space spanned by the base vector group.

  In the distance calculation step, a comparative image can be generated by Equation 5 as an image closest to the input image in the illumination variation space Ψ′j.

  Here, the vector group is a basis vector group obtained by orthonormalizing the vector group.

  The evaluation value D of the distance between the comparison image and the input image can be calculated by Equation 6 as the sum of squares of the difference in luminance value.

  In the collation determination step, the evaluation value D is used as an evaluation value of the similarity between the input image and the registered data. Based on the evaluation value D, it is confirmed whether the target object is a registered object, and which object is the registered object. Judgment processing is performed to perform processing such as searching for a similar object among registered objects. For example, when confirming whether the target object is a registered object by simple threshold processing, a certain threshold value D ′ is set, and if D <D ′, the target object is determined to be a registered object.

  (2) Second Embodiment The object collation method of the invention according to the second embodiment of the present invention is executed by the object collation apparatus of the second embodiment shown in FIG. The object measurement process includes a three-dimensional shape measurement process for measuring the three-dimensional shape of a registered object, and actually sets various illumination conditions and captures luminance and color information at each position on the object surface to generate a texture group. It is composed of a texture photographing process, there is no texture group generating process, and the photographed texture group is used instead.

  In the invention according to the second embodiment, instead of measuring the three-dimensional shape and surface reflectance of a registered object and generating a lighting variation texture group in the registration process, various illumination conditions are actually set and the object image is set. The fact that the illumination fluctuation texture group can be photographed by photographing is utilized.

  That is, in the invention according to the second embodiment, an appropriate number of illumination conditions are set such that sufficient illumination variation texture groups can be captured to generate a texture space that includes variations due to the illumination conditions of the texture of the registered object. However, if image information is taken under the illumination conditions, processing such as setting reflectance conditions and generating shadows by ray tracing without measuring reflectance or image generation by computer graphics, The fact that a texture group for generating a lighting fluctuation space can be generated is used.

  An example of a method that can be used for the texture image photographing process is as follows. A hemispherical tower is installed in front of the registered object, and an appropriate number of lamps are attached at uniform intervals. Then, an image is taken while each lamp is turned on. In addition to this, various methods such as taking an image while attaching and moving the lamp to the manipulator can be used.

  (3) Third Embodiment The object collating method of the invention according to the third embodiment of the present invention is executed by the object collating apparatus of the third embodiment shown in FIG. Alternatively, in the object matching method of the invention according to the second embodiment, an input texture that converts an input image into a texture using the three-dimensional shape and the estimated position and orientation instead of the illumination variation space generation step. There is a generation step, and the distance calculation step is a step of calculating a distance between the input texture and the texture space.

  In the invention according to the third embodiment, instead of converting the texture space and generating the illumination variation space by matching the position and orientation with the input image, the input image is used by using the three-dimensional shape and the estimated position and orientation. The coordinate transformation from the coordinate system of the texture to the coordinate system of the texture space can be obtained, the input image can be transformed into the input texture by the coordinate transformation, and the distance between the input texture and the texture space can be used as an evaluation value for object matching Is used.

  Various methods can be used to obtain the coordinate transformation. As an example, the color uniquely determined by the coordinate system of the texture space is set as the color of the corresponding object surface, and the three-dimensional shape of the registered object is used as the color. An image at the estimated position and orientation is generated. Since this process can be performed only by a standard function of computer graphics, high-speed calculation is possible. Since the color of each pixel of the generated image indicates the texture coordinate corresponding to that pixel, conversion from the coordinate system of the input image to the coordinate system of the texture space can be easily generated.

  (4) Fourth Embodiment The object collating method according to the fourth embodiment of the present invention is executed by the object collating apparatus of the fourth embodiment shown in FIG. In the invention according to the third embodiment, instead of measuring the three-dimensional shapes of a plurality of registered objects in the registration process, only the three-dimensional shapes of one or a few registered objects are measured. The average shape that is the average of the three-dimensional shapes of a small number of registered objects is output, the shapes of all objects to be collated are not measured, and the reflectance measurement is performed on all objects to be collated. Measurement or imaging of image information under one or more illumination conditions is performed, and only the average shape is used as a three-dimensional shape in the subsequent processing.

  In the invention according to the fourth embodiment, in the case of registered objects that are similar in shape to each other, representative three-dimensional shape data is replaced without measuring the three-dimensional shapes of all registered objects. It is used that the position and orientation estimation process and the illumination correction process can be performed.

  An example of a method that can be used to generate an average shape is as follows. Here, an example in which the average shape of two objects is obtained will be described. First, the three-dimensional shapes of the object 1 and the object 2 are measured. Then, as shown in FIG. 11A, the three-dimensional shapes of the two objects are translated so that the centers of gravity coincide with each other, and a cross section perpendicular to the z axis is considered, and the cross sections are moved in the z axis direction at appropriate intervals. While calculating the average shape on each cross-section. As shown in FIG. 11B, an average calculation axis that is a straight line extending from the center of gravity to the outside of the object on the cross section is considered, and intersections between the shapes of the object 1 and the object 2 are P1 and P2. Assume that the three-dimensional coordinates of the point Pm having an average shape average the three-dimensional coordinates (x1, y1, z1) and (x2, y2, z2) of the points P1, P2 on the two object surfaces. By repeating this process while rotating the average calculation axis around the center of gravity at an appropriate interval, the average shapes of the objects 1 and 2 can be generated. In the three-dimensional shape measurement process, the average shape is output.

  This method can be applied even when the number of objects to be collated is three or more. Various other methods for obtaining the average shape can be used.

  In the verification process, the average shape is read from the data storage process, and is substituted for the three-dimensional shape of each registered object.

  (5) Fifth Embodiment The object collation method of the invention according to the fifth embodiment of the present invention is executed by the object collation apparatus of the fifth embodiment shown in FIG. In the process, the registration data is registered by adding position information of a characteristic part of the registered object.

  In the invention according to the seventh embodiment, the position of the feature point of the registered object is extracted and stored as registration data, and if the three-dimensional coordinates of the feature point of the input image can be known from the registration data, The fact that the position and orientation of the target object in the input image and the parameters of the imaging device can be automatically obtained using the position on the image and the three-dimensional coordinates is utilized.

  As a method for automatically obtaining the position and orientation of the target object in the input image and the parameters of the imaging device when the position on the image and the three-dimensional coordinates are known for a plurality of points, a camera calibration method can be used. There are many methods for this, and an example is described in Reference 12 (“An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision”, Roger Y. Tsai, Proc. CVPR '86, pp. 364-374, 1986). However, details are omitted here.

  (6) Sixth Embodiment The object collation method of the invention according to the sixth embodiment of the present invention is executed by the object collation apparatus of the sixth embodiment shown in FIG. In the object collating method of the invention according to the fourth embodiment, the three-dimensional shape of the registered object is not measured, and the three-dimensional shape is separately input from a drawing or the like.

  In the invention according to the sixth embodiment, when the three-dimensional shape of a registered object exists as a drawing or data as a design drawing or the like of the object, the data is obtained from the drawing or the like without measuring the registered object. The fact that the registration process can be performed by inputting is used.

  As an example, if the object to be collated is an object with a design drawing such as an industrial product, or the measurement of a three-dimensional shape of a building or the like is difficult with a general three-dimensional shape measuring device, a separate survey is performed. When measuring the shape, the three-dimensional shape of the registered object can be input from a drawing or the like.

  Further, the object collation method of the invention according to the sixth embodiment does not measure the reflectance of the registered object, but inputs the reflectance and color information separately from a drawing or the like.

  In the invention according to the sixth embodiment, when the reflectance or color information of the object surface exists as a drawing or data as a design drawing of the object, the reflectance or the like is not measured or photographed. In both cases, the registration process can be performed by inputting data from a drawing or the like.

  As an example of this, when the object to be collated is an object whose coating reflectance of each part of the object surface such as an industrial product is known, the fact that the three-dimensional shape of the object can be input from a drawing or the like is used.

  First Embodiment FIG. 1 is a block diagram showing a configuration of an object collating apparatus according to a first embodiment of the present invention. The object collation apparatus according to the present embodiment includes a registration unit 100 including a registered object measuring unit 110, an illumination variation texture group generating unit 120, a texture space generating unit 130, and a data storage unit 140, an imaging unit 210, and a position / orientation estimating unit 220. , Illumination variation space generation means 230, distance calculation means 240, and collation unit 200 including collation determination means 250.

  FIG. 2 is a diagram illustrating a specific configuration example of the object matching apparatus according to the first embodiment. The registration unit 100 includes a three-dimensional shape measuring device, a registration processing device, a database storage device, and the like. The collation unit 200 includes a camera, a collation processing device, and the like.

  As shown in FIG. 1, the registration unit 100 stores a three-dimensional shape and texture space of an object. The matching unit 200 takes a two-dimensional image using an imaging device such as a video camera, loads the image into a processing device that performs matching processing, and performs matching processing with a registered object.

  First, the registration unit 100 stores a three-dimensional shape of an object and a texture space including texture variations under various illumination conditions as registration data used for object collation.

  The registered object measuring means 110 measures the three-dimensional shape of the object and the reflectance of the object surface corresponding to the three-dimensional shape or the color information corresponding thereto using a three-dimensional shape measuring apparatus. For this, for example, the three-dimensional shape measuring apparatus described in Document 9 can be used. In addition, various devices can be used.

  As shown in FIG. 4, the illumination variation texture group generation unit 120 includes a texture coordinate system generation unit 121, an illumination condition change unit 122, and a shadow calculation unit 123.

  The texture coordinate system generation unit 121 sets a texture coordinate system as a coordinate system that represents the position of each point on the object surface regardless of the position and orientation of the object, and the shape (3 of the object surface with respect to the texture coordinates (s, t)) Dimensional coordinates) and reflectivity are output. Various methods can be used for setting the texture coordinate system. Here, as an example, a method using projection onto a spherical surface will be described. As shown in FIG. 3, a sphere covering an object with the center of gravity of the object as a center is considered, and each point P on the surface of the object is projected onto the sphere surface with the center of gravity as the center, and the longitude, latitude ( A technique using s, t) as texture coordinates can be used. This method is merely an example, and various other methods suitable for the shape of the object to be registered can be used.

  The illumination condition changing means 122 sets a sufficient number of illumination condition groups to approximate the illumination variation space. For example, given a point light source at infinity, the direction of the light source is indicated by angles (θ, φ) representing the longitude and latitude of a sphere centered on an object as shown in FIG. φ is changed from −90 ° to 90 ° every 10 °, and 361 kinds of illumination condition groups are set. The type of the light source and how to determine the setting interval and range of the illumination direction are merely examples, and various changes can be made.

  The shadow calculation means 123 reads the three-dimensional shape and reflectance of the object j, and generates an illumination variation texture group in the illumination condition group input from the illumination condition change means 122 using a function such as computer graphics. As an example, this processing can be realized with a basic function of a computer having a graphics function. In image generation using computer graphics, various object surface reflection models, camera models, and the like can be used. As an example, a pinhole camera model can be used as the camera model, and a complete scattering model can be used as the reflection model of the object surface. These models are only examples, and various other reflection models can be used for performing a ray tracing process to add a shadow or to apply a shadow. In this image generation process, collation performance can be improved by making the model of the reflection characteristics of the object surface, the light source, and the like more accurate near reality. Further, this image generation can be realized by numerical calculation without using computer graphics.

  The texture space generation unit 130 calculates a texture space, which is an image space that includes texture variations when the illumination conditions vary in various ways, from the generated illumination variation texture group.

  Various methods can be used for this, but in this embodiment, an example of using a method of representing a texture space as a low-dimensional linear space using principal component analysis will be described.

  The texture space is calculated from the illumination variation texture group generated by the illumination variation texture group generation means 120 according to the above Equation 3, and the calculated basis vector group is output as the object texture space Ψj. In this embodiment, M basis vectors are obtained in descending order of eigenvalues, and are output as the texture space Ψj of the object j. Taking the number M of the basis vectors as an example, in order to determine that the cumulative contribution calculated in Equation 4 exceeds 95%, 361 equal to the number of images in the illumination variation texture group or the number of pixels is less than that. In this case, the number of pixels is N, N eigenvalues are obtained, and the following number M is obtained and determined. The determination method of the number M can be determined by applying various other criteria.

  The data storage unit 140 stores and holds the registered three-dimensional shape and texture space of each object, and reads them out for processing of the collation unit 200 in a timely manner.

  The following collation unit 200 performs object collation processing using an image on the object that has been subjected to the processing of the registration unit 100 described above.

  The verification unit 200 captures an input image of the target object using an imaging device such as a camera, and imports it into a processing device that is a verification unit.

  The imaging unit 210 captures an input image of the target object using an imaging device such as a camera or a video camera.

  The position / orientation estimation unit 220 estimates the object position / orientation, imaging device parameters, and the like, which are imaging conditions when the input image is captured. For example, the parallel movement distance (Tx, Ty, Tz), the rotation angle (Rx, Ry, Rz), the camera focal length f, and the viewing angle α are used as the position and orientation parameters. An interactive interface is provided on the processing device so that the user can manually adjust these parameters while viewing the screen. For example, an image of a registered object generated by computer graphics using the eight parameters and an input image of the target object are superimposed on the screen and displayed by the superimpose method. The user adjusts the values of the eight parameters so that the two images are exactly overlapped, and determines an appropriate parameter. This interactive interface is an example, and various forms can be used. Further, the position and orientation parameters may be automatically calculated without using such an interactive interface.

  The illumination fluctuation space generation means 230 uses the result of the position / orientation estimation means 220 and uses the result of the position / orientation estimation means 220 to create an illumination fluctuation that is an image space that includes image fluctuations of an object under various illumination conditions. Create a space. That is, the illumination variation space is generated by converting the registered texture space Ψj according to the estimated position and orientation. Various methods can be used for this conversion. For example, the following method can be used to generate the illumination fluctuation space by obtaining the transformation from the coordinate system of the texture space to the coordinate system of the input image. is there.

  The value of each element of each basis vector of the registered texture space Ψj is used as the luminance value of the object surface to which the element corresponds, and an image group at the estimated position and orientation is generated using the registered three-dimensional shape. . Since this process can be performed only by a standard function of computer graphics, high-speed calculation is possible. If each generated image is directly used as a base vector, an illumination variation space Ψ′j can be generated as a space spanned by the base vector group. The generated basis vector group is orthonormalized, and a basis vector group forming an orthonormal system is calculated.

  The distance calculation means 240 generates a comparison image as the image closest to the input image in the illumination variation space Ψ′j by the equation 5, and calculates the evaluation value D of the distance between the comparison image and the input image by the equation 6. Then output.

  In the collation determination unit 250, the evaluation value D is used as an evaluation value of the similarity between the input image and the registered data, and based on this, it is confirmed whether the target object is a registered object, and any object among the registered objects. Judgment processing is performed to perform processing such as searching for a certain object or searching for a similar object among registered objects. For example, when confirming whether the target object is a registered object by simple threshold processing, a certain threshold value D ′ is set, and if D <D ′, the target object is determined to be a registered object.

  The above distance calculation method and collation determination method are merely examples, and various other methods can be applied.

  Second Embodiment FIG. 7 is a block diagram showing the configuration of an object collating apparatus according to a second embodiment of the present invention. The object collation apparatus according to the present embodiment includes a registration unit 2100 including a three-dimensional shape measuring unit 2110, a texture photographing unit 2121, a texture space generating unit 130, and a data storage unit 140, a photographing unit 210, a position / orientation estimating unit 220, and an illumination. The variable space generation unit 230, the distance calculation unit 240, and the verification unit 200 including the verification determination unit 250 are configured.

  Compared to the object collating apparatus of the first embodiment, the object collating apparatus of the second embodiment adds a texture photographing unit 2121 instead of measuring the reflectance in the registered object measuring unit 110, and has a plurality of illumination conditions. The difference is that an image is taken in advance and the illumination variation texture group is generated using these images instead of the reflectance, and the illumination condition changing means 122 is not provided.

  First, the registration unit 2100 registers, as registration data for a registered object, the three-dimensional shape of the registered object and image data under a plurality of illumination conditions.

  The three-dimensional shape measuring means 2110 measures the three-dimensional shape of the registered object using the three-dimensional shape measuring apparatus described in Document 9, as with the registered object measuring means 110 in the first embodiment. Do not measure.

  The texture photographing unit 2121 actually sets an illumination condition equivalent to the illumination condition output from the illumination condition changing unit 122 in the first embodiment to photograph the image group of the registered object, and generates the illumination variation texture group generation A texture coordinate system is generated using the three-dimensional shape by a method equivalent to that of the means 120, and the image group is converted into the texture coordinate system to be output as an illumination variation texture group. For example, a hemispherical tower centered on the registered object is installed in front of the registered object, and an appropriate number of lamps are attached at appropriate intervals. As an example, with respect to a registered object, at angles (θ, φ) shown in FIG. 5, lamps are attached at intervals of 15 degrees in the range from −90 degrees to 90 degrees with respect to θ and φ, and each lamp is turned on. Take one image at a time. This photographing method and the setting method of the illumination position are only examples, and various other methods such as photographing an image while attaching and moving the lamp to the manipulator can be used. The texture photographing unit 2121 outputs an image group photographed by the above method as an illumination variation texture group.

  The texture space generation means 130 and the data storage means 140 are exactly the same as those in the first embodiment.

  The following verification unit 200 is equivalent to the verification unit 200 in the first embodiment.

  Third Embodiment FIG. 8 is a block diagram showing the configuration of an object collating apparatus according to a third embodiment of the present invention. The object collation apparatus according to the present embodiment includes a registration unit 100 including a registered object measuring unit 110, an illumination variation texture group generating unit 120, a texture space generating unit 130, and a data storage unit 140, an imaging unit 210, and a position / orientation estimating unit 220. , An input texture generation unit 231, a distance calculation unit 241, and a collation unit 3200 including a collation determination unit 250.

  The registration unit 100 is equivalent to the registration unit 100 in the first embodiment.

  Compared with the verification unit 200 in the first embodiment, the verification unit 3200 includes an input texture generation unit 231 instead of the illumination variation space generation unit 230, and the distance calculation unit 241 converts the input image. The difference is that the distance between the input texture and the texture space is calculated.

  The processing from the photographing means 210 to the position / orientation estimation means 220 is exactly the same as in the first embodiment.

  The input texture generation means 231 uses the result of the position / orientation estimation means 220 to generate an input texture by converting each pixel of the input image into texture coordinates and transforming it. Various methods can be used for this conversion. As an example, an illumination fluctuation space is generated by obtaining a conversion from the coordinate system (u, v) of the input image to the coordinate system (s, t) of the texture space. The following techniques can be used.

  For a registered three-dimensional shape, a virtual object assigned with a color that is uniquely determined by the texture coordinates (s, t) is defined as the color of the object surface, and an image when in the estimated position and orientation is generated. . Since this process can be performed only by a standard function of computer graphics, high-speed calculation is possible. Since the texture coordinates Q (s, t) can be found by looking at the color of each pixel R (u, v) in the generated image, the texture coordinates (u, v) from the coordinate system (u, v) of the input image as shown in FIG. A coordinate transformation to s, t) is required. This transformation is applied to all pixels used for matching on the input image to generate an input texture.

  The distance calculation unit 241 generates a comparison texture as a texture closest to the input texture in the texture space Ψj, and outputs an evaluation value D of the distance between the comparison texture and the input texture.

  The registered texture space Ψj is read, and pixels that correspond to the input image at the time of generation of the input texture, except for the region that is not visible in the input image (region B in FIG. 9), of each basis vector (see FIG. Only the elements corresponding to the region A) 9 are extracted, and a vector in which the elements are arranged is generated.

  The distance D between the texture space Ψj and the input texture can be calculated by the least square method as the minimum value of the error when the input texture is expressed as a linear sum as shown in Equation 7.

  Fourth Embodiment FIG. 10 is a block diagram showing the configuration of an object collating apparatus according to a fourth embodiment of the present invention. The object collating apparatus according to the present embodiment includes a registration unit 4100 including a registered object measuring unit 110, an average shape generating unit 111, an illumination variation texture group generating unit 120, a texture space generating unit 130, and a data storage unit 141, and a photographing unit 210. , Position / orientation estimation means 220, illumination variation space generation means 230, distance calculation means 240, and collation unit 200 including collation determination means 250.

  Compared with the object collating apparatus of the first embodiment, the object collating apparatus of the fourth embodiment is configured such that the registered object measuring unit 110 compares all three registered objects when the registration unit 4100 registers a plurality of registered objects. Instead of measuring the three-dimensional shape, only the three-dimensional shape of one or a few registered objects is measured, and the average shape that is the average of the three-dimensional shapes of the one or a few registered objects is obtained in the average shape generating means 111. The difference is that the shapes of all the objects to be collated are not measured, and the collation unit 200 uses the average three-dimensional shape.

  Below, it demonstrates using the example which registers two registration objects of the object 1 and the object 2. FIG.

  First, the registration unit 4100 registers an average shape and respective texture spaces for two registered objects, the object 1 and the object 2, as registration data used for object matching.

  The registered object measuring means 110 measures the three-dimensional shapes of the object 1 and the object 2 using the three-dimensional shape measuring apparatus described in Document 9.

  In the average shape generation means 111, as shown in FIG. 11A, the three-dimensional shapes of the two objects 1 and 2 are translated so that the centers of gravity coincide with each other, and a cross section perpendicular to the z axis is arranged at an appropriate interval. Set and calculate the average shape on each cross section. As shown in FIG. 11B, a straight line that is an average calculation axis from the center of gravity to the outside of the object on the cross section is considered, and the intersections with the shapes of the objects 1 and 2 are P1 and P2. Assume that the three-dimensional coordinates of the point Pm having an average shape average the three-dimensional coordinates (x1, y1, z1) and (x2, y2, z2) of the points P1, P2 on the two object surfaces. By performing this process at appropriate intervals while rotating the average calculation axis around the center of gravity, the average shapes of the object 1 and the object 2 can be generated. The average shape generation unit 111 outputs the average shape.

  In the following processing, the average shape is used instead of the three-dimensional shape of each object, and the data storage unit 141 is the same as in the first embodiment except that the average shape and the texture space of each object are stored. It is equivalent.

  The processing in the matching unit 200 is different from the first embodiment only in that the three-dimensional shape read from the data storage unit 141 as the shapes of the object 1 and the object 2 is the average shape, and the other processes are all the same. .

  As mentioned above, although the example which memorize | stores the average shape when registering two registration objects was demonstrated in the 4th Example, this is an example to the last, the number of registration objects may be three or more, It is also possible to obtain and use the average shape of an arbitrary number of registered objects by a similar process.

  Fifth Embodiment FIG. 12 is a block diagram showing the configuration of the object collating apparatus according to the fifth embodiment of the present invention. The object collation apparatus according to the present embodiment includes a registration unit 5100 including a registered object measurement unit 110, an illumination variation texture group generation unit 120, a texture space generation unit 130, a data storage unit 143, and a feature point position extraction unit 150, and an imaging unit. 210, a position / orientation estimation unit 221, an illumination variation space generation unit 230, a distance calculation unit 240, and a collation unit 5200 including a collation determination unit 250.

  The object collation apparatus of the fifth embodiment is different from the object collation apparatus of the first embodiment in that, in the registration unit 5100, the position of a characteristic point such as a luminance value greatly changing on the image of the registration object is determined. A feature point position extraction unit 150 for extracting and outputting as a feature point position is added, a point for storing the feature point position in the data storage unit 143, and a data storage unit in the position / orientation estimation unit 221 of the collation unit 5200. The difference is that the feature point position is read from 143 and the position and orientation of the target object are automatically estimated.

  In the fifth embodiment, a human face is used as an example of an object to be collated.

  First, the registration unit 5100 measures the three-dimensional shape and reflectance of the registered object as registration data used for object matching, and obtains the three-dimensional coordinates of the feature points of the registered object from the three-dimensional shape and reflectance. The three-dimensional shape, texture space, and feature point position are registered.

  The registered object measuring means 110 measures the three-dimensional shape and reflectance of the registered object using a three-dimensional shape measuring apparatus. In this embodiment, the three-dimensional shape measuring device described in Document 9 is used as an example of the three-dimensional shape measuring device, but various other devices can be used.

  The feature point position extraction unit 150 detects the position of a characteristic region or point (feature point) such as a portion where the luminance value greatly changes on the image of the registered object, and outputs the three-dimensional coordinates as the feature point position. . As an example, when a person's face is an object to be collated, a part where the reflectance such as the eyes or the mouth changes greatly or a part where the three-dimensional shape such as the head of the nose changes significantly is detected. To do. This can be performed manually, or as a method of automatically performing the above-mentioned document 2 (Japanese Patent No. 2872776 “Face Image Matching Device”) or Document 4 (Japanese Patent Laid-Open No. 6-168317 “Personal Identification Device”). Various methods such as those described can be used. In this embodiment, twelve points (0 to 11) at positions as shown in FIG. 14 are used as feature points. It goes without saying that the definition of these feature points can be changed variously depending on the object to be collated. Below, the feature point position which is the three-dimensional coordinate of these feature points is represented by.

  The data storage unit 143 stores and holds the three-dimensional shape, texture space, and feature point position of each registered object, and reads them out for processing by the matching unit 5200.

  The following verification unit 5200 performs object verification processing using the input image of the target object on the registered object that has been processed by the registration unit 5100 described above.

  The imaging unit 210 captures an input image of the target object using an imaging device such as a camera or a video camera.

  The position / orientation estimation means 221 estimates the position / orientation of the target object and the parameters of the imaging apparatus, which are the imaging conditions when the input image is captured.

  Referring to FIG. 13, the position / orientation estimation unit 221 further includes a feature point extraction unit 222 and a position / orientation calculation unit 223.

  The feature point extraction unit 222 extracts the same feature point position as the feature point group extracted by the feature point position extraction unit 150 in the registration unit 5100 from the input image, and uses the position on the input image as the input image feature point position. Output. This can be input manually by a person looking at the input image displayed on the screen of the processing apparatus, or used by the feature point position extraction means 150 such as the methods described in the literature 4 and literature 5. Various methods similar to those described above can be used. In the present embodiment, a case where a human face is collated is taken as an example. However, for example, when a polyhedral object is a collation target, a vertex can be used as a feature point, and an edge is extracted from an image. The vertex of the polyhedron can be detected as an intersection. Also, when there is a characteristic pattern on the object surface, the position of the pattern can be used.

  The position / orientation calculation means 223 uses the input image feature point position and the feature point position read from the data storage means 143 to calculate the position / orientation of the target object in the input image, the parameters of the imaging device, and the like. Output as posture. For this calculation, various methods such as the method described in the above-mentioned document 5 can be used. In this embodiment, as an example, the parallel movement distance (Tx, Ty, Tz), x, y, z of the target object is used as a position and orientation parameter. The rotation angle (Rx, Ry, Rz) around the axis and the focal length f of the camera are taken, and a pinhole camera is used as the camera model, and the following method is used. As described above, the parameters including the parameters of the imaging device such as the focal length f are referred to as a position and orientation. The relationship between the feature point position and the input image feature point position is expressed by Equation 8.

  Here, a, b, and c are values represented by Equation 9.

  R is a matrix representing the rotation expressed by Equation 10.

  Rx, Ry, Rz, Tx, Ty, Tz, f are set so that the sum of errors between the value calculated by the equation 8 for each of the 12 feature points and the value of the input image feature point position is minimized. Obtained by optimization calculation. Various methods can be used for this optimization calculation. The obtained Rx, Ry, Rz, Tx, Ty, Tz, f are output as position and orientation parameters.

  The position and orientation parameters and the camera model definition and calculation method are merely examples, and various other methods can be used.

  The processing after the illumination variation space generating means 230 is the same as in the first embodiment.

  Sixth Embodiment FIG. 15 is a block diagram showing a configuration of an object collating apparatus according to a sixth embodiment of the present invention. The object collating apparatus according to the present embodiment includes a registration unit 6100 including a registered object input unit 112, an illumination variation texture group generation unit 120, a texture space generation unit 130, and a data storage unit 140, an imaging unit 210, and a position and orientation estimation unit 220. , Illumination variation space generation means 230, distance calculation means 240, and collation unit 200 including collation determination means 250.

  In the object collating apparatus of the sixth embodiment, the object to be collated is an industrial product, the design drawing of the shape is stored as CAD (Computer Aided Design) data, and the surface coating specifications are determined by the design drawing. Suppose that For this reason, the object collating apparatus of the sixth embodiment is compared with the object collating apparatus of the first embodiment shown in FIG. The difference is that the reflectance is read from the design drawing.

  The registered object input unit 112 converts the CAD data of the design drawing into a data format that can be handled by the matching unit 200 and outputs the data as a three-dimensional shape. In addition, the color of each part of the registered object, the method of finishing the surface, etc. are read from the design drawing, converted into reflectance, and output. The verification unit 200 is exactly the same as that in the first embodiment.

  Seventh Embodiment FIG. 16 is a block diagram showing the configuration of the object collating apparatus according to the seventh embodiment of the present invention. The object collating apparatus according to the present embodiment includes a recording medium 1000 in which a registration program is recorded in the registration unit 100, and the collation program is recorded in the collation unit 200, in contrast to the object collation apparatus according to the first embodiment illustrated in FIG. The difference is that the recording medium 2000 is provided. These recording media 1000 and 2000 may be magnetic disks, semiconductor memories, or other recording media.

  The registration program is read from the recording medium 1000 into a computer constituting the registration unit 100, and the operations of the computer are registered object measurement means 110, illumination variation texture group generation means 120, texture space generation means 130, and data storage means 140. Control. Since the operation of the registration unit 100 under the control of the registration program is exactly the same as the operation of the registration unit 100 in the first embodiment, a detailed description thereof will be omitted.

  The collation program is read from the recording medium 2000 to a computer constituting the collation unit 200, and the operation of the computer is performed by the photographing unit 210, the position / orientation estimation unit 220, the illumination variation space generation unit 230, the distance calculation unit 240, and the collation. The collation unit 200 including the determination unit 250 is controlled. Since the operation of the collation unit 200 under the control of the collation program is exactly the same as the operation of the collation unit 200 in the first embodiment, its detailed description is omitted.

  Eighth Embodiment FIG. 17 is a block diagram showing a configuration of an object collating apparatus according to an eighth embodiment of the present invention. The object collation apparatus according to the present embodiment is provided with a recording medium 1010 in which a registration program is recorded in the registration unit 2100, and the collation program is recorded in the collation unit 200, compared to the object collation apparatus according to the second embodiment illustrated in FIG. The difference is that the recording medium 2000 is provided. These recording media 1010 and 2000 may be magnetic disks, semiconductor memories, or other recording media.

  The registration program is read from the recording medium 1010 into a computer constituting the registration unit 2100, and controls the operation of the computer as a three-dimensional shape measuring unit 2110, a texture photographing unit 2121, a texture space generating unit 130, and a data storage unit 140. . Since the operation of the registration unit 2100 under the control of the registration program is exactly the same as the operation of the registration unit 2100 in the second embodiment, its detailed description is omitted.

  The collation program is read from the recording medium 2000 to a computer constituting the collation unit 200, and the operation of the computer is performed by the photographing unit 210, the position / orientation estimation unit 220, the illumination variation space generation unit 230, the distance calculation unit 240, and the collation. The collation unit 200 including the determination unit 250 is controlled. Since the operation of the collation unit 200 under the control of the collation program is exactly the same as the operation of the collation unit 200 in the second embodiment, a detailed description thereof will be omitted.

  Ninth Embodiment FIG. 18 is a block diagram showing a configuration of an object collating apparatus according to an eighth embodiment of the present invention. The object collation apparatus according to the present embodiment is provided with a recording medium 1000 in which a registration program is recorded in the registration unit 100, and the collation program is recorded in the collation unit 3200 with respect to the object collation apparatus according to the third embodiment illustrated in FIG. The recording medium 2010 is different. These recording media 1000 and 2010 may be magnetic disks, semiconductor memories, or other recording media.

  The registration program is read from the recording medium 1000 into a computer constituting the registration unit 100, and the operations of the computer are registered object measurement means 110, illumination variation texture group generation means 120, texture space generation means 130, and data storage means 140. Control. Since the operation of the registration unit 100 under the control of the registration program is exactly the same as the operation of the registration unit 100 in the third embodiment, a detailed description thereof will be omitted.

  The collation program is read from the recording medium 2010 into a computer constituting the collation unit 3200, and the operation of the computer is performed by the photographing unit 210, the position / orientation estimation unit 220, the input texture generation unit 231, the distance calculation unit 241, and the collation determination. The collation unit 3200 including the means 250 is controlled. Since the operation of the collation unit 3200 under the control of the collation program is exactly the same as the operation of the collation unit 3200 in the third embodiment, a detailed description thereof will be omitted.

  Tenth Embodiment FIG. 19 is a block diagram showing a configuration of an object collating apparatus according to a tenth embodiment of the present invention. The object collation apparatus according to the present embodiment is provided with a recording medium 1020 in which a registration program is recorded in the registration unit 4100, and the collation program is recorded in the collation unit 200, compared to the object collation apparatus according to the fourth embodiment illustrated in FIG. The difference is that the recording medium 2000 is provided. These recording media 1020 and 2000 may be magnetic disks, semiconductor memories, or other recording media.

  The registration program is read from the recording medium 1020 into a computer constituting the registration unit 4100, and the operations of the computer are registered object measurement means 110, average shape generation means 111, illumination variation texture group generation means 120, texture space generation means 130, The data storage unit 141 is controlled. Since the operation of the registration unit 4100 under the control of the registration program is exactly the same as the operation of the registration unit 4100 in the fourth embodiment, its detailed description is omitted.

  The collation program is read from the recording medium 2000 to a computer constituting the collation unit 200, and the operation of the computer is performed by the photographing unit 210, the position / orientation estimation unit 220, the illumination variation space generation unit 230, the distance calculation unit 240, and the collation. The collation unit 200 including the determination unit 250 is controlled. Since the operation of the collation unit 200 under the control of the collation program is exactly the same as the operation of the collation unit 200 in the fourth embodiment, a detailed description thereof will be omitted.

  Eleventh Embodiment FIG. 20 is a block diagram showing the configuration of an object collating apparatus according to an eleventh embodiment of the present invention. The object collation apparatus according to the present embodiment is provided with a recording medium 1030 in which a registration program is recorded in the registration unit 5100, and the collation program is recorded in the collation unit 5200 with respect to the object collation apparatus according to the fifth embodiment illustrated in FIG. The difference is that the recording medium 2020 is provided. These recording media 1030 and 2020 may be magnetic disks, semiconductor memories, or other recording media.

  The registration program is read from the recording medium 1030 into a computer constituting the registration unit 5100, and the operation of the computer is registered object measurement means 110, illumination variation texture group generation means 120, texture space generation means 130, data storage means 143, and Control is performed as the feature point position extraction unit 150. Since the operation of the registration unit 5100 under the control of the registration program is exactly the same as the operation of the registration unit 5100 in the fifth embodiment, a detailed description thereof will be omitted.

  The collation program is read from the recording medium 2020 into a computer constituting the collation unit 5200, and the operation of the computer is performed by the photographing unit 210, the position / orientation estimation unit 221, the illumination variation space generation unit 230, the distance calculation unit 240, and the collation. The collation unit 5200 including the determination unit 250 is controlled. Since the operation of the collation unit 5200 under the control of the collation program is exactly the same as the operation of the collation unit 5200 in the fifth embodiment, its detailed description is omitted.

  Twelfth Embodiment FIG. 21 is a block diagram showing the configuration of an object collating apparatus according to a twelfth embodiment of the present invention. The object collation apparatus according to the present embodiment is provided with a recording medium 1040 in which a registration program is recorded in the registration unit 6100 as compared to the object collation apparatus according to the sixth embodiment illustrated in FIG. The difference is that the recording medium 2000 is provided. These recording media 1040 and 2000 may be magnetic disks, semiconductor memories, or other recording media.

  The registration program is read from the recording medium 1040 into a computer constituting the registration unit 6100, and the operations of the computer are registered object input means 113, illumination variation texture group generation means 120, texture space generation means 130, and data storage means 140. Control. Since the operation of the registration unit 6100 under the control of the registration program is exactly the same as the operation of the registration unit 6100 in the sixth embodiment, a detailed description thereof will be omitted.

  The collation program is read from the recording medium 2000 to a computer constituting the collation unit 200, and the operation of the computer is performed by the photographing unit 210, the position / orientation estimation unit 220, the illumination variation space generation unit 230, the distance calculation unit 240, and the collation. The collation unit 200 including the determination unit 250 is controlled. Since the operation of the collation unit 200 under the control of the collation program is exactly the same as the operation of the collation unit 200 in the second embodiment, a detailed description thereof will be omitted.

  As described in the above embodiments, the present invention can be applied to general objects. However, the present invention is particularly effective for applications such as vehicle model / model verification and human face verification.

  Although the present invention has been specifically described above based on the respective embodiments, it is needless to say that the present invention is not limited to the respective embodiments and can be variously modified without departing from the gist thereof.

It is a block diagram which shows the structure of the object collation apparatus of 1st Example of this invention. It is a figure which shows an example of the specific structure of the object collation apparatus of this invention. It is a figure explaining an example of the method of defining a texture coordinate system with respect to an object surface. It is a block diagram which shows the structure of the illumination variation texture group production | generation means in FIG. 1 in detail. It is a figure explaining the angle showing the direction with respect to the object of the illumination which determines illumination conditions. It is the figure which showed the example of the object collation apparatus using an image. It is a block diagram which shows the structure of the object collation apparatus of the 2nd Example of this invention. It is a block diagram which shows the structure of the object collation apparatus of the 3rd Example of this invention. It is a figure explaining the conversion from the input image coordinate to a texture coordinate in the input texture production | generation means in FIG. It is a block diagram which shows the structure of the object collation apparatus of the 4th Example of this invention. It is a figure explaining the production | generation method of the average shape in the average shape production | generation means in FIG. It is a block diagram which shows the structure of the object collation apparatus of the 5th Example of this invention. It is a block diagram which shows the more detailed structure of the position and orientation estimation means in FIG. It is a figure which shows an example of the site | part of the object used as a feature point of the object used as collation object. It is a block diagram which shows the structure of the object collation apparatus of the 6th Example of this invention. It is a block diagram which shows the structure of the object collation apparatus of the 7th Example of this invention. It is a block diagram which shows the structure of the object collation apparatus of the 8th Example of this invention. It is a block diagram which shows the structure of the object collation apparatus of the 9th Example of this invention. It is a block diagram which shows the structure of the object collation apparatus of the 10th Example of this invention. It is a block diagram which shows the structure of the object collation apparatus of the 11th Example of this invention. It is a block diagram which shows the structure of the object collation apparatus of the 12th Example of this invention. It is a figure explaining the structure of the technique which uses only a two-dimensional image both at the time of registration and the time of collation as an example of the conventional object collation technique. It is a figure explaining the structure of the technique which measures a three-dimensional shape both at the time of registration and the time of collation as an example of the conventional object collation technique. As an example of a conventional object matching technique, it is a diagram illustrating a configuration of a technique that takes a two-dimensional image both during registration and during matching and uses a standard three-dimensional shape model for position and orientation correction. It is a figure explaining the structure of the technique which image | photographs and recognizes as an example of the conventional object collation technique under many position and orientation and illumination conditions at the time of registration. It is a figure explaining the structure of the technique which image | photographs a two-dimensional image on several illumination conditions at the time of registration, and corrects illumination conditions as an example of the conventional object collation technique. It is a figure explaining the structure of the technique which image | photographs a two-dimensional image on many illumination conditions at the time of registration, and corrects illumination conditions as an example of the conventional object collation technique.

Explanation of symbols

100, 2100, 4100, 5100, 6100 Registration unit 110 Registered object measurement means 111 Average shape generation means 112 Registered object input means 120 Illumination variation texture group generation means 121 Texture coordinate system generation means 122 Illumination condition change means 123 Shadow calculation means 130 Texture Space generation means 140, 141, 143 Data storage means 150 Feature point position extraction means 200, 3200, 5200 Collation unit 210 Imaging means 220, 221 Position / orientation estimation means 230 Illumination variation space generation means 231 Input texture generation means 240, 241 Distance calculation Means 250 Collation determining means 1000, 1010, 1020, 1030, 1040 Recording medium 2000, 2010, 2020 Recording medium 2110 Three-dimensional shape measuring means 2121 Texture photographing means

Claims (6)

A base vector representing a texture space spanned by texture groups representing brightness and color information at each position on the surface of a registered object under various lighting conditions, and using a 3D shape of the registered object as a target in the input image A step of deforming in accordance with the position and orientation of the object, and whether the target object is a registered object based on a distance between the closest image in the illumination variation space represented by the deformed basis vector and the input image And a step of making a determination. The method further comprises a step of performing at least one of a search for a registered object and a search for a similar object among the registered objects based on the determination. Item 3. The object matching method according to Item 1. The basis vector representing the texture space spanned by texture groups representing the brightness and color information of each position on the surface of the registered object under various lighting conditions, using the 3D shape of the registered object as the target in the input image Whether the target object is a registered object based on a distance between the input image and a means for deforming according to the position and orientation of the object, and a distance between the closest image in the illumination variation space expressed by the deformed basis vector and the input image An object collating apparatus comprising: means for performing determination; And a means for searching for at least one of registered objects or a similar object of registered objects based on the determination. Item 4. The object matching apparatus according to Item 3. Using a 3D shape of the registered object, input a base vector that represents the texture space spanned by a texture group that represents the brightness and color information of each position on the surface of the registered object under various lighting conditions. The target object is a registered object based on the process of deforming according to the position and orientation of the target object in the image and the distance between the closest image in the illumination variation space represented by the deformed basis vector and the input image And a process for determining whether or not the object collation program is executed. And a process of performing at least one of searching for a registered object or searching for a similar object among registered objects based on the determination. The object collation program according to claim 5.

Images