Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
  • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
  • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
  • G06V40/172—Classification, e.g. identification

Definitions

• the present invention relates to an image matching system, an image matching method, and an image matching program using a three-dimensional object model, and more particularly to input of an object (person's face) photographed under different conditions of posture and lighting.
• the present invention relates to an image matching system, an image matching method, and an image matching program that can search for a reference image in a database (DB) after the input image.
• DB database
• the image matching system of the first prior art includes an image input unit 10, an image matching unit 40, a result display unit 80, a reference image storage unit 70, , A reference image registration unit 75.
• the conventional image matching system having such a configuration operates as follows.
• reference images such as reference face images of persons
• the reference image registration unit 75 reference images (such as reference face images of persons) of various objects shot by the reference image registration unit 75 are stored in advance.
• the image input unit 10 is realized by a camera or the like, and stores a captured input image in a memory (not shown).
• the image matching unit 40 compares the input image obtained from the image input unit 10 with each reference image obtained from the reference image storage unit 70, The similarity (or distance value) is calculated, and the reference image with the largest similarity (or the smallest distance) for each object is selected.
• Each image is represented by a gray-scale feature. For example, normalized correlation and Euclidean distance are used for calculating the similarity between features and calculating the distance value.
• the result display unit 80 displays a reference image of the selected object having the highest similarity among the reference images as a comparison result (or displays candidate reference images in descending order of the similarity).
• a conventional image matching system is described in Japanese Patent Application Laid-Open No. 2000-322577 (hereinafter, referred to as a second conventional technology). As shown in FIG. 28, this conventional image matching system includes an image input unit 10, an image conversion unit 35, a partial image matching unit 45, a result display unit 80, and a reference image storage unit.
• the conventional image matching system having such a configuration operates as follows.
• the representative three-dimensional object model storage unit 20 stores one or more representative three-dimensional object models obtained from the three-dimensional object model registration unit 25 in advance.
• the image conversion unit 35 is obtained from the representative three-dimensional object model storage unit 20 with respect to a common partial region of the input image obtained from the image input unit 10 and each reference image obtained from the reference image storage unit 70.
• a partial image is generated by transforming the input image and / or the reference image so that the input conditions (for example, posture conditions) are the same.
• a partial area is a characteristic part such as an eye or a nose or mouth, as shown in Fig. 29, for example, and feature points are specified in advance for each image and 3D object model. Can respond.
• the partial image matching unit 45 compares the converted input image obtained from the image conversion unit 35 with the partial image of each reference image, calculates the average similarity, and determines the reference having the highest similarity for each object. Select an image.
• the result display unit 80 displays an object having the highest similarity of the reference image as a comparison result.
• the input image or the reference image is transformed so that the posture matches, and the comparison is performed.
• the number of reference images of the three-dimensional object model is not sufficient or the posture is largely different,
• the distortion due to the conversion became large and the matching could not be performed correctly.
• it is very difficult to adjust the lighting conditions by the conversion and there is a problem that a common area must exist for comparing images in a common area.
• each conventional technique has a problem that it takes a long time to perform the matching.
• An object of the present invention is to provide an image matching system capable of retrieving a reference image registered in a database from an input image with respect to an image photographed under different conditions of posture and illumination for each object even when only a small number of reference images exist.
• An object of the present invention is to provide a system, an image matching method, and an image matching program.
• Another object of the present invention is to perform collation with a small number of reference images of a three-dimensional object model without performing processing such as transforming an input image or a reference image so that the orientation matches, and to perform an area common to each image. It is an object of the present invention to provide an image collating system, an image collating method, and an image collating program, which can perform collation even if they do not necessarily exist.
• Another object of the present invention is to provide an image matching system, an image matching method, and an image matching program capable of matching images without generating a necessary number of three-dimensional object models for all objects. It is in.
• Still another object of the present invention is to provide an image matching system, an image matching method, and an image matching method that can search at high speed even when reference images related to many objects are registered in a database.
• An image matching program is provided.
• an image matching system that searches for a reference image similar to an input image, a unit that matches the input image with a plurality of representative 3D object models; Means for matching a three-dimensional object model, a result of matching the input image with the plurality of representative three-dimensional object models, and a result of matching the reference image and the plurality of representative three-dimensional object models. And means for searching for the reference image similar to the input image.
• the above-mentioned image matching system comprising: means for obtaining a reference three-dimensional object model corresponding to the reference image similar to the input image; and using the reference three-dimensional object model and the input image to resemble the input image. Means for newly searching for the reference image to be performed.
• the image collating system includes: means for obtaining a reference three-dimensional object model corresponding to the reference image similar to the input image; and, based on the reference three-dimensional object model, at least one of the input image and the reference image.
• a conversion unit that converts the input image and the reference image into input conditions by converting the input image and the reference image, and compares the input image with the input conditions and the reference image to obtain the reference image corresponding to the input image.
• means for retrieving images is provided.
• the conversion unit may convert the reference image in advance and match an input condition of the input image with an input condition of the reference image.
• the image collating system includes: an image input unit that inputs the input image; a representative three-dimensional object model storage unit that stores a plurality of the representative three-dimensional object models; and a representative three-dimensional object model storage unit.
• Image generating means for generating at least one comparative image having a similar input condition to the input image for each representative 3D object model, based on the plurality of the representative 3D object models, the input image, and the image
• the similarity between each comparative image generated by the generating means is calculated, and the maximum similarity is selected for the comparative image corresponding to each representative 3D object model, and the maximum similarity is calculated with the input image and the input image.
• Image matching means for determining the degree of similarity with the representative 3D object model; a reference image storage unit for storing the reference image of each object; and each reference stored in the reference image storage unit
• a result collating unit for extracting the reference image similar to the input image.
• the above image matching system is a three-dimensional object model registration unit that registers each representative three-dimensional object model in the representative three-dimensional object model storage unit, and a reference image registration unit that registers each reference image in the reference image storage unit
• a new representative 3D object model is registered in the representative 3D object model storage unit by the 3D object model registration unit, or a new reference to the reference image storage unit by the reference image registration unit.
• the similarity is calculated by the image matching means for the combination of the reference image newly generated by the registration and the representative 3D object model, and the result of the calculation is referred to as the above.
• a reference image comparison result updating means added to the image comparison result storage unit.
• the image matching unit calculates a similarity between the input image and the representative 3D object model for each partial area
• the reference image matching result storage unit includes the reference image storage unit
• the similarity between each reference image stored in the storage unit and each representative three-dimensional object model stored in the representative three-dimensional object model storage unit is stored for each partial region.
• the reference image similar to the input image may be extracted on the basis of the similarity of each of the regions with the three-dimensional object model.
• the result matching unit may determine a similarity between the input image and each representative 3D object model and a similarity between each reference image and each representative 3D object model. And calculating the similarity between the input image and each of the comparative images and the representative three-dimensional object model.
• the image collating system includes: an image input unit that inputs the input image; a representative three-dimensional object model storage unit that stores a plurality of the representative three-dimensional object models; and a representative three-dimensional object model storage unit.
• Image generating means for generating at least one comparison image having a similar input condition to the input image for each representative 3D object model based on the plurality of stored representative 3D object models; and Calculating the similarity between each comparative image generated by the image generating means, selecting the maximum similarity for the comparative image corresponding to each representative 3D object model, and inputting the maximum similarity to the input.
• Image matching means for determining the degree of similarity between the image and the representative 3D object model; a reference image storage unit for storing the reference image of each object; and each reference image stored in the reference image storage unit.
• a reference image matching result storage unit that stores a similarity between each representative 3D object model stored in the representative 3D object model storage unit; and the input image and each representative that are calculated by the image matching unit.
• a result matching unit that extracts the similar reference image, a reference three-dimensional object model storage unit that stores a reference three-dimensional object model corresponding to each reference image stored in the reference image storage unit, and the result matching unit
• the reference 3D object model corresponding to each reference image extracted from the means is obtained from the reference 3D object model storage unit, and the input image and the input are obtained based on the obtained reference 3D object models.
• Second comparison with close conditions Second image generating means for generating at least one image for each reference three-dimensional object model; similarity between the input image and each second comparative image generated by the second image generating means The maximum similarity is selected for the second comparison image corresponding to each reference 3D object model, and the maximum similarity is defined as the second similarity between the input image and the reference 3D object model.
• the image collating means may be provided.
• the above image matching system is a three-dimensional object model registration unit that registers each representative three-dimensional object model in the representative three-dimensional object model storage unit, and a reference image registration unit that registers each reference image in the reference image storage unit
• a new representative 3D object model is registered in the representative 3D object model storage unit by the 3D object model registration unit, or a reference image is stored in the reference image storage unit by the reference image registration unit.
• the similarity is calculated by the image matching means, and the calculation result is stored in the reference image matching result storage unit.
• the 3 + -dimensional object model generating unit includes: a reference image stored in the reference image storage unit; and a representative three-dimensional object stored in the representative three-dimensional object model storage unit.
• the representative 3D object model stored in the representative 3D object model storage unit is synthesized for each partial area based on the similarity of each partial area with the object model, and the reference corresponding to each reference image is obtained.
• a three-dimensional object model may be generated, and the generated reference three-dimensional object model may be registered in the reference three-dimensional object model storage unit.
• the image matching unit calculates a similarity between the input image and the representative 3D object model for each partial area
• the reference image matching result storage unit includes the reference image storage unit
• the similarity between each reference image stored in the storage unit and each representative three-dimensional object model stored in the representative three-dimensional object model storage unit is stored for each partial area
• the result matching unit includes: The similarity of each partial area between the input image calculated by the image matching means and each representative 3D object model, and the reference image and each representative 3 stored in the reference image matching result storage unit.
• the reference image similar to the input image may be extracted based on a part between the two-dimensional object model and the 9D region.
• the result matching unit may determine a similarity between the input image and each representative 3D object model and a similarity between each reference image and each representative 3D object model. Calculating a similarity between the input image and each comparative image based on the candidate ranking of the similarity between each of the comparative images and each of the representative three-dimensional object models. You can weight it.
• the image collating system includes: an image input unit that inputs the input image; a representative three-dimensional object model storage unit that stores a plurality of the representative three-dimensional object models; and a representative three-dimensional object model storage unit.
• Image generating means for generating at least one comparative image having a similar input condition to the input image for each representative 3D object model, based on the plurality of the representative 3D object models, the input image, and the image The similarity between each comparative image generated by the generating means is calculated, and the maximum similarity is selected for the comparative image corresponding to each representative 3D object model, and the maximum similarity is calculated with the input image and the input image.
• Image matching means for determining the degree of similarity with the representative three-dimensional object model; a reference image storage unit for storing the reference image of each object; each reference image stored in the reference image storage unit; Table A reference image matching result storage unit that stores the similarity between each representative 3D object model stored in the 3D object model storage unit, and the input image calculated by the image matching unit and Based on the similarity between the representative 3D object model and the similarity between each reference image stored in the reference image matching result storage unit and each representative 3D object model, A result comparing unit that extracts the similar reference image, a reference three-dimensional object model storage unit that stores a reference three-dimensional object model corresponding to each reference image stored in the reference image storage unit, and the result matching unit.
• Each reference 3D object model corresponding to each reference image extracted from the reference 3D object model storage unit is obtained from the reference 3D object model storage unit, and based on the obtained reference 3D object models, the input image and the By means of result matching
• the input conditions of the input image and the reference image extracted by the result matching unit are aligned, and the input image and the reference image of which the input conditions are aligned are aligned.
• the above image matching system is a three-dimensional object model registration unit that registers each representative three-dimensional object model in the representative three-dimensional object model storage unit, and a reference image registration unit that registers each reference image in the reference image storage unit
• a new representative 3D object model is registered in the representative 3D object model storage unit by the 3D object model registration means
• the image matching unit compares the similarity with the combination of the reference image newly generated by the registration and the representative 3D object model.
• a reference image comparison result updating unit that stores the calculation result in the reference image comparison result storage unit.
• the reference image comparison result storage unit stores the reference image in the reference image comparison result storage unit.
• the reference 3D object model corresponding to the reference image is stored in the representative 3D object model storage unit based on the similarity.
• the representative 3D object model is generated by synthesizing the reference 3D object model, and the generated reference 3D object model is registered in the reference 3D object model storage unit. Means may be further provided.
• the three-dimensional object model generating means includes: a reference image stored in the reference image storage unit; and a representative three-dimensional object stored in the representative three-dimensional object model storage unit.
• the representative 3D object model stored in the representative 3D object model storage unit is synthesized for each partial region based on the similarity of each partial region with the model, and the reference 3 corresponding to each reference image is synthesized.
• a three-dimensional object model may be generated, and the generated reference three-dimensional object model may be registered in the reference three-dimensional object model storage unit.
• the image matching unit calculates a similarity between the input image and the representative 3D object model / let for each partial area
• the reference image matching result storage unit stores the reference image.
• the similarity between each reference image stored in the storage unit and each representative 3D object model stored in the representative 3D object model storage unit is stored for each partial area
• the result matching unit Is the similarity for each partial region between the input image calculated by the image matching means and each representative 3D object model, and each reference image stored in the reference image matching result storage unit.
• the reference image similar to the input image may be extracted based on the similarity of each region with the representative three-dimensional object model.
• the result matching unit may determine a similarity between the input image and each representative 3D object model and a similarity between each reference image and each representative 3D object model. Calculating the similarity between the input images, Each similarity may be weighted based on the degree of similarity between the comparative image and each representative 3D object model.
• the object may be a human face. According to the present invention, the following effects are achieved.
• the first effect is that even when there is only one or a small number of reference images for each object, search for the same object's reference image is performed for input images of objects captured under different input conditions such as posture and lighting conditions. What you can do.
• image matching can be performed without necessarily generating a predetermined number of 3D object models for all objects.
• the reference image is retrieved by comparing the matching result between the input image and the representative 3D object model and the matching result between the reference image and the representative 3D object model.
• the reference 3D object model is generated by synthesizing the representative 3D object model, and is collated.
• the second effect is that a reference image can be searched at high speed from an input image.
• FIG. 1 is a block diagram showing a configuration of an image matching system according to a first embodiment of the present invention.
• FIG. 2 is a flowchart showing an operation at the time of matching in the first embodiment.
• FIG. 3 is a diagram showing a specific example of a representative three-dimensional object model according to the first embodiment. It is.
• FIG. 4 is a diagram showing a specific example of a reference image according to the first embodiment.
• FIG. 5 is a diagram showing a specific example of a reference image comparison result according to the first embodiment.
• FIG. 6 is a diagram showing a specific example of an input image according to the first embodiment.
• FIG. 7 is a diagram showing a specific example of an input image collation result in the first embodiment.
• FIG. 8 is a diagram showing a specific example of the result matching in the first embodiment.
• FIG. 9 is a block diagram showing the configuration of the image matching system according to the second embodiment of the present invention.
• FIG. 10 is a flowchart showing the operation of registering a three-dimensional object model in the second embodiment.
• FIG. 11 is a flowchart showing an operation at the time of reference image registration in the second embodiment.
• FIG. 12 is a diagram showing a specific example of a result of matching a registered three-dimensional object model according to the second embodiment.
• FIG. 13 is a diagram showing a specific example of updating the reference image matching result in the second embodiment.
• FIG. 14 is a diagram showing a specific example of the result of matching registered reference images according to the second embodiment.
• FIG. 15 is a diagram showing a specific example of updating the reference image comparison result according to the second embodiment.
• FIG. 16 is a block diagram showing a configuration of an image matching system according to the third embodiment of the present invention.
• FIG. 17 is a flowchart showing the operation at the time of matching in the third embodiment.
• FIG. 18 is a diagram showing a specific example of a reference three-dimensional object model according to the third embodiment.
• FIG. 19 is a diagram showing a specific example of a reference image comparison result according to the third embodiment.
• FIG. 20 shows a configuration of an image matching system according to the fourth embodiment of the present invention. It is a block diagram.
• FIG. 21 is a flowchart showing an operation of registering a three-dimensional object model according to the fourth embodiment.
• FIG. 22 is a flowchart showing an operation of registering a reference image according to the fourth embodiment.
• FIG. 23 is a block diagram showing the configuration of the image matching system according to the fifth embodiment of the present invention.
• FIG. 24 is a flowchart showing an operation at the time of matching according to the fifth embodiment.
• FIG. 25 is a block diagram showing the configuration of the image matching system according to the sixth embodiment of the present invention.
• FIG. 26 is a block diagram showing a configuration of an image matching system according to the first conventional technique.
• FIG. 27 is a diagram showing a specific example of the coordinates of the three-dimensional object model.
• FIG. 28 is a block diagram showing a configuration of an image matching system according to a second conventional technique.
• FIG. 2.9 is a diagram showing a specific example of a partial area according to the second conventional technique.
• an image matching system includes an image input unit 10, an image generating unit 30, an image matching unit 40, a result matching unit 60, A result display unit 80, a reference image storage unit 70, a representative three-dimensional object model storage unit 20 and a reference image comparison result storage unit 50 are provided.
• the representative three-dimensional object model storage unit 20 stores a representative three-dimensional object model (the three-dimensional shape of the object and the texture of the object surface).
• a three-dimensional shape measuring device described in Japanese Patent Application Laid-Open No. 2000-125925 or a number of cameras described in Japanese Patent Application Laid-Open No. 9-91436 are used. It can be generated by using a device that restores the three-dimensional shape from the multiple images obtained.
• the three-dimensional object model is composed of the shape P Q , y, z) and the texture T Q (R, G, B) in the three-dimensional space (X, y, z) of the object surface. As information.
• Q represents the index of a point on the object surface. For example, it corresponds to the coordinates of a point Q (s, t) obtained by projecting a point on the object surface from the center of gravity to a sphere centered on the center of gravity of the object.
• a learning CG image under various lighting conditions is generated by computer graphics using various 3D object models in advance, and a base image group is obtained by principal component analysis of the learning CG image. Keep it.
• the image generation unit 30 is based on the representative three-dimensional object model obtained from the representative three-dimensional object model storage unit 20, and assuming posture conditions, the input image obtained from the image input unit 10 and the illumination conditions are close. Generate a plurality of comparison images.
• the generation of the comparison image whose illumination conditions are close to the input image is performed by performing coordinate transformation based on a posture condition assuming a base image group obtained in advance, and a linear sum of the coordinate transformed base image is generated in the input image. It can be realized by finding the coefficient of the linear sum by the least squares method so as to be close.
• the image matching unit 40 compares the input image obtained from the image input unit 10 with each comparative image obtained from the image generating unit 30 and calculates the similarity between the input image and each comparative image. Then, the posture is estimated by selecting the comparison image having the highest similarity for each object.
• the reference image comparison result storage unit 50 stores the reference images in the reference image storage unit 70, which is a database (DB) for storing reference images, as input images, using the image generation unit 30 and the image comparison unit 40.
• DB database
• the result of matching each representative 3D object model in the representative 3D object model storage unit 20 with each reference image is stored in advance.
• the result collating unit 60 compares the result of the collation performed by the image generating unit 30 and the image collating unit 40 on the input image obtained from the image input unit 10 and each reference in the reference image collating result storage unit 50.
• the image is compared with the matching result, and reference images with similar matching results are extracted in descending order of similarity.
• the result display section 80 illuminates the object having the highest similarity. The result is displayed.
• the representative three-dimensional object model storage unit 20 stores a plurality of representative three-dimensional object models.
• the input image is obtained by the image input unit 10 (step 100 in FIG. 2).
• the image generation unit 30 compares the input image with the input conditions such as the posture and illumination, that is, the comparison image that is easy to compare with each representative 3D object model in the representative 3D object model storage unit 20. Is generated (step 101).
• the image matching unit 40 obtains a similarity between the input image and each of the comparison images (Step 102).
• the result matching unit 60 calculates the similarity between the matching result and the matching result of each reference image in the reference image matching result storage unit 50, and extracts reference images having similar matching results in descending order of similarity. (Step 103). Finally, the reference image having a high similarity is displayed (step 104).
• the reference image is searched by comparing the matching result between the input image and the representative 3D object model and the matching result between the reference image and the representative 3D object model. Therefore, even when there is only one or a small number of reference images for each object, a reference image can be searched for an input image of an object photographed under different conditions of posture and lighting conditions.
• the search time depends on the number of image matching.
• the representative three-dimensional object model storage unit 20 stores N representative representative three-dimensional object models Cj (j-1, 2,..., N).
• the reference image matching result storage section 50 by the processing at the time of registration of the reference image, the matching result the representative three-dimensional object models C j of each reference image Ri (similarity) S i3 is stored (in FIG. 5, the similarity is displayed in descending order of similarity, but in actuality, it may be stored in the order of the model;).
• an input image I (u, V) as shown in FIG. 6 has been obtained by the image input unit 10 (step 100 in FIG. 2).
• the image matching unit 40 obtains a similarity S (I, G jk ) between the input image I (u, V) and each of the comparative images G jk (u, v), and
• the maximum similarity S oj ma Xk S (1, G jk ) is calculated (step 102 ).
• the matching result (similarity) S 0 j is, for example, as shown in FIG.
• the result matching unit 60 determines the matching result S. j and the matching result S ij of each reference image in the reference image matching result storage unit 50 (S oj , S H ) is calculated, and reference images with high similarity Di of the matching result are sequentially extracted (step 103).
• the extraction result is, for example, as shown in Fig. 8, which is likely to be an image of the same object as the input image.
• R 5 and R 2 are obtained in order as a reference image.
• the reference image having a high similarity is displayed (step 104).
• a normalized correlation, a rank correlation, or the like can be used as a method of calculating the similarity Di (S. j, S n ) of the matching result.
• Rank correlation is the correlation between the catching ranks of the matching results.
• the Spearman's rank correlation for example, can be obtained from 1 6 ⁇ j (A 0j — Aij) V ⁇ N (N 2 -l) ⁇ .
• the similarity may be calculated after converting each variable (S. 3 ;] . ⁇ 0 into a variable.
• one or both of the candidate ranks A oj and A i3 may be used.
• the weight of the upper candidate becomes higher.
• the image matching system includes an image input unit 10, an image generation unit 30, an image matching unit 40, a result matching unit 60, a result display unit 80, and a reference image storage.
• a reference image registration unit 75, a three-dimensional object model registration unit 25, and a reference image comparison result update unit 55 are added to the configuration of the first embodiment.
• the three-dimensional object model registration unit 25 newly registers the representative three-dimensional object model (the three-dimensional shape of the object and the texture of the object surface) in the representative three-dimensional object model storage unit 20.
• the reference image matching result updating unit 55 5 when a representative 3D object model is registered in the representative 3D object model storage unit 20 and when a reference image is registered in the reference image storage unit 70 by the reference image registration unit 75
• the combination of the representative three-dimensional object models is collated by the image generation unit 30 and the image collation unit 40, and the collation result is added to the reference image collation result storage unit 50.
• the reference image registration unit 75 registers a reference image, which is a two-dimensional image of the object to be searched, in the reference image storage unit 70.
• the reference image to be registered has no restrictions on the input conditions including its lighting and posture, and at least one image is registered for one object (search target).
• the 3D object model registration unit 25 is the same as the 3D object model registration unit 25 in the second related art shown in FIG. 28, and is stored in the representative 3D object model storage unit 20. Represents a representative three-dimensional object model obtained from the three-dimensional object model registration unit 25 in advance.
• the operation at the time of collation of the input image is exactly the same as the operation shown in FIG. 2 of the first embodiment.
• the input image is obtained by the image input unit 10 (step 100 in FIG. 2).
• the image generation unit 30 compares the input image with the input conditions such as posture and illumination, which are close to the input image, that is, a comparative image that can be easily compared with each representative 3D object model in the representative 3D object model storage unit 20. Is generated (step 101).
• the image matching unit 40 obtains a similarity between the input image and each of the comparison images (Step 102).
• the result matching unit 60 calculates the similarity between the matching result and the matching result of each reference image in the reference image matching result storage unit 50, and extracts reference images having similar matching results in descending order of similarity. (Step 103). Finally, the reference image having a high similarity is displayed (step 104).
• the 3D object model registration unit 25 When registering the representative 3D object model (the 3D shape of the object and the texture of the object surface), first, the 3D object model registration unit 25 first stores the new representative 3D object in the representative 3D object model storage unit 20. Register the model (step 200 in FIG. 10). Next, the reference image matching result update unit 55 sends each reference image in the reference image storage unit 70 to the image input unit 10 as an input image, and stores the reference image and the registered representative 3D object. The comparison image by the image generation unit 30 based on the model is collated by the image collation unit 40, and each similarity is obtained (step 201). Finally, the matching result is added to the matching result of each reference image in the reference image matching result storage unit 50 (step 202).
• the reference image registration unit 75 registers a new reference image in the reference image storage unit 70 (step 210 in FIG. 11).
• the reference image matching result updating unit 55 sends the reference image registered in the reference image storage unit 70 to the image input unit 10 as an input image, and the reference image and the representative three-dimensional object model / restore.
• the image comparison unit 40 checks the comparison image by the image generation unit 30 based on the representative three-dimensional object model in the storage unit 20, and obtains each similarity (step 211). Finally, the matching result is added to the reference image matching result storage unit 50 (Step 2 1 2) o
• the second embodiment is configured to search for a reference image by comparing a matching result between an input image and a representative 3D object model and a matching result between a reference image and a representative 3D object model. Therefore, even when there is only one or a small number of reference images for each object, a reference image can be searched for an input image of an object photographed under different conditions of posture and lighting conditions.
• the configuration is such that image matching is performed by comparing with a representative three-dimensional object model having a smaller number of objects than the number of objects, and similarity calculation of the matching result is performed, high-speed search can be performed. Since the time required to calculate the similarity of the matching result is shorter than that of image matching, the search time depends on the number of image matching.
• an input image I (u, v) as shown in FIG. 6 is obtained by the image input unit 10 at the time of input image collation (step 100 in FIG. 2).
• the image matching unit 40 obtains a similarity S (1, G jk ) between the input image I (u, V) and each of the comparative images G jk (u, v), and calculates the similarity S (1, G jk ) for each representative three-dimensional object model.
• Request the maximum similarity similarity S 0 j m a x k S (1, G jk) ( step 1 02).
• the collation result (similarity) S oj is, for example, as shown in FIG.
• the reference images with the higher similarity Di are sequentially extracted (step 103). Extraction result, for example, as FIG. 8, an image is a possibly the reference image of the input image and the same object, R 5, R 2 is found in order. Finally, the reference image having a high similarity is displayed (step 104).
• the 51st new representative 3D object model C 5 i is registered (step 200 in FIG. 10).
• the reference image matching result updating unit 55 sends each reference image R i in the reference image storage unit 70 to the image input unit 10 as an input image, and the reference three-dimensional object registered as the reference image R i is registered.
• the matching result (similarity) Si, 51 is, for example, as shown in FIG.
• the matching result is added to the matching result of each reference image in the reference image matching result storage unit 50 (step 202).
• the reference image comparison result update unit 55 updates the reference image R 1 registered in the reference image storage unit 70. i is sent to the image input unit 10 as an input image to obtain the reference image R. i and the representative 3D object model C j in the representative 3D object model storage unit 20 are collated by the image generation unit 30 and the image collation unit 40, and each similarity is compared. (R 101 , G jk ) (Step 21 l) o
• the matching result (similarity) is as shown in Fig. 14, for example. Finally, as shown in FIG. 15, the matching result is added to the reference image matching result storage unit 50 (step 212).
• an image matching system includes an image input unit 10, an image generating unit 30, an image matching unit 40, a result matching unit 60, a second image Generating unit 31, second image matching unit 41, result display unit 80, reference image storage unit 70, representative 3D object model storage unit 20, reference image matching result storage unit 50, and reference 3D An object model storage unit 21 is provided.
• M works as follows.
• the image storage unit 70 and the representative three-dimensional object model storage unit 20 perform the same processing as the processing in the first embodiment of the present invention shown in FIG.
• the reference 3D object model storage corresponding to the reference image is registered in the reference 3D object model storage unit 21.
• the reference three-dimensional object model is synthesized from the representative three-dimensional object model in the representative three-dimensional object model storage unit 20 based on the information of the reference image comparison result registered in the reference image comparison result storage unit 50. Can be generated by Alternatively, as in the case of the registration of the representative 3D object model described above, if the 3D shape measurement device has generated a 3D object model of the same object as the reference image, the 3D object model is used. May be.
• the second image generation unit 31 refers to each reference image corresponding to the reference image obtained from the reference three-dimensional object model storage unit 21 with respect to the reference image of the top candidate of the matching result obtained from the result matching unit 60. Based on the three-dimensional object model, a comparative image in which the input image obtained from the image input unit 10 is close to the input conditions such as the posture and the lighting conditions is generated.
• the second image matching unit 41 compares the input image obtained from the image input unit 10 with each comparison image obtained from the second image generation unit 31 and calculates the similarity.
• steps 100, 101, 102, and 103 are performed in the first embodiment shown in FIG. It is the same as the operation in the form.
• the second image generation unit 31 refers to each reference image corresponding to the reference image obtained from the reference three-dimensional object model storage unit 21 with respect to the reference image of the top candidate of the matching result obtained from the result matching unit 60.
• a comparative image in which the input image obtained from the image input unit 10 is close to the input conditions such as posture and lighting conditions is generated (step 111).
• the second image collating unit 41 compares the input image obtained from the image input unit 10 with each comparative image obtained from the second image generating unit 31 and calculates the similarity.
• Step 1 1 2 the reference image having a high degree of similarity is displayed (step 104).
• the reference 3D object model generated by combining the representative 3D object models is configured to be collated, even when only one reference image exists for each object, the reference A reference image can be retrieved from an input image of an object captured under different input conditions such as posture and lighting conditions using a three-dimensional object model. Further, in the present embodiment, further, a reference image having high similarity is extracted from the representative three-dimensional object model, and then the upper candidate is compared with the reference three-dimensional object model. As a result, reference images can be searched at high speed.
• the matching result (similarity) of each reference image 1 ⁇ with respect to the representative 3D object model Cj is stored.
• the second image generation unit 31 uses the reference three-dimensional object model storage unit 21 based on the comparison result obtained by the result comparison unit 60, for example, with respect to the reference images of the top three R 1 R 5 and R 2.
• the generation of the comparison image H jk (u, v) is performed in the same manner as in step S101 .
• each reference 3D object model B i (j 1,
• the second image matching unit 41 obtains the similarity S (1, H jk ) between the input image I (u, v) and each of the comparative images H jk (u, v), and calculates the similarity for each model.
• the maximum similarity S 0j maXk S (1, H jk ) is determined (step 112).
• the matching result is, for example, as shown in Fig. 19.
• S 05 > S 01 > S 02 , R 5 , R, and R 2 are reference images that are highly likely to be images of the same object as the input image. In order. Finally, the reference image having a high similarity is displayed (step 104).
• an image matching system according to a fourth embodiment of the present invention will be described in detail with reference to FIG.
• the image collating system includes an image input unit 10, an image generating unit 30, an image collating unit 40, a result collating unit 60, a second Image generation unit 31, second image comparison unit 41, result display unit 80, reference image storage unit 70, reference image registration unit 75, representative 3D object model storage unit 20, and 3D object model It includes a registration unit 25, a reference image comparison result storage unit 50, a reference image comparison result update unit 55, a reference three-dimensional object model storage unit 21, and a three-dimensional object model generation unit 27.
• a reference image registration unit 75 a three-dimensional object model registration unit 25, a reference image comparison result update unit 55, and a three-dimensional object model generation are added to the configuration of the third embodiment. Part 27 is added.
• the model storage unit 20, the three-dimensional object model registration unit 25, and the reference image matching result update unit 55 include the first embodiment of this effort shown in FIG. 1 and the second embodiment shown in FIG. The same processing as the processing in the embodiment is performed. Also, the reference three-dimensional object model storage unit 21, the second image generation unit 31, and the second image comparison unit 41 are the same as the processes in the third embodiment shown in FIG. Perform processing.
• the three-dimensional object model generation unit 27 uses the information of the reference image comparison result. Then, a reference three-dimensional object model corresponding to the reference image is generated by combining the representative three-dimensional object model in the representative three-dimensional object model storage unit 20, and the reference three-dimensional object model is referred to. 21 Registered or referenced in 1 3D object model storage unit 21 Updates the referenced 3D object model in 1.
• the second image generation unit 31 refers to each reference image corresponding to the reference image obtained from the reference three-dimensional object model storage unit 21 with respect to the reference image of the top candidate of the matching result obtained from the result matching unit 60. Based on the three-dimensional object model, a comparative image in which the input image obtained from the image input unit 10 is close to the input conditions such as the posture and the lighting conditions is generated.
• the second image matching unit 41 compares the input image obtained from the image input unit 10 with each comparison image obtained from the second image generation unit 31 and calculates the similarity. Next, the overall operation of the fourth embodiment will be described in detail with reference to FIG. 20 and the flowcharts of FIGS. 17, 21, and 22.
• steps 100, 101, 102, and 103 are performed in the first embodiment shown in FIG. It is the same as the operation in the form.
• the second image generation unit 31 sends a reference image corresponding to the reference image obtained from the reference three-dimensional object model storage unit 21 to the reference image of the higher candidate of the matching result obtained by the result matching unit 60.
• a comparison image in which the input image obtained from the image input unit 10 is close to the input conditions such as the posture and the lighting conditions is generated (step 111).
• the second image collating unit 41 compares the input image obtained from the image input unit 10 with each comparative image obtained from the second image generating unit 31 and calculates the similarity. Step 1 1 2).
• the reference image having a high degree of similarity is displayed (step 104).
• Steps 200, 201, and 202 correspond to the second embodiment shown in FIG. In Operation is the same.
• the 3D object model generation unit 27 represents the reference 3D object model corresponding to each reference image based on the information of each reference image comparison result in the reference image comparison result storage unit 50.
• the reference 3D object model is regenerated by combining the representative 3D object models in the model storage unit 20, and the reference 3D object model is referenced.
• the reference 3D object model registered or stored in the 3D object model storage unit 21 is replaced. Yes (Step 220).
• steps 210, 211, and 212 are performed according to the operation in the second embodiment shown in FIG. Same.
• the 3D object model generation unit 27 represents the reference 3D object model corresponding to the reference image based on the information of the reference image comparison result newly registered in the reference image comparison result storage unit 50.
• the reference three-dimensional object model is generated by synthesizing the representative three-dimensional object model in the three-dimensional object model storage unit 20 and the reference three-dimensional object model is additionally registered in the reference three-dimensional object model storage unit 21 (step 2 21).
• the reference three-dimensional object model is generated and synthesized by combining the representative three-dimensional object models, so that even when there is only one reference image for each object, A reference image can be searched for from an input image of an object taken under different input conditions such as posture and lighting conditions using a reference 3D object model. Further, in the present embodiment, after extracting a reference image having a high similarity based on the representative 3D object model / relation, the upper candidate is compared with the reference 3D object model. , So that reference images can be searched at high speed.
• the reference three-dimensional object model storage unit 21 stores a reference image.
• M reference 3D object models B i (i 1, 2,..., M) corresponding to the reference image R; are stored.
• the second image generator 31, the comparison result obtained from the result matching section 60, R have R 5 is, for example, the top three candidates.
• the reference three-dimensional object models SL ⁇ 21 each reference three-dimensional object model BB 5 to more corresponding, B 2 acquires an image input unit compares the image H 3 near the input image and pose and lighting conditions obtained from 10 k (u, v)
• the second image matching unit 41 calculates the similarity S (1, H jk ) between the input image I (u, v) and each of the comparative images H jk (u, v), and calculates the similarity S for each model.
• the maximum similarity S 0j max k S (1, H jk ) is obtained (step 1 12).
• the matching result is, for example, as shown in FIG. 19, and R 5 , R t , and R 2 are sequentially obtained as reference images that are likely to be images of the same object as the input image. Finally, the reference image having a high similarity is displayed (step 104).
• the reference image matching result update section 5 matching result of each reference image in the reference image matching result storage section 50 to update the S j i (step 20 1, 202).
• the representative 3D object model Cj is regenerated by synthesis, and the reference 3D object model B i is replaced with the reference 3D object model already stored in the reference 3D object model storage unit 21 (step 220). .
• T Qi (R, G, B) ⁇ jf (ST Qj (R, G, B)
• f is monotonically increasing with increasing S i 3
• the reference image registration unit 75 sets the first new reference image to register R 101 (step 2 1 0 of Fig. 22).
• the reference image matching result updating unit 55 causes the reference image R! .
• Corresponding to ⁇ is added to the reference image matching result storage unit 50 (steps 211, 212).
• the three-dimensional object model generation unit 27 generates the reference image R based on the information of the reference image comparison result S101ij in the reference image comparison result storage unit 50 .
• Reference corresponding to i 3D object model B! Is generated by combining the representative three-dimensional object model Cj in the representative three-dimensional object model storage unit 20, and the reference three-dimensional object model B 1 is generated. Is added to the 3D object model storage unit 21 (step 221).
• the image collating system includes an image input unit 10, an image generating unit 30, an image collating unit 40, a result collating unit 60, and an image converting unit 36.
• Image input unit 10 image generation unit 30, image comparison unit 40, result comparison unit 60, result display unit 80, reference image storage unit 70, representative 3D object model
• the storage unit 20 performs the same processing as the processing in the first embodiment of the present invention shown in FIG.
• the image conversion unit 36 generates a reference image corresponding to the reference image obtained from the reference three-dimensional object model storage unit 21 with respect to the reference image of the higher rank of the matching result obtained from the result matching unit 60. Based on the two-dimensional object model, a partial image is generated by transforming the input image and / or the reference image so that the input conditions (for example, posture conditions) are the same.
• the image conversion unit 36 is similar to the second conventional image conversion unit 35 shown in FIG.
• the partial image matching unit 45 compares the converted input image obtained by the image converting unit 36 with the partial image of the reference image, and calculates the similarity between the two. The calculation of the similarity is performed in the same manner as in step S102 described above.
• steps 100, 101, 102, and 103 are performed in the first embodiment shown in FIG. It is the same as the operation in the form.
• the image conversion unit 36 generates a reference three-dimensional image corresponding to the reference image obtained from the reference three-dimensional object model storage unit 21 with respect to the reference image of the higher candidate of the matching result obtained by the result matching unit 60. Based on the object model, a partial image is generated by converting the input image and / or the reference image so that input conditions (for example, posture conditions) are the same. (Steps 1 2 1).
• the partial image matching unit 45 compares the converted input image obtained by the image converting unit 36 with the partial image of the reference image, and calculates the respective iJ i degrees (step 122). Finally, the reference image having a high degree of similarity is displayed (step 104).
• the image conversion unit 36 converts one or any of the input image and the reference image.
• the reference image is converted in advance into standard input conditions (for example, posture Condition), and store the input image in the image conversion unit 36. May be converted into standard input conditions (for example, posture conditions). By doing so, it is not necessary to convert the reference image every time collation is performed, and the collation time can be reduced.
• the image collating system comprises an image input unit 10, an image generating unit 30, an image collating unit 40, and a result collating unit 60.
• Image input unit 10 Image generation unit 30, Image comparison unit 40, Result comparison unit 60, Result display unit 80, Reference image storage unit 70, Reference image registration unit 7 5, a representative three-dimensional object model storage unit 20, a three-dimensional object model registration unit 25, and a reference image matching result update unit 55 are the same as those of the first embodiment of the present invention shown in FIG.
• Reference image storage unit 70 Reference image registration unit 7 5
• a representative three-dimensional object model storage unit 20 a three-dimensional object model registration unit 25, and a reference image matching result update unit 55
• the same processing as the processing in the second embodiment shown in FIG. 9 is performed.
• reference three-dimensional object model storage unit 21 and the three-dimensional object model / record generation unit 27 correspond to the third embodiment shown in FIG. 16 and the third embodiment of the present invention shown in FIG. The same processing as the processing in the fourth embodiment is performed.
• the image conversion unit 36 generates a reference 3D image corresponding to the reference image obtained from the reference 3D object model storage unit 21 with respect to the reference image of the higher candidate of the matching result obtained from the result matching unit 60. Based on the object model, a partial image is generated by transforming the input image and / or the reference image so that input conditions (for example, posture conditions) are the same.
• the partial image matching unit 45 compares the converted input image obtained by the image converting unit 36 with the partial image of the reference image, and calculates the similarity.
• first, steps 100, 101, 102, and 103 are the same as the operations in the first embodiment shown in FIG.
• the image conversion unit 36 generates a reference 3D object model corresponding to the reference image obtained from the reference 3D object model storage unit 21 with respect to the reference image of the higher candidate of the matching result obtained by the result matching unit 60. Based on, the input image and / or the reference image are converted to generate a partial image so that the input conditions (for example, posture conditions) are the same. (Step 121).
• the partial image matching unit 45 compares the converted input image obtained by the image converting unit 36 with the partial image of the reference image, and calculates the similarity (step 122). Finally, the reference image having a high similarity is displayed (step 104).
• the image conversion unit 36 converts both or one of the input image and the reference image.
• the reference image is converted into standard input conditions (for example, posture conditions) in advance.
• the input image may be converted into standard input conditions (for example, posture conditions) in the image conversion unit 36.
• the similarity S (1, G jk ) between the input image I (u, v) and each comparison image G jk (u, v) when seeking, although total seek one similarity S (1, G ik) we obtain the similarity S (1, G 'jkm) for each partial region m, the maximum similarity for each model
• the similarity S ′ 0jB ni aXk S (1, G, j km ) may be obtained.
• the partial area is, for example, an area as shown in FIG.
• the result matching unit 60 ⁇ .Similarity D ⁇ SJD ( S'0jm , S'i .) Between m and the reference result S 'of each reference image in the reference image comparison result storage unit 50; Also, the 3D object model generation unit 27 in the fourth and sixth embodiments may synthesize a representative 3D object model for each partial region.
• the threshold value for determining the similarity between the input image and the specific reference image in the second image matching unit 41 and the partial image matching unit 45 is used. It can be determined by whether it is greater than.
• the image matching system includes an image matching program 500 for realizing the functions of the components described above in a computer, as well as realizing the functions of the components as hardware. It can also be realized by reading and executing.
• the image collation program 500 is stored on a magnetic disk, a semiconductor memory, or another recording medium, and the computer reads the image collation program 500 from the recording medium.
• INDUSTRIAL APPLICABILITY The present invention can be used for identification of a person using an image of a face or the like, personal authentication, and the like.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Human Computer Interaction (AREA)
• Health & Medical Sciences (AREA)
• Oral & Maxillofacial Surgery (AREA)
• Multimedia (AREA)
• General Health & Medical Sciences (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Image Analysis (AREA)
• Processing Or Creating Images (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• Image Processing (AREA)
• Apparatus For Radiation Diagnosis (AREA)
• Length Measuring Devices By Optical Means (AREA)

Priority Applications (9)

Applications Claiming Priority (2)

Related Child Applications (2)

Publications (1)

Family

ID=30112549

Family Applications (1)

Country Status (10)

Cited By (4)

Families Citing this family (30)

Citations (2)

Family Cites Families (25)

• 2003
  • 2003-07-08 JP JP2004521151A patent/JP4560832B2/ja not_active Expired - Lifetime
  • 2003-07-08 CN CN03816308XA patent/CN1669052B/zh not_active Expired - Lifetime
  • 2003-07-08 AT AT03741264T patent/ATE374978T1/de not_active IP Right Cessation
  • 2003-07-08 WO PCT/JP2003/008642 patent/WO2004008392A1/ja active IP Right Grant
  • 2003-07-08 CA CA002491727A patent/CA2491727A1/en not_active Abandoned
  • 2003-07-08 EP EP03741264A patent/EP1530157B1/de not_active Expired - Lifetime
  • 2003-07-08 KR KR1020057000443A patent/KR100651010B1/ko not_active IP Right Cessation
  • 2003-07-08 US US10/520,661 patent/US7545973B2/en active Active
  • 2003-07-08 DE DE60316690T patent/DE60316690T2/de not_active Expired - Lifetime
  • 2003-07-08 AU AU2003281007A patent/AU2003281007B2/en not_active Ceased
• 2009
  • 2009-05-01 US US12/433,951 patent/US7873208B2/en not_active Expired - Lifetime

Patent Citations (2)

Also Published As

Similar Documents

Legal Events