WO2013125494A1

WO2013125494A1 - Image verification device, image verification method, and program

Info

Publication number: WO2013125494A1
Application number: PCT/JP2013/053896
Authority: WO
Inventors: 達勇秋山
Original assignee: 日本電気株式会社
Priority date: 2012-02-24
Filing date: 2013-02-18
Publication date: 2013-08-29

Abstract

[Problem] To make possible the detection of the position of each of a plurality of the same partial image among search images and to increase the versatility of a device. [Solution] An image verification device comprising: a geometric transformation parameter estimation unit (16) that estimates geometric transformation parameters for a search image (QI) relative to a recorded image (RI); a partial region calculation unit (14) that utilizes the estimated geometric transformation parameters and calculates a plurality of partial regions inside the search image (QI); a partial region characteristics value calculation unit (18) that calculates the characteristics of each partial search image (QP) specified by the plurality of partial regions in the search image (QI); a partial image verification unit (20) that determines matches between the partial search images (QP) and a partial recorded image (RP), by comparing the characteristics of each of the plurality of partial search images (QP) to the characteristics of the partial recorded image (RP) specified by the partial region in the recorded image (RI); and an image position detection unit (22) that calculates the position of the partial search image (QP) matched using the geometric transformation parameters.

Description

Image collation apparatus, image collation method and program

The present invention relates to an image matching device, an image matching method, and a program thereof, and in particular, an image matching device capable of detecting even a plurality of registered images to be detected among search images to be matched, The present invention relates to an image matching method and a program thereof.

Image collation is a technique for comparing the feature points of the first image and the feature points of the second image to determine whether the degree of similarity or coincidence is high. By this image collation, an image similar to a registered image (or sample image) registered in advance can be searched from a search image (query image) group. In such feature point comparison, even if there is a geometric transformation between images, the same or similar images can be obtained even if there are differences in shooting angles in three dimensions, etc. You can search.

In relation to this, Patent Documents 1 to 6 shown below are known as examples of an image collating apparatus.
Among these, the image collating apparatus 501 disclosed in Patent Document 1 is a projection affine in which the image collating module 501A, which is the main part, first estimates the affine parameters for registration of all registered images, as shown in FIG. A parameter estimation module 250, a partial area projection module 262 that projects a partial area on the registered image RI onto the search image QI using the estimated affine parameters, and associates the partial area with each other, and a partial area projected on the search image QI A projection image matching module 264 that obtains a feature amount and compares each partial area of the registered image RI and the search image QI to determine the match / mismatch, and the number of times that the projection image matching module 264 determines a match for each registered image The image matching result calculation module 29 for identifying the registered image RI with respect to the search image QI using And it is configured to include and.

The image collation apparatus 501 having such a configuration operates to perform registration processing and search processing.
Among these, in the registration process, after calculating the arrangement of the feature points of the registered image RI and the feature vector of the registered image RI, the partial region feature amount of the registered image RI is calculated.

In the search process, first, projection affine parameters are estimated for all registered images RI, a partial area of the registered image RI is projected onto the search image QI, a feature amount in the projected image partial area is calculated, A check is made for each area to determine whether the partial areas match. Thereafter, a final image matching result is obtained.

Here, when there are a plurality of affine parameters used for determining the projection image partial area, the projection image partial area is determined for each of the plurality of affine parameters. At this time, it is determined that the partial area of the registered image RI matches the partial area of the projection image using any one affine parameter.

When one affine parameter is used to determine the projected image partial area, the projected image partial area is determined for each of a plurality of single affine parameters, and the partial areas are verified based on this. Is called. Note that the above-described affine parameters can be replaced with arbitrary geometric transformation parameters such as a homography matrix representing a correspondence between a plane and a plane.

Patent Document 2 discloses a technique of using features having arbitrary invariance with respect to scale, affine transformation, and perspective distortion for the purpose of searching a partial image of a search image. As the perspective transformation, homography transformation is exemplified.
Furthermore, in Patent Document 3, for the purpose of searching for Japanese in an image, a method is adopted in which part or all of characters are adjusted to be connected components, and the center of gravity of the connected components is obtained as a feature point. Is disclosed.

Patent Document 4 discloses a technique for preventing a registration image and a search image from being erroneously determined to match with a certain condition of conformity.
Further, Patent Document 4 discloses an image collation technique that can collate with high accuracy even when there is a geometric transformation such as affine transformation between one and the other images to be compared.

Further, Patent Document 5 specifies candidate images by extracting local features in order to prevent a decrease in accuracy when the number of local feature points is insufficient while enabling search of images that have been rotated or cut out in the search image. A method for searching for an image by obtaining a geometric transformation parameter that minimizes the distance between a sample feature point and a query feature point, comparing the overall features, and calculating the overall similarity between the local feature and the overall feature feature is disclosed. ing.
This Patent Document 5 discloses a method for obtaining geometric transformation parameters by repeatedly evaluating and voting errors of parameters extracted at random by RANSAC (RANdom SAmple Consensus).

International Publication No. 2010 / 053109A1 Special table 2010-530998 JP 2009-32109 A International Publication No. 2009/060975 JP 2010-266964 A

However, in the above-mentioned Patent Documents 1 to 5 and related technologies combining these, when a plurality of the same registered images are included in the search image, only a maximum of one can be detected. There has been a disadvantage that the position or number of each cannot be detected with high accuracy.

[Object of the invention]
An object of the present invention is to enable detection of a plurality of images when a plurality of the same registered images are included in a search image, thereby expanding the versatility of the apparatus, and an image verification method And providing the program.

In order to achieve the above object, an image collation apparatus according to the present invention includes a storage unit that stores a registered image having a predetermined feature point, and an image processing unit that compares the registered image with a search image to be collated. Image collating device,
The image processing means estimates a geometric transformation parameter based on each feature point and the arrangement of the registered image and the search image, and the geometric transformation parameter estimated by the geometric transformation parameter estimation unit. A partial area determining unit that determines a projected partial image area in the search image corresponding to each partial area image in the registered image by using the plurality of projected partial image areas and each part of the registered image in the search image; A partial region feature amount calculation unit that calculates each feature amount of the region image, and compares the feature amounts of the plurality of partial search images with the feature amount of the partial registration image specified in the partial region of the registered image, respectively. A partial image collating unit for determining a match between the partial search image and the partial registered image, and using the geometric transformation parameter, Characterized by comprising the registered image position detecting unit to detect and calculate the location, the.

In order to achieve the above object, an image matching method according to the present invention includes a storage unit that stores a registered image having a predetermined feature amount, and an image processing unit that compares the registered image with a search image to be compared. In the image verification device,
Based on each feature point and the arrangement of the registered image and the search image, a geometric transformation parameter estimation unit of the image processing means estimates a geometric transformation parameter, and uses the estimated geometric transformation parameter to register a registered image. A plurality of projection partial image areas in the search image corresponding to each of the partial image areas of the image processing means are determined by the partial area determination unit of the image processing means, the plurality of projection partial image areas of the search image and the registered image Each feature quantity of each partial area image is calculated by a partial area feature quantity calculation unit of the image processing means, and the plurality of feature quantities corresponding to the feature quantities of the partial registration image specified in the partial area of the registered image The partial image matching unit of the image processing unit compares the feature amount of the partial search image to determine whether the partial search image matches the partial registered image, and When it is determined that the partial search image and the partial registration image match, the image position detection unit of the image processing means determines the partial search image based on a geometric transformation parameter of the search image with respect to the registration image. The position is calculated and detected.

In order to achieve the above object, an image collation program according to the present invention includes a storage unit that stores a registered image having a predetermined feature amount, and an image processing unit that compares the registered image with a search image to be collated. In the image collation device provided,
A geometric transformation parameter estimation procedure for estimating a geometric transformation parameter based on each feature point of the registered image and the search image and its arrangement;
A partial region determination procedure for determining a plurality of projection partial image regions in a search image corresponding to each partial region image in a registered image using the estimated geometric transformation parameter;
A feature amount calculating procedure for calculating each feature amount of each of the plurality of projection partial image regions of the search image and each of the partial region images of the registered image;
By comparing the feature quantities of the plurality of partial search images corresponding to the feature quantities of the partial registration image specified in the partial area of the registration image, the partial search image and the partial registration image are compared. Partial image matching procedure to determine match,
And an image position for calculating and detecting the position of the matched partial search image using the geometric transformation parameter when it is determined in the partial image matching procedure that the partial search image and the partial registered image match. A detection procedure,
These are executed by a computer provided in the image processing means.

Since the present invention is configured and functions as described above, according to this, the geometric transformation parameter estimation unit estimates the geometric transformation parameters of the search image with respect to the registered image, and the partial image matching unit searches for the feature points and the partial search of the partial registration image. Since it is configured to determine whether or not there is a match by comparing the feature points of the image, and the position calculation unit calculates and determines the position of the matched partial search image using the geometric transformation parameter, a plurality of search images Even when there is a partial search image, the position can be calculated individually, and thereby the versatility of the apparatus can be expanded, and the excellent image matching apparatus, the image matching method, which are not in the related technology described above, And the program can be provided.

It is a block diagram which shows the structure of 1st Embodiment of this invention. It is a block diagram which shows an example of the basic composition of 1st Embodiment. It is a block diagram which shows the structural example of the registration image position detection part disclosed by 1st Embodiment. It is a flowchart which shows the procedure of the image collation process disclosed by 1st Embodiment. It is a flowchart which shows the procedure of the position specific process of the partial search image disclosed by 1st Embodiment. It is explanatory drawing which shows an example of the image collation result by the image processing means disclosed in 1st Embodiment. It is a block diagram which shows the structural example of 2nd Embodiment of this invention. It is a flowchart which shows the procedure of the image collation process in 2nd Embodiment. It is a block diagram which shows the specific example in the case of using a homography matrix in the geometric transformation parameter estimation part disclosed by 2nd Embodiment. It is explanatory drawing which shows an example in the case of calculating the reliability at the time of making the two homography matrix disclosed in FIG. 9 into a geometric transformation parameter. FIG. 11 is an explanatory diagram illustrating an example of a table used when calculating the reliability of a geometric transformation parameter in relation to FIG. 10. It is a block diagram which shows the structural example of 3rd Embodiment of this invention. It is a block diagram which shows the detailed structural example of the integrated process part disclosed by 3rd Embodiment. It is explanatory drawing which shows an example of the integration process using the union or the reliability in FIG. It is explanatory drawing which shows an example of the integration process by vote in relation to FIG. It is a flowchart which shows the procedure of the image collation process in 3rd Embodiment. It is a flowchart which shows an example of the image position calculation process of the partial search image in 3rd Embodiment. It is explanatory drawing which shows an example in the case of carrying out display control of the similarity as display data of 3rd Embodiment. It is explanatory drawing which shows an example in the case of carrying out display control of text data as display data of 3rd Embodiment. It is explanatory drawing which shows an example of the hardware resource which comprises the background of each structure of each embodiment of this invention. It is a block diagram which shows related technology.

[First Embodiment]
(Basic configuration)
Hereinafter, a first embodiment of an image collating apparatus according to the present invention will be described with reference to FIGS. First, the basic configuration of the image collating apparatus according to the present invention will be described, and then the specific contents will be described.

First, in FIG. 1, an image collation apparatus 101 compares a storage unit 10 that stores a registered image RI having a feature quantity with the registered image RI and a search image QI to be collated, and performs an image processing. And processing means 12.
Among these, the storage means 10 stores in advance a registered image RI having a feature quantity and various data (partial registered image RP and the like) calculated from the registered image RI.

The image processing means 12 is estimated by the geometric transformation parameter estimation unit 16 that estimates the geometric transformation parameters based on the feature points and the arrangement of the registered image RI and the search image QI, and the geometric transformation parameter estimation unit 16. In addition, a partial area determination unit 14 that determines a projected partial image area in the search image QI corresponding to each partial area image in the registered image RI using the geometric transformation parameter, and each of the plurality of projections of the search image QI A partial region feature amount calculation unit 18 that calculates each feature amount of the partial image region and each partial region image of the registered image RI, and the feature amount of the partial registered image specified by the partial region of the registered image RI A partial image collation unit 20 that determines a match between the partial search image and the partial registration image by comparing feature quantities of a plurality of partial search images; Comprises an image position detecting section 22 to detect and calculate what the position of the partial search image the matching using the transformation parameters, a.

Among these, the geometric transformation parameter estimation unit 16 includes a parameter estimation function (candidate parameter calculation function 16a) that calculates and estimates the geometric transformation parameters that are candidates for the partial search image for a plurality of sets.

The image position detection unit 22 calculates the degree of coincidence between the partial search image and the partial registered image for each estimated geometric transformation parameter, and sets the reliability of the geometric transformation parameter having a high degree of coincidence. Whole image position estimation for calculating and estimating in which area in the search image QI the image area corresponding to the registered image RI is calculated using the reliability calculation function 22a and this highly reliable geometric transformation parameter. And a function 22b.

Furthermore, the above-described image position detection unit 22 includes a corresponding position calculation function 22c that calculates the position of the partial search image using the geometric transformation parameter and the preset coordinates of the partial registration image.

Therefore, in the first embodiment, the above-described geometric transformation parameter estimation unit estimates the geometric transformation parameter of the search image QI with respect to the registered image RI, and the partial image matching unit uses the feature points of the partial registration image and the feature of the partial search image. Since it is configured to determine whether or not there is a match by comparing the point, and the position calculation unit calculates and determines the position of the matched partial search image using the geometric transformation parameter, a plurality of parts are included in the search image QI. Even if there is a search image, its position can be calculated individually. That is, the position of each of the plurality of partial search images included in the search image QI can be calculated, thereby increasing the versatility of the apparatus.

This will be described in detail below.
(Specific configuration contents)
First, the geometric transformation parameter estimation unit 16 described above first extracts a feature point for each of the registered image RI and the search image QI described above, and also specifies a feature point extraction / specification function that specifies the arrangement position of the extracted feature point. It has. The geometric transformation parameter estimation unit 16 includes a feature vector calculation function for calculating a geometric invariant string (invariant feature vector) that characterizes the arrangement of feature points, and further, the registered image RI and the search image. A geometric transformation parameter estimation function for estimating a geometric transformation parameter from each QI feature point arrangement information is provided.

Here, the geometric transformation parameter is a parameter representing the content and amount of geometric transformation between two images, and examples of the geometric transformation include affine transformation and homography transformation. The affine transformation can represent transformation of movement, scaling, rotation, and skew (oblique distortion) of the feature on the image. The homography transformation can represent trapezoidal distortion.

This geometric transformation parameter is a set of parameters representing a geometric transformation between two images, such as an affine parameter and a homography matrix. The parameters of these geometric transformations include an affine parameter and a homography parameter. In addition, these parameters may be approximated. As the approximate value, the length ratio of the characteristic portion, the area ratio, a ratio of some cumulative value, or the like can be used.

Also, not only linear conversion but also non-linear conversion can be handled by existing image processing methods. In the first embodiment, the search image QI as a whole has non-linear distortion. However, even when partial approximation can be satisfactorily approximated by linear geometric transformation, the geometric transformation parameters for each partial search image QP are obtained and handled. By doing so, it is possible to cope with non-linear conversion by linear conversion for each part.

The geometric transformation parameters between the registered image RI and the search image QI can be obtained by various methods. First, when the external parameter indicating the installation angle of the camera that captured the registered image RI and the external parameter of the camera that captured the search image QI are known in advance, the geometric transformation parameter is determined from the difference between the external parameters. Can be calculated.

Next, the same object (including characters and codes) is captured in the registered image RI and the search image QI, and the object imaged in the search image QI moves with reference to the object imaged in the registered image RI. When there is enlargement / reduction, deformation, etc., the geometric transformation parameters can be obtained by comparing the characteristics of the same object (for example, coordinates of the circumscribed rectangle). When it is known in advance that the same object is imaged as an index in the registered image RI and the search image QI, the geometric transformation parameter can be calculated using the image portion of the index.

Even if it is unclear whether or not the same object is being imaged, it is the same or similar by determining whether or not the invariable amount with respect to geometric transformation is common to the registered image RI and the search image QI. It is possible to determine whether or not the object is being imaged. The geometric transformation invariant is data indicating the feature amount of an image that is not lost by geometric transformation, and is a quantity that can be reproduced using a geometric transformation parameter. The geometric transformation invariant is a feature of the image, and the specific calculation of the feature may be executed by the partial region feature amount calculator 18.

In the first embodiment, the geometric transformation parameter estimation unit 16 estimates the geometric transformation parameters of the search image QI for the registered image RI. The geometric transformation parameter estimation unit 16 determines whether or not the same or similar object is imaged by determining whether there is a geometric transformation invariant between the registered image RI and the search image QI, for example. Furthermore, the geometric transformation parameter can be estimated by calculating a parameter such that the feature portions which are geometric transformation invariants overlap.

In order to estimate the geometric transformation parameter, the degree of coincidence between the registered image RI of the geometric transformation invariant and the search image QI is used as an index, and all combinations or random combinations of the parameters are obtained by voting or the least square method. It can be calculated by optimization calculation or the like. This estimation method can perform estimation even if the external parameters of the camera are not known in advance or an object whose shape or the like is specified in advance in the search image QI is not reliably captured. For this reason, it is suitable for collation processing for determining whether or not an image that is the same as or similar to the registered image RI (part thereof) is included in the search image QI.

In the first embodiment, when a plurality of partial images are included in the search image QI, the geometric transformation parameter estimation unit 16 estimates a geometric transformation parameter for each partial retrieval image QP. Further, instead of estimating only one geometric transformation parameter for one partial search image QP, a plurality of candidate geometric transformation parameters may be estimated.

When the geometric transformation parameters are estimated by voting or optimization using geometric transformation invariants, the geometric transformation parameters adapted to each partial retrieval image QP can be obtained without specifying the coordinates and shape of the partial retrieval image QP in advance. Can be sought.

That is, if the geometric transformation parameter between the registered image RI and the retrieval image QI is estimated by voting or optimization without directly calculating the geometric transformation parameter suitable for each partial retrieval image QP, as a result, The estimated geometric transformation parameter is a value adapted to the partial search image QP. In this case, when there are a plurality of partial search images QP, geometric transformation parameters corresponding to each of the partial search images QP are obtained.

The partial area determination unit 14 described above has a function of specifying a partial registered image RP by calculating a plurality of partial areas of the search image QI.
In calculating the partial area, a circumscribed rectangle or the like may be obtained by applying image processing such as clustering, binarization, and discriminant analysis to the search image QI, or the above-described geometric transformation parameters may be obtained in advance. The partial area on the search image QI may be obtained by projecting the position of the partial registration image RP onto the search image QI using the geometric transformation parameter.

When calculating the partial search image QP from the search image QI, the partial region determination unit 14 may specify the partial search image QP by image processing on the search image QI, but it is registered using the geometric transformation parameter. The partial search image QP may be specified by projecting the coordinates of the partial registration image RP in the image RI onto the search image QI.

Furthermore, when the partial search image QP is specified, the partial area feature amount calculation unit 18 described above calculates the feature amount of each partial search image QP. The feature amount may be a geometric transformation invariant or another type of feature that is not a geometric transformation invariant as long as it is a feature amount for determining a match between the partial registration image RP and the partial search image QP. In the first embodiment, the features of the partial registration image RP are calculated in advance using the partial region feature amount calculation unit 18 and the like, and are stored in the storage unit 10.

Further, the partial image matching unit 20 described above compares the feature amounts of the plurality of partial search images QP with the features of the partial registration image RP specified in the partial region of the registered image RI, so that the partial search image QP. And the partial registration image RP. This determination of coincidence can be performed by information processing for determining whether or not the difference in feature amount or its ratio is within a range indicated by a static or dynamic threshold value.

The image position detection unit 22 has a function of calculating the position of the matched partial search image QP using the geometric transformation parameter. When the partial search image QP matches the partial registration image RP, it is suggested that the geometric transformation parameter is useful for specifying the partial search image QP. When the position of the partial registration image RP is calculated, the position of the partial search image QP can be accurately obtained.

In the first embodiment, with the above-described configuration, a plurality of partial search images QP included in the search image QI can be searched, and the position of the partial search image QP can be calculated. Furthermore, by determining the position of the partial search image QP, the number of partial registration images RP in the search image QI can be calculated.

FIG. 3 is a block diagram illustrating a configuration example in the case of calculating the reliability of the geometric transformation parameter, although a part of the description is around. In this example, the geometric transformation parameter estimation unit 16 includes a candidate parameter calculation function 16a. In FIG. 3, the image position detection unit 22 described later estimates the position of the partial search image QP using the parameter reliability calculation function 22a for determining the reliability of the geometric conversion parameter and the geometric conversion parameter. And a calculated image position estimating function 22b.

The candidate parameter calculation function 16a of the geometric transformation parameter estimation unit 16 calculates a plurality of sets of geometric transformation parameters that are candidates for the partial search image QP.
On the other hand, the parameter reliability calculation function 22a of the image position detection unit 22 calculates the degree of coincidence between the partial search image QP and the partial registered image RP for each candidate geometric transformation parameter, and obtains a geometric transformation parameter having a high degree of coincidence. This is a function for setting a high reliability. The degree of coincidence may be the degree of coincidence of individual partial search images QP, or the ratio of the number of matched partial search images QP to partial search images QP when there are a plurality of partial search images QP subjected to the same geometric transformation. It is also good.

Then, for example, the parameter reliability calculation function 22a determines whether each of the partial registration images RP matches the partial registration image RP using the geometric transformation parameter, and the number of partial search images QP determined to match is determined. If it is above a certain level, it is determined that the geometric transformation parameter is reliable.

On the other hand, the image position estimation function 22b is a function for calculating the position of the partial search image QP using a highly reliable geometric transformation parameter. That is, the image position estimation function 22b uses the geometric transformation parameter determined to have high reliability by the parameter reliability calculation function 22a to determine which partial search image QP corresponding to the partial registration image RP is in the search image QI. It can be estimated whether it is in position.

As described above, it is determined that the reliability is high by comparing the feature amounts of the partial registration image RP and the partial search image QP, not by the geometric conversion parameter based on the comparison of the geometric transformation invariant between the registration image RI and the search image QI. By obtaining the position of the partial search image QP using the geometric transformation parameter, the position of each partial search image QP can be calculated with high accuracy.

In particular, when there are a plurality of geometric transformation parameters obtained from the whole, the geometric transformation parameters derived from the features of the respective partial registration images RP become the optimal geometric transformation parameters for the partial search image QP that matches the partial registration image RP. The optimum geometric transformation parameter for each partial search image QP can be selected based on the reliability, and the position of the partial search image QP can be obtained using the geometric transformation parameter.

Here, as a preferable example, the image position detection unit 22 preferably includes a corresponding position calculation function 22c that calculates the position of the partial search image QP using the geometric transformation parameters and the coordinates of the partial registration image RP ( (See FIG. 1). Thereby, the position of the partial search image QP can be calculated in the coordinate system of the registered image RI, and various image processing corresponding to the coordinates of the registered information can be performed.

(Operation of the first embodiment)
Next, the overall operation of the first embodiment will be described in detail with reference to FIG.
In the example illustrated in FIG. 4, it is assumed that the registered image RI, the partially registered image RP, and the partially registered feature amount RC are stored in the storage unit 10 in advance. Then, the search image QI is received, and a search process for checking whether or not an image identical to or similar to the partially registered image RP exists in the search image QI is started.

First, the geometric transformation parameter estimation unit 16 estimates a geometric transformation parameter (FIG. 4: step S101). In this example, it is preferable to obtain a geometric transformation parameter established by the registered image RI and the search image QI. If the geometric transformation parameters of the registered image RI for the search image QI portion (the portion that becomes the partial search image QP) are locally different, different geometric transformation parameters may be obtained respectively. Conversely, when the geometric transformation parameters of the search image QI for the partial registration image RP are locally different, different geometric transformation parameters may be obtained in the same manner.

Next, the partial region determination unit 14 specifies a partial registration image RP in the registration image RI (FIG. 4: step S102). And the partial registration feature-value RC which is the feature-value of this partial registration image RP is read from the memory | storage means 10 (FIG. 4: step S103).

Then, the partial area determination unit 14 determines a partial area of the search image QI and sets it as the partial search image QP (FIG. 4: step S104). In order to specify the partial search image QP, the region of the partial search image QP may be specified by projecting the coordinates of the partial registration image RP onto the search image QI using the geometric transformation parameter calculated in step S101.

Next, the partial region feature quantity calculation unit 18 calculates the partial search feature quantity QC of the partial search image QP (FIG. 4: step S105). The partial search feature quantity QC is a feature quantity in a format comparable to the partial registration feature quantity RC.
Then, the partial image matching unit 20 determines whether the partial registration image RP in the registration image RI matches the partial search image QP (FIG. 4: step S106).

In the first embodiment, in particular, the image position detection unit 22 has one technical feature in that the position of the partial search image QP (existing area of the registered image RI) is calculated (FIG. 4: step S107). . In the calculation of the position of the partial search image QP, using the geometric transformation parameter itself calculated in step S101 of FIG. 4 or a better geometric transformation parameter for specifying the partial search image QP, the partial search is performed. It is preferable to calculate the position of the image QP.

The example shown in FIG. 5 shows an example in which the position of the partial search image QP is calculated using a highly reliable geometric transformation parameter.
In the example of FIG. 5, first, the parameter reliability calculation function 22a determines the coincidence of each of the plurality of partial search images QP for each of the plurality of geometric transformation parameters (FIG. 5: Step S111). In step S111 of FIG. 5, when there are three geometric transformation parameters and there are four partial search images QP, it is determined whether there are 12 combinations.
Note that the parameter reliability calculation function 22a may count the reliability using the reliability table TB shown in FIG.

Subsequently, the parameter reliability calculation function 22a determines that a geometric transformation parameter having a certain number of partial search images QP determined to match can be trusted (FIG. 5: step S112). When it is determined that the four partial search images QP match for each of the three geometric transformation parameters, the geometric transformation parameter having the largest number of matches is set as a highly reliable geometric transformation parameter.

For example, the first geometric transformation parameter matches in two partial search images QP, the second geometric transformation parameter matches in two partial search images QP, and the third geometric transformation parameter matches in three partial search images QP. When they match, the parameter reliability calculation function 22a determines that the third geometric conversion parameter is a highly reliable geometric conversion parameter.
Then, using the highly reliable geometric transformation parameter, the position of the partial search image QP is finally specified (FIG. 5: step S113), and the coordinates of the partial search image QP based on this geometric transformation parameter are determined as the part. The position of the search image QP is set (FIG. 5: Step S114).

Here, in the first embodiment, the operation content (information processing) of each component and the linkage operation at the time of information processing between the components are programmed, and the image processing means is provided. It may be realized by a computer. The same applies to each embodiment described later.

(Effect of 1st Embodiment)
In the first embodiment, as described above, since the registration image RI is determined so as to determine in which area in the search image QI, the same registration image RI is included in the search image QI. A plurality of images can be detected when a plurality of images are captured. In addition, when the same registered image RI is included in the search image QI, it is possible to detect at which position in the search image QI the registered image RI is included.

FIG. 6 is a diagram for illustrating the effect of the first embodiment. In the example shown in FIG. 6, two partial registration images RP are included in the search image QI.
Patent Document 1 described above discloses a similar content. However, in the prior art shown in Patent Document 1, the partial search image QP [1] and the partial search image QP [2] are in different areas. Since collation is performed without considering that there is, it is difficult to obtain with high accuracy where the partial registration image RP exists in the search image QI.

On the other hand, in the first embodiment, in order to calculate in which region in the search image QI the partial registration image RP is located, a plurality of the same partial registration images RP are included in the search image QI. In addition, a plurality of partial registration images QP [1] and QP [2] can be detected.

As described above, in the first embodiment, in order to calculate in which region in the search image QI the partial registration image RP is located, a plurality of the same registration images RI are included in the search image QI. In some cases, multiple images can be detected.
In addition, when the same registered image RI is included in the search image QI, it is possible to detect at which position in the search image QI the registered image RI is included.
[Second Embodiment]

Next, a second embodiment of the present invention will be described with reference to FIGS.
In the second embodiment shown in FIGS. 7 to 11, an example in which a feature vector is used as a feature amount and an example in which a homography transformation is used as a geometric transformation will be described.
Here, the same code | symbol shall be used about the same component as 1st Embodiment mentioned above.

In FIG. 7, the image collation apparatus 102 according to the second embodiment performs a partial area determination unit 14 configured substantially the same as that of the first embodiment described above in order to collate the registered image RI and the search image QI. A geometric transformation parameter estimation unit 16, a partial region feature amount calculation unit 18, a partial image matching unit 20, and an image position detection unit 22.

The partial region feature quantity calculation unit 18 has a function of calculating feature quantities of the partial registration image RP and the search image QI and calculating a feature vector having a geometric transformation invariant as an element as the partial registration feature quantity RC. The partial region feature amount calculation unit 18 calculates a feature amount for each of the registered image RI and the search image QI. The feature amount of the registered image RI may be calculated in advance and stored in the storage unit 10.

The partial region feature quantity calculation unit 18 generates a feature point of the image and the arrangement of the feature point, and further, a geometric invariant string characterizing the arrangement of the feature point from the generated arrangement of the feature point and its coordinates. (Feature vector) is calculated.
Here, when the partial registration image RP is calculated in advance by the partial region determination unit 14 described above, the partial region feature amount calculation unit 18 has a function of calculating the partial registration feature amount RC of the partial registration image RP. Yes.

Further, the geometric transformation parameter estimation unit 16 has a function of determining whether or not the feature vector calculated from the partial registration image RP matches the feature vector calculated from the search image QI (invariant match determination function 16b). It has. Based on this, the geometric transformation parameter estimation unit 16 estimates a geometric transformation parameter having a matching feature vector. Furthermore, the geometric transformation parameter estimation unit 16 can use a combination of feature points in the vicinity as the feature points and the center of gravity of the connected region after binarization and the feature point arrangement.

The partial area determination unit 14 described above has a function of calculating a partial area of the registered image RI and calculating one or more partial registered images RP. The partial registration image RP may be calculated in advance as a registration process and stored in the storage unit 10 in advance.
The partial region determination unit 14 further has a function of calculating a partial search image QP of the search image QI by projection using a predetermined geometric transformation parameter.

In addition, the partial region determination unit 14 uses the geometric transformation parameter estimated by the geometric transformation parameter estimation unit 16 to obtain a partial retrieval image QP in the retrieval image QI corresponding to each partial registration image RP in the registration image RI. It has a function to decide.
Specifically, the partial search image QP in the search image QI is determined by applying the geometric transformation parameters to the circumscribed rectangle and the coordinates of the outer periphery of the partial registration image RP. If n (n ≧ 1) geometric transformation parameters are obtained, n partial search images QP are obtained as described above.

Further, the partial region feature amount calculation unit 18 includes a function (partial registration feature amount calculation function 18d) for calculating a feature vector (partial search feature amount QC) of the partial search image QP in the search image QI. . According to this, since one partial search image QP is obtained for one partial search region of the search image QI, when n geometric transformation parameters are obtained, the feature vector (partial search feature quantity QC) is also n. Individuals are required. This feature vector is a vector including, for example, a feature point arrangement which is a geometric transformation invariant.

The partial image matching unit 20 described above uses the feature vector (partial registration feature amount RC) of the partial registration image RP of the search image RI specified by using the geometric transformation parameter and the feature vector (partial part of the partial search image QP). It has a function of determining whether or not the search feature value (QC) matches. For example, the partial image matching unit 20 can determine that both images match when the difference or ratio between the feature vectors of the two images is within a predetermined threshold range.

In addition, the image position detection unit 22 calculates the position of the partial search image QP determined to be coincident by the partial image matching unit 20 by using the geometric transformation parameter. As described above, the image position detection unit 22 determines which partial registration image RP is included in which position in the search image QI from the collation result obtained by the partial image collation unit 20.

In order to calculate a feature vector in the second embodiment, the partial region feature quantity calculation unit 18 includes, for example, a feature point extraction function 18a, a feature point arrangement function 18b, and an invariant calculation function 18c. good.

The partial region feature amount calculation unit 18 calculates the feature vector of the registered image RI, partial registered image RP, search image QI, or partial search image QP at each timing.
For example, the feature vectors of the registration image RI and the partial registration image RP may be calculated in advance as registration processing. The feature vector of the search image QI may be calculated when the geometric transformation parameter is initially estimated by the geometric transformation parameter estimation unit 16. Further, the feature vector of the partial search image QP may be calculated after the partial search image QP is specified by the partial region determination unit 14 using the geometric transformation parameter.

Here, the feature point extraction function 18a of the partial area feature amount calculation unit 18 described above calculates the feature points of the registered image RI, the partial registration image RP, the search image QI, or the partial search image QP at each timing. The feature point is the center or the center of gravity of the feature portion. When the feature point is obtained, the feature point arrangement function 18b calculates the arrangement of a plurality of feature points. Further, the invariant calculation function 18c calculates a feature vector having a geometric transformation invariant as an element from the arrangement of the feature points.

Also, as in the case of the first embodiment described above, the image position detection unit 22 may include a parameter reliability calculation function 22a and an image position estimation function 22b. The parameter reliability calculation function 22a calculates the geometric transformation parameter reliability, which is the reliability of the geometric transformation parameter. Then, the image position estimation function 22b estimates the position of the search image QI included at least partially in the registered image RI using the geometric transformation parameter reliability.

Next, an example of image processing using feature vectors in the second embodiment will be described with reference to FIG.
In the example shown in FIG. 8, first, the partial region determination unit 14 calculates a partial region of the registered image RI and calculates one or more partial registered images RP (FIG. 8: Step S201). Then, when the search image QI is received and the search process is started, the partial region feature value calculation unit 18 calculates a feature vector whose elements are geometric invariants of the search image QI and the partial registration image RP.

In the feature vector calculation, the feature point extraction function 18a calculates the feature points of the partial registration image RP and the search image QI (FIG. 8: step S202). Then, the feature point arrangement function 18b calculates the arrangement of a plurality of feature points (FIG. 8: Step S203). Further, the invariant calculation function 18c calculates a feature vector whose elements are geometric transformation invariants from the arrangement and coordinates of the feature points (FIG. 8: step S204).

Subsequently, the geometric transformation parameter estimation unit 16 determines whether or not the feature vector calculated from the partially registered image RP matches the feature vector calculated from the search image QI (FIG. 8: Step S205). Then, the geometric transformation parameter estimation unit 16 estimates a geometric transformation parameter based on the matching of the feature vectors (FIG. 8: Step S206). In the determination of the coincidence and the geometric transformation parameter (FIG. 8: Steps S205 and S206), the coincidence judgment may be repeated while changing the value of the geometric transformation parameter, or as an optimization process by a voting process or a least square method. good.

When the geometric transformation parameter is estimated, the partial region determination unit 14 calculates the partial retrieval image QP of the retrieval image QI by projection using the geometric transformation parameter (FIG. 8: Step S207). When a plurality of geometric transformation parameters are estimated, the same number of partial search images QP as the number of geometric transformation parameters are calculated.

When the partial search image QP is calculated, the partial region feature amount calculation unit 18 calculates again. That is, the partial region feature quantity calculation unit 18 calculates the feature vector of the partial search image QP (FIG. 8: Step S208). Then, the partial image matching unit 20 determines whether or not the feature vector for the partial search image QP specified using the geometric transformation parameter matches the feature vector of the partial registration image RP (FIG. 8). : Step S209). Then, the image position detection unit 22 calculates the position of the partial search image QP determined to be coincident by the partial image matching unit 20 by using the geometric transformation parameter.

In the calculation of the position of the partial search image QP, the parameter reliability calculation function 22a calculates the geometric transformation parameter reliability, which is the reliability of the geometric transformation parameter (FIG. 8: step S210), and the image position estimation function 22b The position of the search image QI included at least partially in the registered image RI may be estimated using the geometric transformation parameter reliability (FIG. 8: step S211).

(Homography matrix)
Next, an example of using a homography matrix as a geometric transformation will be described with reference to FIG.
In the example shown in FIG. 9, the geometric transformation parameter estimation unit 16 includes a homography matrix calculation function 16c that calculates a homography matrix as a geometric transformation parameter.

Then, the partial region feature quantity calculation unit 18 calculates the geometric transformation invariant, which is an invariable quantity even if the partial search image QP is geometrically transformed with the homography matrix with respect to the partial registration image RP, as a feature. A function 18c is provided.

Furthermore, the image position detection unit 22 includes a projection position calculation function 22d that calculates the position of the partial search image QP by projecting a partial region of the registered image RI onto the search image QI using a homography matrix.
Similarly to the first embodiment described above, the image position detection unit 22 may include a parameter reliability calculation function 22a and an image position estimation function 22b.

The homography matrix H is a geometric transformation matrix that associates coordinate values, and is a 3 × 3 matrix that represents the relationship between the position (xr, yr) of the registered image RI and the position (xq, yq) of the search image QI. is there. Specifically, the following expression (1) is satisfied.
Here, the symbol a is a constant determined according to the values of (xr, yr) and (xq, yq).

In this example, the homography matrix calculation function 16c of the geometric transformation parameter estimation unit 16 calculates a homography matrix as a geometric transformation parameter. Then, the invariant calculation function 18c of the partial region feature quantity calculation unit 18 features a geometric transformation invariant that is an invariant quantity even if the partial search image QP is geometrically transformed with the homography matrix with respect to the partial registration image RP. Calculate as Subsequently, the projection position calculation function 22d of the image position detection unit 22 functions, and the position of the partial search image QP is calculated by projecting the partial area of the registered image RI onto the search image QI using the homography matrix.
Thus, by using a homography matrix, it is possible to collate partial images using a stable image processing technique.

This will be described in detail.
FIG. 10 illustrates an example of a match determination result between the partial registration feature quantity RC and the partial search feature quantity QC when two homography matrices (H1, H2) are obtained.

On the left side of FIG. 10, for the circumscribed rectangle surrounding the character “a”, the upper row is a partial registration image RP (RP [1] to RP [8]) of the registration image RI (DB Image) read from the storage means 10. There is a partial search image QP (QP [1] to QP [8], Compensated Image) obtained by projecting the search image QI.
The feature amount calculated from these is indicated by the symbol IF in FIG. These are calculated for each partial registration image RP and each partial search image QP.

An example of a comparison result between the partial registration image RP and the partial search image QP is illustrated on the right side of FIG. In the figure, ◯ represents coincidence and x represents disagreement.
In this example, it is assumed that due to noise or the like, a mismatch determination has occurred for a partial region that originally matches.

According to this figure, the partial search image QP is collated [(number of partial search images QP) × (number of obtained geometric transformation parameters)] times. Specifically, in the example shown in FIG. 10, 16 comparison results of 8 × 2 are obtained.

When the comparison is completed, the parameter reliability calculation function 22a of the image position detection unit 22 fixes the geometric transformation parameter H1 and calculates the partial registration image RP calculated from the registration image RI and the search image QI. The collation results with the partial search image QP are totaled.

In the example shown in FIG. 11, regarding the homography matrix H1, it is determined that seven partial search images QP match the eight partial search images QP calculated as a whole.
As a result, the reliability is calculated as 7/8. Here, the ratio of the number of matching partial search images QP is used as the reliability. However, the reliability may be calculated after weighting the area of each partial search image QP.

The parameter reliability calculation function 22a of the image position detection unit 22 arranges the collation results by the partial image collation unit 20 shown in FIG. 10 into the data structure of the reliability table TB in FIG. 11, and matches the total number of partial search images QP. The parameter reliability is calculated from the number, and it is preferable to determine whether or not the homography matrix as the geometric transformation parameter is reliable based on the reliability.

In the example shown in FIG. 11, the homography matrix H1 is a reliable geometric transformation parameter because it is determined that the reliability is a geometric transformation parameter that has a predetermined reliability, for example, a geometric transformation parameter of 5/8 or more. It is determined that there is.
On the other hand, regarding the homography matrix H2, only one partial search image QP matches eight partial search images QP. In this case, the reliability is 1/8. Since this is smaller than a predetermined value, the homography matrix H2 is not determined as a reliable parameter.

Next, the image position estimation function 22b of the image position detection unit 22 estimates which position in the search image QI the region corresponding to the partially registered image RP of the registered image RI is using the reliable conversion parameter. To do. In the example shown in FIG. 11, only the homography matrix H1 is used as the geometric transformation parameter.

The relationship between the coordinates (xr, yr) in the registered image RI and the point (xq, yq) in the search image QI is expressed by the above-described equation (1).
Using this relationship, the position of the registered image RI (partial registered image RP) existing in the search image QI can be obtained.
The homography matrix projects the line segment in the registered image RI onto the line segment in the search image QI. Therefore, when the partial area of the registered image RI is configured to be a rectangle, if the four sides constituting the rectangle that is the partial area in the registered image RI are projected onto the search image QI, A corresponding area is required.
[Third Embodiment]

Next, a third embodiment of the image collating apparatus according to the present invention will be described with reference to FIGS. Here, the same reference numerals are used for the same constituent members as those of the above-described embodiments.

In the third embodiment, as shown in FIG. 12, an integrated processing unit 24 is newly added to the image position detection unit 22 provided in advance in the image matching device 103. Further, the geometric transformation parameter estimation unit 16 of the image processing means 12 is equipped with a plural parameter calculation function 16d, and the image position detection unit 22 is newly provided with a position candidate calculation function 22e.

The multiple parameter calculation function 16d of the geometric transformation parameter estimation unit 16 estimates a plurality of geometric transformation parameters for each partial search image QP when the partial region determination unit 14 calculates a plurality of partial regions. By this processing, a plurality of geometric transformation parameters are estimated for one partial search image QP.

Then, in the position candidate calculation function 22e of the image position detection unit 22, the image position detection unit 22 calculates a position candidate of the matched partial search image QP as a position candidate for each geometric transformation parameter. By this process, a geometric transformation parameter that satisfies the condition of matching with the partial registration image RP and a partial search image QP that can be specified by this geometric transformation parameter remain.

In the third embodiment, the newly-integrated integration processing unit 24 integrates the position candidates for each overlapping partial search image QP when a plurality of partial search images QP overlap in the search image QI. The position of the partial search image QP is calculated. Thereby, the number of partial search images QP can be calculated with high accuracy.
The integration processing unit 24 integrates two or more position candidates (candidates for the position of the partial search image QP) that are outputs of the image position detection unit 22. This makes it possible to calculate a representative position from slightly different position candidates (geometric transformation parameters).

Referring to FIG. 13, the integration processing unit 24 may be provided with any of the union position function 24a, the reliability position function 24b, or the voting position function 24c. Further, the integration processing unit 24 may include two or more of these three functions, and may be configured to perform integration processing using a plurality of methods in combination.

The union position function 24a calculates a union of a plurality of position candidates and determines the position of the union as a position.
In the example shown in FIG. 14, the union position function 24a indicates that the plurality of partial search images QP [10a], QP [10b], and QP [10c] are calculated as shown in FIG. 13 (1) union. In addition, the sum area (OR area) related to the overlapping areas is integrated to obtain the outer peripheral shape of the partial search image QP [10A].
The position of

Thereby, the characteristic part of the partial search image QP can be detected without omission. Similarly, the union position function 24a integrates a plurality of partial search images QP [11a] and QP [11b] into a partial search image QP [11A].

The search image position determination function 24b calculates the reliability of the geometric transformation parameter and determines the position using the geometric transformation parameter having a high reliability. That is, when there are partial search images QP [11a] and QP [11b] that overlap each other, the search image position determination function 24b performs the respective geometric transformations that specify the partial search images QP [11a] and QP [11b]. Among the parameters, the position candidate based on the geometric transformation parameter having the highest reliability is selected.

Thereby, in the example shown in FIG. 14, the partial search image QP [11B] is selected. Similarly, the search image position determination function 24b has a plurality of partial search images QP [10a], QP [10b], and QP [10c].
Among them, the most reliable geometric transformation parameter is used as a partial search image QP [10B]. Here, the position candidate may be selected using the statistical value of the parameter value PM of the geometric transformation parameter having a certain degree of reliability.

The other image position determination function 24c controls the voting of the geometric transformation parameters to the parameter space, and determines the position using statistical values of a plurality of geometric transformation parameters whose voting results are equal to or greater than a threshold value. The statistical value is a value that can be calculated by statistical processing such as an average value or a median value.

FIG. 15 shows an example of integration processing by another image position determination function 24c. The other image position determination function 24c converts the reliable geometric transformation parameter values PM [1], PM [2],..., PM [n] into ballot boxes VT [1], VT [2],. ., Vote for VT [n], calculate the average of the geometric transformation parameters related to the ballot box VT for which the vote is equal to or greater than a certain value, and obtain the result.

In the example shown in FIG. 15, as a result of voting the parameter values PM of the partial search images QP [10a], QP [10b], and QP [10c], the number of votes for one ballot box VT [1] exceeds a certain number. The position of the partial search image QP [10C] is specified using this geometric transformation parameter. Similarly, the position of the partial search image QP [11C] is specified using the geometric transformation parameter corresponding to the ballot box VT [2].

(Operation of Third Embodiment)
Next, the operation of the third embodiment will be described with reference to FIGS.
Unlike the image processing of the first embodiment described above (see FIG. 4), the position detection processing of the partial search image QP (FIG. 4: step S107) is different from the position detection processing of the partial search image QP in the third embodiment. (FIG. 6: Step S301).

FIG. 17 is a flowchart showing a detailed operation example of the position detection process (FIG. 16: step S301) of the partial search image QP.
Here, the process (FIG. 16: step S301) for detecting the region of the search image QI (position of the partial search image QP) where the partial registration image RP of the registration image RI exists is different from step S107 of FIG. 4 described above. Is that a region integration process (FIG. 17: step S311) is added after the position candidate estimation process (FIG. 5: step S114). In this region integration processing (FIG. 17: step S311), the integration processing unit 24 integrates the regions by the method described above.

As described above, in the third embodiment, since a representative position is calculated from slightly different position candidates, one region can be obtained for each registered image RI included in the search image QI. The possibility increases. Therefore, the number of registered images RI included in the search image QI can be determined with high accuracy.

In particular, since the integration processing unit 24 performs integration processing on the overlapping partial search images QP when a plurality of partial search images QP overlap in the search image QI, the same search result is assumed due to the overlap. It is possible to integrate the search results of the partial search images QP well into one, and further, by using a value obtained by averaging the parameter values PM of geometric transformation parameters having a certain degree of reliability or voting, etc. A position close to the value of can be calculated.

Further, by integrating overlapping partial search images QP by some method, the number of partial registration images RP (corresponding to partial search images QP) in the search image QI can be strictly calculated.

(Information display)
The image processing means 12 in the third embodiment is provided with a display device 95 as shown in FIG. Further, the image processing means 12 is provided with a display control unit 26 for controlling the information display operation on the display device 95.
The display control unit 26 controls display of display data DP related to the partial registration image RP at a position corresponding to the partial search image QP. Thereby, the convenience of an image search can be improved.

The display data DP may be any expression or description as long as the content indicates the result of successful image matching to the user. When searching for the shape of a printed or handwritten character, various display data DP such as the character data, meaning, a link to data containing the character, a translation into another language, etc. are adopted. be able to. If the search result is a sign or a guide map, an explanation of the contents can be displayed.

The display control unit 26 may include a similarity display function 26a. The similarity display function 26a controls the display of the similarity between the partial registration image and the search partial image as display data DP.
FIG. 18 shows an example in which the positions of the partial registration image RP [1] and partial registration image RP [2] detected from the search image QI are displayed to show the correspondence with the search image QI.

In the example of FIG. 18, the display control unit 26 displays the entire search image QI on the display device 95 and associates it with the partial search image QP so that the partial registration images RP [1] and RP [2] that can be searched. The value of similarity is displayed as the display data DP while displaying the type and content.

As a result, the display control unit 26 performs display control of “similarity 0.9” that is the display data DP [1] in association with the partial registration image RP [1], and associates it with the partial registration image RP [2]. Display control is performed on “similarity 0.8” which is display data DP [2]. Thereby, the certainty of the search of the partial search image QP can be displayed in an easily understandable manner. As described above, the accuracy of the search result can be transmitted to the user by the similarity display function 26a.

Further, the display control unit 26 may include a link display function 26b. The link display function 26b displays a link to a predetermined content in association with the partial registration image RP as the display data DP. The link to the content may be content in the same image processing device, or may be a link to content managed by another server device 70 when the image matching device 103 is connected to the network 96.

The link can display any information that can be normally handled by a computer, such as a link to voice or a URL, or a link to any information. Thereby, it is possible to provide not only a simple image display but also an environment in which a user can easily use various media in an integrated manner.

The example shown in FIG. 19 is an example in which information accompanying the registered image RI (or partial registered image RP) stored in advance in the storage means 10 or the like is superimposed and displayed according to the image collation result. In the example shown in FIG. 19, the entire search image QI is displayed on the entire display device 95. In this example, a position candidate related to the registered image RI or an integrated representative area is detected, so that information can be presented around the area.

As shown in FIG. 19, the partial search image QP [3] corresponding to the partial registration image RP is detected from the search image QI, and the information “Explanation 1, explanation 1” (display data DP [3]) is displayed. Has been. The display data DP [3] is a sentence using a text character string. At that time, if sentences representing the same explanation contents in various languages are stored in advance, the language presented by the user can be selected and displayed.

Then, the display control unit 26 performs display control of the display data DP [4] corresponding to the partial search image QP [4], whereby the difference between the partial search image QP [3] and the partial search image QP [4]. Can be clearly displayed. Furthermore, if the explanatory text, which is the display data DP, is used as a link to other contents, the user can access more detailed explanation with an easy operation.

Furthermore, you may display an image instead of just a sentence. For the display of the image, the partially registered image RP may be displayed as it is, or the registered image RI may be transformed into a trapezoid using the value of the homography matrix (for example, disclosed in Patent Document 1). Figure 2).

Further, if there is a speaker (not shown), it is possible to automatically reproduce the sound associated with the registered image RI. A plurality of display / automatic reproductions as described above can be combined.
The third embodiment of the present invention can be applied to an application such as an information providing apparatus that provides guidance information. Further, when the registered image RI is a signboard or a guide sentence, the present invention can be applied to a use such as a parallel translation display system that displays a parallel translation sentence related to the content.
Other configurations and the effects thereof are the same as those of the second embodiment described above.

Here, the configuration and operation of the

image collation apparatuses

101, 102, and 103 described above are examples of the implementation, and the configuration and operation order can be changed within a range that does not impair the principle of image collation. It is.

As described above, in each of the embodiments of the arrangement collation device and the

image collation devices

101, 102, and 103 described above, the case where various means are implemented in the form of software has been described as an example. However, part or all of each means can be configured by hardware.

[Relationship with hardware resources]
Next, the relationship between the information processing common to the

image collating apparatuses

101, 102, and 103 of the first to third embodiments and the hardware resources required for the information will be described by taking the image collating apparatus 101 as an example. (See FIG. 20).

Information processing by the image collating apparatus 101 disclosed in FIGS. 1 to 7 shows specific technical contents in which software and hardware resources cooperate to calculate or process information according to the purpose of use. It is.

As a hardware resource, the image collating apparatus 101 actually has a computer 80 for information processing as shown in FIG. The computer 80 includes calculation means 82 that is a central processing unit (CPU), and main storage means 86 that provides a storage area for the calculation means 82. The computer 80 generally has peripheral devices connected through a data bus and an input / output interface. The peripheral devices are typically a communication unit 88, an external storage unit 90, an input unit 92, and an output unit 94. The whole including peripheral devices may be referred to as a computer 80.

The communication unit 88 controls communication with the server device 70 via a wired or wireless network. The external storage means 90 is a program file 100 or a storage medium that can be installed or carried to store data. The input unit 92 is a keyboard, a touch panel, a pointing device, a scanner, or the like, and inputs data that can be read by the computer 80 in accordance with a user operation. The output means 94 displays and outputs data calculated by the computer 80 such as a display and a printer.

In the first embodiment, in particular, the storage unit 10 uses the external storage unit 90 as a hardware resource, and stores a registered image RI having a feature amount. Further, the storage means 10 stores not only the external storage means 90 directly connected to the computer 80 serving as the image processing means 12 but also the registered image RI and the like from the database 72 of the server device 70 connected via the network 96. It may be used as a wear resource.

The external storage unit 90 or the database 72 stores a registered image RI, a partially registered image RP, and a partially registered feature quantity RC as the storing unit 10. Further, the search image QI, the partial search image QP, the partial search feature quantity QC, the reliability table TB, and the like are stored in the image matching process of each embodiment. In relation to the third embodiment, display data DP related to the partial registration image RP may be stored.

The search image QI may be received from a scanner or facsimile receiving device which is the input means 92 connected to the computer 80, or may be received from another server device 70 or the like. The image collation result is displayed directly on the display device 95 of the output means 94, stored in the external storage means 90 or the database 72, and transmitted as data in response to access from another server device 70 or the like. Also good.

The above-described image processing means 12 compares the registered image RI with the search image QI to be collated using the computing means 82 which is a CPU as a hardware resource. The main storage means 86 temporarily stores various data necessary for the calculation of the CPU. Such data is data that is compared and generated in each step of the flowchart showing the image processing example of each embodiment. For example, in addition to the partial registration image RP and the partial search image QP, the feature amount and the geometric transformation parameter For example, the parameter value PM.

Each part and each function of the image collating apparatus shown in FIG. 1 and the like perform information processing specifically realized by software using hardware resources. Information processing is a combination of processing such as calculation (logical operation), comparison, conditional branching, repetition, and determination. This software has program procedures (instructions and codes) corresponding to the execution environment of the computer 80. The code group is generally stored in the external storage unit 90 as a program file 100. Further, the program file 100 may be downloaded from the server device 70 in response to a request from the computer 80. Of the code groups included in the program file 100, a program for image collation is an image collation program.

Information processing combined as means, units, functions, and the like of the image collation apparatus is realized by cooperation with the hardware resources by causing the computer 80 included in the image collation apparatus to execute a program for image collation. Can do.
The same applies to the other

image matching devices

102 and 103.

As described above, the present invention has been exemplified by the first to third embodiments. However, the present invention is not limited to the technical contents of these embodiments, and As long as equivalent effects are produced, these are included.

For each of the embodiments described above, the main points of the new technical contents are summarized as follows. In addition, although part or all of the said embodiment is put together as follows as a novel technique, this invention is not necessarily limited to this.

[Appendix 1]
An image collating apparatus comprising a storage unit 10 that stores a registered image having a predetermined feature point, and an image processing unit 12 that compares the registered image with a search image to be collated.
The image processing means 12 is
A geometric transformation parameter estimation unit 16 that estimates a geometric transformation parameter based on each feature point and the arrangement of the registered image and the search image;
A partial region determination unit 14 for determining a projected partial image region in a search image corresponding to each partial region image in a registered image using the geometric conversion parameter estimated by the geometric conversion parameter estimation unit 16;
A partial region feature amount calculation unit 18 that calculates each feature amount of each of the plurality of projection partial image regions of the search image and each partial region image of the registered image;
A match between the partial search image and the partial registration image is determined by comparing feature amounts of the plurality of partial search images with the feature amount of the partial registration image specified in the partial area of the registration image. A partial image matching unit 20;
An image collating apparatus comprising: a registered image position detecting unit 22 that calculates and detects the position of the matched partial search image using the geometric transformation parameter.

[Appendix 2]
In the image collation device according to attachment 1,
The geometric transformation parameter estimation unit 16 includes a parameter estimation function (candidate parameter calculation function 16a) that calculates and estimates the geometric transformation parameters that are candidates for the partial search image for a plurality of sets.
The image position detection unit 22 calculates the degree of coincidence between the partial search image and the partial registered image for each estimated geometric transformation parameter, and sets the reliability of the geometric transformation parameter having a high degree of coincidence to a high level. Degree calculation function 22a and whole image position estimation function 22b for calculating and estimating in which area of the search image the image area corresponding to the registered image is located using the geometric transformation parameter having high reliability And an image collating apparatus.

[Appendix 3]
In the image collation device according to

appendix

1 or 2,
The image position detection unit 22 includes a corresponding position calculation function 22c that calculates the position of the partial search image using the geometric transformation parameter and preset coordinates of the partial registration image. Image collation device.

[Appendix 4]
In the image collating device according to

appendix

1, 2, or 3,
The geometric transformation parameter estimation unit 16 includes a multiple parameter estimation function 16d that estimates a plurality of geometric transformation parameters for each partial search image when a plurality of partial regions are calculated by the partial region determination unit 14.
The image position detection unit 22 includes a position candidate calculation function (22e) that calculates the position candidate of the matched partial search image as a position candidate for each geometric transformation parameter,
An integration processing unit that calculates the position of the partial search image by integrating the position candidates for each of the overlapping partial search images when a plurality of the partial search images overlap in the search image. An image collating apparatus characterized by having 24.

[Appendix 5]
In the image collating device according to attachment 4,
The image collation characterized in that the integration processing unit 24 includes a union position determination function 24a that calculates a union of the plurality of position candidates and determines the position of the union as the position of the partial search image. apparatus.

[Appendix 6]
In the image collating device according to attachment 4,
The integration processing unit 24 includes a search image position determination function 24b that calculates the reliability of the geometric conversion parameter and determines the position of the partial search image using the geometric conversion parameter having the high reliability. An image matching apparatus characterized by the above.

[Appendix 7]
In the image collating device according to attachment 4,
The integration processing unit 24 controls voting of the geometric transformation parameters to the parameter space, and determines the position of the partial search image using a plurality of statistical values of the geometric transformation parameters whose voting results are equal to or greater than a threshold value. The image collating apparatus according to claim 4, further comprising a second search image position determination function 24 b.

[Appendix 8]
In the image collating device according to any one of appendices 1 to 7,
The image processing means 12 includes a display control unit 26 that performs display control at a position corresponding to the partial search image of a display device 95 that is preliminarily equipped with display data DP related to the partial registration image. Image matching device.

[Appendix 9]
An image collating apparatus comprising a storage unit that stores a registered image having a predetermined feature amount, and an image processing unit that compares the registered image with a search target search image,
Based on each feature point and the arrangement of the registered image and the search image, the geometric transformation parameter estimation unit 16 of the image processing means estimates a geometric transformation parameter,
The partial region determination unit 14 of the image processing means determines a plurality of projection partial image regions in the search image corresponding to each partial region image in the registered image using the estimated geometric transformation parameter,
A partial region feature amount calculation unit 18 of the image processing means calculates each feature amount of each of the plurality of projection partial image regions of the search image and each partial region image of the registered image,
The partial image matching unit 20 of the image processing unit compares the feature quantities of the plurality of partial search images corresponding to the feature quantities of the partial registration image specified in the partial area of the registration image, respectively. Determine whether or not the partial search image and the partial registration image match,
When the partial image collation unit 20 determines that the partial search image matches the partial registration image, the image position detection unit 22 of the image processing unit performs geometric conversion parameters of the search image with respect to the registration image. An image collating method characterized in that the position of the partial search image is calculated and detected based on the above.

[Appendix 10]
An image collating apparatus comprising a storage unit that stores a registered image having a predetermined feature amount, and an image processing unit that compares the registered image with a search target search image,
A geometric transformation parameter estimation procedure for estimating a geometric transformation parameter based on each feature point of the registered image and the search image and its arrangement;
A partial region determination procedure for determining a plurality of projection partial image regions in a search image corresponding to each partial region image in a registered image using the estimated geometric transformation parameter;
A feature amount calculating procedure for calculating each feature amount of each of the plurality of projection partial image regions of the search image and each of the partial region images of the registered image;
By comparing the feature quantities of the plurality of partial search images corresponding to the feature quantities of the partial registration image specified in the partial area of the registration image, the partial search image and the partial registration image are compared. Partial image matching procedure to determine match,
And an image position for calculating and detecting the position of the matched partial search image using the geometric transformation parameter when it is determined in the partial image matching procedure that the partial search image and the partial registered image match. A detection procedure,
An image collation processing program characterized by causing a computer provided in the image processing means to execute this.

This application claims priority based on Japanese Patent Application No. 2012-038255 filed on February 24, 2012, the entire disclosure of which is incorporated herein.

The present invention can be used for all information processing apparatuses that search for a part that is the same as or similar to a part of the registered image RI from the search image.

DESCRIPTION OF SYMBOLS 10 Memory | storage means 12 Image processing means 14 Partial area determination part 16 Geometric transformation parameter estimation part 16a Candidate parameter calculation function 16d Multiple parameter calculation function 18 Partial area feature-value calculation part 20 Partial image collation part 22 Image position detection part 22a Parameter reliability calculation Function 22b Image position estimation function 22c Corresponding position calculation function 22d Projection position calculation function 22e Position candidate calculation function 24 Integration processing unit 24a Union position determination function 24b Search image position determination function 24c Other image position determination function 26 Display control unit 95 Display device

Claims

An image collating apparatus comprising storage means for storing a registered image having a predetermined feature point, and image processing means for comparing the registered image with a search image to be collated.
The image processing means
A geometric transformation parameter estimation unit that estimates a geometric transformation parameter based on each feature point and the arrangement of the registered image and the search image;
A partial region determination unit that determines a projection partial image region in a search image corresponding to each partial region image in a registered image using the geometric conversion parameter estimated by the geometric conversion parameter estimation unit;
A partial region feature amount calculation unit for calculating each feature amount of each of the plurality of projection partial image regions of the search image and each of the partial region images of the registered image;
A match between the partial search image and the partial registration image is determined by comparing feature amounts of the plurality of partial search images with the feature amount of the partial registration image specified in the partial area of the registration image. A partial image matching unit;
A registered image position detection unit that calculates and detects the position of the matched partial search image using the geometric transformation parameter.
The image collating apparatus according to claim 1,
The geometric transformation parameter estimation unit includes a parameter estimation function for calculating and estimating the geometric transformation parameters as candidates for the partial search image for a plurality of sets,
The image position detecting unit calculates the degree of coincidence between the partial search image and the partial registered image for each estimated geometric transformation parameter, and sets the reliability of the geometric transformation parameter having a high degree of coincidence to be high. A degree calculation function, and an image position estimation function for calculating and estimating in which area in the search image an image area corresponding to the registered image is located using the geometric transformation parameter with high reliability. An image collating apparatus characterized by comprising:
In the image collation device according to claim 1 or 2,
The image position detection unit includes a corresponding position calculation function for calculating the position of the partial search image using the geometric transformation parameter and the preset coordinates of the partial registration image. Verification device.
In the image collation device according to claim 1, 2, or 3,
The geometric transformation parameter estimation unit includes a multiple parameter estimation function for estimating a plurality of geometric transformation parameters for each partial search image when a plurality of partial regions are calculated by the partial region determination unit,
The image position detection unit includes a position candidate calculation function for calculating a position candidate of the matched partial search image as a position candidate for each geometric transformation parameter,
An integration processing unit that calculates the position of the partial search image by integrating the position candidates for each of the overlapping partial search images when the plurality of partial search images overlap in the search image. An image collation device characterized in that it is also provided.
The image collating apparatus according to claim 4,
An image collating apparatus, wherein the integration processing unit includes a union position determination function that calculates a union of the plurality of position candidates and determines a position of the union as a position of the partial search image.
The image collating apparatus according to claim 4,
The integrated processing unit includes a search image position determination function that calculates the reliability of the geometric conversion parameter and determines the position of the partial search image using the geometric conversion parameter with the high reliability. Image collation device.
The image collating apparatus according to claim 4,
The integration processing unit controls voting of the geometric transformation parameters to the parameter space, and determines the position of the partial search image using a plurality of statistical values of the geometric transformation parameters whose voting results are equal to or greater than a threshold value. An image collating apparatus comprising a second search image position determining function.
In the image collating device according to any one of claims 1 to 7,
The image collating apparatus, wherein the image processing means includes a display control unit that performs display control at a position corresponding to the partial search image of a display device that is preliminarily equipped with display data DP related to the partial registration image. .
An image collating apparatus comprising a storage unit that stores a registered image having a predetermined feature amount, and an image processing unit that compares the registered image with a search target search image,
Based on each feature point and the arrangement of the registered image and the search image, a geometric transformation parameter estimation unit of the image processing means estimates a geometric transformation parameter,
The partial region determination unit of the image processing means determines a plurality of projection partial image regions in the search image corresponding to each partial region image in the registered image using the estimated geometric transformation parameter,
A partial region feature amount calculation unit of the image processing means calculates each feature amount of each of the plurality of projection partial image regions of the search image and each of the partial region images of the registered image,
The partial image collation unit of the image processing unit compares the feature amounts of the plurality of partial search images corresponding to the feature amounts of the partial registration image specified in the partial region of the registration image, respectively. Determine whether there is a match between the search image and the partial registration image,
When the partial image collation unit determines that the partial search image and the partial registration image match, the image position detection unit of the image processing unit is based on a geometric transformation parameter of the search image with respect to the registration image. An image collating method characterized in that the position of the partial search image is calculated and detected.
An image collating apparatus comprising a storage unit that stores a registered image having a predetermined feature amount, and an image processing unit that compares the registered image with a search target search image,
A geometric transformation parameter estimation procedure for estimating a geometric transformation parameter based on each feature point of the registered image and the search image and its arrangement;
A partial region determination procedure for determining a plurality of projection partial image regions in a search image corresponding to each partial region image in a registered image using the estimated geometric transformation parameter;
A partial region feature amount calculation procedure for calculating each feature amount of each of the plurality of projection partial image regions of the search image and each partial region image of the registered image;
By comparing the feature quantities of the plurality of partial search images corresponding to the feature quantities of the partial registration image specified in the partial area of the registration image, the partial search image and the partial registration image are compared. Partial image matching procedure to determine match,
And an image position for calculating and detecting the position of the matched partial search image using the geometric transformation parameter when it is determined in the partial image matching procedure that the partial search image and the partial registered image match. A detection procedure,
An image collation processing program characterized by causing a computer provided in the image processing means to execute this.