JP2019028700A - Verification device, method, and program - Google Patents

Abstract

To provide a verification device, a method, and a program capable of highly accurately verifying more diverse objects.SOLUTION: A feature extraction unit 120 applies a convolution neural network to each of two images and obtains an output of a convolution layer for each partial area. A correspondence candidate calculation unit 130 obtains a cosine similarity of an output of the convolution layer for each combination of the partial area of one image and the partial area of the other image, when the cosine similarity becomes a value higher than a threshold value and a partial region of an other image having a maximum cosine similarity with respect to a partial region A coincides with a partial region B and a partial region of one image having a maximum cosine similarity with respect to the partial region B coincides with the partial region A, a combination of the partial region A of one image and the partial region B of the other image is respectively selected as a corresponding candidate. A verification unit 140 determines suitability of the correspondence candidate based on position coordinates of a partial area for each of the correspondence candidates and outputs the correspondence candidate as a correspondence when the correspondence candidate is appropriate.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a verification apparatus, method, and program, and more particularly, to a verification apparatus, method, and program for verifying correspondence between two images.

  The progress of image recognition technology is remarkable. Conventionally, the use areas where the recognition target and environment are limited, such as face / fingerprint authentication and factory automation, have been the focus. Recently, with the spread of small imaging devices such as smartphones, there is an increasing industrial demand for recognition of free shot images in which a general user has shot an arbitrary object in a free place or environment. Expectations are particularly high for O2O services that connect products in the real world and the web world, information guidance / navigation services that provide information by recognizing various landmarks in the real environment, and robot agents.

  There are several forms of image recognition technology for such new applications, but one of the typical ones is recognition technology based on image retrieval. That is, a database of images (referred to as reference images) obtained by capturing an object to be recognized in advance is constructed, and a reference image in the reference database is searched for an image similar to the captured query image. Thus, an object existing in the query image is specified.

  In order to achieve the above object, it is not sufficient to simply search for a similar image, and it is necessary to have a function for accurately searching for an image showing the same object. Normally, the same object is not always captured in the same position, orientation (partial area angle), and size in every image, but is usually captured from various viewpoints. is there. In particular, it is almost impossible to know in advance what kind of viewpoint an object is shot in advance for images that are taken freely by general users, and the appearance of images changes greatly. There are many cases. Therefore, there is a problem that desired image recognition cannot be realized even if a search is performed by simply measuring the similarity between images.

  In view of such problems, verification techniques for verifying whether the same object exists regardless of the shooting viewpoint and realizing effective search have been invented and disclosed.

  Non-Patent Document 1 discloses a verification method based on a Scale Invariant Feature Transform (SIFT) feature and a generalized Hough transform. First, by analyzing the luminance value of each image, a large number of partial areas with significant luminance changes are extracted, and the luminance changes of each partial area are invariant to size and rotation. Expressed as a quantity vector (SIFT feature). Next, the Euclidean distance between SIFT features is measured for partial areas included in two different images, and partial areas between different images having a small value are obtained as correspondence candidates. Furthermore, for partial areas obtained from the same object, based on the assumption that changes in position, posture, and size between corresponding partial areas on the object are consistent regardless of the shooting viewpoint, the candidate for correspondence The “deviation” of the position / posture / size between the partial areas is obtained. Assuming that the set of corresponding subregions obtained from the same object constitutes a histogram of deviation under the assumption that this deviation is consistent, these may be concentrated in a very small number of bins. is assumed. Accordingly, only the correspondence candidates distributed in the bins with high frequency are regarded as truly effective correspondences, and the other candidates are deleted as not being effective correspondences. As a result, an image having a large number of effective correspondences is searched as an image having the same object.

  Patent Document 1 discloses a technique obtained by improving Non-Patent Document 1. It is the same as finding the corresponding candidate for the partial area based on the SIFT feature and calculating the deviation of the position / posture / size to determine the suitability of the correspondence. However, when evaluating the deviation, the three-dimensional rotation angle is set. thinking. As a result, a finer verification than that of Non-Patent Document 1 is possible.

  The technique disclosed in Non-Patent Document 2 is the same until the corresponding candidates of the partial areas between different images are obtained using the SIFT feature, but the position of the partial area when a plurality of corresponding candidates are viewed as a set. This is a method of deleting a non-valid correspondence candidate by regarding only a correspondence candidate that is restricted by a specific linear transformation as an effective correspondence.

JP-A-2015-95156

D.G.Lowe, “Distinctive Image Features from Scale-Invariant Keypoints”, International Journal of Computer Vision, pp.91-110, 2004 J. Philbin, O. Chum, M. Isard, Josef Sivic and Andrew Zisserman.Object retrieval with large vocabularies and fast spatial matching 1470-1477, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2007.

  From a broad perspective, the existing invention first obtains a corresponding candidate based on the distance between SIFT features, and then verifies the suitability of the corresponding candidate by analyzing the displacement of the position of the partial region.

  However, the verification based on the SIFT feature has a problem that it cannot perform high-precision verification for various objects. That is, since the verification according to the prior art is premised on the basis of feature vector correspondence candidates, the accuracy depends on the ability to express SIFT features. SIFT features (or other local features similar to this) have the characteristic of describing significant luminance changes, for example, there are patterns (textures) that are easy to distinguish, so that significant luminance changes are likely to occur. Although very accurate verification is possible for a simple object, it is impossible to obtain an accurate response for an object with no characteristic pattern or many flat parts. Highly accurate verification could not be realized.

  That is, until now, no verification technique has been invented that can verify the presence or absence of the same object with high accuracy for various objects.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to provide a verification device, method, and program that enable highly accurate verification of a wider variety of objects.

  In order to achieve the above object, a verification apparatus according to the present invention is a verification apparatus for verifying correspondence between a first image and a second image, each of the first image and the second image. And applying a convolutional neural network including at least one convolutional layer to obtain an output of the convolutional layer for each partial region of the image, and for each of the first image and the second image Based on the output of the convolution layer determined for each partial region, for each combination of the partial region of the first image and each of the partial region of the second image, the convolution layer An output cosine similarity is obtained, and the cosine similarity is higher than a predetermined threshold for a combination of the partial area A of the first image and the partial area B of the second image. In addition, the partial area of the second image having the maximum cosine similarity with respect to the partial area A coincides with the partial area B and has the maximum cosine similarity with respect to the partial area B. Correspondence candidate calculation for selecting each combination of the partial area A of the first image and the partial area B of the second image as a corresponding candidate when the partial area of the first image matches the partial area A Verification of determining whether or not the correspondence candidate is appropriate based on the position coordinates in the image of the partial area for each of the correspondence candidates and the correspondence candidate, and outputting the correspondence candidate as a correspondence if the correspondence candidate is appropriate Part.

  The verification method according to the present invention is a verification method in a verification device for verifying correspondence between a first image and a second image, wherein the feature extraction unit includes the first image and the second image. For each of the above, a convolutional neural network including at least one convolutional layer is applied, the output of the convolutional layer is obtained for each partial region of the image, and the correspondence candidate calculation unit is configured to output the first image and the second image. For each combination of each of the partial regions of the first image and each of the partial regions of the second image, based on the output of the convolution layer determined for each of the partial regions for each of The cosine similarity of the output of the convolution layer is obtained, and the cosine similarity is greater than a predetermined threshold for the combination of the partial area A of the first image and the partial area B of the second image. The partial area of the second image that has a high value and the maximum cosine similarity with respect to the partial area A coincides with the partial area B and has the maximum cosine similarity with respect to the partial area B When the partial area of the first image that coincides with the partial area A, a combination of the partial area A of the first image and the partial area B of the second image is selected as a corresponding candidate. The verification unit determines whether or not the corresponding candidate is appropriate based on the position coordinates in the partial region image for each of the corresponding candidates. If the corresponding candidate is appropriate, the corresponding candidate is determined to be corresponding. It is characterized by outputting.

  Further, the program according to the present invention is a program for causing a computer to function as each unit of the verification device according to the above invention.

  According to the verification apparatus, method, and program of the present invention, a convolutional neural network is applied to each of the first image and the second image to obtain an output of the convolution layer for each partial region, and For each combination of the partial areas of one image and each of the partial areas of the second image, the cosine similarity of the output of the convolution layer is obtained, and the cosine similarity is lower than a predetermined threshold value. The partial area of the second image that has a high value and the maximum cosine similarity with respect to the partial area A coincides with the partial area B and has the maximum cosine similarity with respect to the partial area B When the partial area of the first image that coincides with the partial area A corresponds to the combination of the partial area A of the first image and the partial area B of the second image Each is selected as a complement, and the suitability of the correspondence candidate is determined based on the position coordinates in the image of the partial area for each of the correspondence candidates. If the correspondence candidate is appropriate, the correspondence candidate is regarded as correspondence By outputting, it is possible to obtain an effect of enabling highly accurate verification with respect to a wider variety of objects.

It is a block diagram which shows the structure of the verification apparatus which concerns on embodiment of this invention. It is a flowchart which shows the verification process routine in the verification apparatus which concerns on embodiment of this invention. It is a figure which shows the output of a convolution layer. It is a figure which shows the geometric relationship of the geometric information of the partial area | region of two images.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Principle according to the embodiment of the present invention>
First, the principle in the embodiment of the present invention will be described.

  In the embodiment of the present invention, the feature of each partial region in the image is expressed by using the output of the convolution layer of the convolutional neural network. In general, a convolutional neural network can be configured with a plurality of complex convolution filters, thereby realizing high expression capability. As a result, an accurate correspondence candidate can be obtained even for an object that does not have a noticeable luminance change, such as a conventional feature such as a SIFT feature that cannot be targeted.

  Furthermore, in the embodiment of the present invention, in obtaining this correspondence candidate, the output of the convolution layer for each partial region (that is, the response of one or more convolution filters constituting the convolution layer) between the two images. A cosine similarity is obtained, and a combination of partial areas in which the similarity is equal to or greater than a threshold value and has a maximum cosine similarity when compared with two images is determined as a corresponding candidate. The cosine similarity can more accurately determine the similarity of the filter response of the convolutional neural network than other various distance measures, and in particular, it exceeds a certain value (for example, 0.5 or more). Can give a corresponding candidate with high accuracy. In addition, if the partial area is the same object (or part of it), it should have the maximum cosine similarity with each other. Therefore, a more reliable response candidate can be obtained. Furthermore, in the embodiment of the present invention, it is possible to perform highly accurate verification as a result by determining the geometric consistency of the correspondence candidates and obtaining the final correspondence.

  The embodiment of the present invention is not realized by a simple combination of a general convolutional neural network and a conventional verification method. First, in a conventional convolutional neural network, as described in Reference Document 1, Reference Document 2, and Reference Document 3, for example, several convolution layers made up of one or more convolution filters are applied to an input image. After obtaining the output of the convolution layer, the output is aggregated in the spatial direction and expressed as a fixed-dimensional vector. For example, in Reference 1, all the elements of the output of the convolution layer are linearly combined, in Reference 2, by taking an average in the spatial direction, and in Reference 3, a specific space Aggregation in the spatial direction is performed by taking the maximum value or the sum of the regions.

[Reference 1] Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton: ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of the Neural Information Processing Systems, pp. 1106-1114, 2012.

[Reference 2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun: Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778., 2016.

[Reference 3] Giorgos Tolias, Ronan Sicre, Herve Jegou: Particular Object Retrieval with Integral Max-Pooling of CNN Activations, arXiv: 1511.05879. 2015.

  This method of aggregation in the spatial direction can reduce the final representation size (vector dimension) while reducing the spatial representation performance of the output of the convolution layer. It is not an appropriate expression for technical requirements. Therefore, in the embodiment of the present invention, the aggregation processing for the output of the convolution layer is eliminated and used as it is. Such a procedure is not disclosed in any of the above references 1 to 3.

  In addition, the output of the convolution layer is extracted with uniform size and spacing in the image, and is different from the SIFT feature in that it is a high-dimensional and sparse non-negative vector. To achieve this, a procedure suitable for the output characteristics of the convolutional layer is required. In the embodiment of the present invention, in consideration of the fact that the output of the convolution layer is a sparse high-dimensional vector and the norm of the vector is not useful in obtaining the correspondence candidate, the correspondence between the convolution layers is obtained using the cosine similarity. Only when this exceeds a certain value and the two images have the maximum similarity between the two images is left as a corresponding candidate. With such a procedure, it is possible to obtain correspondence candidates with high accuracy.

  In addition, the convolution layer output extracted at a uniform size and spacing gives a homogeneous and exhaustive representation of the entire image, while subregions that are not characteristic (such as the sky and roads in the background). Since the output is also obtained from a partial area that is not important for distinguishing the object), it often causes erroneous correspondence. Therefore, in the embodiment of the present invention, an output of a representative convolution layer calculated from a partial area that is not characteristic is obtained in advance, and a process of removing the partial area corresponding to this is applied in advance. This suppresses mishandling. Such an idea is not described in any of Reference Documents 1 to 3 and Prior Art Documents.

  As described above, according to the embodiment of the present invention, it is possible to perform highly accurate verification on various objects.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

<< Overall structure >>
FIG. 1 is a block diagram showing an example of the configuration of the verification apparatus 100 according to the embodiment of the present invention. The verification apparatus 100 illustrated in FIG. 1 includes an input unit 110, a feature extraction unit 120, a correspondence candidate calculation unit 130, a verification unit 140, and an output unit 150.

  The verification apparatus 100 is connected to the reference database 160 via the input unit 110 via a communication unit and communicates information with each other, so that arbitrary image information can be registered in the database or image information can be read out. Use a possible configuration.

  The image information here includes the image itself (image file), the geometric information of the partial area of the image, and the feature vector, and if they relate to the same image, they are associated with each other. To do. In particular, it is assumed that the reference database 160 includes image information related to a reference image to be searched for a query image. The partial region of the image may be determined in any way as long as it is a partial region of the image, but preferably a partial region defined by a convolution filter of a convolution layer of a convolutional neural network is used (details will be described later). .

  Further, the geometric information of the partial area includes the position and size of the partial area. However, in the example of the embodiment of the present invention, at least the position may be included.

  The reference database 160 can be configured by, for example, a file system implemented on a general general-purpose computer. Each reference image file shall be given an identifier (for example, ID by serial number, unique file name, etc.) that can uniquely identify each file, and a file that describes a partial area defined in the image and a feature vector Are stored in association with the identifier of the image. Alternatively, it may also be implemented and configured by RDBMS (Relational Database Management System) or the like. In addition, the metadata may include, for example, data representing the content of the image (image title, summary text, or keyword), data related to the image format (image data amount, thumbnail size, etc.), and the like. Although not required, it is not essential in the practice of the present invention.

  The reference database 160 may be inside or outside the verification apparatus 100, and any known communication means can be used. However, in the present embodiment, the communication is assumed to be outside. The means is assumed to be connected to communicate via the Internet or TCP / IP.

  Further, each unit included in the verification apparatus 100 and the reference database 160 may be configured by a computer, a server, or the like including an arithmetic processing device, a storage device, and the like, and the processing of each unit may be executed by a program. This program is stored in a storage device included in the verification apparatus 100 or the reference database 160, and can be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or provided through a network. Of course, any other components need not be realized by a single computer or server, but may be realized by being distributed to a plurality of computers connected by a network.

  Note that the image information itself does not necessarily have to be stored in the reference database 160. For example, the image information itself may be directly input from the outside via the input unit 110 as appropriate. In such a configuration, for example, when the present invention is used for object search, the reference image is stored in the reference database 160 after performing necessary processing for the reference image, and the query image 170 is suitable for an application in which an input of a query image is received from the outside at an inquiry timing as appropriate to obtain processing image information. As a specific example, in the example of the configuration of the verification apparatus 100 illustrated in FIG. 1, image information related to one or more reference images is stored in the reference database 160 in advance, and these are as described above. Are connected in a form that can be read / registered with each other. In addition, a configuration is adopted in which a query image 170 input as an inquiry can be received from the outside.

  Hereinafter, in an example of an embodiment of the present invention, when verifying the correspondence between two images, one of the reference images registered in the reference database 160 and a query, particularly with a view to use in object search, A case where the correspondence with one query image input as the image 170 is verified will be described as an example. When the correspondence between a plurality of sets of reference images and query images is verified, the processing described below may be repeated as many times as necessary for the number of sets to be verified. Note that the reference image and the query image are examples of the first image and the second image.

<< Processor >>
Each processing unit of the verification apparatus 100 in the present embodiment will be described.

  The input unit 110 is an interface that accepts input from the outside to the verification apparatus 100, reads a reference image or feature amount file from the reference database 160, and accepts an image such as a query image 170 from the outside to receive each processing unit. Communicate to.

  The feature extraction unit 120 receives a reference image or a query image via the input unit 110, and applies a convolutional neural network to the image to extract a feature vector for each partial region. The extracted partial region and feature quantity vector are transmitted to the correspondence candidate calculation unit 130 or the output unit 150.

  The correspondence candidate calculation unit 130 selects one image based on image information (at least a feature amount vector for each partial region) of two images (for example, a reference image and a query image) received from the input unit 110 or the feature extraction unit 120. The cosine similarity of the feature amount vector is obtained for each combination of each of the partial areas of the image and the partial area of the other image, and a correspondence candidate between the two images is obtained based on the cosine similarity. .

  The verification unit 140 obtains the geometric relationship for each of the sets of geometric information of the partial areas that are correspondence candidates for the two images, and determines whether or not the correspondence candidate is appropriate based on the geometric relationship. If appropriate, a determination result corresponding to this is output.

<< Process overview >>
Next, the process of the verification apparatus 100 in this Embodiment is demonstrated. Processing in the embodiment of the present invention is roughly divided into offline processing and online processing. The former is a process that needs to be performed at least once for each reference image stored in the reference database 160, and the latter is a process that is performed by using the input of a query image as a trigger when performing an actual search. It is. Hereinafter, this will be described in order.

  The off-line processing is simple, and a feature vector for each partial region is extracted by applying a convolutional neural network to each of the reference images registered in the reference database 160 and stored in the reference database 160.

  Since online processing has more steps than offline processing, it will be described with reference to FIG.

  First, in step S201, when a query image is received, a convolutional neural network is applied to the query image to extract a feature vector for each partial region.

  Subsequently, in step S202, the cosine similarity between the feature quantity vector of each partial area of the query image and the feature quantity vector of each partial area of the reference image stored in the reference database 160 is obtained, and based on this cosine similarity degree. Then, correspondence candidates that are combinations of partial areas between the query image and the reference image are respectively obtained.

  Subsequently, in step S203, for each correspondence candidate, a geometric relationship is obtained with respect to a combination of the geometric information of the partial area on the reference image side and the geometric information of the partial area on the query image side that is the corresponding candidate, Based on the geometric relationship obtained for each correspondence candidate, it is determined whether or not each correspondence candidate is appropriate. 180 is output by the output unit 150.

  With the above processing, the correspondence between the input query image and the partial region of the reference image can be verified.

<< Details of each process >>
Hereinafter, an example of the detailed processing of each processing will be described in the present embodiment.

[Feature extraction processing]
First, a method for extracting a partial region and a feature vector from an input image will be described.

  In the embodiment of the present invention, a convolutional neural network is applied to an image to extract a feature vector of each partial region. Various known variations have been proposed for convolutional neural networks, but they are configured using at least one convolutional layer (a layer of a neural network defined by a set of convolutional filters having the same size and skip width). Any of those described in Reference Document 1, Reference Document 2, etc. may be used.

  Here, no matter what convolutional neural network is used, the output of the convolutional layer is obtained as the termination output in the embodiment of the present invention. For example, since many convolutional neural networks including those described in Reference 1 include a pooling layer and a fully connected layer after a plurality of convolutional layers, the final output of the convolutional neural network is the output of the fully connected layer. It will be. However, with this method of aggregation in the spatial direction, the final representation size (vector dimension) can be kept small, while the spatial representation performance of the output of the convolutional layer is impaired. There's a problem. Therefore, in the embodiment of the present invention, the aggregation processing for the output of the convolution layer is eliminated and used as it is. More specifically, the output of a certain convolution layer constituting the convolutional neural network is obtained and used as a feature vector. Preferably, the output of the convolutional layer close to the final output layer of the convolutional neural network is used.

  FIG. 3 illustrates the output of a formal convolution layer. The output of the convolution layer is usually expressed as a third-order tensor having a height (h), a width (w), and a depth (d). That is, it is a three-dimensional array having h × w × d elements. In other words, this takes a partial area of height h × width w for the input image and outputs a d-dimensional vector in each partial area. In the embodiment of the present invention, this is exactly obtained as a feature vector for each partial region.

  Strictly speaking, as to which partial area of the original input image the position of each output element of the convolution layer (each of the cells divided in the grid of FIG. 3) corresponds to the convolutional neural network. It is determined depending on the configuration of the convolutional layer of the network. In the embodiment of the present invention, the output element at the position corresponding to each partial area may be taken after dividing the original input image into h × w.

  Preferably, L2 normalization is performed on each d-dimensional feature vector. This is preferable because the subsequent cosine similarity is equivalent to the inner product operation.

  As described above, the feature vector for each partial region can be obtained for the input image.

[Correspondence candidate calculation processing]
Next, processing for obtaining a correspondence candidate between partial areas defined between two different images will be described. In an example of the embodiment of the present invention, the process is used to determine a corresponding partial area between a query image and each reference image.

A partial region extracted from the query image (that is, the output element of the previous convolution layer) is represented as Q _i , and a partial region extracted from the reference image is represented as R _j . Hereinafter, taking the partial areas Q _i and R _j as an example, processing for determining whether or not these are correspondence candidates will be described.

Each partial region is associated with a feature vector that expresses the partial region extracted by the convolutional neural network. Assume that a feature vector describing the partial region Q _i is represented by q, and a feature vector describing R _j is represented by r. At this time, sim (Q _i , R _j ) is obtained from the following equation for the cosine similarity between the partial regions.

... (1)

  Here, || q || represents the L2 norm of q. If, in the feature extraction process, each feature vector is L2 normalized, || q || = || r || = 1, the above equation (1) is equivalent to the following equation: is there.

  Usually, in verification using SIFT features or the like, correspondence candidates are often obtained using the Euclidean distance ratio (for example, Non-Patent Document 1). However, the output of the convolution layer is often a very sparse high-dimensional vector compared to SIFT features, etc., and it is often not useful to evaluate including the norm of the vector in order to obtain a corresponding candidate. . Therefore, in the embodiment of the present invention, in order to obtain a more accurate similarity, cosine similarity is used, and when there are combinations of partial areas that satisfy both of the following two conditions, they are regarded as correspondence candidates. To do.

<<< Condition 1: High similarity >>>
The cosine similarity sim (Q _i , R _j ) takes a value from −1 to 1, and the larger the value, the closer the feature vector is. In the embodiment of the present invention, an object is to find a combination of partial regions having close feature vector values, and therefore only those having a high value need be considered. From this viewpoint, the cosine similarity sim (Q _i , R _j ) is required to have a value equal to or greater than a certain threshold value. This threshold is preferably set to 0.5, for example.

<<< Condition 2: Interactivity >>>
When attention is paid to the partial region R _j of the reference image, it is assumed that the partial region of the query image closest to this (the highest similarity) is Q _i . At this time, conversely, when the partial area Q _i of the query image is focused, the partial area of the reference image closest to this is also assumed to be R _j .

  By executing the above calculation for all combinations of partial areas between the query and the reference image, it is possible to obtain partial areas that are correspondence candidates.

  Note that, when features are extracted by a convolutional neural network, the partial regions are extracted uniformly in the normal image. As a result, in the later correspondence candidate calculation process, non-characteristic areas in the image where no object exists, such as the sky, correspond to each other, which may adversely affect recognition accuracy. In order to prevent such an undesirable response, a “taboo partial area” may be configured.

  This tabu partial area is configured by a feature vector extracted from an area that should not be a corresponding candidate in advance. For example, it is assumed that a feature vector that tends to appear in the sky is to be excluded so as not to become a corresponding candidate. At this time, the feature quantity vector of the partial area extracted from the partial area corresponding to the sky in advance is stored as a taboo partial area. If there are a large number of tabu partial areas to be extracted, a representative feature vector is selected using a clustering method (k-means, etc.) as necessary, and only the selected representative feature vector is tabued. It may be stored as a partial area.

  Thereafter, when the partial region and the feature vector are extracted from the reference image or the query image, the cosine similarity of the feature vector for the combination of the extracted partial region and the taboo partial region is obtained, and this cosine similarity is When the conditions 1 and 2 are satisfied, the correspondence candidate including the partial region is removed.

  That is, the taboo portion in which the cosine similarity is higher than a predetermined threshold for the combination of the partial region C and the tabular partial region D of the reference image or the query image, and the maximum cosine similarity is obtained with respect to the partial region C. The partial area C is included when the area matches the tabu partial area D and the partial area of the reference image or query image that has the maximum cosine similarity with the tabu partial area D matches the partial area C. Remove correspondence candidates.

  When selecting a taboo partial area, it is more preferable if it can be expressed as a category as in the previous sky. This is because, for example, an area that falls within a certain category can be automatically detected by an image recognition method called known semantic segmentation, such as Reference 4, so that it is possible to reduce the labor of manually selecting a tabu partial area. .

[Reference 4] Jonathan Long, Evan Shelhamer, Trevor Darrell: Fully Convolutional Networks for Semantic Segmentation. In Proc. Conference on Computer Vision Pattern Recognition, pp. 3431-3440, 2015.

[Verification processing]
Subsequently, based on the geometric information of the partial area, it is determined whether or not the obtained correspondence candidate of the partial area is appropriate. That is, among the correspondence candidates obtained by the correspondence candidate calculation unit 130, correspondence candidates that are combinations of partial areas that are considered not to be effective correspondences (that is, not correspondences between the partial areas extracted from the object) are deleted. .

  Suppose that the query image and the reference image include the same object. If the objects have approximately the same shape, the object in the query image and the object in the reference image are only taken from different viewpoints. To give consistency. In other words, if the partial areas that are candidates for correspondence are a combination of partial areas that are correctly present on the same object, the geometric information of the partial area on the query image side and the reference that is a candidate for correspondence The geometric relationship (displacement) with the geometric information of the partial area on the image side is consistent with other appropriate correspondence candidates. Therefore, it is only necessary to consider only correspondence candidates that are consistent in this way of deviation as effective correspondences, and reject those that are not.

  This will be described in an easy-to-understand manner with reference to FIG. FIG. 4 shows two images A and B including the same object. Each of the two types of patterns 41A, 41B, 42A, and 42B surrounded by a broken line is defined as a partial area, and a combination of partial areas (for example, 41A and 41B) represented by the same number is used as a corresponding candidate. Assume that it has been determined. The purpose is to determine only a combination of partial areas existing on the same object (in this case 41A and 41B and 42A and 42B) as an effective correspondence.

  As can be seen from FIG. 4, the positions of 41A and 41B and 42A and 42B on the same object change in the same way depending on only the viewpoint, so the relative positional relationship is approximately the same. It can be seen that it is. Therefore, by determining whether the relative positional relationship of the combination of partial areas that are correspondence candidates is consistent with the relative positional relationship of the combination of partial areas that are other appropriate correspondences, It is possible to determine suitability.

  Therefore, in the embodiment of the present invention, this is realized by using a geometric relationship verification method. For example, it is known that the geometric relationship between partial areas on the same object is constrained by linear transformation under appropriate conditions. There are known effective methods such as the RANSAC algorithm described in Reference 5 and the LO-RANSAC algorithm described in Reference 6 as a method for obtaining a correspondence candidate having such a linear transformation and a geometric relationship according to the linear transformation. These may be used.

[Reference 5] MA Fischler and RC Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” Comm. ACM, vol. 24, no. 6, pp. 381-395, 1981 .

[Reference 6] O. Chum, J. Matas, and S. Obdrzalek, “Enhancing RANSAC by generalized model optimization,” Proceedings of Asian Conference on Computer Vision, pp. 812-817, 2004.

  According to the above procedure, the suitability of the correspondence candidate is determined for each of the correspondence candidates between the query image and the reference image, and if the correspondence candidate is appropriate, the correspondence candidate can be output as correspondence, and the same It can be determined whether or not an object is included.

  As described above, according to the verification device according to the embodiment of the present invention, the convolutional neural network is applied to each of the query image and the reference image, the output of the convolution layer is obtained for each partial region, and the query image For each combination of each of the partial areas and each of the partial areas of the reference image, the cosine similarity of the output of the convolution layer is obtained, the cosine similarity is a value higher than a predetermined threshold, and the partial area A When the partial area of the reference image having the maximum cosine similarity matches the partial area B, and the partial area of the query image having the maximum cosine similarity with respect to the partial area B matches the partial area A , A combination of the partial area A of the query image and the partial area B of the reference image is selected as a corresponding candidate, Based on the position coordinates, to determine the appropriateness of the candidate corresponding, if the corresponding candidate is appropriate by outputting the corresponding candidate as the corresponding, it enables high-accuracy verification against a wider variety of objects.

  In addition, by using a convolutional neural network that has higher expressive capabilities than ordinary feature vectors, and verifying the geometric consistency of the output of the convolutional layer, it enables high-precision verification for a wider variety of objects. .

  Heretofore, an example of the configuration of the verification apparatus according to an example of the embodiment of the present invention has been described in detail. The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 100 Verification apparatus 110 Input part 120 Feature extraction part 130 Correspondence candidate calculation part 140 Verification part 150 Output part 160 Reference database 170 Query image

Claims (6)

A verification device for verifying correspondence between a first image and a second image,
Applying a convolutional neural network including at least one convolution layer for each of the first image and the second image, and a feature extraction unit for obtaining an output of the convolution layer for each partial region of the image;
Based on the output of the convolution layer determined for each partial region for each of the first image and the second image, each of the partial regions of the first image and the second image of the second image For each combination with each of the partial regions, obtain the cosine similarity of the output of the convolution layer,
For the combination of the partial area A of the first image and the partial area B of the second image, the cosine similarity is a value higher than a predetermined threshold, and the cosine similarity is maximum with respect to the partial area A. The partial area of the second image that coincides with the partial area B and the partial area of the first image that has the maximum cosine similarity with the partial area B is the partial area A. A correspondence candidate calculation unit that selects a combination of the partial area A of the first image and the partial area B of the second image as a correspondence candidate when they match,
A verification unit that determines the suitability of the correspondence candidate based on the position coordinates in the image of the partial region for each of the correspondence candidates, and outputs the correspondence candidate as a correspondence when the correspondence candidate is appropriate;
A verification apparatus comprising: The correspondence candidate calculation unit stores, as a tabu partial region, a set of partial regions that are not appropriate for correspondence and a corresponding convolution layer output,
Furthermore, based on the output of the convolutional layer obtained for each partial area for each of the first image and the second image, each of the partial areas of the first image and the tabu partial areas For each combination with each and each combination of the partial region of the second image and each of the tabu partial regions, obtain the cosine similarity of the output of the convolution layer,
For the combination of the partial area C and the taboo partial area D of the first image or the second image, the cosine similarity is a value higher than a predetermined threshold, and the maximum cosine with respect to the partial area C The tabu partial area that is the similarity is the tabu partial area D, and the partial area of the first image or the second image that has the maximum cosine similarity with respect to the tabu partial area D is the The verification apparatus according to claim 1, wherein, when matching with the partial area C, a combination including the partial area C is excluded from the correspondence candidates.   The verification unit, based on the position coordinates in the image of the partial area for each of the corresponding candidates, according to the consistency with the relative positional relationship in the combination of the partial areas for the other corresponding candidates The verification apparatus according to claim 1, wherein the suitability of the candidate is determined. A verification method in a verification device for verifying correspondence between a first image and a second image,
A feature extraction unit applies a convolutional neural network including at least one convolutional layer for each of the first image and the second image, and obtains an output of the convolutional layer for each partial region of the image,
Each of the partial areas of the first image based on the output of the convolutional layer obtained for each partial area for each of the first image and the second image; For each combination with each of the partial regions of the second image, determine the cosine similarity of the output of the convolution layer,
For the combination of the partial area A of the first image and the partial area B of the second image, the cosine similarity is a value higher than a predetermined threshold, and the cosine similarity is maximum with respect to the partial area A. The partial area of the second image that coincides with the partial area B and the partial area of the first image that has the maximum cosine similarity with the partial area B is the partial area A. If they match, each of the combinations of the partial area A of the first image and the partial area B of the second image is selected as a corresponding candidate,
The verification unit determines the suitability of the correspondence candidate based on the position coordinates in the partial region image for each of the correspondence candidates, and outputs the correspondence candidate as a correspondence when the correspondence candidate is appropriate. A verification method characterized by that. By selecting a correspondence candidate by the correspondence candidate calculation unit, a set of partial areas that are not appropriate for correspondence and a corresponding convolution layer output are stored as tabu partial areas,
Furthermore, based on the output of the convolutional layer obtained for each partial area for each of the first image and the second image, each of the partial areas of the first image and the tabu partial areas For each combination with each and each combination of the partial region of the second image and each of the tabu partial regions, obtain the cosine similarity of the output of the convolution layer,
For the combination of the partial area C and the taboo partial area D of the first image or the second image, the cosine similarity is a value higher than a predetermined threshold, and the maximum cosine with respect to the partial area C The tabu partial area that is the similarity is the tabu partial area D, and the partial area of the first image or the second image that has the maximum cosine similarity with respect to the tabu partial area D is the The verification method according to claim 4, wherein, when matching with the partial area C, a combination including the partial area C is excluded from the correspondence candidates.   The program for functioning a computer as each part of the verification apparatus of any one of Claims 1-3.