JP6758250B2 - Local feature expression learning device and method - Google Patents

Description

The present invention relates to a local feature expression learning device and a method, and more particularly to a local feature expression learning device and a method for learning a local feature expression from an image data set including a plurality of images.

Local feature expression technology that features an arbitrary small area (hereinafter referred to as "patch") extracted from an image has numerous applications such as image search, stereo matching, and image editing of images and images. This is the basic technology for pattern recognition. There are two types of local feature expression technology: one in which the local feature expression algorithm represented by SIFT descriptor and SURF descriptor is designed manually, and one in which feature expression is acquired by machine learning from a huge data set. It can be roughly divided.

In Non-Patent Document 1, the L2 distance (norm) of a patch pair that captures the same physical region (same label) becomes smaller and different physical regions are captured from a large-scale patch data set with a label (the same physical region). A method of learning a feature expression using a neural network so that the L2 distance of a patch pair (with different labels) becomes sufficiently large is disclosed.

On the other hand, in Non-Patent Document 2, labels having different L2 distances between patches of the same label, with a triplet (triple set) composed of two patches belonging to the same label and patches of different labels as the minimum input unit. A method of learning a feature expression smaller than the L2 distance between patches by a neural network is disclosed.

On the other hand, Non-Patent Document 3 discloses a method of learning a local feature expression by inputting a data set to which only an image unit label is assigned. Specifically, the similarity calculated between the local feature sets extracted from each image constituting the image pair is defined, and the similarity between the image pairs of the same label is higher than the similarity between the images of different labels. A method of learning a neural network that is expensive has been proposed.

E. Simo-Serra et al., Discriminative Learning of Deep Convolutional Feature Point Descriptors, in ICCV, 2015. Internet <URL: http://cvlabwww.epfl.ch/~trulls/pdf/iccv-2015-deepdesc.pdf> V. Balntas et al., Learning local feature descriptors with triplets and shallow convolutional neural networks, in BMVC, 2016. Internet <URL: http://www.bmva.org/bmvc/2016/papers/paper119/paper119.pdf> Nenad Markus et al., Learning Local Descriptors by Optimizing the Keypoint-Correspondence Criterion, in ICPR, 2016.

However, both methods of Non-Patent Document 1 and Non-Patent Document 2 assume the existence of training data to which correct labels are assigned at the patch level, and a huge number of patches can be applied directly. There is a problem that it has to be labeled manually.

Further, in Non-Patent Document 3, the learning of the local feature expression is realized without using the label of the patch unit by introducing the similarity of the image unit, but on the other hand, this similarity is the total of the local features. It is defined based only on hit comparisons and does not take into account the spatial relationships of local features. The spatial relationship of local features is a great clue to discover patches that are extracted from the same physical region but look different, for example due to geometric and optical factors. It is considered that using these as training data is indispensable for acquiring local feature representations that are robust to geometrical and optical deformations.

From the above, there is a problem that the known technique for acquiring a local feature expression robust to geometrical and optical deformation requires a huge manual cost for labeling a patch for learning. In addition, the known technology that does not require patch-based annotation does not include a mechanism that considers the spatial relationship between local features, and the information potentially possessed by the training data cannot be fully utilized. There was a problem.

The present invention has been made in consideration of the above problems, and local feature expression learning that can learn local feature descriptors that are robust to geometrical and optical deformations without using patch-based annotations. It is an object of the present invention to provide an apparatus and a method.

In order to achieve the above object, the first aspect of the present disclosure is a local feature expression learning device that learns a local feature expression from an image data set containing a plurality of images, and is included in the input image data set. A patch extraction unit that extracts a plurality of patches from each image and an arbitrary image pair included in the image data set are determined to be geometrically identical, and geometric deformation is performed from the image pairs determined to be the same. The same region patch pair extraction unit that extracts the pair of patches that capture the same physical region using parameters, a set of patches consisting of the plurality of patches extracted by the patch extraction unit, and the same region patch pair extraction unit A learning data construction unit that constructs learning data for learning a local feature expression using the extracted pair of patches, and a learning data construction unit that constructs a local feature expression using the training data constructed by the learning data construction unit. Processing from the same region patch pair extraction unit to the local feature expression learning unit using the local feature expression learning unit to be performed and the local feature expression learned by the local feature expression learning unit until a predetermined criterion is satisfied. It is provided with an end determination unit for repeatedly performing the above.

In the second aspect of the present disclosure, in the local feature expression learning device of the first aspect, the same region patch pair extraction unit expresses the local feature expression for each of the plurality of patches extracted by the patch extraction unit. Determining the geometrical identity of the image pair based on the local feature of each patch expressed by the local feature description unit and the image pair included in the image data set. , And the patch pair that captures the same physical region using the deformation parameters for the deformation estimation unit that estimates the geometric deformation parameters and the image pair that is determined to be the same by the deformation estimation unit. It is provided with the same area determination unit for extracting the image.

A third aspect of the present disclosure is the local feature expression learning device of the first aspect or the second aspect, wherein the local feature expression learning unit is the same in the same region patch pair extraction unit using the learning data. Patches are placed in a feature space where the similarity of the patch pairs determined to capture the physical region is high and the similarity of the patch pairs determined to capture a different physical region is low. Learn the neural network to map.

According to the fourth aspect of the present disclosure, in the local feature expression learning device of any one aspect from the first aspect to the third aspect, it is determined that the learning data construction unit does not capture the same physical region. When the patch pair is used as the training data, the determination result of the sameness in the same area patch pair extraction unit is used.

According to the fifth aspect of the present disclosure, in the local feature expression learning device according to any one of the first to third aspects, it is determined that the learning data construction unit does not capture the same physical region. When the pair of patches is used as the training data, the information of the label of the image unit attached to the input image data set is used.

Further, in order to achieve the above object, the sixth aspect of the present disclosure is a local feature expression learning method in a local feature expression learning device that learns a local feature expression from an image data set including a plurality of images, and is a patch. The extraction unit extracts a plurality of patches from each image included in the input image data set, and the same area patch pair extraction unit determines the geometrical identity of any image pair included in the image data set. The step of extracting the pair of patches that capture the same physical region using geometric deformation parameters from the image pairs that are determined to be the same, and the learning data construction unit extracts them with the patch extraction unit. A step of constructing learning data for learning a local feature expression using the set of patches composed of the plurality of patches and the pair of the patches extracted by the same region patch pair extraction unit, and a local feature expression. The step in which the learning unit learns the local feature expression using the learning data constructed by the learning data construction unit, and the end determination unit learns in the local feature expression learning unit until it satisfies a predetermined criterion. It includes a step of repeating the process from the same region patch pair extraction unit to the local feature expression learning unit using the local feature expression.

According to the present disclosure, it is possible to learn a local feature descriptor that is robust to geometrical and optical deformations without using patch-based annotation.

It is a block diagram which shows an example of the image feature apparatus in embodiment. It is a figure for demonstrating an example of the patch extraction method in the patch extraction part of embodiment. It is a figure for demonstrating another example of the patch extraction method in the patch extraction part of embodiment. It is a block diagram which shows an example of the structure of the same area patch pair extraction part of embodiment. It is a figure for demonstrating the operation of the local feature description part, the deformation estimation part, and the same area determination part of the same area patch pair extraction part of embodiment. It is a figure which shows an example of the Siamese network. It is a figure which shows another example of the Siamese network. It is a figure which shows an example of a Triplet network. It is a block diagram which shows the specific configuration example of an example of the Siamese network. It is a block diagram which shows the specific configuration example of an example of a Triplet network. It is a flowchart which shows an example of the local feature expression learning processing routine in the local feature expression learning apparatus of this embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The present embodiment does not limit the present invention.

<Outline of the Embodiment of the present disclosure>

In the present disclosure, in the same region patch pair extraction unit, geometrical identity is determined for any image pair in the image data set, and the same is the same using the deformation parameters between the images determined to be the same. Extract a pair of patches that are thought to capture the physical area (hereinafter simply referred to as the "same area"). Further, the local feature expression learning unit learns a local feature expression (local feature descriptor) in which the patch pairs extracted by the same region patch pair extraction unit are similar to each other (or the distance is reduced) by the neural network. Furthermore, the same region patch pair extracted by the same region patch pair extraction unit is updated using the local feature descriptor learned by the local feature expression learning unit, and the obtained learning data is used as a neural network in the local feature expression learning unit. The iterative process of updating is repeated until a predetermined condition is satisfied.

In the present disclosure, by extracting patch pairs (capturing the same region) using the determination result of identity between images, it is possible to learn local feature expressions without using annotations for each patch, and non-patent documents. It is not necessary to prepare annotations for each patch in advance as in 1 and Non-Patent Document 2. In addition, by extracting patch pairs using the deformation parameters obtained as a result of determining the identity between images, instead of comparing the similarity between patches based on a simple known local feature representation, the known local feature representation It is possible to include patch pairs that cannot be extracted by, for example, patch pairs that capture the same area but are strongly affected by at least one of geometric and optical deformations in one image, for example. , It is possible to learn a local feature descriptor that is more robust to the above geometric and optical deformations as compared with the technique disclosed in Non-Patent Document 3.

<Structure of local feature expression learning device according to the embodiment of the present disclosure>

The local feature expression learning device of the present embodiment outputs a local feature descriptor by inputting an arbitrary image data set including a plurality of arbitrary images. FIG. 1 shows a configuration diagram of an example of the image feature device of the present embodiment.

In the local feature expression learning device 20 shown in FIG. 1, as an example, a plurality of images 12 are labeled with a plurality of categories (in FIG. 1, categories 1 to 3 are shown) (labeled). ) The case where the image data set 10 is input is shown.

The image data set 10 is input to the patch extraction unit 30, and the patch extraction unit 30 extracts the patch to be the target of local feature extraction from each of the images 12 included in the image data set 10. The patch extraction method in the patch extraction unit 30 is arbitrary. For example, as shown in FIG. 2, a plurality of patches 102 can be extracted by dividing the image 100 into a grid shape. Further, for example, as shown in FIG. 3, a plurality of patches 112 can be extracted from the image 110 by using a known local feature detector as shown in Non-Patent Document 1 and Non-Patent Document 2.

The patch extraction unit 30 outputs a set of extracted patches.

A set of patches is input from the patch extraction unit 30 to the same area patch pair extraction unit 32. The same area patch pair extraction unit 32 determines geometrical identity of any image pair with the same label from the set of input patches, and uses the resulting deformation parameters to capture the same area. (Same area patch pair) is extracted.

As shown in FIG. 4, the same region patch pair extraction unit 32 of the present embodiment includes a local feature description unit 40, a deformation estimation unit 42, and the same region determination unit 44, as shown in FIG. 4 as an example. The same region patch pair extraction unit 32 shown in FIG. 4 shows a configuration for extracting the same region patch pair when homography is assumed as a geometric deformation parameter between images.

A set of patches is input from the patch extraction unit 30 to the local feature description unit 40. The local feature description unit 40 features each patch using the local feature descriptor, and associates the patches between the images based on at least one of the similarity and the distance of the obtained local features. As the local feature descriptor, for example, those obtained by the immediately preceding processing in the local feature description unit 40, or those obtained in Non-Patent Document 1, Reference 1, and Reference 2 can be used. it can. The local feature description unit 40 outputs information indicating the correspondence of local features.

[Reference 1] DG Lowe, Distinctive Image Features from Scale-Invariant Keypoints, in IJCV, 2004.
[Reference 2] K. Mikolajczyk et al., A Comparison of Affine Region Detectors, in IJCV, 2005.

Information indicating the correspondence of local features is input to the deformation estimation unit 42 from the local feature description unit 40. Since the correspondence obtained by the local feature description unit 40 may include outliers, they are removed by a robust estimation method such as RANSAC, and geometric deformation between images is estimated.

For example, in the example shown in FIG. 5, as shown in (A), the local feature description unit 40 uses the homography transformation matrix H as a geometric deformation parameter between the image 130 and the image 132, specifically. Is

To get. The deformation estimation unit 42 outputs the obtained deformation parameters.

Deformation parameters are input from the deformation estimation unit 42 to the same area determination unit 44. The same area determination unit 44 projects the representative coordinates of each patch included in one image onto the other image by using the deformation parameter, and some of the coordinates are the closest and the distance is smaller than a predetermined threshold value. If so, those patch pairs are determined to be the same region patch pair 138.

In the patch pair strongly subjected to at least one of the geometrical and optical deformations, which could not be extracted by the deformation estimation unit 42 by the local feature description unit 40 by a simple comparison of the local features, for example, in the example shown in FIG. , (B), patch pair 134, patch pair 136 and the like can be extracted. In addition, the deformation estimation unit 42 can remove patch pairs extracted from different physical regions, that is, outliers, although they have similar simple appearances, and can extract patch pairs that capture the same region.

In addition to the homography transformation matrix H, an affine transformation matrix or a thin plate partial transformation may be used as the transformation parameters between the images, and when the external parameters of the camera that captured each image 12 are also known. May use a camera pose matrix. Further, when extracting the patch corresponding to the patch of one image from the other image, the conversion matrix is applied to the patch itself in one image without using the patch obtained by the patch extraction unit 30, and the area is defined. The patch pair may be extracted by extracting.

The same area determination unit 44 of the same area patch pair extraction unit 32 outputs a set of the extracted same area patch pair and the patches input from the patch extraction unit 30.

The same region patch pair and the set of patches output from the same region patch pair extraction unit 32 are input to the learning data construction unit 34. The learning data construction unit 34 constructs the data actually used for learning in the local feature expression learning unit 36 in the subsequent stage by using the same area patch pair and the set of patches. The data construction differs depending on the type of the error function of the neural network used in the local feature expression learning unit 36.

When the error function of the neural network used in the local feature expression learning unit 36 inputs a patch pair, the set of patch pairs in the same region may be a positive pair, and the patch pair that does not capture the same region may be a negative pair. Good. For both the positive pair and the negative pair, the training data and all the patch pairs input from the same region patch pair extraction unit 32 may be used as training data, or a predetermined number of them may be sampled and obtained. The patch pair may be used as training data. Further, the obtained patch pair may be subjected to a simple modification called so-called data augmentation to inflate the data.

When the neural network used in the local feature expression learning unit 36 inputs a patch triplet (triplet), the same area patch pair set is set as a positive patch pair and extracted from the positive patch pair and a physical area different from them. Make up a triplet from the patch. In this case as well, as for the number of triplets, all the triplets obtained from the training data may be used as the training data, or the triplets obtained by sampling a predetermined number of the triplets may be used as the training data.

In any case where patch pairs and triplets are used as training data, when extracting negative patch pairs extracted from different physical regions, if the label information of each image can be referred to, it is used. You may. The geometrical identity verification performed by the same region patch pair extraction unit 32 may not be determined to be the same even for images having the same label. In that case, if a negative patch is extracted by randomly sampling, there is a problem that, in rare cases, the patch is determined to be negative even though the patch is originally extracted from the same region. This can be avoided by using the label information of the image. If the input image data set 10 is not labeled, the identity of the image pair cannot be determined exactly. Therefore, for example, the negative pair is extracted from the image pair having a very small number of corresponding points. Can be done.

The learned learning data is output from the learning data construction unit 34.

Learning data is input from the learning data construction unit 34 to the local feature expression learning unit 36. The local feature expression learning unit 36 uses the input learning data to have a high degree of similarity between patches extracted from the same physical area (small distance), and a degree of similarity between patches extracted from different physical areas. Learn a local feature descriptor that maps each patch to a feature space where is low (not similar). The local feature descriptor learned here is a neural network in which patch pairs or triplets are used as the minimum input unit, and the positive patches obtained by the training data construction unit 34 are similar, and the negative patches are not similar. Suppose you want to learn a network.

In the present embodiment, any neural network that satisfies the above conditions is not particularly limited, and any neural network can be used. For example, the Siamese network 150 shown in FIG. 6, the Siamese network 152 shown in FIG. 7, the Triplet network 156 shown in FIG. 8, and the like can be used.

In the Siamese networks 150 and 152 as shown in FIGS. 6 and 7, an error function is defined from the distance or similarity between intermediate representations obtained by inputting each patch constituting the patch pair into the forward propagation network. It is a neural network like. As an example, as shown in FIG. 6, the error function (i) is defined in the Siamese network 150, and the error function (ii) is defined in the Siamese network 152.

The weight W of the forward propagation network may or may not be shared by the patch pair. Also, regarding the distance or similarity, the L2 distance or cosine similarity without parameters may be used, or a layer for learning the metric itself may be stacked on the forward propagation network.

As the error function (i), for example, a Contrastive loss function as represented by the following equation (1) can be used.

... (1)

Further, as the error function (ii), for example, a Binary cross entropy function as represented by the following equation (2) can be used.

... (2)

FIG. 9 shows a specific configuration example of an example of the Siamese network 150 as a specific example of the Siamese network actually used. The Siamese network is not limited to the configuration, and any form of forward propagation network can be adopted.

The Siamese network 150 shown in FIG. 9 has a filter size of (h × w × c). In addition, Conv represents a convolutional layer, MaxPolling represents a pooling layer, ReLu represents an activation function, Flatten represents a flattening layer, and FC represents a total connecting layer. Further, FIG. 9 shows, as an example, a case where the size of the input patch is (32 × 32 × 1).

On the other hand, in the Triplet network 156 as shown in FIG. 8, the similarity between the positive patches is negative for the intermediate feature table obtained by inputting each patch constituting the triplet to the forward propagation network. It is a neural network in which an error function is defined so as to be higher than the similarity of (in the case of a distance, the distance between positive patches is smaller than the distance between negative patches). As an example, as shown in FIG. 8, the Triplet network 156 defines an error function (iii).

As the error function (iii), for example, a Ratio loss function or a Margin ranking loss function as represented by the following equation (3) can be used.

... (3)

FIG. 10 shows a specific configuration example of an example of the Triplet network 156 actually used. The Triplet network 156 is not limited to the configuration, and any form of forward propagation network can be adopted.

The optimization method for minimizing the error function of the neural network described above is arbitrary, for example, SGD, momentum SGD, RMSProp, AdaGrad disclosed in Reference 3, Adadelta disclosed in Reference 4, and the like. Adam and the like disclosed in Reference 5 can be used. The learning rate and the weight decay are also optional. Of the neural networks learned here, any intermediate representation output from the single-patch forward propagation network can be used as the local feature descriptor.

[Reference 3] J. Duchi et al., Adaptive Subgradient Methods for Online Learning and Stochastic Optimization, in JMLR, 2011.
[Reference 4] MD Zeiler, ADADELTA: An Adaptive Learning Rate Method, in CoRR, 2012.
[Reference 5] DP Kingma et al., ADAM: A Method for Stochastic Optimization, in Proc. ICLR, 2015.

The local feature expression learning unit 36 outputs the learned local feature descriptor.

A local feature descriptor is input from the local feature expression learning unit 36 to the end determination unit 38. When the local feature expression learning unit 36 performs the iterative processing from the same region patch pair extraction unit 32, the end determination unit 38 outputs the local feature descriptor to the same region patch pair extraction unit 32. In this case, the same region patch pair extraction unit 32 can extract more patch pairs by using the local feature descriptor input from the end determination unit 38. Therefore, the local feature descriptor itself can be updated by the local feature expression learning unit 36 by using the learning data reconstructed from the patch pair extracted by the learning data construction unit 34.

The end determination unit 38 performs an end determination process for the above-mentioned iterative process. The criteria for determining the end is arbitrary. For example, the number of repetitions of the above repetition process is cut off at a predetermined number, or the patch pair set obtained by the same area patch pair extraction unit 32 and the patch pair set obtained immediately before are different. It may be based on the fact that it has disappeared.

The local feature expression learning device 20 of the present embodiment includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM that stores a local feature expression learning program and various data (Read Only Memory). Can be configured with including computers. When the CPU of the present embodiment executes the local feature expression learning program, the patch extraction unit 30 of the local feature expression learning device 20, the same area patch pair extraction unit 32, the learning data construction unit 34, the local feature expression learning unit 36, and It functions as each of the end determination units 38.

<Action of local feature expression learning device 20 of this embodiment>

Next, the operation of the local feature expression learning device 20 of the present embodiment will be described.

When the image data set 10 is input to the patch extraction unit 30, the local feature expression learning device 20 executes the local feature expression learning processing routine shown in FIG. 11 as an example.

As described above, in step S100 shown in FIG. 11, the patch extraction unit 30 extracts the patch to be the target of local feature extraction from each of the images 12 included in the image data set 10, and the extracted patch Output the set.

In the next step S102, the same area patch pair extraction unit 32 determines the geometric identity of any image pair with the same label from the set of input patches, as described above, and the resulting deformation. The same area patch pair is extracted and output using homography as a parameter.

In the next step S104, the learning data construction unit 34 uses the input same region patch pair and the set of patches as described above, and the learning data construction unit 34 uses the same region patch pair and the set of patches in the latter stage. The learning data corresponding to the type of the error function of the neural network used in the local feature expression learning unit 36 of is constructed and output.

In the next step S106, as described above, the local feature expression learning unit 36 has high similarity between patches extracted from the same physical region and is extracted from different physical regions using the input learning data. Between patches, a local feature descriptor that maps each patch to a feature space with low similarity is learned, and the learned local feature descriptor (local feature expression) is output.

In the next step S108, the end determination unit 38 determines whether or not to perform the iterative process based on a predetermined criterion as described above. In the present embodiment, as an example, if the predetermined criteria are not satisfied, the repetitive processing is performed. Therefore, the determination in step S108 becomes a negative determination, the process returns to step S102, and the processing in each of the above steps is repeated. In this case, the end determination unit 38 outputs the input local feature descriptor to the same area patch pair extraction unit 32.

On the other hand, when the predetermined criteria are satisfied, the determination in step S108 becomes an affirmative determination, and the local feature expression learning processing routine ends. In this case, the end determination unit 38 outputs the input local feature descriptor (local feature representation) to the outside of the local feature expression learning device 20.

As described above, the local feature expression learning device 20 of the present embodiment is a local feature expression learning device 20 that learns a local feature expression from an image data set 10 including a plurality of images 12, and is input image data. An image determined to be the same by determining the geometrical identity of the patch extraction unit 30 that extracts a plurality of patches from each image 12 included in the set 10 and an arbitrary image pair included in the image data set 10. The same region patch pair extraction unit that extracts a pair of patches that capture the same physical region using geometric deformation parameters from the pair, a set of patches consisting of a plurality of patches extracted by the patch extraction unit 30, and the same region. A learning data construction unit that constructs training data for learning a local feature expression using the patch pair extracted by the patch pair extraction unit 32, and a local feature expression using the training data constructed by the training data construction unit. The processing from the same area patch pair extraction unit to the local feature expression learning unit is performed using the local feature expression learning unit that learns the data and the local feature expression learned by the local feature expression learning unit until a predetermined standard is satisfied. It is provided with an end determination unit for repeating the operation.

Therefore, according to the local feature expression learning device 20 of the present embodiment, the local feature descriptor robust to geometrical and optical deformation can be learned without using patch-based annotation. Further, according to the local feature expression learning device 20 of the present embodiment, since the plurality of patches are automatically labeled without being manually labeled, the manual labor can be saved.

In the above embodiment, the image data set 10 input to the local feature expression learning device 20 is, for example, an image data set 10 in which a plurality of categories of labels are assigned to a plurality of images 12. As described above, the image data set 10 is not limited to this. For example, it may include a plurality of unlabeled images 12.

It should be noted that this embodiment is an example, and the specific configuration is not limited to this embodiment, but includes a design and the like within a range that does not deviate from the gist of the present invention, and may be changed depending on the situation. Needless to say.

20 Local feature expression learning device 30 Patch extraction unit 32 Same area patch pair extraction unit 34 Learning data construction unit 36 Local feature expression learning unit 38 End judgment unit 40 Local feature description unit 42 Deformation estimation unit 44 Same area judgment unit

Claims (6)

A local feature expression learning device that learns local feature expressions from an image data set containing multiple images.
A patch extractor that extracts multiple patches from each image contained in the input image dataset,
The pair of patches that captures the same physical region using geometric deformation parameters from the image pairs that are determined to be identical by determining the geometric identity of any image pair included in the image dataset. The same area patch pair extractor that extracts
Using a set of patches composed of the plurality of patches extracted by the patch extraction unit and the patch pairs extracted by the same region patch pair extraction unit, learning data for learning a local feature expression is constructed. Learning data construction department and
A local feature expression learning unit that learns a local feature expression using the learning data constructed by the learning data construction unit, and a local feature expression learning unit.
An end determination unit that repeatedly performs processing from the same region patch pair extraction unit to the local feature expression learning unit using the local feature expression learned by the local feature expression learning unit until a predetermined criterion is satisfied. ,
A local feature expression learning device including. The same region patch pair extraction unit
A local feature description section that features each of the plurality of patches extracted by the patch extraction section using the local feature representation, and a local feature description section.
For the image pair included in the image data set, the geometrical identity of the image pair is determined and the geometric deformation parameter is estimated based on the local feature of each patch expressed by the local feature description unit. Deformation estimation unit to be performed and
For the image pair determined to be the same by the deformation estimation unit, the same region determination unit that extracts the patch pair that captures the same physical region using the deformation parameter,
The local feature expression learning device according to claim 1. The local feature expression learning unit
Using the learning data, the patch determined to capture the same physical region by the same region patch pair extraction unit has a high degree of similarity and is determined to capture a different physical region. Learn a neural network that maps patches to a feature space where the similarity of pairs is low,
The local feature expression learning device according to claim 1 or 2. The learning data construction unit
In using the pair of patches determined not to capture the same physical region as learning data, the determination result of identity in the same region patch pair extraction unit is used.
The local feature expression learning device according to any one of claims 1 to 3. The learning data construction unit
When the pair of patches determined not to capture the same physical region is used as training data, the information of the label of each image assigned to the input image data set is used.
The local feature expression learning device according to any one of claims 1 to 3. It is a local feature expression learning method in a local feature expression learning device that learns a local feature expression from an image data set containing a plurality of images.
A step in which the patch extractor extracts multiple patches from each image contained in the input image dataset.
The same area patch pair extraction unit determines the geometrical identity of any image pair included in the image data set, and from the image pairs determined to be the same, the same physical area using geometric deformation parameters. And the step of extracting the pair of the patches that captured
The learning data construction unit uses the set of patches composed of the plurality of patches extracted by the patch extraction unit and the patch pair extracted by the same region patch pair extraction unit to learn the local feature expression. Steps to build training data for
A step in which the local feature expression learning unit learns a local feature expression using the learning data constructed by the learning data construction unit.
The process from the same region patch pair extraction unit to the local feature expression learning unit is repeated using the local feature expression learned by the local feature expression learning unit until the end determination unit satisfies a predetermined criterion. Steps to make
Local feature expression learning method including.