KR20230034887A - Keypoint extraction method based on deep learning and learning method for keypoint extraction - Google Patents

Abstract

Disclosed are a keypoint extraction method robust to a geometrical change in an image and a learning method for keypoint extraction. The keypoint extraction method based on deep learning includes the steps of: receiving a target image; generating a first feature map of a target image by using a pre-trained first artificial neural network; and generating a plurality of second feature maps having different receptive field scales from the first feature map by using a pre-trained second artificial neural network and generating and extracting a keypoint and a descriptor for the target image from each of the second feature maps.

Description

Deep learning-based feature point extraction method and learning method for feature point extraction {KEYPOINT EXTRACTION METHOD BASED ON DEEP LEARNING AND LEARNING METHOD FOR KEYPOINT EXTRACTION}

The present invention relates to a method for extracting feature points based on deep learning and a learning method for extracting feature points.

Keypoint Extraction is a key technology used in the first step in various computer vision fields such as image registration, object recognition, simultaneous localization and mapping (SLAM). In the case of this technology, it is very important to find the same point even if the shape, size, or position of an object changes without being affected by changes in camera viewpoint or lighting.

For feature point extraction, there is a traditional handcraft method and a deep learning-based method that is being actively researched recently. The handcraft method mathematically defines a location with high corneriness in an image and extracts a feature point based on it. Therefore, it is mainly extracted from the corner or the center point of a blob with a large change in pixel value within the image. On the other hand, the deep learning-based method learns the points obtained through SfM (Structure from Motion) as the correct answer (Ground Truth) or learns the characteristics of points with high repeatability between images, showing higher performance than the traditional method. .

However, conventional deep learning-based feature point extraction methods are mainly trained for robust operation in situations such as viewpoint or lighting change, and there is a limitation that performance in a situation with a relatively large geometric scale change is similar to that of the handcraft method. do. In order to overcome this limitation, a method of extracting feature points for each scale image by composing one image with images of various scales has been proposed.

As related prior literature, Korean Patent Publication No. 2021-0074163, a patent document, non-patent document, "Dusmanu, M., Rocco, I., Pajdla, T., Pollefeys, M., Sivic, J., Torii, A., & Sattler, T. (2019). "D2-net: A trainable cnn for joint description and detection of local features. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp.8092-8101)." there is.

An object of the present invention is to provide a method for extracting feature points that is robust to geometric changes in an image and a learning method for extracting feature points.

According to one embodiment of the present invention for achieving the above object, receiving a target image; generating a first feature map of a target image by using a pre-learned first artificial neural network; and generating a plurality of second feature maps having different receptive field scales from the first feature map by using a second artificial neural network trained in advance, and generating information about the target image from each of the second feature maps. A method for extracting feature points based on deep learning including extracting feature points and descriptors is provided.

In addition, according to another embodiment of the present invention for achieving the above object, generating a conversion image for the training image; Performing learning so that a deep learning model extracts feature points for the training image and the transformed image and descriptors for the feature points using the training image, the converted image, and correct answer values; The correct answer value includes position values of feature points for the training image and the converted image, and the descriptor includes an identifier assigned differently to each feature point. A learning method for extracting feature points is provided.

According to an embodiment of the present invention, feature points and descriptors are extracted from feature maps generated by a plurality of receptive fields having different scales, so that feature points and descriptors can be effectively extracted even in a situation where the scale of the target image is changed.

1 is a diagram for explaining a feature point extraction method based on deep learning according to an embodiment of the present invention.
2 is a diagram for explaining dilated convolution.
3 is a diagram for explaining a deep learning model according to an embodiment of the present invention.
4 is a diagram showing a score map according to an embodiment of the present invention.
5 is a diagram showing feature points extracted according to an embodiment of the present invention.
6 is a diagram for explaining a learning method for feature point extraction according to an embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. Like reference numerals have been used for like elements throughout the description of each figure.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining a feature point extraction method based on deep learning according to an embodiment of the present invention, and FIG. 2 is a diagram for explaining dilation convolution.

A feature point extraction method according to an embodiment of the present invention may be performed in a computing device including a processor and a memory.

Referring to FIG. 1 , a computing device according to an embodiment of the present invention receives a target image (S110), receives an input, and extracts feature points and descriptors from the target image. In this case, the computing device may extract a keypoint and a descriptor using a deep learning model including the first and second artificial neural networks. Here, the descriptor corresponds to an identifier that is differently assigned to each feature point and supports identification of the feature point.

The computing device generates a first feature map of the target image by using the pre-learned first artificial neural network (S120). Then, a plurality of second feature maps are generated from the first feature map using the pre-learned second artificial neural network, and feature points and descriptors of the target image are extracted from each of the second feature maps (S130). Here, each of the second feature maps is a feature map having a different receptive field scale. In other words, each of the second feature maps is a feature map generated by receptive fields of different scales.

The first and second artificial neural networks may be neural networks based on a convolutional neural network (CNN), and the second artificial neural network may include a feature point extraction network for extracting feature points and a descriptor extraction network for extracting descriptors. Also, the second feature map may be generated through dilated convolution or an artificial neural network such as ReS2Net.

Dilatation convolution is a convolution method that adjusts the scale of the receptive field by adding zero padding inside the convolution filter. As shown in FIG. 2, the scale of the receptive field varies according to the value of the dilation ratio. 2 shows the scale of the receiving field when the value of the expansion ratio is 1, 2, and 3, respectively. As the value of the expansion ratio increases, the zero padding increases and the scale of the receptive field increases.

And even when convolution operations are continuously applied to the input image using convolutional layers arranged consecutively, such as ReS2Net, the scale of the receptive field increases in proportion to the number of convolution operations.

Accordingly, the computing device according to an embodiment of the present invention may generate second feature maps having different scales by performing dilation convolution using different expansion ratios, or may re-create the previously generated second feature maps. By performing the convolution operation, a plurality of second feature maps having different receptive field scales from the previous second feature maps may be generated.

In this way, according to one embodiment of the present invention, feature maps are generated for each scale of the receptacle, and feature points and descriptors are extracted from these feature maps, so that feature points and descriptors can be effectively extracted even in a situation where the scale of the target image varies. can

In step S130, candidate feature points for the target image are extracted for each second feature map by the feature point extraction network. A reliability value is also output for each candidate feature point, and the computing device may determine a feature point having a reliability value equal to or greater than a threshold value among the candidate feature points as a feature point for the target image.

Also in the descriptor extraction network, candidate descriptors for feature points are extracted for each second feature map. The computing device may determine a descriptor corresponding to a location of a feature point among candidate descriptors as a descriptor for the feature point.

3 is a diagram for explaining a deep learning model according to an embodiment of the present invention, and FIG. 3 shows a feature map generated by the deep learning model according to an embodiment of the present invention.

As described above, the deep learning model according to an embodiment of the present invention includes first and second artificial neural networks, and the second artificial neural network includes a feature point extraction network and a descriptor extraction network.

And, the first artificial neural network includes a plurality of convolutional layers. As an example, the first artificial neural network may include 9 convolutional layers. The first and second convolutional layers may generate feature maps 310 having the same size as the target image 300, and the third and fourth convolutional layers may generate feature maps generated by the first and second convolutional layers ( 310), a feature map 320 having a smaller size than the target image may be generated. In addition, the fifth to ninth convolutional layers receive feature maps 320 generated by the third and fourth convolutional layers, and feature sizes smaller than those of the feature maps 320 generated by the third and fourth convolutional layers. A map 330 may be created.

The feature maps 320 generated from the third and fourth convolutional layers and the feature maps 330 generated from the fifth to ninth convolutional layers are converted to the same size as the target image 300, and the first to ninth convolutional layers The first feature map 340 is created by concatenating the feature maps 310 , 320 , and 330 generated in the convolution layer. The first feature map 340 may be a 3D feature map.

That is, the computing device may generate the first feature map 340 by repeatedly performing a convolution operation on the target image to generate third feature maps having different sizes, and connecting the third feature maps.

The feature point extraction network may extract feature points using Res2Net as an embodiment. The feature point extraction network continuously performs a convolution operation on the first feature map 340 using a convolution layer to generate second feature maps 351 to 354 having different scales of receptive fields. 3 shows an embodiment in which the second feature maps 351 to 354 generated by four different receptive field scales are used for feature point extraction. The feature point extraction network may generate second feature maps 351 to 354 to have the same size as the first feature map 340 .

As the convolution operation is repeatedly performed, the second feature maps 351 to 354 are generated in the order from the upper feature map to the lower feature map. In other words, among the second feature maps 351 to 354, the upper feature map is the previous second feature map generated relatively first, and the lower feature map is the feature map generated from the previous second feature map. In this way, the feature point extraction network performs a convolution operation on the previous second feature map to generate a second feature map in which the scale of the receptacle increases, and the scale of the receptacle increases downward in the second feature maps 351 to 354. It increases.

The feature point extraction network may extract candidate feature points from each of the second feature maps 351 to 354 and generate a score map including a reliability value for the candidate feature points. In addition, a feature point having a reliability value equal to or greater than a threshold value in the score map may be determined as a feature point for the target image. In FIG. 3 , black points displayed on the second feature maps 351 to 354 represent feature points of the target image. The union of the feature points determined in each of the second feature maps 351 to 354 corresponds to the feature points of the target image.

The descriptor extraction network may extract descriptors using dilated convolution. As an example, the descriptor extraction network can generate four second feature maps 355 to 358 having different scales of the receptive field from the first feature map 340 using expansion ratios of 1, 2, 3, and 8. there is. The larger the value of the expansion ratio, the larger the scale of the receptive field. The descriptor extraction network may generate second feature maps 355 to 358 to have the same size as the first feature map 340 . In addition, to correspond to the fact that the feature point extraction network generates the second feature maps 351 to 354 of four different receptacle scales, the descriptor extraction network also generates second feature maps 355 to 358 of four different receptacle scales. can create That is, the network may be configured such that the number of second feature maps of different receptive field scales generated by the feature point extraction network and the descriptor extraction network is the same.

The descriptor extraction network extracts candidate descriptors for each of the second feature maps 355 to 358, and may determine a descriptor at a location corresponding to a feature point determined in the feature point extraction network as a descriptor for the feature point. In FIG. 3, it can be seen that black points displayed on the second feature maps 355 to 358 represent descriptors for feature points, and the positions of the descriptors correspond to the positions of feature points.

4 is a diagram showing a score map according to an embodiment of the present invention, and FIG. 5 is a diagram showing feature points extracted according to an embodiment of the present invention.

4 is a score map for the target image of FIG. 5, and the 9 score maps shown in FIG. 4 are score maps obtained from the second feature map generated by 9 different receptive field scales. A bright pixel in the score map represents a higher score, that is, a reliability value, than a dark pixel. And, in FIG. 5, the feature point is located at the center of the red circle, and the size of the red circle corresponds to the size of the receptive field scale.

As shown in FIG. 4, the location of the high score appears differently in the foreground and background areas for each receptacle scale. When using a feature map generated by one receptive field scale, feature points may not be extracted from the foreground or background area, but according to an embodiment of the present invention, feature maps generated by a plurality of receptive fields having different scales. By using this, it can be seen that feature points for the target image are effectively extracted without omission as shown in FIG. 5 .

6 is a diagram for explaining a learning method for extracting feature points according to an embodiment of the present invention, and is a diagram for explaining the learning method for the aforementioned deep learning model.

Referring to FIG. 6 , the computing device according to an embodiment of the present invention generates a converted image for a training image (S610). In order to train a deep learning model so that robust feature points can be extracted even with geometric changes in the target image, the computing device may generate an image in which a view point of the training image is converted, as an example. The computing device may generate an image in which a viewpoint of a training image is transformed by using a homography transformation matrix.

In addition, the computing device may generate an image in which the color of the training image is converted, as an example, in order to train a deep learning model so that a feature point robust to a change in lighting of an environment in which the target image is photographed can be extracted.

The computing device performs learning so that the feature point extraction model extracts feature points for the training image and converted image and descriptors for the feature points using the training image, the converted image, and the correct answer value (S620). Here, the correct answer value (ground truth) includes the location value of the feature point for the training image and the converted image.

Since the descriptors include identifiers that are allocated differently for each feature point, the location value of the descriptor for each feature point does not have to be included in the correct answer value, and learning can be performed such that different descriptors are assigned to each feature point.

The technical contents described above may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiments or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. A hardware device may be configured to act as one or more software modules to perform the operations of the embodiments and vice versa.

As described above, the present invention has been described by specific details such as specific components and limited embodiments and drawings, but these are provided to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , Those skilled in the art in the field to which the present invention belongs can make various modifications and variations from these descriptions. Therefore, the spirit of the present invention should not be limited to the described embodiments, and it will be said that not only the claims to be described later, but also all modifications equivalent or equivalent to these claims belong to the scope of the present invention. .

Claims (6)

receiving a target image;
generating a first feature map of a target image by using a pre-learned first artificial neural network; and
A plurality of second feature maps having different receptive field scales are generated from the first feature map using a second artificial neural network trained in advance, and feature points of the target image are generated in each of the second feature maps. and extracting descriptors
Deep learning-based feature point extraction method including.
According to claim 1,
The second artificial neural network is
A feature point extraction network for extracting the feature point and a descriptor extraction network for extracting the descriptor;
The sizes of the first and second feature maps are
same size as the target image
Deep learning-based feature point extraction method.
According to claim 2,
The feature point extraction network or the descriptor extraction network
Generating the second feature map using dilation convolution or performing a convolution operation on the previous second feature map to generate a second feature map having a different receptive field scale from the previous second feature map
Deep learning-based feature point extraction method.
According to claim 1,
The step of extracting the feature points and descriptors is
Extracting candidate descriptors for the feature point, and determining a descriptor corresponding to the position of the feature point among the candidate descriptors as a descriptor for the feature point
Deep learning-based feature point extraction method.
generating a conversion image for the training image;
Performing learning so that a deep learning model extracts feature points for the training image and the transformed image and descriptors for the feature points using the training image, the converted image, and correct answer values;
The correct answer value includes position values of feature points for the training image and the converted image,
The descriptor includes an identifier that is differently assigned to each feature point.
Learning method for extracting feature points.
According to claim 5,
The step of generating the converted image is
Creating an image in which the viewpoint or color of the training image is converted
Learning method for extracting feature points.

Images