TWI749821B - Image feature comparison processing method and system - Google Patents

Abstract

An image feature comparison processing method is provided. The method comprises: obtaining a first image and applying a feature matching neural network to process the first image; using a detector to find a plurality of feature points from the first image for matching, and applying a deconvolution algorithm to amplify the plurality of feature points; applying a descriptor to detect feature positions of the plurality of feature points, and applying the deconvolution algorithm to effectively describe the feature positions; and generating a second image according to the plurality of feature points and the effective position description.

Description

Image feature comparison processing method and system

The present invention relates to an image feature comparison processing method and system, in particular, an image feature comparison processing method and system that is applied to the field of computer vision and uses deep learning in artificial intelligence to solve the problem of feature matching .

The existing image feature comparison technology will cause some problems. The main problem lies in the training method of the feature detector and the resolution of the feature descriptor. In the training method of the feature detector, it is necessary to directly provide keypoint ground truth information in order to train the deep network. Generally speaking, the method of generating these key facts is usually performed by hand-crafted detectors (hand-crafted detectors). ), which usually makes the deep network to be trained just to "imitate" the performance of the existing manual detection, and it is difficult to learn to "explore" new and more representative feature points. In addition, although SuperPoint uses virtual data generated by itself to allow the deep network to learn the information of general feature points, such as T-junction or L-junction, it does not have the above Problem, but because the above training is only in the domain of virtual data, it is very difficult to directly use it in real-life applications. Therefore, to train a super-point network with generalization, one is needed. The complex domain-adaptation process not only increases the complexity of training deep networks, but also indirectly limits the capabilities of the network (because this adaptation process is completely based on homography transformation, But the real images are not completely based on homography conversion of. In the resolution part of the feature descriptor, although the prior art can avoid the lack of using key facts, in the descriptor resolution, because the prior art uses a rectangular grid to approximate The position of the descriptor, so in the location detection of the key point, the position will not be accurately detected, which will cause a mismatch in the final match.

Therefore, the present invention proposes an image feature comparison processing method and system, which mainly uses a deep neural network to achieve the feature matching function.

An embodiment of the present invention provides an image feature comparison processing system. This system includes: a feature matching neural network to obtain a first image and process the first image; a detector to find multiple feature points from the first image for matching; a descriptor to detect multiple The feature location of the feature point; the processor receives multiple feature points, applies deconvolution algorithm to amplify multiple feature points, receives the feature location and applies deconvolution algorithm to describe the feature location effectively, and then lets the processor according to A plurality of feature points and the effective position description are used to generate a second image.

In one embodiment, the feature matching neural network applies a multi-view stereo algorithm to process the first image.

In one embodiment, the feature matching neural network uses a key point perception loss algorithm to learn and distinguish multiple feature points, where the feature matching neural network applies a corner detection algorithm and a loss function to generate the key point perception loss algorithm Method, wherein the feature matching neural network uses a corner detection algorithm to detect the edges and corners of the first image.

In one embodiment, the processor applies a deconvolution algorithm to describe the effective positions of the multiple feature points.

An embodiment of the present invention provides an image feature comparison processing method. This method includes: obtaining a first image, and applying a feature matching neural network to process the first image; applying a detector to find multiple feature points from the first image for matching, and applying a deconvolution algorithm Amplify multiple feature points; apply a descriptor to detect the feature positions of multiple feature points, apply deconvolution algorithm to describe the feature locations effectively; and generate a second feature based on multiple feature points and effective location descriptions image.

In one embodiment, a multi-view stereo algorithm is used to process the first image.

In one embodiment, the key point perception loss algorithm is applied to allow the feature matching neural network to learn and distinguish multiple feature points, wherein the corner detection algorithm and loss function are applied to generate the key point perception loss algorithm, wherein, The corner detection algorithm is used to detect the edges and corners of the first image.

In one embodiment, the deconvolution algorithm is applied to describe the effective positions of multiple feature points.

In order to further understand the features and technical content of the present invention, please refer to the following detailed description and drawings about the present invention. However, the drawings provided are only for reference and description, and are not used to limit the present invention.

11: Feature matching neural network

12: Detector

13: Descriptor

14: processor

S201~S205: steps

A, A1: target

Fig. 1 shows an image feature comparison processing system according to an embodiment of the present invention.

Fig. 2 shows an image feature comparison processing method according to an embodiment of the present invention.

Figure 3A shows a diagram generated without applying the present invention.

FIG. 3B shows a diagram generated by applying the image comparison processing method of the present invention.

Fig. 1 shows an image feature comparison processing system according to an embodiment of the present invention. The image feature comparison processing system 1 includes: a feature matching neural network 11, a detector 12, a descriptor 13, and a processor 14. The feature matching neural network 11 obtains the first image and processes the first image; the detector 12 finds multiple feature points from the first image for matching; the descriptor 13 detects the feature positions of the multiple feature points ; The processor 14 receives multiple feature points, applies the deconvolution algorithm to amplify multiple feature points, and receives the feature positions at the same time, applies the deconvolution algorithm to describe the feature positions effectively, and then lets the processor 14 according to the multiple Feature points and effective location descriptions to produce a second image. In this embodiment, the feature matching neural network 11 uses a multi-view stereo algorithm to process the first image, and the feature matching neural network 11 can apply a key point perception loss algorithm to learn and distinguish multiple feature points, and the feature matching neural network The network 11 applies a corner detection algorithm and a loss function to generate a key point perception loss algorithm, and the feature matching neural network 11 uses a corner detection algorithm to detect the edges and corners of the first image. The processor 14 applies a deconvolution algorithm to describe the effective positions of the multiple feature points.

Fig. 2 shows an image feature comparison processing method according to an embodiment of the present invention. 1 and 2 at the same time, the image feature processing method includes: obtaining a first image, and applying a feature matching neural network 11 to process the first image (step S201); applying the detector 12 from the first image For example, find multiple feature points for matching, and apply the deconvolution algorithm to amplify multiple feature points (step S202); apply the descriptor 13 to detect the feature positions of the multiple feature points, and apply the deconvolution algorithm to the features Perform an effective location description for the location (step S203); and generate a second image based on multiple feature points and effective location descriptions (step S204).

In this embodiment, the image feature processing method further includes applying a multi-view stereo algorithm to process the first image, and applying a key point perception loss algorithm to match the features to the neural network The path learns and distinguishes a plurality of the feature points, wherein a corner detection algorithm and a loss function are applied to generate a key point perception loss algorithm, wherein the corner detection algorithm is applied to detect the edges and corners of the first image. In particular, the deconvolution algorithm can be applied to describe the effective positions of a plurality of the feature points.

Figure 3A shows a diagram generated without applying the present invention. FIG. 3B shows a diagram generated by applying the image comparison processing method of the present invention. As shown in FIG. 3A, the image without applying the present invention has fewer feature points and their corresponding feature point positions during the conversion and comparison of the image. Therefore, the image is prone to distortion and clearness and lack of recognition. High, it may even be distorted or broken. For example, the target A shown in FIG. 3A may be distorted and disconnected during the conversion process to the target A1. However, after the conversion of the present invention, as shown in Figure 3B, the feature comparison network trained by the feature comparison processing method of the present invention can have more feature points and display the desired graphics more accurately, and Improve the clarity of graphics. In addition, because there are many feature points and corresponding feature locations, the image is less likely to be distorted, and the clarity and recognition are also high. For example, when the target object A is converted to the target object A1, since the feature points and effective feature positions are many and dense, the converted target object A1 will be clearer and less likely to be distorted and disconnected. The present invention uses deconvolution to amplify the descriptor tensor back to the resolution of the original input image, which can train feature detectors efficiently, and can detect more accurately The position of the pixel described by each descriptor.

As shown in FIG. 3B, in this embodiment, the image feature comparison processing method of the present invention is applied to obtain the target A in the first image (the left image of FIG. 3B), and the feature matching neural network is applied to the first image The target A in the image is processed. The application detector 12 fairly and randomly finds multiple feature points from the target A in the first image for matching to the target in the second illustration (the right image of FIG. 3B) A1, and apply the deconvolution algorithm to amplify multiple characteristic points on the target A. The application descriptor 13 detects the feature positions of multiple feature points, and applies the deconvolution algorithm to effectively describe the feature locations, and according to the multiple feature points and effective location descriptions, the corresponding and matched target A1 is generated to generate The second image of the target A1.

In this embodiment, a multi-view stereo algorithm can also be used to process the target A. For example, a camera device is used to scan and depict the target A in multiple directions to depict the target A. Then use the corner detection algorithm to scan and calculate the edges and corners of the target A. Then, the corner detection algorithm and the loss function are used to generate the key point perception loss algorithm. The aforementioned loss function is used to calculate the confidence level of whether each pixel is a key point. According to the key point perception algorithm, the feature matching neural network learns and distinguishes multiple feature points, and applies the deconvolution algorithm to describe the effective positions of multiple feature points, and then generates the target A1.

In this embodiment, the present invention uses multiple feature points as key points, and then learns, distinguishes, and collects a large amount of data through the neural network. Therefore, it can be restored to a complete icon based on only part of the icon, such as As shown in Figure 3B, if only part of the target A (for example, only the part of the cylinder and the inclined plane) is marked with feature points and feature positions, it can be completely restored to the icon of the target A1 through the discrimination of the neural network .

Unlike traditional feature matching, the present invention separates feature point detection (feature detector) and feature description (feature descriptor) into two irrelevant tasks. The present invention combines the above two tasks into one end-to-end (end-to-end) task. -end) learning process; in addition, the traditional method of detector (detector) and descriptor (descriptor) must rely on specific domain knowledge (domain knowledge) to design, the timing of the use of this method will be limited and impossible It has good generalization. On the contrary, the feature matching neural network based on deep learning proposed by the present invention does not There is no such limitation. Instead, the deep network can learn by itself from the given data and collect a large amount of data at the same time, so that it can be used in a wider range of fields.

The problem to be solved by the present invention can be divided into two points: The first is that it can effectively detect features without providing keypoint ground truth and without relying on complex domain-adaptation. Point feature detector (feature detector). The second is to solve the problem that the feature descriptor (feature descriptor) gradually decreases in the resolution of the feature map, and the keypoint positioning is not accurate enough.

The present invention uses the proposed feature matching neural network to extract reliable dense feature correspondences from images with obvious appearance differences. All previous related studies have relied on hand-crafted features, which are trained from the sparse feature correspondences of Structure-from-Motion (SfM). Therefore, the knowledge of the prior art is limited to these hand-crafted features. In addition, in some existing methods, the resulting feature description tensor resolution will result in features that cannot be accurately detected. In order to overcome the above-mentioned problems, the present invention proposes a novel concept of simultaneously learning detectors and descriptors and dense correspondence from Multi-View Stereo (MVS). The present invention does not use any conditions to select pixels in advance, but selects them fairly and randomly. In addition, the present invention proposes a keypoint-aware loss, which aims to help the detector to learn intensively. In terms of feature description, unlike existing methods, the detected feature position is approximated by a square grid. The present invention adopts a deconvolution operation to learn a more effective position description. From the application knot It turns out that learning detectors and descriptors from dense correspondences can significantly improve the quality of image matching.

The present invention does not require the use of keypoint ground truth and any domain-adaptation, and uses deconvolution to enlarge the descriptor tensor back to the original input image. With high resolution, this can efficiently train feature detectors, and can more accurately detect the pixel positions described by each descriptor. In addition, the present invention uses the concept of the Harris corner detector (Harris corner detector) and the operation of Convolutional Neural Networks (CNN) to design a keypoint-aware loss function (keypoint-aware loss). function), this loss function is used to calculate the confidence of whether each pixel is a keypoint, which can effectively improve the probability that those feature points are detected.

More specifically, the present invention does not need to provide keypoint ground truth in advance to train the feature detector, which makes the detector of the present invention not limited to the performance of the existing detector. Furthermore, the deep network of the present invention is not like SuperPoint. It needs to be trained on virtual data first, and then a complex domain-adaptation method is used to allow the network to be applied in the real world. In real scenes, it will not only perform well between images with homography pairs. In addition, the present invention additionally uses a deconvolution method to allow the reduced feature tensor (feature tensor) to learn to zoom back to the original image size, which can avoid the use of rectangular grids like traditional techniques The approximate process is lost, and the accuracy of locating feature points is lost.

The key point perception loss proposed by the present invention may not need to provide additional ground truth for the key point of the convolutional neural network (CNN). The reason is that the present invention uses the reciprocal of the response value R in the corner detection algorithm as the coefficient in the kp map. The meaning of each pixel in the key point map is This pixel belongs to the confidence index of a key point. The higher the index, the more confident CNN is that the pixel belongs to a key point. At this time, if CNN detects that the response of a certain key point is very low, which leads to the triplet margin loss commonly used in the prior art and is difficult to optimize, a large coefficient will be generated at this time (Because R is a very small value, but it will be very large after the countdown). At this time, CNN will reduce the value of the key point in the key point icon, which is to reduce the confidence index of CNN that this pixel is a key point. In this way, CNN can learn what is the key point pixel by itself.

The image feature comparison processing method and system of the present invention can use existing computer software and other related electronic hardware equipment to realize deep neural network, detection, description, multi-view stereo algorithm, corner detection algorithm, and rewind Technologies such as product algorithm, loss function, and key point perception loss are used to process computer vision images, and use deep learning in artificial intelligence to solve the problem of feature matching.

The content disclosed above is only a preferred and feasible embodiment of the present invention, and does not limit the scope of the patent application of the present invention. Therefore, all equivalent technical changes made using the description and schematic content of the present invention fall into the application of the present invention. Within the scope of the patent.

S 201~S205: steps

Claims (4)

An image feature comparison processing method includes: obtaining a first image through a feature matching neural network and processing the first image through a multi-view stereo algorithm to obtain a dense feature correspondence, and The feature matching neural network uses a key point perception loss algorithm to allow the feature matching neural network to learn and distinguish multiple feature points, wherein a corner detection algorithm and a loss function are used to generate the key point perception Loss algorithm, the corner detection algorithm is used to detect the edges and corners of the first image; the multi-view stereo algorithm through a detector processes the first image and finds it from the first image A number of feature points are generated for matching; the feature positions of multiple feature points are detected by a descriptor, and feature tensors are performed on the feature points through a deconvolution algorithm in a processor to amplify multiple features The point and the feature location are effectively described; and the processor generates a second image according to the plurality of feature points, the effective location description, and the corresponding relationship between the dense feature. The image feature comparison processing method of claim 1, wherein the deconvolution algorithm is applied to describe the effective positions of a plurality of the feature points. An image feature comparison processing system includes: a feature matching neural network to obtain a first image, and the feature matching neural network should process the first image through a multi-view stereo algorithm to obtain A dense feature correspondence, where the feature matching neural network learns and distinguishes multiple features through a key point perception loss algorithm, and the feature matching neural network uses a corner point detection algorithm and a loss function Number to generate the key point perception loss algorithm, the feature matching neural network detects the edges and corners of the first image through the corner detection algorithm; a detector uses the multi-view stereo algorithm for the first image An image is processed and a plurality of feature points are found from the first image for matching; a descriptor detects the feature positions of a plurality of the feature points; and a processor receives a plurality of feature points and applies a reverse The convolution algorithm performs a feature tensor on the feature point to amplify a plurality of the feature points, receives the feature location and applies the deconvolution algorithm to perform an effective location description of the feature location, and then allows the processor to perform an effective location description of the feature location based on the multiple features. The feature point and the effective location description and the corresponding relationship between the dense feature to generate a second image. The image feature comparison processing system according to claim 3, wherein the processor applies the deconvolution algorithm to describe the effective positions of the multiple feature points.

Images