CN116051873A - Key point matching method and device and electronic equipment - Google Patents
Key point matching method and device and electronic equipment Download PDFInfo
- Publication number
- CN116051873A CN116051873A CN202310093986.0A CN202310093986A CN116051873A CN 116051873 A CN116051873 A CN 116051873A CN 202310093986 A CN202310093986 A CN 202310093986A CN 116051873 A CN116051873 A CN 116051873A
- Authority
- CN
- China
- Prior art keywords
- image
- key point
- sub
- detection model
- key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000001514 detection method Methods 0.000 claims description 118
- 238000012549 training Methods 0.000 claims description 59
- 230000003287 optical effect Effects 0.000 claims description 55
- 230000006870 function Effects 0.000 claims description 21
- 238000002372 labelling Methods 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 15
- 238000004422 calculation algorithm Methods 0.000 description 11
- 238000013135 deep learning Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 7
- 238000013139 quantization Methods 0.000 description 7
- 238000006243 chemical reaction Methods 0.000 description 5
- 238000003062 neural network model Methods 0.000 description 5
- 238000010606 normalization Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/757—Matching configurations of points or features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
The disclosure provides a key point matching method, a device and electronic equipment, wherein the method comprises the following steps: detecting key points of a first image and a second image to be matched to obtain the key points in the first image and the key points in the second image, wherein the first image and the second image have a superposition area; for any key point in the first image, determining a first sub-image with a set size corresponding to the any key point according to the first position information of the any key point in the first image as a center; acquiring second position information matched with the first position information in the second image, and determining a second sub-image with a set size by taking the second position as the center; and matching the key points in the first sub-image and the key points in the second sub-image, so that the key points of the first image and the second image can be efficiently and accurately matched.
Description
Technical Field
The disclosure relates to the technical field of image processing, and in particular relates to a key point matching method, a device and electronic equipment.
Background
Keypoint matching is a key issue in the field of computer vision. The task of the key point matching is that for a given two pictures (which may be color pictures or black-and-white pictures) that are partially overlapped, the matching needs to be completed by accurately finding the same points in the two pictures, and the recognition degree of the found matching needs to be high, such as corner points, edge points, and the like, so that the matching of the non-texture repeated area is reduced. The key point matching can be widely applied to the fields of instant localization and mapping (SLAM) and moving object tracking, so how to accurately perform the key point matching is very important.
Disclosure of Invention
The present disclosure aims to solve, at least to some extent, one of the technical problems in the related art.
Therefore, the present disclosure provides a method, an apparatus, and an electronic device for matching key points, which can detect key points of a first image and a second image to be matched, determine a first sub-image with a set size according to each detected key point in the first image as a center, determine a second sub-image with the same size as the first sub-image at the same position in the second image, and match the key points in the first sub-image with the key points in the second sub-image, thereby, under the condition that the key points in the first sub-image and the second sub-image are obtained, performing key point matching on the first sub-image with each key point in the first image as a center and the second sub-image with the same position in the second image, and realizing the key point matching of the first image and the second image efficiently and accurately.
According to a first aspect of the present disclosure, there is provided a key point matching method, including: detecting key points of a first image and a second image to be matched so as to obtain the key points in the first image and the key points in the second image, wherein an overlapping area exists between the first image and the second image; determining a first sub-image with a set size corresponding to any key point according to the first position information of any key point in the first image as a center aiming at any key point in the first image; acquiring second position information matched with the first position information in the second image, and determining key points in a second sub-image with the second position information as a center set size from the key points of the second image; and matching the key points in the first sub-image with the key points in the second sub-image.
According to a second aspect of the embodiments of the present disclosure, there is provided a keypoint matching apparatus, including: the detection module is used for carrying out key point detection on a first image and a second image to be matched so as to obtain key points in the first image and key points in the second image, wherein a superposition area exists between the first image and the second image; the first determining module is used for determining a first sub-image with a set size corresponding to any key point according to the first position information of the any key point in the first image as a center for any key point in the first image; the second determining module is used for acquiring second position information matched with the first position information in the second image and determining key points in a second sub-image with the second position information as a center set size from the key points of the second image; and the matching module is used for matching the key points in the first sub-image and the key points in the second sub-image.
According to a third aspect of the present disclosure, there is provided an electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the keypoint matching method set forth in the embodiment of the first aspect of the disclosure.
According to a fourth aspect of the present disclosure, there is provided a computer readable storage medium, which when executed by a processor of an electronic device, causes the electronic device to perform the keypoint matching method set forth in the embodiments of the first aspect of the present disclosure.
According to a fifth aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor of an electronic device, enables the electronic device to perform the keypoint matching method as set forth in the embodiments of the first aspect.
According to the technical scheme, key point detection is carried out on the first image and the second image to be matched, so that key points in the first image and key points in the second image are obtained, and a superposition area exists between the first image and the second image; for any key point in the first image, determining a first sub-image with a set size corresponding to the any key point according to the first position information of the any key point in the first image as a center; acquiring second position information matched with the first position information in the second image, and determining key points in a second sub-image with the second position information as a center set size from the key points of the second image; and matching the key points in the first sub-image with the key points in the second sub-image, so that the key points of the first image and the second image can be efficiently and accurately matched by matching the key points of the first sub-image with the set size and the same position in the second image by taking each key point in the first image as the center under the condition of acquiring the key points in the first sub-image and the second sub-image.
Additional aspects and advantages of the disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the disclosure.
Drawings
The foregoing and/or additional aspects and advantages of the present disclosure will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:
FIG. 1 is a flow diagram illustrating a keypoint matching method in accordance with an exemplary embodiment;
FIG. 2 is a flow diagram illustrating a keypoint matching method in accordance with an exemplary embodiment;
FIG. 3 is a flow diagram illustrating a keypoint matching method in accordance with an exemplary embodiment;
FIG. 4 is a flow diagram illustrating a keypoint matching method in accordance with an exemplary embodiment;
FIG. 5 is a flow diagram illustrating a keypoint matching method in accordance with an exemplary embodiment;
FIG. 6 is a schematic diagram of a key point matching device according to an exemplary embodiment;
FIG. 7 is a block diagram of an electronic device showing a keypoint match, according to an exemplary embodiment.
Detailed Description
Embodiments of the present disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and intended for the purpose of explaining the present disclosure and are not to be construed as limiting the present disclosure.
It should be appreciated that the methods of keypoint matching can be divided into two categories: deep learning methods and conventional methods. The method for matching the key points in the deep learning firstly completes the detection of the key points in the picture through a deep neural network, and simultaneously predicts a feature descriptor for each key point; and then, performing key point matching between the two pictures by using a deep learning or mathematical similarity calculation mode. The traditional key point matching method is generally divided into two steps, namely, firstly, corner points, edge points and the like of a given picture are found out through the traditional method such as scale-invariant feature transformation; the location of the point in the other picture is then found, using a conventional point matching algorithm.
However, if the deep learning method is to be deployed on a special chip for deep learning, a quantization process is often needed, the current main quantization process is post-training quantization (PTQ, post Training Quantization), the model precision is irreversibly lost by the current main deep learning method, meanwhile, the current main deep learning method is dependent on using a feature descriptor for performing key point matching, the feature descriptor is a high-dimensional feature vector sensitive to precision abnormality, the quantization can bring relatively large precision loss, and the post-quantization performance of the high-dimensional feature descriptor is not ideal, so that the precision of the key point matching is reduced. Conventional keypoint detection algorithms require sufficiently differentiated keypoint features such as large gradients, color changes, etc. Secondly, the conventional matching method needs a large number of fitting optimization processes, the processes have high calculation force requirements on a CPU, and the processes are not suitable for being deployed on a low-cost deep learning special chip for SLAM and other tasks, so that the hardware cost is increased.
Aiming at the problems, the disclosure provides a key point matching method, a device and electronic equipment.
The following describes in detail a key point matching method, a device and an electronic device provided by the present disclosure with reference to the accompanying drawings.
Fig. 1 is a flow diagram illustrating a keypoint matching method in accordance with an exemplary embodiment. It should be noted that the keypoint matching method may be applied to a keypoint matching apparatus. Wherein the keypoint match may be configured in the electronic device. The electronic device may be a mobile terminal, such as a mobile phone, a tablet computer, a personal digital assistant, or other hardware devices with various operating systems.
As shown in fig. 1, the key point matching method includes the following steps:
and step 101, detecting key points of the first image and the second image to be matched so as to acquire the key points in the first image and the key points in the second image.
Wherein the first image and the second image have a region of coincidence.
In the embodiment of the present disclosure, a keypoint detection algorithm may be used to detect keypoints of a first image and a second image to be matched, so as to obtain keypoints in the first image and keypoints in the second image, where the keypoint detection algorithm may include: SIFT algorithm, FAST algorithm, ORB algorithm, etc. Key points may include: image corner points, edge points, etc.
In practical application, in order to facilitate hardware deployment and improve accuracy of key point detection, as an example, a lightweight neural network model may be used for key point detection, where the lightweight neural network model for key point detection has learned to obtain a correspondence between an image and a key point.
In the embodiment of the present disclosure, for any key point in the first image, the position information of the any key point in the first image is taken as first position information, the first position information is taken as the center, a first sub-image with a set size is determined in the first image, for example, the first position information is (x, y), and in the first image, the first sub-image with the set size of 60 x 60 is determined taking the (x, y) as the center.
That is, in the second image, second position information matching the first position information of any one of the key points is acquired, for example, the first position information is (x, y), in the second image, the second position information is determined as (x ', y'), and further, in the second image, a second sub-image of a set size is determined centering on the second position information, wherein the first position information (x, y) in the first image matches the second position information (x ', y') in the second image.
And 104, matching the key points in the first sub-image with the key points in the second sub-image.
Further, a keypoint matching algorithm may be employed to match keypoints in the first sub-image with keypoints in the second sub-image. The keypoint matching algorithm may include: optical flow matching algorithms, stereo matching algorithms, etc.
In practice, to facilitate hardware deployment, as an example, a lightweight trained keypoint matching model may be used to match keypoints in the first sub-image with keypoints in the second sub-image.
In summary, key point detection is performed on a first image and a second image to be matched to obtain key points in the first image and key points in the second image, wherein a superposition area exists between the first image and the second image; determining a first sub-image with a set size corresponding to any key point according to the first position information of any key point in the first image as a center aiming at any key point in the first image; acquiring second position information matched with the first position information in the second image, and determining key points in a second sub-image with the second position information as a center set size from the key points of the second image; and matching the key points in the first sub-image with the key points in the second sub-image, so that the key points of the first image and the second image can be efficiently and accurately matched by matching the key points of the first sub-image with the set size and the same position in the second image by taking each key point in the first image as the center under the condition of acquiring the key points in the first sub-image and the second sub-image.
In practical applications, in order to facilitate hardware deployment and improve the accuracy of keypoint matching, in embodiments of the disclosure, a trained optical flow-based keypoint matching model may be used to perform keypoint matching on the first sub-image and the second sub-image. Wherein the trained optical flow-based keypoint matching model may be a lightweight neural network model. The following is a detailed description with reference to fig. 2.
Fig. 2 is a flow diagram illustrating a keypoint matching method in accordance with an exemplary embodiment.
As shown in fig. 2, the keypoint matching method may include the steps of:
Wherein the first image and the second image have a region of coincidence.
In embodiments of the present disclosure, a trained optical flow-based keypoint matching model may be employed to match keypoints in a first sub-image with keypoints in a second sub-image, and optical flow values for keypoints in the second sub-image that match any keypoint relative to any keypoint in the first sub-image may be output. Wherein the trained optical flow-based keypoint matching model may be a lightweight neural network model. That is, the data input and output by the trained optical flow-based keypoint matching model are of quantized low-bit type, and in order to improve the computational efficiency on the deep learning-specific chip, the trained optical flow-based keypoint matching model does not include other complex network layers except a convolution layer and a bulk normalization (Batch Normalization, BN) layer.
It should be noted that the initial optical flow-based keypoint matching model may be trained prior to matching the keypoints in the first sub-image with the keypoints in the second sub-image using the trained optical flow-based keypoint matching model.
Optionally, obtaining an initial optical flow-based keypoint matching model; acquiring a first training data set; wherein the first training data set comprises: the device comprises a first sample image and a second sample image, wherein the first sample image and the second sample image contain key points; labeling optical flow values of the key points in the second sample image and the matched key points in the first sample image to obtain a labeling optical flow value; inputting the first training data set into an initial optical flow-based key point matching model to obtain a predicted optical flow value output by the initial optical flow-based key point matching model; generating a first loss function value according to the difference between the predicted light flow value and the marked light flow value; the initial optical flow-based keypoint matching model is trained based on the first loss function value.
In addition, it should be noted that the complexity of the key point matching model based on the optical flow is relatively low compared with that of the deep learning model, so that the model parameters in the trained key point matching model based on the optical flow can be converted into the set hardware format according to the set conversion tool, thereby the trained key point matching model can be deployed on hardware, and the hardware deployment cost can be reduced.
In summary, by inputting the first sub-image and the second sub-image into the trained optical flow-based key point matching model, so that the trained optical flow-based key point matching model matches key points in the first sub-image with key points in the second sub-image, and optical flow values of the key points in the second sub-image relative to the key points in the first sub-image are output, thereby, the accuracy of the key point matching result can be improved by adopting the optical flow-based key point matching model to perform key point matching, and the lightweight optical flow-based key point matching model is convenient to deploy on hardware, and the deployment cost on hardware is reduced.
In practical applications, in order to facilitate hardware deployment and improve the accuracy of keypoint detection, as an example, a lightweight trained keypoint detection model may be used to perform keypoint detection on the first sub-image and the second sub-image. The details are described below in connection with fig. 3.
Fig. 3 is a flow diagram illustrating a keypoint matching method in accordance with an exemplary embodiment.
As shown in fig. 3, the keypoint matching method may include the steps of:
Wherein the first image and the second image have a region of coincidence.
In the embodiment of the disclosure, the first image and the second image may be subjected to keypoint detection by using a trained keypoint detection model, that is, the first image and the second image may be input to the trained keypoint detection model respectively, so that the keypoint detection model detects the keypoints in the first image and the keypoints in the second image. The key point detection model may be a lightweight neural network model. The data input and output by the key point detection model are of quantized low-bit type, and in order to improve the calculation efficiency on the deep learning special chip, the key point detection does not comprise other complex network layers except a convolution layer and a batch normalization (Batch Normalization, BN) layer.
Thus, keypoints in the first image and keypoints in the second image output by the trained keypoint detection model can be acquired. The trained key point detection model is learned to obtain the corresponding relation between the image and the key points.
In summary, the first image and the second image are respectively input into a trained keypoint detection model, so that the trained keypoint detection model detects keypoints in the first image and keypoints in the second image; the key points in the first image and the key points in the second image which are output by the trained key point detection model are obtained, so that the key point detection is performed by adopting the key point detection model, the accuracy of the key point detection can be improved, the key point detection model is a lightweight model, the lightweight model is convenient to deploy on hardware, and the deployment cost of the key point detection model on hardware is reduced.
It should be noted that, before the trained keypoint detection model is used to detect the keypoints of the first image and the second image, the initial keypoint detection model may be trained. The details are described below in connection with fig. 4.
Fig. 4 is a flow diagram illustrating a keypoint matching method in accordance with an exemplary embodiment.
As shown in fig. 4, the keypoint matching method may include the steps of:
The third sample image carries the first labeling key point.
In the embodiment of the disclosure, the third sample image may be a triangle image, a quadrilateral image, and other polygon images, and the key points in the third sample image may be labeled, so as to obtain the first labeled key points in the third sample image.
As an example, inputting the third sample image into the initial keypoint detection model, and acquiring a first predicted keypoint output by the initial keypoint detection model; generating a second loss function value according to the difference between the first predicted key point and the corresponding first marked key point; and performing first training on the initial key point detection model by adopting the second loss function value to obtain a key point detection model after the first training.
In step 405, in response to a user operation, the target key point is updated.
It should be appreciated that, since the first trained keypoint detection model is obtained by training a small amount of third sample images, the keypoint detection accuracy of the first trained keypoint detection model is poor, and therefore, there may be an error in inputting the fourth sample image into the first trained keypoint detection model, and the target keypoints output by the first trained keypoint detection model, and therefore, in the embodiment of the present disclosure, in response to a user operation, the target keypoints may be updated, and updated target keypoints may be obtained.
And 406, performing second training on the first trained keypoint detection model by using the fourth sample image and the updated target keypoints to obtain a trained keypoint detection model.
Inputting the fourth sample image into the first trained key point detection model, and obtaining a second predicted key point output by the first trained key point detection model; generating a third loss function value by using the difference between the second predicted key point and the corresponding updated target key point; and performing second training on the key point detection model subjected to the first training by adopting the third loss function value so as to obtain a trained key point detection model.
In step 410, second position information matching the first position information is acquired in the second image, and a second sub-image of a set size is determined centering on the second position.
In step 411, the keypoints in the first sub-image and the keypoints in the second sub-image are matched.
To sum up, a third sample image with a set shape is obtained; performing first training on the initial key point detection model by adopting the third sample image and the first marked key points carried by the third sample image to obtain a key point detection model after the first training; acquiring a fourth sample image from the set image database; inputting the fourth sample image into the key point detection model subjected to the first training to obtain a target key point of the fourth sample image; responding to user operation, and updating the target key points; and performing second training on the key point detection model subjected to the first training by adopting the fourth sample image and the updated target key point to obtain a key point detection model subjected to the training, so that the key point detection model is subjected to the first training by adopting a small amount of third sample images, the target key point on the fourth sample is determined by adopting the key point detection model subjected to the first training, and further, the key point detection model subjected to the first training is subjected to the second training by adopting the fourth sample and the updated target key point, thereby enhancing the training on the key point detection model and improving the accuracy of the key point detection model.
In any embodiment of the disclosure, as shown in fig. 5, the keypoint matching method may also be implemented based on the following steps:
1. the first stage is a keypoint detection network (keypoint detection model) that takes as input a complete picture and outputs all the keypoints on the picture. In order to facilitate the description of the subsequent process, in the schematic diagram of the upper diagram, two pictures to be matched are respectively input, and key point detection results of the two pictures are respectively output;
2. the second stage is an optical flow detection network (a key point matching model based on optical flow), traversing all key points in the I_a for two pictures I_a and I_b to be matched, and setting coordinates of one key point kp_a as (x, y);
3. taking (x, y) of the picture I_a as a center, digging out a picture patch_a with the size of 60x60, and simultaneously taking (x ', y') of the picture I_b as a center, digging out another picture patch_b with the size of 60x60, wherein the position information of (x, y) in the picture I_a is matched with the position information of (x ', y') in the picture I_b;
4. the two patch input optical flow detection quantized neural networks are calculated to obtain optical flow from a central point of the picture patch_a to the picture patch_b, so that the matching from a key point of a first picture to a second picture is completed;
5. Repeating the steps 3 and 4 for all the detected key points;
6. training a low-bit model, completing the conversion of the model by the trained model parameters through a chip quantization conversion tool chain, obtaining a format which can be deployed on a chip, and completing hardware deployment.
According to the key point matching method, key points in the first image and the key points in the second image are obtained by detecting the key points of the first image and the second image to be matched, wherein the first image and the second image have a superposition area; determining a first sub-image with a set size corresponding to any key point according to the first position information of the any key point in the first image as a center aiming at any key point in the first image; acquiring second position information matched with the first position information in the second image, and determining key points in a second sub-image with the second position information as a center set size from the key points of the second image; and matching the key points in the first sub-image with the key points in the second sub-image, so that the key points of the first image and the second image can be efficiently and accurately matched by matching the key points of the first sub-image with the set size and the same position in the second image by taking each key point in the first image as the center under the condition of acquiring the key points in the first sub-image and the second sub-image.
In order to implement the above embodiment, the present disclosure proposes a key point matching device.
Fig. 6 is a schematic diagram of a structure of a key point matching device according to an exemplary embodiment.
As shown in fig. 6, the keypoint matching device 600 includes: the detection module 610, the first determination module 620, the second determination module 630, and the matching module 640.
The detection module 610 is configured to perform keypoint detection on a first image and a second image to be matched, so as to obtain a keypoint in the first image and a keypoint in the second image; a first determining module 620, configured to determine, for any key point in the first image, a first sub-image of a set size corresponding to the any key point according to first position information of the any key point in the first image as a center; a second determining module 630, configured to obtain, in the second image, second position information that matches the first position information, and determine a second sub-image with a set size with the second position as a center; and a matching module 640, configured to match the keypoints in the first sub-image with the keypoints in the second sub-image.
As one possible implementation of an embodiment of the disclosure, the matching module 640 is configured to input the first sub-image and the second sub-image into a trained optical flow-based keypoint matching model, so that the trained optical flow-based keypoint matching model matches any keypoint in the first sub-image with a keypoint in the second sub-image, and output an optical flow value of a keypoint in the second sub-image that matches any of the keypoints with respect to any of the keypoints in the first sub-image.
As one possible implementation of the embodiment of the present disclosure, the keypoint matching apparatus 600 further includes: the system comprises a first acquisition module, a second acquisition module, a labeling module, a first input module, a first generation module and a first training module.
The first acquisition module is used for acquiring an initial key point matching model based on optical flow; the second acquisition module is used for acquiring the first training data set; wherein the first training data set comprises: the first sample image and the second sample image comprise key points; the labeling module is used for labeling optical flow values of the key points in the second sample image and the matched key points in the first sample image to obtain labeling optical flow values; the first input module is used for inputting the first training data set into an initial optical flow-based key point matching network so as to obtain an initial optical flow-based key point matching output predicted optical flow value; the first generation module is used for generating a first loss function value according to the difference between the predicted light current value and the marked light current value; and the first training module is used for training the initial key point matching based on the optical flow according to the first loss function value.
As one possible implementation of the embodiment of the present disclosure, the keypoint matching apparatus 600 further includes: a first conversion module.
The first conversion module is used for converting model parameters in the trained optical flow-based key point matching model into a set hardware format so as to deploy the trained key point matching model on the set hardware.
As one possible implementation of the embodiments of the present disclosure, the detection module 610 is configured to: inputting the first image and the second image into a trained key point detection model respectively, so that the key point detection model detects the key points in the first image and the key points in the second image; and acquiring the key points in the first image and the key points in the second image which are output by the trained key point detection model.
As one possible implementation of the embodiment of the present disclosure, the keypoint matching apparatus 600 further includes: the system comprises a third acquisition module, a second training module, a fourth acquisition module, a second input module, an updating module and a third training module.
The third acquisition module is used for acquiring a third sample image with a set shape; the third sample image carries a first marked key point; the second training module is used for carrying out first training on the initial key point detection model by adopting the third sample image and the first marked key points carried by the third sample image so as to obtain a key point detection model after the first training; the fourth acquisition module is used for acquiring a fourth sample image from the set image database; the second input module is used for inputting a fourth sample image into the key point detection model subjected to the first training so as to obtain a target key point of the fourth sample image; the updating module is used for responding to the user operation and updating the target key points; and the third training module is used for carrying out second training on the key point detection model subjected to the first training by adopting the fourth sample image and the updated target key points so as to obtain a trained key point detection model.
As one possible implementation manner of the embodiments of the present disclosure, a second training module is configured to: inputting the third sample image into an initial key point detection model, and acquiring a first predicted key point output by the initial key point detection model; generating a second loss function value according to the difference between the first predicted key point and the corresponding first marked key point; and performing first training on the initial key point detection model by adopting the second loss function value to obtain a key point detection model after the first training.
As a possible implementation manner of the embodiments of the present disclosure, a third training module is configured to: inputting a fourth sample image into the first trained key point detection model, and acquiring a second predicted key point output by the first trained key point detection model; generating a third loss function value according to the difference between the second predicted key point and the corresponding updated target key point; and performing second training on the key point detection model subjected to the first training by adopting the third loss function value so as to obtain a trained key point detection model.
According to the key point matching device, key points in the first image and the key points in the second image are obtained by detecting the key points of the first image and the second image to be matched, wherein a superposition area exists between the first image and the second image; determining a first sub-image with a set size corresponding to any key point according to the first position information of the any key point in the first image as a center aiming at any key point in the first image; acquiring second position information matched with the first position information in the second image, and determining key points in a second sub-image with the second position information as a center set size from the key points of the second image; and matching the key points in the first sub-image with the key points in the second sub-image, so that the key points of the first image and the second image can be efficiently and accurately matched by matching the key points of the first sub-image with the set size and the same position in the second image by taking each key point in the first image as the center under the condition of acquiring the key points in the first sub-image and the second sub-image.
In order to implement the above-described embodiments, the present disclosure further proposes an electronic device, as shown in fig. 7, and fig. 7 is a block diagram of an electronic device for key point matching, which is shown according to an exemplary embodiment. As shown in fig. 7, the electronic device 700 may include:
A program/utility 780 having a set (at least one) of program modules 770 may be stored in, for example, memory 710, such program modules 770 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment. Program modules 770 typically carry out the functions and/or methods of the embodiments described in this disclosure.
The electronic device 700 may also communicate with one or more external devices 790 (e.g., keyboard, pointing device, display 791, etc.), one or more devices that enable a user to interact with the electronic device 700, and/or any device (e.g., network card, modem, etc.) that enables the electronic device 700 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 792. Also, the electronic device 700 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet, through a network adapter 793. As shown in fig. 7, the network adapter 793 communicates with other modules of the electronic device 700 over the bus 730. It should be appreciated that although not shown in fig. 7, other hardware and/or software modules may be used in connection with electronic device 700, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.
The processor 720 executes various functional applications and data processing by running programs stored in the memory 710.
It should be noted that, the implementation process and the technical principle of the electronic device in this embodiment refer to the foregoing explanation of the key point matching in the embodiments of the present disclosure, and are not repeated herein.
To achieve the above embodiments, the embodiments of the present disclosure also propose a computer readable storage medium.
Wherein the instructions in the computer-readable storage medium, when executed by the processor of the electronic device, enable the electronic device to perform the keypoint matching method as described previously.
To implement the above embodiments, the present disclosure also provides a computer program product which, when executed by a processor of an electronic device, enables the electronic device to perform the keypoint matching method as described previously.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any adaptations, uses, or adaptations of the disclosure following the general principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (13)
1. A key point matching method, comprising:
detecting key points of a first image and a second image to be matched so as to obtain the key points in the first image and the key points in the second image, wherein an overlapping area exists between the first image and the second image;
determining a first sub-image with a set size corresponding to any key point according to the first position information of any key point in the first image as a center aiming at any key point in the first image;
acquiring second position information matched with the first position information in the second image, and determining a second sub-image with a set size by taking the second position as the center;
and matching the key points in the first sub-image with the key points in the second sub-image.
2. The method of claim 1, wherein the matching keypoints in the first sub-image and keypoints in the second sub-image comprises:
Inputting the first sub-image and the second sub-image into a trained optical flow-based key point matching model, so that the optical flow-based key point matching model matches any one of the key points in the first sub-image and the key points in the second sub-image, and outputting an optical flow value of the key point matched with any one of the key points in the second sub-image relative to any one of the key points in the first sub-image.
3. The method according to claim 2, wherein the method further comprises:
acquiring an initial key point matching model based on optical flow;
acquiring a first training data set; wherein the first training data set comprises: a first sample image and a second sample image, wherein the first sample image and the second sample image contain key points;
labeling optical flow values of the key points in the second sample image and the matched key points in the first sample image to obtain a labeling optical flow value;
inputting the first training data set into the initial optical flow-based key point matching model to obtain a predicted light value output by the initial optical flow-based key point matching model;
Generating a first loss function value according to the difference between the predicted light flow value and the noted light flow value;
training the initial optical flow-based keypoint matching model according to the first loss function value.
4. A method according to claim 2 or 3, characterized in that the method further comprises:
model parameters in the trained optical flow-based keypoint match model are converted to a set hardware format to deploy the trained keypoint match model on set hardware.
5. The method of claim 1, wherein performing keypoint detection on the first image and the second image to be matched to obtain keypoints in the first image and keypoints in the second image comprises:
inputting the first image and the second image into a trained keypoint detection model respectively, so that the trained keypoint detection model detects keypoints in the first image and keypoints in the second image;
and acquiring key points in the first image and key points in the second image output by the trained key point detection model.
6. The method of claim 5, wherein the method further comprises:
acquiring a third sample image with a set shape; the third sample image carries a first marked key point;
performing first training on the initial key point detection model by adopting a third sample image and a first marked key point carried by the third sample image to obtain a key point detection model after the first training;
acquiring a fourth sample image from the set image database;
inputting the fourth sample image into a first trained key point detection model to obtain a target key point of the fourth sample image;
responding to user operation, and updating the target key points;
and performing second training on the first trained keypoint detection model by adopting the fourth sample image and the updated target keypoints to obtain a trained keypoint detection model.
7. The method of claim 6, wherein the performing a first training on the initial keypoint detection model using the third sample image and the first labeled keypoints carried by the third sample image to obtain a first trained keypoint detection model comprises:
Inputting the third sample image into an initial key point detection model, and acquiring a first predicted key point output by the initial key point detection model;
generating a second loss function value according to the difference between the first predicted key point and the corresponding first marked key point;
and performing first training on the initial key point detection model by adopting the second loss function value to obtain a key point detection model after the first training.
8. The method of claim 6, wherein the second training the first trained keypoint detection model using the fourth sample image and the updated target keypoints to obtain a trained keypoint detection model comprises:
inputting the fourth sample image into the first trained key point detection model, and acquiring a second predicted key point output by the first trained key point detection model;
generating a third loss function value according to the difference between the second predicted key point and the corresponding updated target key point;
and performing second training on the key point detection model subjected to the first training by adopting the third loss function value so as to obtain a trained key point detection model.
9. The method according to any one of claims 6-8, further comprising:
model parameters in the trained keypoint detection model are converted to a set hardware format to deploy the trained keypoint detection model on set hardware.
10. A keypoint matching device, comprising:
the detection module is used for carrying out key point detection on a first image and a second image to be matched so as to obtain key points in the first image and key points in the second image;
the first determining module is used for determining a first sub-image with a set size corresponding to any key point according to the first position information of the any key point in the first image as a center for any key point in the first image;
a second determining module, configured to obtain second position information matched with the first position information in the second image, and determine a second sub-image with a set size with the second position as a center;
and the matching module is used for matching the key points in the first sub-image and the key points in the second sub-image.
11. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the keypoint matching method of any one of claims 1-9.
12. A computer readable storage medium, which when executed by a processor of an electronic device, causes the electronic device to perform the keypoint matching method of any of claims 1-9.
13. A computer program product comprising a computer program which, when executed by a processor of an electronic device, enables the electronic device to perform the keypoint matching method as claimed in any of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310093986.0A CN116051873A (en) | 2023-02-03 | 2023-02-03 | Key point matching method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310093986.0A CN116051873A (en) | 2023-02-03 | 2023-02-03 | Key point matching method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116051873A true CN116051873A (en) | 2023-05-02 |
Family
ID=86119912
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310093986.0A Pending CN116051873A (en) | 2023-02-03 | 2023-02-03 | Key point matching method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116051873A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116309760A (en) * | 2023-05-26 | 2023-06-23 | 安徽高哲信息技术有限公司 | Cereal image alignment method and cereal detection equipment |
-
2023
- 2023-02-03 CN CN202310093986.0A patent/CN116051873A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116309760A (en) * | 2023-05-26 | 2023-06-23 | 安徽高哲信息技术有限公司 | Cereal image alignment method and cereal detection equipment |
| CN116309760B (en) * | 2023-05-26 | 2023-09-19 | 安徽高哲信息技术有限公司 | Cereal image alignment method and cereal detection equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110322500B (en) | Optimization method and device for instant positioning and map construction, medium and electronic equipment | |
| US8483440B2 (en) | Methods and systems for verifying automatic license plate recognition results | |
| US8467596B2 (en) | Method and apparatus for object pose estimation | |
| CN108960211B (en) | Multi-target human body posture detection method and system | |
| US9691132B2 (en) | Method and apparatus for inferring facial composite | |
| WO2017199141A1 (en) | Point cloud matching method | |
| JP5261501B2 (en) | Permanent visual scene and object recognition | |
| US20110134221A1 (en) | Object recognition system using left and right images and method | |
| CN110349212B (en) | Optimization method and device for instant positioning and map construction, medium and electronic equipment | |
| GB2526342A (en) | Point cloud matching method | |
| JP6997369B2 (en) | Programs, ranging methods, and ranging devices | |
| CN112990228A (en) | Image feature matching method and related device, equipment and storage medium | |
| CN112198878B (en) | Instant map construction method and device, robot and storage medium | |
| CN116051873A (en) | Key point matching method and device and electronic equipment | |
| CN112465876A (en) | Stereo matching method and equipment | |
| CN113793370B (en) | Three-dimensional point cloud registration method and device, electronic equipment and readable medium | |
| CN111832634A (en) | Foreign matter detection method, system, terminal device and storage medium | |
| CN115239776B (en) | Point cloud registration method, device, equipment and medium | |
| Zhao et al. | Improved ORB based image registration acceleration algorithm in visual-inertial navigation system | |
| CN111951211B (en) | Target detection method, device and computer readable storage medium | |
| KR20220057691A (en) | Image registration method and apparatus using siamese random forest | |
| Tang et al. | Image matching algorithm based on improved AKAZE and Gaussian mixture model | |
| CN113570667B (en) | Visual inertial navigation compensation method and device and storage medium | |
| US20240153139A1 (en) | Object pose estimation in the context of neural networks | |
| Gu et al. | Pose ambiguity elimination algorithm for 3c components assembly pose estimation in point cloud |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Country or region after: China Address after: Room 59, 17th Floor, Science and Technology Innovation Building, No. 777 Zhongguan West Road, Zhuangshi Street, Ningbo City, Zhejiang Province, 315200 Applicant after: Aixin Yuanzhi Semiconductor Co.,Ltd. Address before: Room 59, 17th Floor, Science and Technology Innovation Building, No. 777 Zhongguan West Road, Zhuangshi Street, Zhenhai District, Ningbo City, Zhejiang Province, 315200 Applicant before: Aixin Yuanzhi Semiconductor (Ningbo) Co.,Ltd. Country or region before: China Country or region after: China Address after: Room 59, 17th Floor, Science and Technology Innovation Building, No. 777 Zhongguan West Road, Zhuangshi Street, Zhenhai District, Ningbo City, Zhejiang Province, 315200 Applicant after: Aixin Yuanzhi Semiconductor (Ningbo) Co.,Ltd. Address before: 201702 room 1190, zone B, floor 11, building 1, No. 158 Shuanglian Road, Qingpu District, Shanghai Applicant before: Aisin Yuanzhi semiconductor (Shanghai) Co.,Ltd. Country or region before: China |