CN113033578B - Image calibration method, system, terminal and medium based on multi-scale feature matching - Google Patents
Image calibration method, system, terminal and medium based on multi-scale feature matching Download PDFInfo
- Publication number
- CN113033578B CN113033578B CN202110341817.5A CN202110341817A CN113033578B CN 113033578 B CN113033578 B CN 113033578B CN 202110341817 A CN202110341817 A CN 202110341817A CN 113033578 B CN113033578 B CN 113033578B
- Authority
- CN
- China
- Prior art keywords
- image
- feature
- characteristic
- matching
- identified
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/24—Aligning, centring, orientation detection or correction of the image
- G06V10/243—Aligning, centring, orientation detection or correction of the image by compensating for image skew or non-uniform image deformations
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention provides an image calibration method and system based on multi-scale feature matching, which are used for processing a target image to generate feature points and feature vectors of the target image; processing the image to be identified to generate a characteristic point and a characteristic vector group queue of the image to be identified; based on the characteristic points and characteristic vector group queues of the image to be identified and the characteristic points and characteristic vectors of the target image, performing multi-distance standard threshold matching to generate characteristic point pair group queues; performing matching degree calculation on each group of characteristic point pairs in the characteristic point pair group queue, obtaining matching characteristic point pairs with matching degree larger than a set threshold value, performing transformation matrix calculation, and generating a transformation image queue; and filtering the transformed images in the transformed image queue to obtain the optimal corrected image, and completing image calibration. A terminal and a medium are also provided. The method has the advantages of high detection speed of the feature points, strong self-adaption, low data and time cost and strong operability.
Description
Technical Field
The invention relates to the technical field of computer vision, in particular to an image calibration method based on GPU multi-scale feature matching.
Background
The high-quality image is the basis of computer vision, and the identification image can be restored to be in the same spatial distribution as the target image by utilizing the image correction technology, so that the image identification key points can be positioned conveniently, and the final identification quality can be improved. The common image correction flow is to acquire a perspective transformation matrix of the identification image and the target image by utilizing a certain strategy, and restore the identification image into a similar target image through inverse transformation. The existing method for acquiring the perspective transformation matrix mainly comprises the following two steps:
1. and (5) utilizing a SIFT, SURF, ORB characteristic point detection method and then obtaining a perspective transformation matrix. This type of approach often requires selection of matching algorithm parameters and is highly time-complex based on CPU algorithms.
2. Training to obtain a transformation matrix network by using a deep learning method. But this method is costly in terms of data and time.
The search finds that:
the Chinese patent application with publication number of CN110197185A discloses a method and a system for monitoring the space under the bridge based on a scale-invariant feature transformation algorithm, based on a framework of the scale-invariant feature transformation algorithm, automatically shooting and uploading pictures in the space under the bridge to be adjusted to be matched with an original image, generating a calibration image by combining an image comparison algorithm, labeling different areas of the uploaded image and the original image, finally calculating SSIM structural similarity of the original image and the calibration image, and setting an alarm threshold value, thereby realizing the monitoring of the space under the bridge. The technology can detect the object change of any size, such as stacking garbage, straw and the like, and the lane occupation operation behavior of a vendor, and can compare images with scale deformation and inconsistent shooting angles shot by an unfixed camera or an approximately fixed angle after rotation, scaling and cutting, thereby reducing the false alarm rate of direct comparison. The technology uses a scale-invariant-based feature transformation algorithm to realize feature point detection, then performs feature point matching, generates a perspective transformation matrix, and generates a calibration image. However, this technique still has the following drawbacks:
the time complexity of the scale-invariant feature transformation algorithm is high; if the unified threshold is used for feature point matching, the image with few feature points cannot be calibrated in the article; the final calibration image is not calibrated with an evaluation standard, and whether the photographed image is accurately calibrated cannot be determined.
The Chinese patent application with publication number of CN112150359A discloses a quick unmanned aerial vehicle image splicing method based on machine learning and feature point recognition, firstly, a reference image is selected, the reference image and 9 adjacent images around the reference image are subjected to feature point extraction by using an improved SURF algorithm based on GPU parallel acceleration optimization, the time consumed by extracting the feature points for multiple times of the reference image is reduced, the image continuous multiplication accumulated error is reduced, and the description feature vector of the SURF algorithm is improved by using the description feature vector calculated by machine learning, so that the precision of feature point matching is greatly improved, and image registration is realized. The invention constructs a rapid and efficient processing method for the remote sensing image of the unmanned aerial vehicle based on SURF algorithm, machine learning, GPU, PROSAC algorithm and image block splicing technology, which has the advantages of faster operation, greatly improved precision and more outstanding exertion in real-time performance compared with the traditional SURF algorithm. However, the technology is an improved SURF algorithm based on GPU parallel acceleration optimization, but the SURF algorithm has poor scale invariance and rotation invariance, and cannot really meet the requirement of image calibration.
No description or report of similar technology is found at present, and similar data at home and abroad are not collected.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides an image calibration method, an image calibration system, a terminal and a medium based on GPU multi-scale feature matching.
According to one aspect of the present invention, there is provided an image calibration method based on multi-scale feature matching, comprising:
processing the target image to generate characteristic points and characteristic vectors representing the target image;
processing the image to be identified to generate a characteristic point and characteristic vector group queue representing the image to be identified; the image to be identified and the target image are shot on the same entity target;
performing multi-distance standard threshold matching based on the characteristic points and characteristic vector group queues of the image to be identified and the characteristic points and characteristic vectors of the target image to generate characteristic point pair group queues;
performing matching degree calculation on each group of characteristic point pairs in the characteristic point pair group queue, obtaining matching characteristic point pairs with matching degree larger than a set threshold value, performing transformation matrix calculation, and generating a transformation image queue;
and filtering the transformed images in the transformed image queue to obtain the optimal corrected image, and completing image calibration.
Preferably, the processing the target image to generate a feature vector representing the target image includes:
selecting a target image, sequentially using a plurality of small-to-large floating point precision SIFT feature point detection algorithms realized based on the GPU to detect feature points, and generating a plurality of groups of feature points and feature vectors of the feature points;
comparing the number of the obtained feature points under various floating point precision, and obtaining the feature points under the minimum floating point precision and the feature vectors of the feature points under the condition that the number of the feature points is higher than the preset number of the feature points;
and taking the feature points and the feature vectors of the feature points under the minimum floating point precision as the feature points and the feature vectors of a group of target images and storing the feature points and the feature vectors of the feature points into a target feature library.
Preferably, three small-to-large floating point precision SIFT feature point detection algorithms based on GPU are sequentially used for feature point detection, namely, int8, fp16 and fp32, and three groups of feature points and feature vectors of the feature points are generated.
Preferably, the storage form of the feature points is a two-dimensional vector for recording coordinates of the map points; the feature vectors of the feature points are stored in 128-dimensional vectors.
Preferably, the processing the image to be identified to generate the feature point and feature vector group queue of the image to be identified includes:
preprocessing an image to be identified;
obtaining floating point precision of a SIFT feature point detection algorithm based on GPU (graphics processing unit) adopted when generating a feature vector of the target image;
and based on the floating point precision, performing feature point detection on the image to be identified by adopting a SIFT feature point detection algorithm realized based on the GPU, obtaining a group of feature points and feature vectors of the image to be identified under each scale, and generating a feature point and feature vector group queue of the image to be identified.
Preferably, the preprocessing the image to be identified includes:
if the scale of the target image is not different from the scale of the image to be identified, performing scale transformation, and generating an image queue containing only one image; otherwise, performing scale transformation on the image to be identified by using a plurality of scales to generate an image queue containing a plurality of scale images; wherein:
the scale of the target image is not greatly different from the scale of the image to be identified, and the method is determined by the following steps:
recording the scale of the target image as h1 and the scale of the image to be identified as h2; if |h1-h2|/h1<10%, the scale of the target image is considered to be not much different from the scale of the image to be identified.
Preferably, the generating the feature point pair group queue based on the feature point and feature vector group queue of the image to be identified and the feature point and feature vector of the target image by performing multi-distance standard threshold matching includes:
obtaining each group of feature vectors in the feature vector group queue of the image to be identified, and performing distance calculation on each group of feature vectors and the feature vector of the target image by using a plurality of distance standard thresholds to obtain different feature point pairs under different distance standard thresholds, so as to generate a plurality of groups of feature point pairs;
and combining the plurality of groups of characteristic point pairs to generate a characteristic point pair group queue based on multi-scale transformation and multi-threshold matching.
Preferably, the calculating the matching degree of each group of feature point pairs in the feature point pair group queue, obtaining a matching feature point pair with a matching degree greater than a set threshold value, and calculating a transformation matrix to generate a transformed image queue, includes:
acquiring each group of characteristic point pairs in the plurality of groups of characteristic point pair group queues, carrying out characteristic point pair matching by using a RANSAC characteristic point matching algorithm realized based on a GPU, and acquiring a plurality of groups of matching characteristic point pairs with matching degree larger than a set threshold;
performing transformation matrix calculation on each group of matching characteristic point pairs in the plurality of groups of matching characteristic point pairs to generate a perspective transformation matrix;
generating a perspective transformation inverse matrix for each perspective transformation matrix; and synthesizing the transformed image by utilizing the perspective transformation inverse matrix, and finally generating a transformed image queue based on multi-scale transformation and multi-threshold matching.
Preferably, the filtering the transformed images in the transformed image queue to obtain the best rectified image includes:
acquiring each image in the transformed image queue, and detecting the characteristic points of the transformed image by using a SIFT characteristic point detection algorithm realized based on a GPU;
acquiring a matching point pair of the characteristic point of each transformation image and the characteristic point of the target image by using a RANSAC algorithm realized based on a GPU, and calculating the distance between the matching point pairs by using Euler distances;
and comparing the distances between the matching point pairs, wherein the transformed image with the smallest distance is the optimal correction image.
According to another aspect of the present invention, there is provided an image calibration system based on multi-scale feature matching, comprising:
the target image processing module is used for processing the target image and generating characteristic points and characteristic vectors representing the target image;
the image processing module to be identified processes the image to be identified to generate a characteristic point and a characteristic vector group queue representing the image to be identified; the image to be identified and the target image are shot on the same entity target;
the characteristic point pair generating module is used for carrying out multi-distance standard threshold matching based on the characteristic points and characteristic vector group queues of the image to be identified and the characteristic points and characteristic vectors of the target image to generate characteristic point pair group queues;
the transformation image generation module is used for carrying out matching degree calculation on each group of characteristic point pairs in the characteristic point pair array, obtaining matching characteristic point pairs with matching degree larger than a set threshold value, carrying out transformation matrix calculation, and generating a transformation image array;
and the image calibration module is used for filtering the transformed images in the transformed image queue, acquiring the optimal correction image and completing the image calibration.
According to a third aspect of the present invention there is provided a terminal comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor being operable to perform the method of any one of the preceding claims when executing the program.
According to a fourth aspect of the present invention there is provided a computer readable storage medium having stored thereon a computer program which when executed by a processor is operable to perform a method as any one of the above.
Due to the adoption of the technical scheme, compared with the prior art, the invention has at least one of the following beneficial effects:
the invention solves the problem of low speed of SIFT feature point detection algorithm.
The invention solves the problem that a large amount of data cost and time cost are required in deep learning.
The method solves the problems that in the prior art, self-adaptive parameter selection and automation are not available, and whether the corrected image is really corrected or not is needed to be determined manually.
The invention has high detection speed on the characteristic points: the SIFT feature point detection algorithm realized based on the GPU is faster than the traditional CPU. Aiming at images with different complexity, different floating point precision is used, so that a better detection result is achieved.
The invention has strong self-adaption: according to the invention, based on the feature matching algorithm, manual parameter matching is not needed, and the optimal parameters can be obtained through self-adaptive comparison, so that the best calibration effect is obtained. Good calibration effect can be obtained when the image feature points are fewer.
The pretreatment cost of the invention is low: compared with a deep learning method, the method does not need a large amount of data cost and time cost, and has strong operability.
The method comprises the following steps that step 1, a GPU is utilized to realize a scale-invariant feature transformation algorithm, so that the feature point detection speed is increased; and step 3, solving the problem that images with few characteristic points can still obtain very good by utilizing a multi-distance threshold, and in step 5, including calibration evaluation standards, ensuring that the calibration images are accurate.
The invention uses an improved SIFT algorithm based on GPU acceleration optimization, and SIFT has higher scale invariance and rotation invariance.
Drawings
Other features, objects and advantages of the present invention will become more apparent upon reading of the detailed description of non-limiting embodiments, given with reference to the accompanying drawings in which:
FIG. 1 is a flow chart of an image calibration method based on multi-scale feature matching in an embodiment of the invention;
FIG. 2 is a flow chart of an image calibration method based on multi-scale feature matching in a preferred embodiment of the present invention;
FIG. 3 is a schematic diagram of an image calibration system based on multi-scale feature matching in an embodiment of the present invention.
Detailed Description
The following describes embodiments of the present invention in detail: the embodiment is implemented on the premise of the technical scheme of the invention, and detailed implementation modes and specific operation processes are given. It should be noted that variations and modifications can be made by those skilled in the art without departing from the spirit of the invention, which falls within the scope of the invention.
Fig. 1 is a flowchart of an image calibration method based on multi-scale feature matching according to an embodiment of the present invention. The method is an image calibration technology based on GPU multi-scale characteristic self-adaptive matching.
As shown in fig. 1, the image calibration method based on multi-scale feature matching provided in this embodiment may include the following steps:
s100, processing a target image to generate feature points and feature vectors representing the target image;
s200, processing the image to be identified to generate a characteristic point and a characteristic vector group queue which represent the image to be identified; the image to be identified and the target image are shot on the same physical target;
s300, performing multi-distance standard threshold matching based on the characteristic points and characteristic vector group queues of the image to be identified and the characteristic points and characteristic vectors of the target image, and generating characteristic point pair group queues;
s400, performing matching degree calculation on each group of characteristic point pairs in the characteristic point pair group queue, obtaining matching characteristic point pairs with matching degree larger than a set threshold value, performing transformation matrix calculation, and generating a transformation image queue;
s500, filtering the transformed images in the transformed image queue to obtain the optimal corrected image, and completing image calibration.
In S100 of this embodiment, as a preferred embodiment, processing the target image to generate a feature vector representing the target image may include the following steps:
s101, selecting a target image, sequentially using SIFT feature point detection algorithms which are realized based on the GPU and have various floating point precision from small to large to detect feature points, and generating a plurality of groups of feature points and feature vectors of the feature points;
s102, comparing the number of the obtained feature points under various floating point precision, and obtaining the feature points under the minimum floating point precision and the feature vectors of the feature points under the condition that the number of the feature points is ensured to be more than the preset number of the feature points;
and S103, taking the feature points and the feature vectors of the feature points under the minimum floating point precision as the feature points and the feature vectors of a group of target images and storing the feature points and the feature vectors into a target feature library.
In S101 of this embodiment, as a specific application example, three kinds of SIFT feature point detection algorithms of small-to-large floating point precision based on GPU implementation may be sequentially used for feature point detection, so as to generate three kinds of feature points and feature vectors of the feature points.
In S101 of this embodiment, as a specific application example, the storage form of the feature points may be a two-dimensional vector in which coordinates of the map points are recorded; the feature vectors of the feature points may be stored in the form of 128-dimensional vectors.
In S200 of this embodiment, as a preferred embodiment, processing the image to be identified to generate the feature points and the feature vector group queue of the image to be identified may include the following steps:
s201, preprocessing an image to be identified;
s202, floating point precision of a SIFT feature point detection algorithm based on GPU (graphics processing unit) adopted when generating a feature vector of a target image is obtained;
s203, based on floating point precision, performing feature point detection on the image to be identified by adopting a SIFT feature point detection algorithm realized based on a GPU, obtaining a group of feature points and feature vectors of the image to be identified under each scale, and generating a feature point and feature vector group queue of the image to be identified.
In S201 of this embodiment, as a preferred embodiment, preprocessing the image to be identified may include the steps of:
if the scale of the target image is not different from the scale of the image to be identified, performing scale transformation, and generating an image queue containing only one image; otherwise, performing scale transformation (scaling) on the image to be identified by using a plurality of scales to generate an image queue containing a plurality of scale images; wherein:
the scale of the target image is not much different from the scale of the image to be identified, and is determined by the following method:
recording the scale of the target image as h1 and the scale of the image to be identified as h2; if |h1-h2|/h1<10%, the scale of the target image is considered to be not much different from the scale of the image to be identified.
In S300 of this embodiment, as a preferred embodiment, based on the feature points and feature vector group queues of the image to be identified and the feature points and feature vectors of the target image, performing multi-distance standard threshold matching, and generating a feature point pair group queue, the method may include the following steps:
s301, obtaining each group of feature vectors in a feature vector group queue of an image to be identified, and performing distance calculation on each group of feature vectors and the feature vectors of the target image by using a plurality of distance standard thresholds to obtain different feature point pairs under different distance standard thresholds, so as to generate a plurality of groups of feature point pairs;
s302, combining a plurality of groups of characteristic point pairs to generate a characteristic point pair group queue based on multi-scale transformation and multi-threshold matching.
In S400 of this embodiment, as a preferred embodiment, performing matching degree calculation on each set of feature point pairs in the feature point pair array, obtaining matching feature point pairs with matching degree greater than a set threshold, and performing transformation matrix calculation, to generate a transformed image array, may include the following steps:
s401, obtaining each group of characteristic point pairs in a plurality of groups of characteristic point pair group queues, carrying out characteristic point pair matching by using a RANSAC characteristic point matching algorithm realized based on a GPU, and obtaining a plurality of groups of matching characteristic point pairs with matching degree larger than a set threshold;
s402, performing transformation matrix calculation on each group of matching characteristic point pairs in a plurality of groups of matching characteristic point pairs to generate a perspective transformation matrix;
s403, generating a perspective transformation inverse matrix for each perspective transformation matrix; and synthesizing the transformed image by utilizing the perspective transformation inverse matrix, and finally generating a transformed image queue based on multi-scale transformation and multi-threshold matching.
In S500 of this embodiment, as a preferred embodiment, filtering the transformed images in the transformed image queue to obtain the best corrected image may include the following steps:
s501, acquiring each image in a transformed image queue, and detecting the characteristic points of the transformed image by using a SIFT characteristic point detection algorithm realized based on a GPU;
s502, acquiring a matching point pair of the characteristic point of each transformation image and the characteristic point of the target image by using a RANSAC algorithm realized based on the GPU, and calculating the distance between the matching point pairs by using the Euler distance;
s503, comparing the distances between the matched point pairs, wherein the transformed image with the smallest distance is the best correction image.
Fig. 2 is a flowchart of an image calibration method based on multi-scale feature matching according to a preferred embodiment of the present invention.
As shown in fig. 2, the image calibration method based on multi-scale feature matching provided in the preferred embodiment may include the following steps:
step 1: the target image is processed. And generating feature points and feature vectors representing the target image, and storing the feature points and the feature vectors into a target feature library.
As a preferred embodiment, step 1 comprises the steps of:
step 1.1: selecting a target image, sequentially using three GPUSIFTs of int8, fp16 and fp32 from small to large floating point precision to detect characteristic points, recording the characteristic points and characteristic vectors of the characteristic points, and generating three groups of characteristic points and characteristic vectors in the step;
step 1.2: comparing the number of the feature points under the three floating point precision obtained in the step 1.1, and recording the feature points under the minimum floating point precision and the feature vectors of the feature points under the condition that the number of the feature points is ensured to be more than the preset number of the feature points;
step 1.3: and (3) taking the feature points and the feature vectors of the feature points under the minimum floating point precision generated in the step (1.2) as the feature points and the feature vectors of a group of target images in the step (1.1) to store into a target feature library.
In some embodiments of the present invention, the GPUSIFT in step 1.1 is a SIFT feature point detection algorithm implemented based on a GPU, and the algorithm principle is implemented according to the SIFT feature point algorithm, where the running environment is a GPU of Nvidia Jetson Xavier NX and CUDA10.2.
In some embodiments of the present invention, in step 1.2, a larger floating point precision is used for the image with fewer feature points, so that more accurate feature information can be obtained.
In some embodiments of the present invention, the feature point storage in step 1.3 is a two-dimensional vector recording coordinates of the map point, and the feature vector of the feature point is stored in a 128-dimensional vector.
Step 2: and processing the image to be identified. And generating a characteristic point and characteristic vector group queue representing the image to be identified. Wherein the content of the image to be identified and the content of the target image should be photographed on the same physical target.
As a preferred embodiment, step 2 comprises the steps of:
step 2.1: and (5) preprocessing an image. If the scale of the target image is not different from the scale of the image to be identified, no scale transformation is needed to generate an image queue containing only one image, otherwise, the image is scaled by a plurality of scales to generate an image queue containing a plurality of scale images;
step 2.2: acquiring floating point precision of a SIFT feature point detection algorithm realized based on the GPU from the feature information of the target image in the step 1.3;
step 2.3: and (3) performing special detection on the preprocessed image queue in the step (2.1) by using GPUSIFT based on the floating point precision obtained in the step (2.2), performing feature point detection on each image in the multi-scale image queue, obtaining a group of feature points and feature vectors under each scale, and generating a feature point and feature vector group queue of the image to be identified.
In some embodiments of the present invention, the scale transformation in step 2.1 marks the target image scale as h1, and the image scale to be identified as h2. If the ratio of the I h1 to the h 2I/h 1 is less than 10%, the target image scale is not greatly different from the to-be-identified image scale, and the scale transformation is not needed; otherwise, the difference between the target image scale and the image scale to be identified is divided into a plurality of scale gradients, and recorded
delta_h＝(h1-h2)/5
h＝h2+n*delta_h，n＝0，1，2，3，4，5
The image to be identified is transformed by 6 scales of h1- > h using the library function cv2.reserve () in opencv.
In some embodiments of the present invention, the feature points and feature vector groups of the image to be identified in step 2.3 are obtained according to step 2.1, and the feature points and feature vector groups of the image to be identified in step 2.3 are feature points and feature vector groups under 6 scales.
Step 3: and (3) performing multi-distance standard threshold matching based on the characteristic points and characteristic vector group queues of the image to be identified in the step (2.3) and the characteristic points and characteristic vectors of the target image in the step (1.3), and generating characteristic point pair group queues.
As a preferred embodiment, step 3 comprises the steps of:
step 3.1: taking out a group of feature vectors of the feature vector group queue of the image to be identified in the step 2.3, performing distance calculation on the feature vector and the feature vector of the target image in the step 1.3 by using a plurality of distance standard thresholds, and obtaining different feature point pairs under different thresholds by using different distance standard thresholds;
step 3.2: carrying out the related operation of the step 3.1 on each group of feature vectors in the feature vector group queue of the image to be identified in the step 2.3 to generate a plurality of groups of feature point pairs;
step 3.3: and (3) merging the plurality of groups of characteristic point pairs generated in the step (3.2) to generate a characteristic point pair group queue based on multi-scale transformation and multi-threshold matching.
In some embodiments of the present invention, for the standard threshold values of different distances in step 3.1, the distance is generally considered to be a trusted matching result between 0.4 and 0.8, where the smaller the distance, the more trustworthy the feature point pair matching.
In some embodiments of the present invention, for images with few feature points, the threshold is set too low, the feature point matching quality requirement is too high, so that feature point pairs cannot be found, and finally feature point pair matching cannot be performed,
in some embodiments of the present invention, according to step 3.1 and step 3.2, different matching results are obtained by setting a plurality of thresholds from 0.4 to 0.8, and the matching results of less than 4 feature point pairs are filtered. A total of 5 thresholds of 0.4,0.5,0.6,0.7,0.8 are set in the preferred embodiment.
In some embodiments of the present invention, according to step 3.3 and step 3.2, on the premise of not filtering the matching results of less than 4 feature point pairs, the feature point pair group queues of the multi-scale transformation and multi-threshold matching in step 3.3 have 6*5 total 30 feature point pair groups.
Step 4: and carrying out feature point matching on the feature vector of each group of feature point pairs in the feature vector group queue and the feature vector of the target image to obtain feature point pairs which can be matched, calculating a transformation matrix, and generating a transformation image queue.
As a preferred embodiment, step 4 comprises the steps of:
step 4.1: taking out a group of characteristic point pairs in the characteristic point pair group queue based on the step 3.3, and carrying out algorithm matching by utilizing a characteristic point matching algorithm based on GPU_RANSAC to obtain the best 4 matching characteristic point pairs;
step 4.2: carrying out the related operation of the step 4.1 on each group of characteristic point pairs in the characteristic point pair group queue of the identification image in the step 3.3 to generate a plurality of groups of matched characteristic point pairs;
step 4.3: generating a perspective transformation matrix for each group of the best 4 matching characteristic point pairs generated in the step 4.2;
step 4.4: generating a perspective transformation inverse matrix for each perspective transformation matrix in the step 4.3, synthesizing a transformation image by utilizing the perspective transformation inverse matrix, and finally generating a transformation image queue based on multi-scale transformation and multi-threshold matching.
In some embodiments of the present invention, the gpu_ransac in step 4.1 is a RANSAC algorithm implemented based on a GPU, and the algorithm principle is implemented according to the RANSAC algorithm, where the running environment is a GPU of Nvidia Jetson Xavier NX and CUDA10.2.
In some embodiments of the present invention, at least 4 feature points are required to generate the perspective transformation matrix in step 4.3, so in step 3.2, matching results of less than 4 feature points are filtered out.
In some embodiments of the present invention, according to step 4 and step 3.4, on the premise of not filtering the matching results of less than 4 feature points, the final multi-scale transformed, multi-threshold transformed image queue contains 30 transformed images.
Step 5: and (4) filtering the transformed images in the transformed image queue in the step (4.4) to obtain the optimal corrected image, and completing the image calibration process.
As a preferred embodiment, step 5 comprises the steps of:
step 5.1: and (4) taking out one image in the transformed image queue in the step (4.4), and detecting the characteristic points by using a characteristic point detection algorithm based on GPUSIFT.
Step 5.2: the feature points of the transformed image in the step 5.1 and the feature points of the target image in the target feature library in the step 1.3 are utilized to acquire matching point pairs based on the GPU_RANSAC algorithm in the step 4; the euler distance is used to calculate the matching point pair distance.
Step 5.3: and (3) comparing the distances between the feature point pairs of all the transformed images obtained in the step (5.2) and the feature points of the target images in the target feature library in the step (1.3), wherein the transformed image with the minimum distance between the feature point pairs of the transformed images and the feature points of the target images is the optimal correction image.
Fig. 3 is a schematic diagram of an image calibration system based on multi-scale feature matching according to an embodiment of the present invention.
As shown in fig. 3, the image calibration system based on multi-scale feature matching provided in this embodiment may include: the device comprises a target image processing module, an image processing module to be identified, a characteristic point pair generating module, a transformation image generating module and an image calibration module; wherein:
the target image processing module is used for processing the target image and generating characteristic points and characteristic vectors representing the target image;
the image processing module to be identified processes the image to be identified to generate a characteristic point and a characteristic vector group queue representing the image to be identified; the image to be identified and the target image are shot on the same physical target;
the characteristic point pair generating module is used for carrying out multi-distance standard threshold matching based on characteristic points and characteristic vector group queues of the image to be identified and characteristic points and characteristic vectors of the target image to generate characteristic point pair group queues;
the transformation image generation module is used for carrying out matching degree calculation on each group of characteristic point pairs in the characteristic point pair array, obtaining the matching characteristic point pairs with the matching degree larger than a set threshold value, carrying out transformation matrix calculation, and generating a transformation image array;
and the image calibration module is used for filtering the transformed images in the transformed image queue, acquiring the optimal correction image and completing the image calibration.
An embodiment of the present invention provides a terminal including a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor being operable to perform the method of any of the above embodiments when the program is executed by the processor.
Optionally, a memory for storing a program; memory, which may include volatile memory (english) such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (Double Data Rate Synchronous Dynamic Random Access Memory, DDR SDRAM), and the like; the memory may also include a non-volatile memory (English) such as a flash memory (English). The memory is used to store computer programs (e.g., application programs, functional modules, etc. that implement the methods described above), computer instructions, etc., which may be stored in one or more memories in a partitioned manner. And the above-described computer programs, computer instructions, data, etc. may be invoked by a processor.
The computer programs, computer instructions, etc. described above may be stored in one or more memories in partitions. And the above-described computer programs, computer instructions, data, etc. may be invoked by a processor.
A processor for executing the computer program stored in the memory to implement the steps in the method according to the above embodiment. Reference may be made in particular to the description of the embodiments of the method described above.
The processor and the memory may be separate structures or may be integrated structures that are integrated together. When the processor and the memory are separate structures, the memory and the processor may be connected by a bus coupling.
An embodiment of the present invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, is operable to perform a method according to any of the above embodiments.
The image calibration method, the system, the terminal and the medium based on multi-scale feature matching provided by the embodiment of the invention solve the problem of low SIFT feature point detection algorithm speed; the problem that a large amount of data cost and time cost are required in deep learning is solved; the method solves the problems that in the prior art, self-adaptive parameter selection cannot be realized, automation cannot be realized, and whether the corrected image is really corrected or not needs to be manually determined. The image calibration method, the system, the terminal and the medium based on the multi-scale feature matching provided by the embodiment of the invention are faster than the traditional CPU (Central processing Unit) implementation based on the SIFT feature point detection algorithm realized by the GPU. Aiming at images with different complexity, different floating point precision is used, so that a better detection result is achieved; based on the feature matching algorithm, the optimal parameters can be obtained through self-adaptive comparison without manual parameter matching, and the best calibration effect is obtained. Good calibration effect can be obtained when the image feature points are fewer; compared with a deep learning method, the method has the advantages that a large amount of data cost and time cost are not needed, and the operability is high. According to the image calibration method, the system, the terminal and the medium based on multi-scale feature matching, which are provided by the embodiment of the invention, step 1, a GPU is utilized to realize a scale-invariant feature transformation algorithm, so that the feature point detection speed is increased; step 3, the problem that images with few feature points can still be obtained well by utilizing a multi-distance threshold is solved, and in step 5, calibration evaluation standards are included, so that the calibration images can be ensured to be accurate; using an improved SIFT algorithm based on GPU-accelerated optimization, SIFT has higher scale invariance as well as rotational invariance.
It should be noted that, the steps in the method provided by the present invention may be implemented by using corresponding modules, devices, units, etc. in the system, and those skilled in the art may refer to a technical solution of the method to implement the composition of the system, that is, the embodiment in the method may be understood as a preferred example of constructing the system, which is not described herein.
Those skilled in the art will appreciate that the invention provides a system and its individual devices that can be implemented entirely by logic programming of method steps, in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc., in addition to the system and its individual devices being implemented in pure computer readable program code. Therefore, the system and various devices thereof provided by the present invention may be considered as a hardware component, and the devices included therein for implementing various functions may also be considered as structures within the hardware component; means for achieving the various functions may also be considered as being either a software module that implements the method or a structure within a hardware component.
The foregoing describes specific embodiments of the present invention. It is to be understood that the invention is not limited to the particular embodiments described above, and that various changes and modifications may be made by one skilled in the art within the scope of the claims without affecting the spirit of the invention.
Claims (8)
1. An image calibration method based on multi-scale feature matching, comprising:
processing the target image to generate characteristic points and characteristic vectors representing the target image; comprising the following steps:
selecting a target image, sequentially using a plurality of small-to-large floating point precision SIFT feature point detection algorithms realized based on the GPU to detect feature points, and generating a plurality of groups of feature points and feature vectors of the feature points;
comparing the number of the obtained feature points under various floating point precision, and obtaining the feature points under the minimum floating point precision and the feature vectors of the feature points under the condition that the number of the feature points is higher than the preset number of the feature points;
storing the feature points and feature vectors of the feature points under the minimum floating point precision as the feature points and feature vectors of a group of target images into a target feature library;
processing the image to be identified to generate a characteristic point and characteristic vector group queue representing the image to be identified; the image to be identified and the target image are shot on the same entity target;
performing multi-distance standard threshold matching based on the characteristic points and characteristic vector group queues of the image to be identified and the characteristic points and characteristic vectors of the target image to generate characteristic point pair group queues; comprising the following steps:
obtaining each group of feature vectors in the feature vector group queue of the image to be identified, and performing distance calculation on each group of feature vectors and the feature vector of the target image by using a plurality of distance standard thresholds to obtain different feature point pairs under different distance standard thresholds, so as to generate a plurality of groups of feature point pairs;
combining the multiple groups of characteristic point pairs to generate a characteristic point pair group queue based on multi-scale transformation and multi-threshold matching;
performing matching degree calculation on each group of characteristic point pairs in the characteristic point pair group queue, obtaining matching characteristic point pairs with matching degree larger than a set threshold value, performing transformation matrix calculation, and generating a transformation image queue;
filtering the transformed images in the transformed image queue to obtain an optimal corrected image, and completing image calibration; comprising the following steps:
acquiring each image in the transformed image queue, and detecting the characteristic points of the transformed image by using a SIFT characteristic point detection algorithm realized based on a GPU;
acquiring a matching point pair of the characteristic point of each transformation image and the characteristic point of the target image by using a RANSAC algorithm realized based on a GPU, and calculating the distance between the matching point pairs by using Euler distances;
and comparing the distances between the matching point pairs, wherein the transformed image with the smallest distance is the optimal correction image.
2. The image calibration method based on multi-scale feature matching according to claim 1, wherein three kinds of small-to-large floating point precision SIFT feature point detection algorithms based on GPU are sequentially used for feature point detection, and three kinds of feature points and feature vectors of the feature points are generated; and/or
The storage form of the characteristic points is a two-dimensional vector for recording coordinates of the map points; the feature vectors of the feature points are stored in 128-dimensional vectors.
3. The image calibration method based on multi-scale feature matching according to claim 1, wherein the processing the image to be identified to generate the feature points and the feature vector group queues of the image to be identified comprises:
preprocessing an image to be identified;
obtaining floating point precision of a SIFT feature point detection algorithm based on GPU (graphics processing unit) adopted when generating a feature vector of the target image;
and based on the floating point precision, performing feature point detection on the image to be identified by adopting a SIFT feature point detection algorithm realized based on the GPU, obtaining a group of feature points and feature vectors of the image to be identified under each scale, and generating a feature point and feature vector group queue of the image to be identified.
4. A multi-scale feature matching based image calibration method according to claim 3, wherein the preprocessing of the image to be identified comprises:
if the scale of the target image is not different from the scale of the image to be identified, performing scale transformation, and generating an image queue containing only one image; otherwise, performing scale transformation on the image to be identified by using a plurality of scales to generate an image queue containing a plurality of scale images; wherein:
the scale of the target image is not greatly different from the scale of the image to be identified, and the method is determined by the following steps:
recording the scale of the target image as h1 and the scale of the image to be identified as h2; if |h1-h2|/h1<10%, the scale of the target image is considered to be not much different from the scale of the image to be identified.
5. The method for calibrating an image based on multi-scale feature matching according to claim 1, wherein the computing the matching degree of each set of feature point pairs in the feature point pair array, obtaining the matching feature point pairs with the matching degree greater than a set threshold value, and computing a transformation matrix to generate a transformed image array, includes:
acquiring each group of characteristic point pairs in the plurality of groups of characteristic point pair group queues, carrying out characteristic point pair matching by using a RANSAC characteristic point matching algorithm realized based on a GPU, and acquiring a plurality of groups of matching characteristic point pairs with matching degree larger than a set threshold;
performing transformation matrix calculation on each group of matching characteristic point pairs in the plurality of groups of matching characteristic point pairs to generate a perspective transformation matrix;
generating a perspective transformation inverse matrix for each perspective transformation matrix; and synthesizing the transformed image by utilizing the perspective transformation inverse matrix, and finally generating a transformed image queue based on multi-scale transformation and multi-threshold matching.
6. An image calibration system based on multi-scale feature matching, comprising:
the target image processing module is used for processing the target image and generating characteristic points and characteristic vectors representing the target image; selecting a target image, sequentially detecting characteristic points by using a plurality of SIFT characteristic point detection algorithms which are realized from small to large and are based on the GPU, and generating a plurality of groups of characteristic points and characteristic vectors of the characteristic points; comparing the number of the obtained feature points under various floating point precision, and obtaining the feature points under the minimum floating point precision and the feature vectors of the feature points under the condition that the number of the feature points is higher than the preset number of the feature points; storing the feature points and feature vectors of the feature points under the minimum floating point precision as the feature points and feature vectors of a group of target images into a target feature library;
the image processing module to be identified processes the image to be identified to generate a characteristic point and a characteristic vector group queue representing the image to be identified; the image to be identified and the target image are shot on the same entity target;
the characteristic point pair generating module is used for carrying out multi-distance standard threshold matching based on the characteristic points and characteristic vector group queues of the image to be identified and the characteristic points and characteristic vectors of the target image to generate characteristic point pair group queues; each group of feature vectors in the feature vector group queue of the image to be identified is obtained, a plurality of distance standard thresholds are used for carrying out distance calculation on each group of feature vectors and the feature vector of the target image, different feature point pairs under different distance standard thresholds are obtained, and a plurality of groups of feature point pairs are generated; combining the multiple groups of characteristic point pairs to generate a characteristic point pair group queue based on multi-scale transformation and multi-threshold matching;
the transformation image generation module is used for carrying out matching degree calculation on each group of characteristic point pairs in the characteristic point pair array, obtaining matching characteristic point pairs with matching degree larger than a set threshold value, carrying out transformation matrix calculation, and generating a transformation image array;
the image calibration module filters the transformed images in the transformed image queue to obtain an optimal correction image and complete image calibration; each image in the transformed image queue is obtained, and feature point detection of the transformed image is carried out by using a SIFT feature point detection algorithm realized based on a GPU; acquiring a matching point pair of the characteristic point of each transformation image and the characteristic point of the target image by using a RANSAC algorithm realized based on a GPU, and calculating the distance between the matching point pairs by using Euler distances; and comparing the distances between the matching point pairs, wherein the transformed image with the smallest distance is the optimal correction image.
7. A terminal comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor is operable to perform the method of any of claims 1-5 when the program is executed.
8. A computer readable storage medium having stored thereon a computer program, which when executed by a processor is operable to perform the method of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110341817.5A CN113033578B (en) | 2021-03-30 | 2021-03-30 | Image calibration method, system, terminal and medium based on multi-scale feature matching |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110341817.5A CN113033578B (en) | 2021-03-30 | 2021-03-30 | Image calibration method, system, terminal and medium based on multi-scale feature matching |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113033578A CN113033578A (en) | 2021-06-25 |
| CN113033578B true CN113033578B (en) | 2023-05-16 |
Family
ID=76452948
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110341817.5A Active CN113033578B (en) | 2021-03-30 | 2021-03-30 | Image calibration method, system, terminal and medium based on multi-scale feature matching |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113033578B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114220068B (en) * | 2021-11-08 | 2023-09-01 | 珠海优特电力科技股份有限公司 | Method, device, equipment, medium and product for determining disconnecting link switching state |
Family Cites Families (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105654421B (en) * | 2015-12-21 | 2019-03-26 | 西安电子科技大学 | Based on the projective transformation image matching method for converting constant low-rank texture |
| CN106845490A (en) * | 2016-12-30 | 2017-06-13 | 国网河南省电力公司检修公司 | Based on the super-pressure converting equipment infrared image localization method that GPU accelerates |
| CN107610164B (en) * | 2017-09-11 | 2020-07-14 | 北京空间飞行器总体设计部 | High-resolution four-number image registration method based on multi-feature mixing |
| CN107967496B (en) * | 2017-12-22 | 2020-05-19 | 华中科技大学 | Image feature matching method based on geometric constraint and GPU cascade hash |
| CN108596867A (en) * | 2018-05-09 | 2018-09-28 | 五邑大学 | A kind of picture bearing calibration and system based on ORB algorithms |
| CN109558844A (en) * | 2018-11-30 | 2019-04-02 | 厦门商集网络科技有限责任公司 | The method and apparatus of self-defined template discrimination is promoted based on image normalization |
| CN109544484B (en) * | 2019-02-20 | 2019-06-07 | 上海赫千电子科技有限公司 | A kind of method for correcting image and device |
| CN110070568A (en) * | 2019-04-28 | 2019-07-30 | 重庆学析优科技有限公司 | A kind of picture antidote and system |
| CN111832634B (en) * | 2020-06-28 | 2023-09-08 | 深圳市优必选科技股份有限公司 | Foreign matter detection method, foreign matter detection system, terminal device and storage medium |
| CN112150520A (en) * | 2020-08-18 | 2020-12-29 | 徐州华讯科技有限公司 | Image registration method based on feature points |
| CN111898589B (en) * | 2020-08-26 | 2023-11-14 | 中国水利水电科学研究院 | Unmanned aerial vehicle image rapid registration method based on GPU+feature recognition |
| CN112102381A (en) * | 2020-09-11 | 2020-12-18 | 西安电子科技大学 | Hardware Trojan horse image registration method based on R-SIFT, storage medium and equipment |
-
2021
- 2021-03-30 CN CN202110341817.5A patent/CN113033578B/en active Active
Non-Patent Citations (1)
| Title |
|---|
| 基于FPGA的图像多尺度特征点提取及匹配;刘桂华;陈林宇;肖得胜;;电视技术(第09期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113033578A (en) | 2021-06-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111161349B (en) | Object posture estimation method, device and equipment | |
| US10621446B2 (en) | Handling perspective magnification in optical flow processing | |
| CN111814794B (en) | Text detection method and device, electronic equipment and storage medium | |
| CN107329962B (en) | Image retrieval database generation method, and method and device for enhancing reality | |
| CN111914908B (en) | Image recognition model training method, image recognition method and related equipment | |
| CN112085033A (en) | Template matching method and device, electronic equipment and storage medium | |
| CN112966725B (en) | Method and device for matching template images and terminal equipment | |
| CN109685830B (en) | Target tracking method, device and equipment and computer storage medium | |
| CN113743385A (en) | Unmanned ship water surface target detection method and device and unmanned ship | |
| CN112348116A (en) | Target detection method and device using spatial context and computer equipment | |
| CN111950370B (en) | Dynamic environment offline visual milemeter expansion method | |
| CN116246119A (en) | 3D target detection method, electronic device and storage medium | |
| CN114119992A (en) | Multi-mode three-dimensional target detection method and device based on image and point cloud fusion | |
| CN112541394A (en) | Black eye and rhinitis identification method, system and computer medium | |
| CN113033578B (en) | Image calibration method, system, terminal and medium based on multi-scale feature matching | |
| Lin et al. | SAN: Scale-aware network for semantic segmentation of high-resolution aerial images | |
| CN103841340A (en) | Image sensor and operating method thereof | |
| CN110880003B (en) | Image matching method and device, storage medium and automobile | |
| CN113704276A (en) | Map updating method and device, electronic equipment and computer readable storage medium | |
| CN115861595B (en) | Multi-scale domain self-adaptive heterogeneous image matching method based on deep learning | |
| CN114898306B (en) | Method and device for detecting target orientation and electronic equipment | |
| CN116977671A (en) | Target tracking method, device, equipment and storage medium based on image space positioning | |
| CN111582013A (en) | Ship retrieval method and device based on gray level co-occurrence matrix characteristics | |
| CN114549429B (en) | Depth data quality evaluation method and device based on hypergraph structure | |
| CN111986299A (en) | Point cloud data processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |