CN113643180B - Image registration method, device, equipment and medium - Google Patents
Image registration method, device, equipment and medium Download PDFInfo
- Publication number
- CN113643180B CN113643180B CN202110952011.XA CN202110952011A CN113643180B CN 113643180 B CN113643180 B CN 113643180B CN 202110952011 A CN202110952011 A CN 202110952011A CN 113643180 B CN113643180 B CN 113643180B
- Authority
- CN
- China
- Prior art keywords
- vector
- pixel
- feature
- image
- dimensional vector
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 239000013598 vector Substances 0.000 claims abstract description 269
- 238000012545 processing Methods 0.000 claims abstract description 27
- 230000009466 transformation Effects 0.000 claims abstract description 21
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims abstract description 20
- 230000006835 compression Effects 0.000 claims description 34
- 238000007906 compression Methods 0.000 claims description 34
- 230000008859 change Effects 0.000 claims description 21
- 239000011159 matrix material Substances 0.000 claims description 18
- 238000004422 calculation algorithm Methods 0.000 claims description 12
- 238000000605 extraction Methods 0.000 claims description 10
- 238000012360 testing method Methods 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 238000005070 sampling Methods 0.000 claims description 3
- 238000004590 computer program Methods 0.000 description 11
- 238000004891 communication Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 8
- 230000000694 effects Effects 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 6
- 238000001514 detection method Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000005286 illumination Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 230000003993 interaction Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 206010028980 Neoplasm Diseases 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000002583 angiography Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000010267 cellular communication Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000007429 general method Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 230000003902 lesion Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000035515 penetration Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/14—Transformations for image registration, e.g. adjusting or mapping for alignment of images
- G06T3/147—Transformations for image registration, e.g. adjusting or mapping for alignment of images using affine transformations
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The present disclosure relates to an image registration method, apparatus, device, and medium; wherein the method comprises the following steps: acquiring a reference image and an original image to be registered; respectively converting the reference image and the original image according to the Hilbert curve to obtain a first one-dimensional vector corresponding to the reference image and a second one-dimensional vector corresponding to the original image; extracting a first feature vector from the first one-dimensional vector, and extracting a second feature vector from the second one-dimensional vector; performing feature point matching on the first feature vector and the second feature vector to obtain feature point pairs; and carrying out affine transformation processing on the reference image according to the characteristic point pairs to obtain a target image registered with the original image. The present disclosure can increase the speed and accuracy of image registration.
Description
Technical Field
The disclosure relates to the technical field of image processing, and in particular relates to an image registration method, an image registration device and an image registration medium.
Background
Image registration is the process of matching and overlaying two or more images of the same object under different conditions (e.g., device, time, illumination, angle, etc.). Currently, the general method of image registration is: firstly, matching is carried out by utilizing image feature points, and then homography matrixes of two images to be registered are calculated by methods such as random sampling, so that image registration is realized. However, the above-described general image registration method is poor in effect against images having complicated features, such as uneven illumination, angular distortion, and the like, so that the image registration accuracy between the images is low.
Disclosure of Invention
In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides an image registration method, apparatus, device and medium.
The present disclosure provides an image registration method, comprising:
Acquiring a reference image and an original image to be registered; converting the reference image and the original image according to a Hilbert curve respectively to obtain a first one-dimensional vector corresponding to the reference image and a second one-dimensional vector corresponding to the original image; the first one-dimensional vector is used for representing the sequence relation between the pixel points in the reference image, and the second one-dimensional vector is used for representing the sequence relation between the pixel points in the original image; extracting a first feature vector from the first one-dimensional vector, and extracting a second feature vector from the second one-dimensional vector; performing feature point matching on the first feature vector and the second feature vector to obtain feature point pairs; and carrying out affine transformation processing on the reference image according to the characteristic point pairs so as to obtain a target image registered with the original image.
The present disclosure provides an image registration apparatus, comprising:
The image acquisition module is used for acquiring a reference image and an original image to be registered; the image conversion module is used for respectively converting the reference image and the original image according to the Hilbert curve to obtain a first one-dimensional vector corresponding to the reference image and a second one-dimensional vector corresponding to the original image; the first one-dimensional vector is used for representing the sequence relation between the pixel points in the reference image, and the second one-dimensional vector is used for representing the sequence relation between the pixel points in the original image; the feature extraction module is used for extracting a first feature vector from the first one-dimensional vector and extracting a second feature vector from the second one-dimensional vector; the matching module is used for matching the characteristic points of the first characteristic vector and the second characteristic vector to obtain characteristic point pairs; and the registration module is used for carrying out affine transformation processing on the reference image according to the characteristic point pairs so as to obtain a target image registered with the original image.
The present disclosure provides an electronic device including: a processor; and a memory storing a program, wherein the program comprises instructions that when executed by the processor cause the processor to perform a method of image registration according to the above.
The present disclosure provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform a method of registration according to an image.
The present disclosure provides a computer program product comprising a computer program which, when executed by a processor, implements the above-described image registration method.
Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has the following advantages:
According to the image registration method, the device, the equipment and the medium, firstly, the acquired reference image and the original image are respectively converted according to the Hilbert curve to obtain a corresponding first one-dimensional vector and a corresponding second one-dimensional vector; then extracting a first characteristic vector from the first one-dimensional vector, and extracting a second characteristic vector from the second one-dimensional vector; and finally, carrying out feature point matching on the first feature vector and the second feature vector, and carrying out affine transformation processing on the reference image according to the obtained feature point pairs so as to obtain a target image registered with the original image. The method and the device can effectively increase the speed and accuracy of image registration.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure.
In order to more clearly illustrate the embodiments of the present disclosure or the solutions in the prior art, the drawings that are required for the description of the embodiments or the prior art will be briefly described below, and it will be obvious to those skilled in the art that other drawings can be obtained from these drawings without inventive effort.
Fig. 1 is a flowchart of an image registration method provided in an embodiment of the present disclosure;
FIG. 2 is a schematic illustration of an original image and a reference image provided by an embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of an image registration apparatus provided in an embodiment of the present disclosure;
Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the disclosure.
Detailed Description
In order that the above objects, features and advantages of the present disclosure may be more clearly understood, embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure have been shown in the accompanying drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.
It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order and/or performed in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.
The term "including" and variations thereof as used herein are intended to be open-ended, i.e., including, but not limited to. The term "based on" is based at least in part on. The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments. Related definitions of other terms will be given in the description below. It should be noted that the terms "first," "second," and the like in this disclosure are merely used to distinguish between different devices, modules, or units and are not used to define an order or interdependence of functions performed by the devices, modules, or units.
It should be noted that references to "one", "a plurality" and "a plurality" in this disclosure are intended to be illustrative rather than limiting, and those of ordinary skill in the art will appreciate that "one or more" is intended to be understood as "one or more" unless the context clearly indicates otherwise.
The names of messages or information interacted between the various devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
When the existing image registration method is adopted to process images with complex features, the accuracy of image registration is low and the effect is poor. For example, in a common application scenario, a user generally adopts a mode of combining photographing questions with caption libraries to modify questions (such as selecting, filling in, judging questions) with semantic information. Due to various reasons such as writing habit and photographing scene, the text image photographed by the user may have a lot of problems such as back penetration (i.e. the other side is affected due to writing on both sides of the same page), uneven illumination, offset and improper photographing angle (refer to the original image shown in fig. 2); thus, in this way, how to more accurately correspond the text image shot by the user to the answer area of the image in the question bank has a great influence on the accuracy of shooting the questions. The effective mode of realizing the region correspondence is to use image registration, so that the accuracy of the photographing judgment is seriously dependent on the effect of the image registration, however, the current image registration method has poor registration effect on text images with complex changes, and further the accuracy of the photographing judgment is restricted.
Based on the above consideration, the embodiment of the disclosure provides an image registration method, device, equipment and medium, and the technology can be widely applied to the fields of target detection, model reconstruction, motion estimation, feature matching, tumor detection, lesion positioning, angiography, geological exploration, aviation reconnaissance and the like. For ease of understanding, embodiments of the present disclosure are described in detail below.
Referring to a flowchart of an image registration method provided in fig. 1, the method may include the steps of:
Step S102, acquiring a reference image and an original image to be registered.
The reference image and the original image in this embodiment may be images in various scenes where image registration is required, such as image enhancement, shape analysis of images, and image stitching. The reference image and the original image are two-dimensional images containing the same content, wherein the original image is an image that needs to be registered with the reference image as a registration target. In practical application, the original image may be an image obtained by a user through an image selection operation, an image shooting operation or an image uploading operation in a terminal, as shown in fig. 2, specifically, for example, a text image obtained by shooting a target test question, and correspondingly, the reference image is specifically, for example, an image including the target test question in a preset test question library.
In some implementations, the reference image and the original image may both be grayscale images that have been previously grayscale processed, i.e., the reference image and the original image are grayscale images, respectively.
Step S104, respectively converting the reference image and the original image according to the Hilbert curve to obtain a first one-dimensional vector corresponding to the reference image and a second one-dimensional vector corresponding to the original image; the first one-dimensional vector is used for representing the sequence relation among the pixel points in the reference image, and the second one-dimensional vector is used for representing the sequence relation among the pixel points in the original image.
The Hilbert curve (Hilbert) is a space-filling curve; a space-filling curve refers to a functional curve that includes the entire two-dimensional or even multidimensional space by means of a one-dimensional curve. The discrete approximation representation method of the Hilbert curve is very practical, the spatial proximity can be well reserved when the multidimensional space is converted into the one-dimensional space, a two-dimensional plane can be filled with the high-order Hilbert curve, then the curve is unfolded, and adjacent pixel points on the two-dimensional plane are still positioned on adjacent positions on the one-dimensional Hilbert curve.
In the present embodiment, the manner of converting the reference image and the original image according to the hilbert curve is the same, and the conversion manner will be described herein only by taking the original image as an example. The Hilbert curve can be used for representing regular text or irregular text in text recognition scenes of any shape; according to the method, a Hilbert curve with a proper order can be selected according to the text intensity or the pixel size of an original image, all pixels in the original image are arranged into one-dimensional vector according to the scanning sequence of the Hilbert curve, and a second one-dimensional vector corresponding to the original image is obtained. It can be understood that the elements with the sequence relationship in the second one-dimensional vector are pixel points in the original image, the pixel points in the original image are in one-to-one correspondence with the elements in the second one-dimensional vector, and the numerical values of the elements are pixel values of the corresponding pixel points; wherein the pixel values may be represented in gray scale values.
According to the embodiment, the two-dimensional reference image and the original image are converted into the corresponding one-dimensional vectors, so that adjacent pixel points in the two-dimensional space in the image become adjacent pixel points in the one-dimensional space which are arranged in sequence in the one-dimensional vector, the registration problem of the two-dimensional image, namely the problem of solving a space transformation matrix, is converted into the problem of matching the vector similarity in the one-dimensional space, and the image registration problem becomes a matching problem, so that the homography matrix can be prevented from being obtained based on simple descriptor screening pairing. In addition, the image is generally divided into a key region including the object content and a background region not including the object content, and in the one-dimensional vector, adjacent pixels of the background region and adjacent pixels of the key region are arranged separately, so that the influence of background noise caused by factors such as uneven illumination, angle deformity, back transmission and the like can be ignored in the one-dimensional vector.
Step S106, extracting a first feature vector from the first one-dimensional vector and extracting a second feature vector from the second one-dimensional vector.
In some implementations, the manner in which the first feature vector is extracted from the first one-dimensional vector may be referred to as follows: sliding windows on the basis of windows with preset sizes, extracting fourth pixel points with severe pixel value changes in each window according to pixel values of the pixel points in the first one-dimensional vectors, and obtaining a first feature vector according to a sequence relation of the fourth pixel points in the first one-dimensional vectors; the first feature vector is used for representing the sequence relation between the fourth pixel points. Similarly, the way to extract the second feature vector from the second one-dimensional vector includes: sliding a window on the second one-dimensional vector based on a window with a preset size, and extracting a third pixel point with severe change of the pixel value in each window according to the pixel value of the pixel point in the second one-dimensional vector; and determining a second feature vector based on the third pixel point.
And S108, performing feature point matching on the first feature vector and the second feature vector to obtain feature point pairs.
In this embodiment, feature points between the first feature vector and the second feature vector may be matched according to a feature point detection algorithm, so as to obtain feature points on the two feature vectors, where the feature points represent the same pixel point, thereby forming a feature point pair, or alternatively, the feature point pair may represent one pixel point on the original image and one pixel point on the reference image. Among them, feature point detection algorithms are, for example, SIFT (Scale-INVARIANT FEATURE TRANSFORM, scale invariant feature transform) feature detection algorithm, SURF (Speeded Up Robust Features, accelerated robust feature) algorithm, orb (Oriented FAST and Rotated BRIEF) algorithm, and the like.
Step S110, affine transformation processing is performed on the reference image according to the feature point pairs to obtain a target image registered with the original image.
After the feature point pairs are obtained, affine transformation processing can be performed on all pixel points in the reference image based on the matched feature point pairs in the first feature vector and the second feature vector, so as to obtain a target image aligned with the preset image. For example, a homography matrix of the reference image with respect to the original image is calculated based on the feature point pairs, and affine transformation processing is performed on the reference image based on the homography matrix, so as to obtain a target image registered with the original image.
The homography matrix is used for describing the position mapping relation of the object between the world coordinate system and the pixel coordinate system; in this embodiment, the homography matrix is used to represent a positional mapping relationship between the object and the reference image and the original image, and by performing image registration on the reference image and the original image, a feature point pair matched with the two images is obtained, and then the homography matrix is calculated based on the position of the feature point pair on the image.
According to the image registration method provided by the embodiment of the disclosure, firstly, the acquired reference image and the original image are respectively converted according to the Hilbert curve to obtain a corresponding first one-dimensional vector and a corresponding second one-dimensional vector; then extracting a first characteristic vector from the first one-dimensional vector, and extracting a second characteristic vector from the second one-dimensional vector; and finally, carrying out feature point matching on the first feature vector and the second feature vector, and carrying out affine transformation processing on the reference image according to the obtained feature point pairs so as to obtain a target image registered with the original image. In the technical scheme, the one-dimensional vector obtained by using the Hilbert curve ensures that adjacent pixel points on the two-dimensional image are adjacent in one-dimensional space, and the calculation of the characteristic points can represent local information; meanwhile, in the feature vectors extracted from the one-dimensional vectors, each feature point is more representative and has a small number, compared with the two-dimensional image, the feature vectors remove noise and redundancy, only a few key information is reserved, and the speed and accuracy of image registration are effectively increased.
For ease of understanding, the following detailed description is presented for an image registration method provided by embodiments of the present disclosure.
For the above step S106, the present embodiment takes the first one-dimensional vector as an example, and provides a method for extracting the first feature vector from the first one-dimensional vector, which includes the following steps:
Step 1, inputting a first one-dimensional vector to a preset encoder, and compressing the first one-dimensional vector by the encoder to obtain a first compressed vector with low dimensionality corresponding to the first one-dimensional vector.
The encoder is obtained by training a preset VAE (Variational Autoencoder, variation self-encoder) model, wherein the convolution kernel and the deconvolution kernel of the VAE model are one-dimensional.
The first one-dimensional vector is usually higher in dimension, which results in a very large calculation amount, and based on this, the embodiment can adopt an encoder to encode and compress the first one-dimensional vector with high dimension, output a first compressed vector with lower dimension, and reduce the vector length. In a specific implementation, the compression ratio of the first compression vector relative to the first one-dimensional vector may be set according to practical situations, for example, the compression ratio of the encoder is set to be 1/4, and then the vector length of the first compression vector output by the encoder is 1/4 of the first one-dimensional vector. The embodiment uses the encoder to compress the vectors, so that the calculated amount of the vectors can be reduced, the calculation speed is increased, the calculation deviation is reduced, and the faster and more accurate image registration is realized.
And 2, extracting pixel points with severe pixel value changes from the first compression vector according to the pixel values of the pixel points in the first compression vector, and determining the extracted pixel points as target feature points.
In a specific embodiment, this can be achieved by the following steps 2.1 to 2.4:
Step 2.1, performing differential operation on the first compression vector to obtain a differential vector; the differential vector is used to represent the variation of pixel values between pixel points. Specifically, the difference operation is performed on the first compression vector to obtain a gradient vector of the first compression vector, thereby obtaining a difference vector.
And 2.2, sliding windows on the first compression vectors by using preset first windows, and extracting first pixel points with the largest pixel value change in each first window according to the difference vectors.
In a specific embodiment, the first window is, for example, a window comprising 32 elements; in this case, first windows of length 32 are used, sliding without overlap on the first compression vectors, and the first pixels with the greatest pixel value variation within each first window are found from the variation of the pixel values represented by the differential vectors.
Step 2.3, sliding windows on the local vectors by using preset second windows, and extracting second pixel points with the largest difference between the pixel values of the second windows and the pixel values of the first pixel points according to the differential vectors; the local vector is a vector of a preset size centered on the first pixel point in the first compressed vector.
In the first compression vector, a local vector with a preset size is determined by taking the position of the first pixel point as the center, for example: 32 elements left and 32 elements right by taking the position of the first pixel point as the center, thereby determining the local vector with the preset length of 64. The second window is, for example, a window comprising 4 elements; based on the above, a second window with the length of 4 is used, the window is slid on the local vector, and a second pixel point with the largest difference between the pixel value of each second window and the pixel value of the first pixel point is found according to the change of the pixel value represented by the differential vector. Based on the above, 16 second pixel points corresponding to the first pixel point can be found on the local vector.
And 2.4, taking the extracted first pixel point and the second pixel point as pixel points with severe pixel value changes.
It is easy to understand that the first pixel point and the second pixel point obtained on the first compression vector are multiple, and all the first pixel point and the second pixel point are determined as the target feature point in this embodiment.
And step 3, determining a first feature vector based on the target feature points. Specifically, the target feature point may be used as an element of the first feature vector, and the first feature vector is obtained according to the sequence relationship of the target feature point in the first compressed vector; and characterizing the sequence relation among the target feature points by using the first feature vector.
The method for extracting the second feature vector from the second one-dimensional vector is the same as the method for extracting the first feature vector from the first one-dimensional vector described in the above embodiment, and will not be described here.
After extracting the first feature vector from the first one-dimensional vector and extracting the second feature vector from the second one-dimensional vector according to the above embodiment, feature point matching is performed on the first feature vector and the second feature vector according to the following embodiment, including:
Firstly, determining the matching degree of feature points between a first feature vector and a second feature vector according to a preset matching degree algorithm, and determining the feature points with the matching degree higher than a preset matching degree threshold value as candidate feature point pairs; and then optimizing the candidate feature point pairs according to a RANSAC (Random Sample Consensus, random sampling) algorithm to obtain final feature point pairs.
Before affine transformation processing is performed on the reference image according to the feature point pairs, the embodiment may also record in advance a first correspondence relationship between the position of the same pixel point on the reference image and the position on the first one-dimensional vector; and recording a second correspondence between the position of the same pixel point on the original image and the position on the second one-dimensional vector.
In a specific embodiment, in the process of respectively converting the reference image and the original image according to the hilbert curve, one-to-one correspondence between the pixel points on the two-dimensional image and the elements on the converted one-dimensional vector can be converted into a dictionary, a first correspondence between the pixel points on the reference image and the elements on the first one-dimensional vector is recorded through a first dictionary, and a second correspondence between the pixel points on the original image and the elements on the second one-dimensional vector is recorded through a second dictionary.
In practical application, according to the first corresponding relation, the position coordinates of the feature points on the first feature vector, which are mapped on the reference image, can be determined; and according to the second corresponding relation, the position coordinates of the feature points on the second feature vector, which are mapped on the original image, can be determined.
According to the above-described embodiment, a description is given of a specific implementation of affine transformation processing of a reference image according to feature point pairs, including:
and calculating the position coordinates of each feature point in the feature point pair mapped on the corresponding image according to the first corresponding relation and the second corresponding relation. Wherein, the characteristic points in the characteristic point pair comprise: a first feature point belonging to a first feature vector and a second feature point belonging to a second feature vector. In the specific calculation process of the position coordinates, the position coordinates of the first feature point in the feature point pair mapped on the reference image can be determined according to the first corresponding relation and the compression ratio between the first compression vector and the first one-dimensional vector. Of course, the same manner may be adopted to determine the position coordinates of the second feature point in the pair of feature points mapped on the original image.
Calculating a homography matrix of the reference image relative to the original image based on the position coordinates of each feature point in the feature point pair; and carrying out affine transformation processing on the reference image based on the homography matrix to obtain a target image registered with the original image.
For example, any two or more candidate feature point pairs are selected from the feature point pairs; calculating a candidate homography matrix of the reference image relative to the original image based on the candidate feature point pairs, and carrying out affine transformation processing on feature points in the reference image based on the candidate homography matrix; judging whether the accuracy of the feature points obtained by affine transformation processing is greater than a preset threshold value or not according to the feature points in the original image; if not, returning to execute the selection of any two or more candidate feature point pairs from the feature point pairs; if yes, taking the candidate homography matrix as a final homography matrix; and then, carrying out affine transformation processing on the reference image based on the final homography matrix to obtain a target image registered with the original image.
In order to enable the encoder in the above embodiment to be used directly for vector compression, the VAE model containing the encoder needs to be trained in advance; the purpose of training the VAE model is to finally determine parameters that can meet the requirements. With the trained parameters, the encoder is able to achieve the desired vector compression. The embodiment provides a training method of a VAE model, which is shown by the following steps:
(1) Acquiring a sample image; the sample images are multiple, and each sample image is converted into a first sample vector according to the Hilbert curve.
(2) The first sample vector is input to an encoder of the VAE model to be trained to output a first sample compression vector.
(3) And inputting the first sample compression vector to a decoder of the VAE model to be trained, and decoding the first sample compression vector by the decoder to generate a restored image corresponding to the sample image.
(4) And calculating a loss function value between the sample image and the restored image according to the L1 loss function, and carrying out parameter adjustment on the VAE model to be trained according to the loss function value until the loss function value is converged to a preset value, ending training, and obtaining the trained VAE model.
(5) The encoders in the trained VAE model are retained.
Through the above steps, an encoder that can be directly used for vector compression is obtained.
In summary, according to the image registration method provided by the embodiment of the present disclosure, the one-dimensional vector obtained by using the hilbert curve ensures that adjacent pixel points on the two-dimensional image are adjacent in the one-dimensional space, so that the calculation of the feature points is ensured to be capable of representing local information; meanwhile, in the feature vectors extracted from the one-dimensional vectors, each feature point is more representative and has a smaller number, and meanwhile, after the vectors are compressed, the length of the vectors is reduced, and meanwhile, key information is reserved, so that compared with a two-dimensional image, the feature vectors remove noise and redundancy, only a few key information is reserved, the effect is better in image registration, and the speed and accuracy of image registration are effectively increased.
On the basis of increasing the image registration effect in the above manner, further, beneficial effects can be obtained in practical application, such as facing text images with complex changes, and the accuracy of photographing judgment can be obviously improved.
Based on the image registration method provided in the above embodiment, the present embodiment provides an image registration apparatus. Referring to a schematic structural view of an image registration apparatus shown in fig. 3, the apparatus includes:
an image acquisition module 302, configured to acquire a reference image and an original image to be registered;
The image conversion module 304 is configured to convert the reference image and the original image according to the hilbert curve, so as to obtain a first one-dimensional vector corresponding to the reference image and a second one-dimensional vector corresponding to the original image; the first one-dimensional vector is used for representing the sequence relation between the pixel points in the reference image, and the second one-dimensional vector is used for representing the sequence relation between the pixel points in the original image;
A feature extraction module 306, configured to extract a first feature vector from the first one-dimensional vector and extract a second feature vector from the second one-dimensional vector;
the matching module 308 is configured to perform feature point matching on the first feature vector and the second feature vector to obtain feature point pairs;
the registration module 310 is configured to perform affine transformation processing on the reference image according to the feature point pairs to obtain a target image registered with the original image.
In one embodiment, the feature extraction module 306 includes:
the vector compression unit is used for inputting the first one-dimensional vector to a preset encoder, and compressing the first one-dimensional vector through the encoder to obtain a first compressed vector with low dimensionality corresponding to the first one-dimensional vector;
A first feature point extracting unit, configured to extract, from the first compression vector, a pixel point with a severe change in a pixel value according to a pixel value of the pixel point in the first compression vector, and determine the extracted pixel point as a target feature point;
And the vector determining unit is used for determining a first feature vector based on the target feature points.
In one embodiment, the feature point extraction unit is specifically configured to:
performing differential operation on the first compression vector to obtain a differential vector; the differential vector is used for representing the change of pixel values among pixel points;
Sliding windows on the first compression vectors by using preset first windows, and extracting first pixel points with the largest pixel value change in each first window according to the differential vectors;
Sliding windows on local vectors by using preset second windows, and extracting second pixel points with the largest difference between pixel values of the second windows and the first pixel points according to the differential vectors; the local vector is a vector of a preset size taking the first pixel point as a center in the first compression vector;
and taking the extracted first pixel point and the second pixel point as pixel points with severe pixel value change.
In one embodiment, the feature extraction module 306 further comprises a second feature point extraction unit for:
Sliding a window on the second one-dimensional vector based on a window with a preset size, and extracting a third pixel point with severe change of the pixel value in each window according to the pixel value of the pixel point in the second one-dimensional vector; and determining a second feature vector based on the third pixel point.
In one embodiment, the matching module 308 is specifically configured to:
determining the matching degree of the feature points between the first feature vector and the second feature vector according to a preset matching degree algorithm;
Determining the feature points with the matching degree higher than a preset matching degree threshold as candidate feature point pairs;
and optimizing the candidate feature point pairs according to the RANSAC algorithm to obtain final feature point pairs.
In one embodiment, the image registration apparatus further comprises a relationship recording module for:
Recording a first correspondence between the position of the same pixel point on the reference image and the position on the first one-dimensional vector;
a second correspondence between the position of the same pixel on the original image and the position on the second one-dimensional vector is recorded.
In one embodiment, the registration module 310 is specifically configured to:
calculating the position coordinates of each feature point in the feature point pair mapped on the corresponding image according to the first corresponding relation and the second corresponding relation;
Calculating a homography matrix of the reference image relative to the original image based on the position coordinates of each feature point in the feature point pair;
And carrying out affine transformation processing on the reference image based on the homography matrix to obtain a target image registered with the original image.
The device provided in this embodiment has the same implementation principle and technical effects as those of the foregoing method embodiment, and for brevity, reference may be made to the corresponding content of the foregoing method embodiment where the device embodiment is not mentioned.
The exemplary embodiments of the present disclosure also provide an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor. The memory stores a computer program executable by the at least one processor for causing the electronic device to perform a method according to embodiments of the present disclosure when executed by the at least one processor.
The present disclosure also provides a computer program product comprising a computer program, wherein the computer program, when executed by a processor of a computer, is for causing the computer to perform a method according to embodiments of the disclosure.
Referring to fig. 4, a block diagram of an electronic device 400 that may be a server or a client of the present disclosure, which is an example of a hardware device that may be applied to aspects of the present disclosure, will now be described. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 4, the electronic device 400 includes a computing unit 401 that can perform various suitable actions and processes according to a computer program stored in a Read Only Memory (ROM) 402 or a computer program loaded from a storage unit 408 into a Random Access Memory (RAM) 403. In RAM 403, various programs and data required for the operation of device 400 may also be stored. The computing unit 401, ROM 402, and RAM 403 are connected to each other by a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.
Various components in electronic device 400 are connected to I/O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 may be any type of device capable of inputting information to the electronic device 400, and the input unit 406 may receive input numeric or character information and generate key signal inputs related to user settings and/or function controls of the electronic device. The output unit 407 may be any type of device capable of presenting information and may include, but is not limited to, a display, speakers, video/audio output terminals, vibrators, and/or printers. Storage unit 404 may include, but is not limited to, magnetic disks, optical disks. The communication unit 409 allows the electronic device 400 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and/or chipsets, such as bluetooth (TM) devices, wiFi devices, wiMax devices, cellular communication devices, and/or the like.
The computing unit 401 may be a variety of general purpose and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 401 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the respective methods and processes described above. For example, in some embodiments, the image registration method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as the storage unit 408. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 400 via the ROM 402 and/or the communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the image registration method by any other suitable means (e.g. by means of firmware).
Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
The foregoing is merely a specific embodiment of the disclosure to enable one skilled in the art to understand or practice the disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown and described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (12)
1. A method of image registration, comprising:
acquiring a reference image and an original image to be registered;
Converting the reference image and the original image according to a Hilbert curve respectively to obtain a first one-dimensional vector corresponding to the reference image and a second one-dimensional vector corresponding to the original image; the first one-dimensional vector is used for representing the sequence relation between the pixel points in the reference image, and the second one-dimensional vector is used for representing the sequence relation between the pixel points in the original image;
Extracting a first feature vector from the first one-dimensional vector, and extracting a second feature vector from the second one-dimensional vector; wherein the extracting a first feature vector from the first one-dimensional vector comprises: sliding a window on the first one-dimensional vector based on a window with a preset size, extracting a fourth pixel point with severe pixel value change in each window according to the pixel value of the pixel point in the first one-dimensional vector, and obtaining a first feature vector according to the sequence relation of the fourth pixel point in the first one-dimensional vector; the fourth pixel point with the severe pixel value change comprises: a first pixel point with the largest pixel value change between the pixel points in a preset first window, and a second pixel point with the largest pixel value difference between the pixel points in a preset second window and the first pixel point; the extracting a second feature vector from the second one-dimensional vector includes: sliding a window on the second one-dimensional vector based on a window with a preset size, and extracting a third pixel point with severe change of the pixel value in each window according to the pixel value of the pixel point in the second one-dimensional vector; determining a second feature vector based on the third pixel point; the extraction mode of the third pixel point with the severe pixel value change is the same as that of the fourth pixel point;
According to the matching degree being higher than a preset matching degree threshold, carrying out feature point matching on the first feature vector and the second feature vector to obtain feature point pairs;
And carrying out affine transformation processing on the reference image according to the characteristic point pairs so as to obtain a target image registered with the original image.
2. The method of claim 1, wherein the extracting a first feature vector from the first one-dimensional vector comprises:
inputting the first one-dimensional vector to a preset encoder, and compressing the first one-dimensional vector by the encoder to obtain a first compressed vector with low dimensionality corresponding to the first one-dimensional vector;
extracting pixel points with severe pixel value changes from the first compression vector according to the pixel values of the pixel points in the first compression vector, and determining the extracted pixel points as target feature points;
a first feature vector is determined based on the target feature points.
3. The method of claim 2, wherein the extracting pixel points with strongly varying pixel values from the first compression vector based on the pixel values of the pixel points in the first compression vector comprises:
performing differential operation on the first compression vector to obtain a differential vector; the differential vector is used for representing the change of pixel values among pixel points;
Sliding windows on the first compression vectors by using preset first windows, and extracting first pixel points with the largest pixel value change in each first window according to the differential vectors;
Sliding windows on local vectors by using preset second windows, and extracting second pixel points with the largest difference between pixel values of the second windows and the first pixel points according to the differential vectors; the local vector is a vector of a preset size taking the first pixel point as a center in the first compression vector;
and taking the extracted first pixel point and the second pixel point as pixel points with severe pixel value change.
4. The method of claim 1, wherein the performing feature point matching on the first feature vector and the second feature vector according to the matching degree being higher than a preset matching degree threshold value to obtain a feature point pair includes:
determining the matching degree of the feature points between the first feature vector and the second feature vector according to a preset matching degree algorithm;
Determining the feature points with the matching degree higher than a preset matching degree threshold as candidate feature point pairs;
And optimizing the candidate feature point pairs according to a random sampling RANSAC algorithm to obtain final feature point pairs.
5. The method according to claim 1, wherein the method further comprises:
Recording a first correspondence between the position of the same pixel point on the reference image and the position on the first one-dimensional vector;
a second correspondence between the position of the same pixel on the original image and the position on the second one-dimensional vector is recorded.
6. The method according to claim 5, wherein performing affine transformation processing on the reference image according to the feature point pairs to obtain a target image registered with the original image, comprises:
calculating the position coordinates of each feature point in the feature point pair mapped on the corresponding image according to the first corresponding relation and the second corresponding relation;
Calculating a homography matrix of the reference image relative to the original image based on the position coordinates of each feature point in the feature point pair;
And carrying out affine transformation processing on the reference image based on the homography matrix to obtain a target image registered with the original image.
7. The method of claim 2, wherein the encoder is trained on a pre-set VAE model, wherein the convolution kernel and the deconvolution kernel of the VAE model are one-dimensional.
8. The method of claim 1, wherein the reference image and the original image are each a grayscale image.
9. The method of claim 1, wherein the original image comprises: shooting a text image obtained by a target test question, wherein the reference image comprises: the preset test question library comprises images of the target test questions.
10. An image registration apparatus, comprising:
the image acquisition module is used for acquiring a reference image and an original image to be registered;
The image conversion module is used for respectively converting the reference image and the original image according to the Hilbert curve to obtain a first one-dimensional vector corresponding to the reference image and a second one-dimensional vector corresponding to the original image; the first one-dimensional vector is used for representing the sequence relation between the pixel points in the reference image, and the second one-dimensional vector is used for representing the sequence relation between the pixel points in the original image;
The feature extraction module is used for extracting a first feature vector from the first one-dimensional vector and extracting a second feature vector from the second one-dimensional vector; wherein, the feature extraction module is further configured to: sliding a window on the first one-dimensional vector based on a window with a preset size, extracting a fourth pixel point with severe pixel value change in each window according to the pixel value of the pixel point in the first one-dimensional vector, and obtaining a first feature vector according to the sequence relation of the fourth pixel point in the first one-dimensional vector; the fourth pixel point with the severe pixel value change comprises: a first pixel point with the largest pixel value change between the pixel points in a preset first window, and a second pixel point with the largest pixel value difference between the pixel points in a preset second window and the first pixel point; sliding a window on the second one-dimensional vector based on a window with a preset size, and extracting a third pixel point with severe change of the pixel value in each window according to the pixel value of the pixel point in the second one-dimensional vector; determining a second feature vector based on the third pixel point; the extraction mode of the third pixel point with the severe pixel value change is the same as that of the fourth pixel point;
The matching module is used for matching the characteristic points of the first characteristic vector and the second characteristic vector according to the fact that the matching degree is higher than a preset matching degree threshold value, so as to obtain characteristic point pairs;
And the registration module is used for carrying out affine transformation processing on the reference image according to the characteristic point pairs so as to obtain a target image registered with the original image.
11. An electronic device, the electronic device comprising:
A processor; and
A memory in which a program is stored,
Wherein the program comprises instructions which, when executed by the processor, cause the processor to perform the image registration method according to any one of claims 1 to 9.
12. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the image registration method according to any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110952011.XA CN113643180B (en) | 2021-08-19 | 2021-08-19 | Image registration method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110952011.XA CN113643180B (en) | 2021-08-19 | 2021-08-19 | Image registration method, device, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113643180A CN113643180A (en) | 2021-11-12 |
| CN113643180B true CN113643180B (en) | 2024-06-21 |
Family
ID=78422799
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110952011.XA Active CN113643180B (en) | 2021-08-19 | 2021-08-19 | Image registration method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113643180B (en) |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1538843A3 (en) * | 1994-04-20 | 2006-06-07 | Oki Electric Industry Company, Limited | Image Encoding and Decoding Method and Apparatus Using Edge Synthesis and Inverse Wavelet Transform |
| US20070061735A1 (en) * | 1995-06-06 | 2007-03-15 | Hoffberg Steven M | Ergonomic man-machine interface incorporating adaptive pattern recognition based control system |
| US6597818B2 (en) * | 1997-05-09 | 2003-07-22 | Sarnoff Corporation | Method and apparatus for performing geo-spatial registration of imagery |
| US20040175057A1 (en) * | 2003-03-04 | 2004-09-09 | Thomas Tsao | Affine transformation analysis system and method for image matching |
-
2021
- 2021-08-19 CN CN202110952011.XA patent/CN113643180B/en active Active
Non-Patent Citations (2)
| Title |
|---|
| Robust Feature Matching for Remote Sensing Image Registration via Locally Linear Transforming;Jiayi Ma;《IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING》;全文 * |
| 基于双特征的丘陵山区耕地低空遥感图像配准算法;宋飞;杨扬;杨昆;张愫;毕东升;;北京航空航天大学学报(09);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113643180A (en) | 2021-11-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11475238B2 (en) | Keypoint unwarping for machine vision applications | |
| CN108022212B (en) | High-resolution picture generation method, generation device and storage medium | |
| CN107330439B (en) | Method for determining posture of object in image, client and server | |
| CN112991180B (en) | Image stitching method, device, equipment and storage medium | |
| CN111079764B (en) | Low-illumination license plate image recognition method and device based on deep learning | |
| CN114155546B (en) | Image correction method and device, electronic equipment and storage medium | |
| CN112929695B (en) | Video duplicate removal method and device, electronic equipment and storage medium | |
| CN109816694B (en) | Target tracking method and device and electronic equipment | |
| CN114255337A (en) | Method and device for correcting document image, electronic equipment and storage medium | |
| CN118158329A (en) | Image steganography and extraction information processing method and electronic equipment | |
| CN113658065B (en) | Image noise reduction method and device, computer readable medium and electronic equipment | |
| WO2017070841A1 (en) | Image processing method and apparatus | |
| CN111429371A (en) | Image processing method and device and terminal equipment | |
| CN110598139A (en) | Web browser augmented reality real-time positioning method based on 5G cloud computing | |
| CN115131714A (en) | Intelligent detection and analysis method and system for video image | |
| CN114511041A (en) | Model training method, image processing method, device, equipment and storage medium | |
| US20230005171A1 (en) | Visual positioning method, related apparatus and computer program product | |
| CN105979283A (en) | Video transcoding method and device | |
| CN111080683A (en) | Image processing method, image processing device, storage medium and electronic equipment | |
| CN113822871A (en) | Target detection method and device based on dynamic detection head, storage medium and equipment | |
| CN111243046B (en) | Image quality detection method, device, electronic equipment and storage medium | |
| CN113516697A (en) | Image registration method and device, electronic equipment and computer-readable storage medium | |
| CN113643180B (en) | Image registration method, device, equipment and medium | |
| CN108682021A (en) | Rapid hand tracking, device, terminal and storage medium | |
| CN112991451B (en) | Image recognition method, related device and computer program product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |