CN111833384B - Method and device for rapidly registering visible light and infrared images - Google Patents
Method and device for rapidly registering visible light and infrared images Download PDFInfo
- Publication number
- CN111833384B CN111833384B CN202010474030.1A CN202010474030A CN111833384B CN 111833384 B CN111833384 B CN 111833384B CN 202010474030 A CN202010474030 A CN 202010474030A CN 111833384 B CN111833384 B CN 111833384B
- Authority
- CN
- China
- Prior art keywords
- image
- infrared
- visible light
- edge
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000013519 translation Methods 0.000 claims abstract description 73
- 230000009466 transformation Effects 0.000 claims abstract description 9
- 239000011159 matrix material Substances 0.000 claims description 18
- 238000004364 calculation method Methods 0.000 claims description 11
- 238000012545 processing Methods 0.000 claims description 11
- 238000000605 extraction Methods 0.000 claims description 10
- 239000000284 extract Substances 0.000 claims description 3
- 230000008569 process Effects 0.000 description 12
- 230000000694 effects Effects 0.000 description 8
- 230000004927 fusion Effects 0.000 description 8
- 238000004422 calculation algorithm Methods 0.000 description 7
- 238000003384 imaging method Methods 0.000 description 4
- 230000007547 defect Effects 0.000 description 3
- 230000002093 peripheral effect Effects 0.000 description 3
- 238000009499 grossing Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000002411 adverse Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000005855 radiation Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
A method and a device for fast registering visible light and infrared images, the method comprises the following steps: inputting a visible light image and an infrared image, respectively carrying out graying, obtaining gray level images of the two images, and respectively extracting edge information of the two gray level images; performing first-stage translation traversal on the infrared image edge map, calculating the coincidence degree of the infrared image edge map and the visible light image edge map in the current state after each translation until the first-stage translation traversal is finished, and recording the offset when the coincidence degree reaches the maximum; and carrying out translation traversal of a second stage in a preset range around the position offset obtained by the translation traversal of the first stage serving as a reference, finding the offset when the coincidence degree of the infrared image edge map and the visible light image edge map in the range is maximum, carrying out translation transformation on the infrared image by utilizing the offset obtained by the translation traversal of the second stage, and aligning the infrared image with the visible light image to finish image registration.
Description
Technical Field
The invention relates to the field of image registration in image processing application, in particular to a method and a device for rapidly registering visible light and infrared images.
Background
Image fusion is a technology for extracting advantageous information in respective channels from image data about the same target acquired by a multi-source channel through image processing, and finally synthesizing a high-quality image. In recent years, unmanned aerial vehicle technology development is rapid, and an onboard photoelectric imaging system can provide the function of acquiring aerial images, so that the unmanned aerial vehicle has wide application in the fields of aerial reconnaissance, traffic monitoring, field search and rescue and the like, and a visible light camera and an infrared thermal imager are generally used for acquiring images on an onboard photoelectric platform. Compared with a visible light image, the infrared image reflects the difference of outward radiation energy of a target and a background, and has obvious advantages at night and in haze weather. Meanwhile, the infrared image also has the defects of low pixel resolution, poor contrast, blurred edges and the like. The visible light image and the infrared image are fused, and the respective advantages of the two images can be combined, so that the photographed image of the unmanned aerial vehicle has higher image quality and better environmental adaptability.
When an unmanned aerial vehicle imaging platform acquires a target image, the visible light image and the infrared image acquired by the same scene can have slight difference in space under the influence of factors such as a machine body design, camera installation, atmospheric refraction and the like, and the positions, the directions and the like of certain targets on the images are particularly inconsistent, so that adverse effects can be generated on the effect of image fusion, and therefore, the visible light image and the infrared image must be registered with high precision before fusion.
Image registration refers to searching a mapping relation between source images, and estimating relative motion parameters between the images so that the images to be registered acquired in the same scene reach complete consistency of corresponding point positions on real space pixel positions. The most common registration mode in the prior art is image registration based on feature matching, specifically, feature point detection and matching are carried out on an infrared image and a visible light image, a transformation matrix of the two images is established by finding out the most matched feature point pair, registration of the two images is realized, and then fusion of the subsequent corresponding pixel points is carried out. In order to ensure the matching effect, the existing feature matching algorithm such as SIFT and SURF has complex flow, generally, feature point extraction and descriptor calculation are firstly carried out, then the relationship of descriptors is utilized to match feature points between two images, and after the matching result is obtained, excellent matching screening and optimization are also required to be carried out by utilizing a certain criterion. The registration process has two obvious defects, namely, the whole process has more steps and higher complexity, and the descriptor calculation and feature point matching of the traditional feature matching algorithm are very time-consuming, so that the real-time processing of the unmanned aerial vehicle on-board embedded equipment is not facilitated; secondly, applicability is not wide enough, the feature matching algorithm has good scene effects of clear object edges and obvious features, but in some special scenes, the gray gradient direction in the neighborhood of the corresponding homonymous point of the heterogeneous image can be reverse, the accuracy of feature point matching is low, and the registration and fusion effects are greatly reduced.
Aiming at the airborne imaging platform with higher real-time requirement, the other simple and efficient registration thought is to compare the similarity of the visible light image edge and the infrared image edge in a certain space offset range, and the offset when the maximum similarity is obtained by searching is used as a registration result, but in order to obtain the offset of pixel-level precision, searching is required to be carried out in the transverse direction and the longitudinal direction, a large number of searching times can increase the time cost of an algorithm, the requirement of the real-time performance of the platform cannot be met, the operation speed of registration is improved while the fusion effect is ensured, and the method is a research direction of image registration and fusion algorithm in the unmanned plane field.
Disclosure of Invention
In view of the technical defects and technical drawbacks existing in the prior art, embodiments of the present invention provide a method and apparatus for fast registration of visible and infrared images, which overcome or at least partially solve the above problems, and a fast automatic registration acceleration algorithm for visible and infrared images based on two-stage search, which aims to solve the problem of slow registration speed of visible and infrared images, and specifically comprises the following steps:
as a first aspect of the present invention, there is provided a method of rapid registration of visible and infrared images, the method comprising:
step 1, inputting a visible light image and an infrared image, respectively graying the visible light image and the infrared image, respectively obtaining gray images of the visible light image and the infrared image, respectively extracting edge information of two gray images, and obtaining edge images of the two gray images, namely a visible light image edge image and an infrared image edge image;
step 2, performing first-stage translation traversal on the infrared image edge map, calculating the coincidence degree of the infrared image edge map and the visible light image edge map in the current state after each translation until the first-stage translation traversal is finished, and recording the position offset (x 1, y 1) when the coincidence degree reaches the maximum;
and step 3, carrying out translation traversal of the second stage in a preset range around the position offset obtained by translation traversal of the first stage serving as a reference, and finding the offset when the coincidence degree of the infrared image edge map and the visible light image edge map in the range is maximum. And the translation traversal of the second stage is similar to the translation traversal of the first stage, namely, the coincidence degree of the infrared image edge map and the visible light image edge map in the current state is calculated after each translation, and the offset when the coincidence degree of the infrared image edge map and the visible light image edge map in the preset range is maximum is obtained to be used as a final registration result.
And 4, performing translation transformation on the infrared image by using the offset obtained in the step 3, and aligning the infrared image with the visible light image to finish image registration.
Further, in step 1, edge information of two gray-scale images is extracted by using Sobel operator, specifically:
let the element value matrix of the gray level image of the visible light image or the infrared image be I;
and (3) respectively carrying out convolution on the I and two convolution kernels with odd sizes to calculate a horizontal gradient graph Gx and a vertical gradient graph Gy of the gray level graph, wherein the specific formulas are as follows:
wherein I is an element value matrix of the gray image, and represents convolution operation, and f (x, y) is set as a pixel value of a pixel point with coordinates (x, y) in the element value matrix, and the value is 0-255, and then the convolution result of the (x, y) point is specifically calculated as follows:
G x (x，y)＝(-1)*f(x-1，y-1)+0*f(x，y-1)+1*f(x+1，y+1)
+(-2)*f(x-1，y)+0*f(x，y)+2*f(x+1，y)
+(-1)*f(x-1，y+1)+0*f(x，y+1)+1*f(x+1，y+1)；
calculation of G by the same method y (x, y) in obtaining G x (x, y) and G y After (x, y), the approximate gradient of the (x, y) point is calculated as:
and calculating the approximate gradient of each pixel point based on the formula, thereby obtaining the edge information of the whole gray scale image.
Further, in step 2, for the time of block-added translation traversal, the translation traversal in the first stage uses preset m pixels as step length to perform rough traversal with large step length, and after the position offset when the fitness reaches the maximum is obtained;
further, for the time of block adding translation traversal, the translation traversal of the first stage carries out rough traversal of a large step length by taking preset m pixels as step length, after the position offset when the coincidence degree reaches the maximum is obtained, the translation traversal of the second stage carries out precision traversal of a small step length by taking preset n pixels as step length, and the offset (x 2, y 2) when the coincidence degree of the infrared image edge map and the visible light image edge map is maximum is found in a peripheral preset range by taking (x 1, y 1) as a center, so as to be used as a final offset.
Further, m is 5n or more.
Further, the integral difference of pixel values is adopted in translation traversal to measure the coincidence degree of two edge graphs, specifically: after each translation, comparing pixel values of each pixel point of the visible light edge map and the translated infrared edge map one by one, wherein the difference of the statistical pixel values is smaller than the number of preset values, and the more the number is, the more similar the two edge images are.
As a second aspect of the present invention, there is provided a device for rapid registration of visible and infrared images, the device comprising an image input module, a graying processing module, an edge image extraction module, a first stage traversal module, a second stage traversal module, and a registration module;
the image input module is used for inputting visible light images and infrared images;
the graying processing module is used for respectively graying the visible light image and the infrared image to obtain gray images of the visible light image and the infrared image;
the edge image extraction module is used for respectively extracting edge information of two gray images, and acquiring edge images of the two gray images, namely a visible light image edge image and an infrared image edge image;
the first-stage traversing module is used for carrying out first-stage translation traversal on the infrared image edge map, calculating the coincidence degree of the infrared image edge map and the visible light image edge map in the current state after each translation until the translation traversal of the first stage is finished, and recording the offset (x 1, y 1) when the coincidence degree reaches the maximum;
the second-stage traversing module is used for carrying out second-stage translational traversing in a preset range around the first-stage translational traversing module by taking the position offset obtained by the first-stage translational traversing as a reference, and finding the offset when the coincidence degree of the infrared image edge map and the visible image edge map in the range is maximum;
the registering module is used for performing translation transformation on the infrared image by using the offset obtained by the second-stage traversing module, and aligning the infrared image with the visible light image to finish image registering.
Further, the edge image extraction module respectively extracts edge information of two gray images by using a Sobel operator, specifically:
let the element value matrix of the gray level image of the visible light image or the infrared image be I;
and (3) respectively carrying out convolution on the I and two convolution kernels with odd sizes to calculate a horizontal gradient graph Gx and a vertical gradient graph Gy of the gray level graph, wherein the specific formulas are as follows:
wherein I is an element value matrix of the gray image, and represents convolution operation, and f (x, y) is set as a pixel value of a pixel point with coordinates (x, y) in the element value matrix, and the value is 0-255, and then the convolution result of the (x, y) point is specifically calculated as follows:
G x (x，y)＝(-1)*f(x-1，y-1)+0*f(x，y-1)+1*f(x+1，y+1)
+(-2)*f(x-1，y)+0*f(x，y)+2*f(x+1，y)
+(-1)*f(x-1，y+1)+0*f(x，y+1)+1*f(x+1，y+1)；
calculation of G by the same method y (x, y) in obtaining G x (x, y) and G y After (x, y), the approximate gradient of the (x, y) point is calculated as:
and calculating the approximate gradient of each pixel point based on the formula, thereby obtaining the edge information of the whole gray scale image.
Further, for the time of block adding translation traversal, the translation traversal of the first stage carries out rough traversal of a large step length by taking preset m pixels as step length, after the position offset when the coincidence degree reaches the maximum is obtained, the translation traversal of the second stage carries out precision traversal of a small step length by taking preset n pixels as step length, and the offset (x 2, y 2) when the coincidence degree of the infrared image edge map and the visible light image edge map is maximum is found in a peripheral preset range by taking (x 1, y 1) as a center, so as to be used as a final offset.
Further, m is 5n or more.
Further, the integral difference of pixel values is adopted in translation traversal to measure the coincidence degree of two edge graphs, specifically: after each translation, comparing pixel values of each pixel point of the visible light edge map and the translated infrared edge map one by one, wherein the difference of the statistical pixel values is smaller than the number of preset values, and the more the number is, the more similar the two edge images are.
The invention has the following beneficial effects:
the method for quickly registering the visible light and the infrared image based on the two-stage search provided by the invention eliminates the complex characteristic matching process, firstly utilizes a large step length to quickly search in the translation traversal process, then subdivides and searches in a small area, simplifies the registering process and improves the operation speed of an algorithm, in addition, adopts simple and efficient pixel value integral differences to measure the edge fitness, is superior to the traditional registering method in performance, has good effect in a simple scene in terms of applicability, and can be suitable for special complex scenes which are difficult to process.
Drawings
Fig. 1 is a schematic flow chart of a method for fast registering visible and infrared images according to an embodiment of the present invention;
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
As shown in fig. 1, as a first aspect of the present invention, there is provided a method for rapid registration of visible and infrared images, the method comprising:
step 1, inputting a visible light image and an infrared image, respectively graying the visible light image and the infrared image, respectively obtaining gray images of the visible light image and the infrared image, respectively extracting edge information of two gray images, and obtaining edge images of the two gray images, namely the visible light image edge image and the infrared image edge image.
Specifically, a visible light image and an infrared image photographed under the same scene at the same time are input, wherein the visible light image is a three-channel color image, and the infrared image is a single-channel image.
In order to facilitate processing and edge extraction, the visible light image is firstly subjected to gray scale processing, the input visible light image is in YUV format in the embodiment, wherein the Y channel represents brightness of the image, and the component of the Y channel is extracted to be the pixel value of the gray scale image.
Limited by the imaging principle of the camera, the photographed visible light and infrared images have slight differences in spatial positions, and the visible light and infrared images need to be registered to be fused further accurately, and in the embodiment, the default horizontal offset and the default vertical offset are both in a range of 40 pixels.
Specifically, image edges are extracted using Sobel operator
Specifically, the image edge refers to the place where the pixel value is in transition, namely the place where the change rate is the largest and the derivative is the largest, the image is imagined as a continuous function, and the pixel value of the edge part is obviously different from the pixel beside, so that the edge information of the whole image can be obtained by locally obtaining the extreme value of the image. However, the image is a two-dimensional discrete function and the derivative becomes a difference, which is called the gradient of the image.
The Sobel operator is a discrete differential operator that can be used to calculate the approximate gradient of the image gray, the greater the gradient the more likely the edge is. The function of the Soble operator integrates Gaussian smoothing and differential derivation, which is also called a first-order differential operator, and derives in the horizontal direction and the vertical direction, and the obtained gradient images of the image in the x direction and the y direction are obtained respectively.
The operator expands the difference through the weight, the Sobel operator utilizes two convolution kernels with weight to perform gradient calculation, a horizontal gradient map and a vertical gradient map of a gray level map of a visible light image or an infrared image are respectively Gx and Gy, and the I is respectively convolved with two convolution kernels with odd sizes to obtain Gx and Gy:
wherein I is an element value matrix of the gray image, the convolution operation is expressed, f (x, y) is set as the pixel value of a pixel point with coordinates (x, y) in the element value matrix, and the value is 0-255, G is x The convolution results of (x, y) are calculated as follows:
calculation of G by the same method y (x, y) in obtaining G x (x, y) and G y After (x, y), the approximate gradient of the (x, y) point is calculated as:
calculating the approximate gradient of each pixel point based on the formula, so as to obtain the edge information of the whole gray scale image;
the Sobel operator detects the edge according to the gray weighting difference of the adjacent points up and down and left and right of the pixel point, and the phenomenon that the extreme value is reached at the edge, has a smoothing effect on noise, provides more accurate edge direction information, and is a more common edge detection method.
Because the visible light and infrared images show different picture details, only the edge information of the objects in the pictures is unified, and therefore the gray-level images are further extracted to register.
Step 2, performing first-stage translation traversal on the infrared image edge map, calculating the coincidence degree of the infrared image edge map and the visible light image edge map in the current state after each translation until the first-stage translation traversal is finished, and recording the position offset (x 1, y 1) when the coincidence degree reaches the maximum;
and 3, carrying out translation traversal of a second stage in a preset range around the position offset obtained by translation traversal of the first stage serving as a reference, finding the offset when the coincidence degree of the infrared image edge map and the visible light image edge map in the range is maximum, and calculating the coincidence degree of the infrared image edge map and the visible light image edge map in the current state after each translation traversal in the translation traversal of the second stage, so as to obtain the offset when the coincidence degree of the infrared image edge map and the visible light image edge map in the preset range is maximum as a final registration result.
Specifically, after obtaining the edge map of the visible light and infrared images, formally entering a registration process, and finding out the optimal offset through two layers of traversal cycles, so that the coincidence degree of the visible light and infrared edge images is highest. For the two edge images obtained in the last step, the most matching of the edges is to make the two images most similar, so that the calculation is simple and efficient, the time consumption of the multi-time traversal process is reduced, and the similarity of the two edge images in the traversal process is measured by adopting the index of pixel value difference in the embodiment.
In one traversal, all pixel points of the visible light and infrared edge graph are successively compared, the number of points with the difference value smaller than 20 corresponding to the pixel values is counted, the proportion of the points accounting for all the pixel points is used as an index for judging the similarity, and a similarity Sim calculation formula is as follows:
in the above formula, width and Height respectively represent the Width and Height of the image, num represents the number of pixels with the difference of the pixel values of the corresponding points smaller than 20 in the two edge images, and the larger the Sim value is, the higher the similarity of the two images is, and the higher the edge matching degree is.
Specifically, the translation traversal is divided into two phases.
The method comprises the steps of firstly setting two variables to respectively represent offset values of x and y in translation traversal in a first stage, gradually taking m pixels as step length to take values from-40 to 40 in a first process, wherein m is preferably 5, traversing by adopting a double-layer cycle, carrying out translation transformation on an infrared edge image according to current cycle variables x0 and y0 for any cycle, then calculating similarity Sim values of the translated image and a visible light edge image, and recording offset (x 1 and y 1) when Sim is maximum after traversing, wherein the step length of the first process is 5, so that rough estimated offset is obtained at the moment;
the translation traversal of the second stage takes the offset of (x 1, y 1) as a reference, and carries out translation traversal again by taking m pixels as step length within the range of 5 pixels around the offset, wherein m is preferably 1, the similarity is calculated, and finally an accurate offset is obtained; specifically, taking X1 and Y1 as centers, enabling two secondary offset value variables at the stage to be from-10 to 10, taking 1 pixel as a step length to gradually take values, similarly calculating similarity Sim values of the image and the visible light edge image after each translation, and recording global offset (X2 and Y2) when Sim is maximum as a final registration result after traversing.
The infrared original image is aligned with the visible light image by using the space offset (x 2, y 2) obtained by two-stage traversal to carry out translation transformation, thus completing the registration process and preparing for image fusion.
As a second embodiment of the present invention, there is provided a device for rapid registration of visible and infrared images, the device including an image input module, a graying processing module, an edge image extraction module, a first stage traversal module, a second stage traversal module, and a registration module;
the image input module is used for inputting visible light images and infrared images;
the graying processing module is used for respectively graying the visible light image and the infrared image to obtain gray images of the visible light image and the infrared image;
the edge image extraction module is used for respectively extracting edge information of two gray images, and acquiring edge images of the two gray images, namely a visible light image edge image and an infrared image edge image;
the first-stage traversing module is used for carrying out first-stage translation traversal on the infrared image edge map, calculating the coincidence degree of the infrared image edge map and the visible light image edge map in the current state after each translation until the translation traversal of the first stage is finished, and recording the offset (x 1, y 1) when the coincidence degree reaches the maximum;
the second-stage traversing module is used for carrying out second-stage translational traversing in a preset range around the first-stage translational traversing module by taking the position offset obtained by the first-stage translational traversing as a reference, and finding the offset when the coincidence degree of the infrared image edge map and the visible image edge map in the range is maximum;
the registering module is used for performing translation transformation on the infrared image by using the offset obtained by the second-stage traversing module, and aligning the infrared image with the visible light image to finish image registering.
Preferably, the edge image extracting module respectively extracts edge information of two gray images by using a Sobel operator, specifically:
let the element value matrix of the gray level image of the visible light image or the infrared image be I;
and (3) respectively carrying out convolution on the I and two convolution kernels with odd sizes to calculate a horizontal gradient graph Gx and a vertical gradient graph Gy of the gray level graph, wherein the specific formulas are as follows:
wherein I is an element value matrix of the gray image, and represents convolution operation, and f (x, y) is set as a pixel value of a pixel point with coordinates (x, y) in the element value matrix, and the value is 0-255, and then the convolution result of the (x, y) point is specifically calculated as follows:
G x (x，y)＝(-1)*f(x-1，y-1)+0*f(x，y-1)+1*f(x+1，y+1)
+(-2)*f(x-1，y)+0*f(x，y)+2*f(x+1，y)
+(-1)*f(x-1，y+1)+0*f(x，y+1)+1*f(x+1，y+1)；
calculation of G by the same method y (x, y) in obtaining G x (x, y) and G y After (x, y), the approximate gradient of the (x, y) point is calculated as:
and calculating the approximate gradient of each pixel point based on the formula, thereby obtaining the edge information of the whole gray scale image.
Preferably, for the time of the block-adding translation traversal, the translation traversal of the first stage performs rough traversal of a large step length by taking preset m pixels as step length, and after the position offset when the coincidence degree reaches the maximum is obtained, the translation traversal of the second stage performs precision traversal of a small step length by taking preset n pixels as step length, and finds the offset (x 2, y 2) when the coincidence degree of the infrared image edge map and the visible light image edge map is maximum in a peripheral preset range by taking (x 1, y 1) as a center, and the offset is used as a final offset.
Preferably, m is 5n or more.
Preferably, the integral difference of pixel values is adopted in the translation traversal to measure the coincidence degree of two edge graphs, specifically: after each translation, comparing pixel values of each pixel point of the visible light edge map and the translated infrared edge map one by one, wherein the difference of the statistical pixel values is smaller than the number of preset values, and the more the number is, the more similar the two edge images are.
The foregoing description of the preferred embodiments of the invention is not intended to limit the invention to the precise form disclosed, and any such modifications, equivalents, and alternatives falling within the spirit and scope of the invention are intended to be included within the scope of the invention.
Claims (10)
1. A method for rapid registration of visible and infrared images, the method comprising:
step 1, inputting a visible light image and an infrared image, respectively graying the visible light image and the infrared image, respectively obtaining gray images of the visible light image and the infrared image, respectively extracting edge information of two gray images, and obtaining edge images of the two gray images, namely a visible light image edge image and an infrared image edge image;
step 2, performing first-stage translation traversal on the infrared image edge map, calculating the coincidence degree of the infrared image edge map and the visible light image edge map in the current state after each translation until the first-stage translation traversal is finished, and recording the offset (x 1, y 1) when the coincidence degree reaches the maximum;
step 3, performing translation traversal of a second stage in a preset range around the position offset obtained by translation traversal of the first stage serving as a reference, and finding the offset when the coincidence degree of the infrared image edge map and the visible light image edge map in the range is maximum;
and 4, performing translation transformation on the infrared image by using the offset obtained in the step 3, and aligning the infrared image with the visible light image to finish image registration.
2. The method for rapid registration of visible light and infrared images according to claim 1, wherein in step 1, edge information of two gray images is extracted by using Sobel operator respectively, specifically:
let the element value matrix of the gray level image of the visible light image or the infrared image be I;
and (3) respectively carrying out convolution on the I and two convolution kernels with odd sizes to calculate a horizontal gradient graph Gx and a vertical gradient graph Gy of the gray level graph, wherein the specific formulas are as follows:
wherein I is an element value matrix of the gray image, and represents convolution operation, and f (x, y) is set as a pixel value of a pixel point with coordinates (x, y) in the element value matrix, and the value is 0-255, and then the convolution result of the (x, y) point is specifically calculated as follows:
G x (x，y)＝(-1)*f(x-1，y-1)+0*f(x，y-1)+1*f(x+1，y+1)+(-2)*f(x-1，y)+0*f(x，y)+2*f(x+1，y)+(-1)*f(x-1，y+1)+0*f(x，y+1)+1*f(x+1，y+1)；
calculation of G by the same method y (x, y) in obtaining G x (x, y) and G y After (x, y), the approximate gradient of the (x, y) point is calculated as:
and calculating the approximate gradient of each pixel point based on the formula, thereby obtaining the edge information of the whole gray scale image.
3. The method according to claim 1, wherein, to accelerate the time of the translation traversal, the first stage of the translation traversal uses m preset pixels as a step length to perform a rough traversal with a large step length, and after the position offset when the coincidence degree reaches the maximum is obtained, the second stage of the translation traversal uses n preset pixels as a step length to perform a precision traversal with a small step length, and uses (x 1, y 1) as a center, to find the offset (x 2, y 2) when the coincidence degree of the infrared image edge map and the visible image edge map is the maximum in a surrounding preset range, as a final offset.
4. A method of rapid registration of visible and infrared images according to claim 3 wherein m is 5n or more.
5. The method for rapid registration of visible and infrared images according to claim 1, wherein the overall difference of pixel values is used in the translation traversal to measure the coincidence degree of two edge maps, specifically: after each translation, comparing pixel values of each pixel point of the visible light edge map and the translated infrared edge map one by one, wherein the difference of the statistical pixel values is smaller than the number of preset values, and the more the number is, the more similar the two edge images are.
6. The device is characterized by comprising an image input module, a graying processing module, an edge image extraction module, a first-stage traversing module, a second-stage traversing module and a registration module;
the image input module is used for inputting visible light images and infrared images;
the graying processing module is used for respectively graying the visible light image and the infrared image to obtain gray images of the visible light image and the infrared image;
the edge image extraction module is used for respectively extracting edge information of two gray images, and acquiring edge images of the two gray images, namely a visible light image edge image and an infrared image edge image;
the first-stage traversing module is used for carrying out first-stage translation traversal on the infrared image edge map, calculating the coincidence degree of the infrared image edge map and the visible light image edge map in the current state after each translation until the translation traversal of the first stage is finished, and recording the offset (x 1, y 1) when the coincidence degree reaches the maximum;
the second-stage traversing module is used for carrying out second-stage translational traversing in a preset range around the first-stage translational traversing module by taking the position offset obtained by the first-stage translational traversing as a reference, and finding the offset when the coincidence degree of the infrared image edge map and the visible image edge map in the range is maximum;
the registering module is used for performing translation transformation on the infrared image by using the offset obtained by the second-stage traversing module, and aligning the infrared image with the visible light image to finish image registering.
7. The device for rapid registration of visible and infrared images according to claim 5, wherein the edge image extraction module extracts edge information of two gray images by using Sobel operators respectively, specifically:
let the element value matrix of the gray level image of the visible light image or the infrared image be I;
and (3) respectively carrying out convolution on the I and two convolution kernels with odd sizes to calculate a horizontal gradient graph Gx and a vertical gradient graph Gy of the gray level graph, wherein the specific formulas are as follows:
wherein I is an element value matrix of the gray image, and represents convolution operation, and f (x, y) is set as a pixel value of a pixel point with coordinates (x, y) in the element value matrix, and the value is 0-255, and then the convolution result of the (x, y) point is specifically calculated as follows:
calculation of G by the same method y (x, y) in obtaining G x (x, y) and G y After (x, y), the approximate gradient of the (x, y) point is calculated as:
and calculating the approximate gradient of each pixel point based on the formula, thereby obtaining the edge information of the whole gray scale image.
8. The device according to claim 5, wherein for the time of the block-added translational traversal, the first translational traversal uses preset m pixels as step length to perform rough traversal with large step length, after the position offset when the coincidence degree reaches the maximum is obtained, the second translational traversal uses preset n pixels as step length to perform precision traversal with small step length, and uses (x 1, y 1) as a center to find the offset (x 2, y 2) when the coincidence degree of the infrared image edge map and the visible image edge map is maximum in the surrounding preset range as the final offset.
9. The device for rapid registration of visible and infrared images of claim 8, wherein m is 5n or greater.
10. The device for rapid registration of visible and infrared images according to claim 5, wherein the overall difference of pixel values is used in the translation traversal to measure the coincidence degree of two edge maps, specifically: after each translation, comparing pixel values of each pixel point of the visible light edge map and the translated infrared edge map one by one, wherein the difference of the statistical pixel values is smaller than the number of preset values, and the more the number is, the more similar the two edge images are.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010474030.1A CN111833384B (en) | 2020-05-29 | 2020-05-29 | Method and device for rapidly registering visible light and infrared images |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010474030.1A CN111833384B (en) | 2020-05-29 | 2020-05-29 | Method and device for rapidly registering visible light and infrared images |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111833384A CN111833384A (en) | 2020-10-27 |
| CN111833384B true CN111833384B (en) | 2023-12-26 |
Family
ID=72914049
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010474030.1A Active CN111833384B (en) | 2020-05-29 | 2020-05-29 | Method and device for rapidly registering visible light and infrared images |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111833384B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116246225A (en) * | 2023-05-12 | 2023-06-09 | 青岛农业大学 | Crop breeding monitoring method and system based on image processing |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101976434A (en) * | 2010-08-27 | 2011-02-16 | 浙江大学 | Frequency domain weighting correlation method for image registration |
| WO2014023231A1 (en) * | 2012-08-07 | 2014-02-13 | 泰邦泰平科技（北京）有限公司 | Wide-view-field ultrahigh-resolution optical imaging system and method |
| CN109146930A (en) * | 2018-09-20 | 2019-01-04 | 河海大学常州校区 | A kind of electric power calculator room equipment is infrared and visible light image registration method |
| CN110956640A (en) * | 2019-12-04 | 2020-04-03 | 国网上海市电力公司 | Heterogeneous image edge point detection and registration method |
-
2020
- 2020-05-29 CN CN202010474030.1A patent/CN111833384B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101976434A (en) * | 2010-08-27 | 2011-02-16 | 浙江大学 | Frequency domain weighting correlation method for image registration |
| WO2014023231A1 (en) * | 2012-08-07 | 2014-02-13 | 泰邦泰平科技（北京）有限公司 | Wide-view-field ultrahigh-resolution optical imaging system and method |
| CN109146930A (en) * | 2018-09-20 | 2019-01-04 | 河海大学常州校区 | A kind of electric power calculator room equipment is infrared and visible light image registration method |
| CN110956640A (en) * | 2019-12-04 | 2020-04-03 | 国网上海市电力公司 | Heterogeneous image edge point detection and registration method |
Non-Patent Citations (2)
| Title |
|---|
| 一种从粗到精的红外和可见光卫星图像配准方法;胡永利;王亮;刘蓉;张丽;段福庆;;光谱学与光谱分析(第11期);全文 * |
| 基于特征点的红外与可见光图像配准研究;彭逸月;何伟基;顾国华;刘晖;张连东;陈钱;;激光与红外(第06期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111833384A (en) | 2020-10-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111209810B (en) | Boundary frame segmentation supervision deep neural network architecture for accurately detecting pedestrians in real time through visible light and infrared images | |
| US10462362B2 (en) | Feature based high resolution motion estimation from low resolution images captured using an array source | |
| EP3499414B1 (en) | Lightweight 3d vision camera with intelligent segmentation engine for machine vision and auto identification | |
| CN111462128B (en) | Pixel-level image segmentation system and method based on multi-mode spectrum image | |
| CN111079556A (en) | Multi-temporal unmanned aerial vehicle video image change area detection and classification method | |
| US11398053B2 (en) | Multispectral camera external parameter self-calibration algorithm based on edge features | |
| CN110956661B (en) | Method for calculating dynamic pose of visible light and infrared camera based on bidirectional homography matrix | |
| CN104601964A (en) | Non-overlap vision field trans-camera indoor pedestrian target tracking method and non-overlap vision field trans-camera indoor pedestrian target tracking system | |
| CN109376641B (en) | Moving vehicle detection method based on unmanned aerial vehicle aerial video | |
| CN111965636A (en) | Night target detection method based on millimeter wave radar and vision fusion | |
| CN112164117A (en) | V-SLAM pose estimation method based on Kinect camera | |
| CN109544635B (en) | Camera automatic calibration method based on enumeration heuristic | |
| CN112016478B (en) | Complex scene recognition method and system based on multispectral image fusion | |
| CN110569861A (en) | Image matching positioning method based on point feature and contour feature fusion | |
| CN111738033B (en) | Vehicle driving information determination method and device based on plane segmentation and vehicle-mounted terminal | |
| CN111080709A (en) | Multispectral stereo camera self-calibration algorithm based on track feature registration | |
| CN112946679B (en) | Unmanned aerial vehicle mapping jelly effect detection method and system based on artificial intelligence | |
| CN114549871A (en) | Unmanned aerial vehicle aerial image and satellite image matching method | |
| CN107045630B (en) | RGBD-based pedestrian detection and identity recognition method and system | |
| CN111833384B (en) | Method and device for rapidly registering visible light and infrared images | |
| Wu et al. | Study of multiple moving targets’ detection in fisheye video based on the moving blob model | |
| CN113096016A (en) | Low-altitude aerial image splicing method and system | |
| CN111583315A (en) | Novel visible light image and infrared image registration method and device | |
| CN113642430B (en) | VGG+ NetVLAD-based high-precision visual positioning method and system for underground parking garage | |
| CN110688876A (en) | Lane line detection method and device based on vision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address |
Address after: 430073 No. 006, 20th floor, business project (China Pharmaceutical Technology Trading Market), No. 1, xiyaojian Road, north of Gaoxin Avenue and Heying Road, East Lake New Technology Development Zone, Wuhan City, Hubei Province Patentee after: Wuhan Zhuomu Technology Co.,Ltd. Country or region after: China Address before: A2-32-05, 2nd floor, Guannan science and Technology Industrial Park, Guandong street, Donghu New Technology Development Zone, Wuhan, Hubei Province Patentee before: WUHAN ZMVISION TECHNOLOGY Co.,Ltd. Country or region before: China |
|
| CP03 | Change of name, title or address |