CN112132753A - Infrared image super-resolution method and system for multi-scale structure guide image - Google Patents
Infrared image super-resolution method and system for multi-scale structure guide image Download PDFInfo
- Publication number
- CN112132753A CN112132753A CN202011226916.0A CN202011226916A CN112132753A CN 112132753 A CN112132753 A CN 112132753A CN 202011226916 A CN202011226916 A CN 202011226916A CN 112132753 A CN112132753 A CN 112132753A
- Authority
- CN
- China
- Prior art keywords
- image
- infrared
- sampling
- resolution
- vis
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000005070 sampling Methods 0.000 claims abstract description 63
- 238000001914 filtration Methods 0.000 claims abstract description 40
- 230000002146 bilateral effect Effects 0.000 claims abstract description 7
- 238000003672 processing method Methods 0.000 claims abstract description 5
- 238000004590 computer program Methods 0.000 claims description 12
- 239000011159 matrix material Substances 0.000 claims description 11
- 238000003384 imaging method Methods 0.000 claims description 9
- 230000006870 function Effects 0.000 claims description 7
- 238000003860 storage Methods 0.000 claims description 7
- 125000004432 carbon atom Chemical group C* 0.000 claims description 3
- 238000001514 detection method Methods 0.000 claims description 3
- 238000005315 distribution function Methods 0.000 claims description 3
- 230000009191 jumping Effects 0.000 claims description 3
- 230000000007 visual effect Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000005855 radiation Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 238000001931 thermography Methods 0.000 description 2
- 230000032683 aging Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012634 optical imaging Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
-
- G06T5/70—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20024—Filtering details
- G06T2207/20028—Bilateral filtering
Abstract
The invention discloses an infrared image super-resolution method and system for a multi-scale structure guide imageI nir And visible light imagesI vis Registering to obtain registered infrared imageI nir‑reg (ii) a Image processing methodI vis By passingnObtaining multiple down-sampling scales by down-samplingiVisible light image ofI i vis An image is formedI nir‑reg Down-sampling to obtain an imageI nir‑ds (ii) a Image processing methodI i vis Converting into HSV color space, and setting colorThe color threshold T will perform target classification; then the image is takenI nir‑ds As the initial current image to be filtered, the method proceedsnMultiple down-sampling scales of the sub-iterationiVisible light image ofI i vis And performing combined bilateral filtering as a guide to obtain an infrared image super-resolution image. The invention can effectively improve the resolution of the infrared image by guiding the image through the visible light, improves the visual effect and has higher practical application value.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to an infrared image super-resolution method and system for a multi-scale structure guide image.
Background
Infrared thermal imaging has its unique advantages over traditional imaging modalities. The intelligent temperature sensor is strong in anti-interference capability, less affected by environment, sensitive to thermal radiation change and capable of effectively acquiring temperature information in a scene. The infrared thermal imaging technology mainly receives thermal radiation energy from a target through an infrared detector and an optical imaging objective lens, and reflects the thermal radiation energy onto a photosensitive element of the infrared detector, so that a camera can acquire temperature information of different targets at the same time. With the aging of the imaging sensor technology, the cost is gradually reduced, so that the application range of the imaging sensor is wider, and the imaging sensor has good application value in multiple fields of transportation, construction, safety and the like. However, the density of most infrared detector arrays is low, which results in low imaging resolution and difficulty in meeting the requirement of people for high-resolution infrared images. If the high-resolution infrared image is acquired by improving the hardware, the cost of the camera can be greatly increased, the infrared image is subjected to super-resolution by utilizing the algorithm, the resolution of the infrared image can be effectively improved under the condition that the cost of the camera hardware is not increased, and the requirement for acquiring the high-resolution infrared image is met.
Currently, super-resolution techniques can be divided into two categories, one is a learning-based method and the other is a reconstruction-based method. The learning-based method needs a large number of high-resolution images to construct a learning library to learn a model, a mapping relation between a low-resolution image and a corresponding high-resolution image is searched or established by means of pre-training and learning, and high-frequency information is extracted, so that under the condition of giving the low-resolution image, the corresponding high-resolution image is obtained by an optimization method; whereas reconstruction-based methods predict a high-resolution image from a single or multiple low-resolution images. The learning-based method has large dependence on data, higher requirement on hardware and more obvious limitation. Considering that the visible light image and the infrared image in the same scene have a certain corresponding relation, the low-resolution infrared image super-resolution of the guide image of the visible light image in the same scene is beneficial to restoring high-frequency information of the low-resolution infrared image, and the requirement for the infrared image super-resolution is realized.
Disclosure of Invention
The technical problems to be solved by the invention are as follows: aiming at the problems in the prior art, the invention provides the infrared image super-resolution method and the infrared image super-resolution system for the multi-scale structure guide image.
In order to solve the technical problems, the invention adopts the technical scheme that:
a super-resolution method for infrared images of multi-scale structure guide images comprises the following steps:
1) will infrared imageI nir And visible light images of the same sceneI vis Registering to obtain registered infrared imageI nir-reg ；
2) Imaging visible lightI vis By passingnSub-down sampling to obtain multiple down-sampling scalesiVisible light image ofI i vis The registered infrared image is processedI nir-reg Down-sampling to obtain an imageI nir-ds The imageI nir-ds And minimum down-sampling scalei min Visible light image ofI i-min vis Are equal in size;
3) respectively down-sampling multiple scalesiVisible light image ofI i vis Converting into HSV color space, setting color threshold T and down sampling in different sizesiVisible light image ofI i vis Classifying the target in (1), and storing the targets of different classes respectivelycAt a plurality of down-sampling scalesiVisible light image ofI i vis Index information in (1)d i c Designing a corresponding filtering kernel; image processing methodI nir-ds As the initial current image to be filtered, initializing the sampling scalekIs 1;
4) aiming at the current image to be filtered, selecting a designed filtering kernel according to the target type, and then carrying out down-sampling on multiple scalesiVisible light image ofI i vis As a guide, respectively performing joint bilateral filtering according to the index informationd i c Extracting pixels of different guide images under the corresponding coordinates of the image after filtering the current image to be filtered, and replacing the pixels of the corresponding positions on the original current image to be filtered before filtering, thereby generating a sampling scalekLower infrared super-resolution imageI i SR ；
5) Judging sampling scalekIs equal to the number of downsampling timesnIf not, sampling scale is determinedkLower infrared super-resolution imageI i SR Up-sampling to a sampling scalek+1Visible light image ofI i vis The size of the image is taken as a new current image to be filtered, and the sampling scale is adjustedkAdding 1, and jumping to execute the step 4); otherwise, the finally obtained sampling scalekLower infrared super-resolution imageI i SR And outputting as a result.
Optionally, the infrared image is processed in step 1)I nir And visible light images of the same sceneI vis Registration refers to the registration of infrared imagesI nir Multiplication by the registration matrixHObtaining the registered infrared imageI nir-reg 。
Optionally, step 1) is preceded by generating a registration matrixHThe steps of (1): will infrared imageI nir And visible light images of the same sceneI vis Respectively divided into image blocks with specified sizes, and then each image block is subjected to feature detection according to the infrared imageI nir And visible light images of the same sceneI vis Calculating the proportional relation between the characteristic points to obtain a registration matrixH。
Optionally, the visible light image is processed in step 2)I vis By passingnThe sub-down sampling is performed 2 times, and an image obtained by each down sampling is reduced to 1/2 of the original image.
Optionally, the filter kernel in step 3) is a gaussian filter kernel, and the designed parameters of the gaussian filter kernel include the size of the gaussian filter kernel, a spatial standard deviation of the filter kernel to the target, and a standard deviation of a gaussian range.
Optionally, when the corresponding filter kernel is designed in step 3), the target includes vegetation and non-vegetation, and the gaussian filter kernel parameter designed for vegetation is: the size of the filtering kernel is 3 x 3, the spatial standard deviation of the vegetation filtering kernel is 3, and the standard deviation of the Gaussian range is 0.03; the gaussian filter kernel parameters designed for non-vegetation are: the filter kernel size is 3 x 3, the spatial standard deviation of the non-vegetation filter kernel is 10, and the gaussian range standard deviation is 0.03.
Optionally, the function expression of the joint bilateral filtering performed in step 4) is as follows:
in the above formula, the first and second carbon atoms are,J p representing target pixel locationpThe output of the (c) is,W p represents the weight coefficient of the neighborhood pixels, omega represents the group of pixels around the target pixel,I q representing the position of a target pixel in a current image to be filteredpSurrounding pixel locationqThe number of pixels of (a) is,f,grespectively, are gaussian weight distribution functions,pin order to be the target pixel position,qis the target pixel positionpOne pixel position of the surroundings is,I guide-p for guiding the position of target pixel in imagepThe number of pixels of (a) is,I guide-q for guiding the position of target pixel in imagepSurrounding pixel locationqThe pixel of (2).
In addition, the invention also provides an infrared image super-resolution system of the multi-scale structure guide image, which comprises a microprocessor and a memory which are connected with each other, wherein the microprocessor is programmed or configured to execute the steps of the infrared image super-resolution method of the multi-scale structure guide image.
In addition, the invention also provides an infrared image super-resolution system of the multi-scale structure guide image, which comprises a microprocessor and a memory which are connected with each other, wherein the memory is stored with a computer program which is programmed or configured to execute the infrared image super-resolution method of the multi-scale structure guide image.
Furthermore, the present invention also provides a computer-readable storage medium having stored therein a computer program programmed or configured to execute the infrared image super-resolution method of the multi-scale structure guide image.
Compared with the prior art, the invention has the following advantages: the method can optimize the infrared image according to the characteristics of low resolution, low contrast, imaging blur and the like. In consideration of the fact that the high-resolution visible light image has rich details and edge texture information, the method adopts a multi-scale structure guiding mode and introduces the self-adaptive filtering kernel, and meets the requirement of realizing self-adaptive filtering on different characteristic targets in the scene. The guiding is carried out in a multi-scale mode, the similarity between the guiding image and the original image can be effectively utilized, the infrared image resolution is gradually improved in a layer-by-layer progressive mode, the introduced self-adaptive filtering kernel can carry out targeted processing according to different characteristics of targets in a scene, the loss of edge textures and detail information of the targets is reduced, the visual effect of the infrared image is effectively improved, the image definition is enhanced, and the infrared image resolution is remarkably improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application and are not intended to limit the application.
FIG. 1 is a basic flow diagram of a method according to an embodiment of the present invention.
Fig. 2 is a flow chart of multi-scale guided filtering in an embodiment of the present invention.
FIG. 3 is an infrared image inputted in an embodiment of the present inventionI nir
FIG. 4 is a visible light image inputted in an embodiment of the present inventionI vis
Fig. 5 is an infrared super-resolution image output in the embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described and explained in detail below with reference to flowcharts and embodiments, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As shown in fig. 1 and fig. 2, the infrared image super-resolution method for a multi-scale structure guide image of the present embodiment includes:
1) will infrared imageI nir And visible light images of the same sceneI vis Registering to obtain registered infrared imageI nir-reg ；
2) Imaging visible lightI vis By passingnSub-down sampling to obtain multiple down-sampling scalesiVisible light image ofI i vis The registered infrared image is processedI nir-reg Down-sampling to obtain an imageI nir-ds The imageI nir-ds And minimum down-sampling scalei min Visible light image ofI i-min vis Are equal in size;
3) respectively down-sampling multiple scalesiVisible light image ofI i vis Converting into HSV color space, setting color threshold T and down sampling in different sizesiVisible light image ofI i vis Classifying the target in (1), and storing the targets of different classes respectivelycAt a plurality of down-sampling scalesiVisible light image ofI i vis Index information in (1)d i c Designing a corresponding filtering kernel; image processing methodI nir-ds As the initial current image to be filtered, initializing the sampling scalekIs 1;
4) aiming at the current image to be filtered, selecting a designed filtering kernel according to the target type, and then carrying out down-sampling on multiple scalesiVisible light image ofI i vis As a guide, respectively performing joint bilateral filtering according to the index informationd i c Extracting pixels of different guide images under the corresponding coordinates of the image after filtering the current image to be filtered, and replacing the pixels of the corresponding positions on the original current image to be filtered before filtering, thereby generating a sampling scalekLower infrared super-resolution imageI i SR ；
5) Judging sampling scalekIs equal to the number of downsampling timesnIf not, sampling scale is determinedkLower infrared super-resolution imageI i SR Up-sampling to a sampling scalek+1Visible light image ofI i vis The size of the image is taken as a new current image to be filtered, and the sampling scale is adjustedkAdding 1, and jumping to execute the step 4); otherwise, the finally obtained sampling scalekLower infrared super-resolution imageI i SR And outputting as a result.
Infrared image in this exampleI nir And visible light images of the same sceneI vis Can be obtained by using the 2-double-light-version unmanned aerial vehicle in the world, wherein infrared imagesI nir 640 x 480 x 3 size, visible light image of the same sceneI vis Size 4056 3040 3.
In this embodiment, the infrared image is obtained in step 1)I nir And visible light images of the same sceneI vis The registration refers to registration by means of block registration, namely: will infrared imageI nir Multiplication by the registration matrixHObtaining the registered infrared imageI nir-reg . In this embodiment, the relative positions of the two modal cameras on the unmanned aerial vehicle are fixed, so that only the corresponding registration matrices at different heights need to be acquiredHI.e. the infrared and visible images acquired at different heights can be registered.
Considering that the infrared visible light image has a large resolution difference and the infrared image is blurred, the registration is difficult to be realized directly by selecting the feature points, and therefore, in the embodiment, the step 1) includes generating the registration matrix beforeHThe steps of (1): will infrared imageI nir And visible light images of the same sceneI vis Respectively divided into image blocks of specified size (in this embodiment, the image is divided into image blocks of 4 × 4 uniform size), and feature detection is performed on each image block, according to the infrared imageI nir And visible light images of the same sceneI vis Calculating the proportional relation between the characteristic points to obtain a registration matrixH。
Will infrared imageI nir Multiplication by the registration matrixHObtaining registered infrared imageI nir-reg The functional expression of (a) is:
in the above formula (1)x 1,y 1,1)TRepresenting visible light imagesI vis Pixel point of (1), (b)x 2,y 2,1)TRepresenting infrared imagesI nir The pixel point in (2). According to the registration matrixHThe infrared image can be displayedI nir Transforming visible light imagesI vis Obtaining registered infrared imageI nir-reg 。
This example converts visible light image in step 2)I vis By passingnObtaining multiple down-sampling scales by down-samplingiVisible light image ofI i vis Whereini=1,2,3,…,n+1 is a visible imageI vis The scale information of (a) is obtained,nis the number of downsamplings. This example converts visible light image in step 2)I vis By passingnThe sub-down sampling is performed 2 times, and an image obtained by each down sampling is reduced to 1/2 of the original image.
In the embodiment, in the step 3), a plurality of down-sampling scales are adoptediVisible light image ofI i vis After converting to HSV color space, multiple down-sampling scales are obtained by setting color threshold TiVisible light image ofI i vis Classifying the target in (1), and storing the targets of different classes respectivelycAt a plurality of down-sampling scalesiVisible light image ofI i vis Index information in (1)d i c Designing a corresponding filtering kernel; different filtering parameters are used according to the richness of different kinds of target edge texture information, and the problem that part of target edge texture information is lost due to the fact that single filtering kernel filtering is used for different targets is solved.
In this embodiment, the filter kernel in step 3) is a gaussian filter kernel, and the designed parameters of the gaussian filter kernel include the size of the gaussian filter kernel, a spatial standard deviation of the filter kernel to the target, and a standard deviation of a gaussian range.
As an alternative embodiment, the HSV color space corresponds to a pixel range of
0.1176<H<0.3137，S>0.1569，V>0.1569
The image target is divided into vegetation and non-vegetation by setting the color threshold T, the spatial standard deviation of the vegetation during filtering is properly reduced, and the influence of distant pixels on central pixels is reduced, so that the loss of vegetation temperature information is reduced. Specifically, when the corresponding filter kernel is designed in step 3) of this embodiment, the target includes vegetation and non-vegetation, and the gaussian filter kernel parameter designed for vegetation is: the size of the filtering kernel is 3 x 3, the spatial standard deviation of the vegetation filtering kernel is 3, and the standard deviation of the Gaussian range is 0.03; the gaussian filter kernel parameters designed for non-vegetation are: the filter kernel size is 3 x 3, the spatial standard deviation of the non-vegetation filter kernel is 10, and the gaussian range standard deviation is 0.03.
In this embodiment, the function expression of the joint bilateral filtering performed in step 4) is shown as follows:
in the above formula, the first and second carbon atoms are,J p representing target pixel locationpThe output of the (c) is,W p represents the weight coefficient of the neighborhood pixels, omega represents the group of pixels around the target pixel,I q representing the position of a target pixel in a current image to be filteredpSurrounding pixel locationqThe number of pixels of (a) is,f,grespectively, are gaussian weight distribution functions,pin order to be the target pixel position,qis the target pixel positionpOne pixel position of the surroundings is,I guide-p for guiding the position of target pixel in imagepThe number of pixels of (a) is,I guide-q for guiding the position of target pixel in imagepSurrounding pixel locationqThe pixel of (2).
In this embodiment, an infrared image is inputI nir As shown in fig. 3, the input visible light image of the same sceneI vis As shown in fig. 4, is finally performed by iterationn+The final result (infrared super-resolution image) obtained after step 4) is 1 time (specifically, 3 times in this embodiment) is shown in fig. 5.
In summary, in the infrared image super-resolution method for the multi-scale structure guide image of the embodiment, the visible light and the infrared image are firstly acquired and registered; secondly, downsampling the visible light image for multiple times to obtain visible light images under different scales, and downsampling the infrared image to be the same as the size of the visible light minimum scale image; the self-adaptive filtering kernel is designed, so that the problem that part of target edge texture information is lost due to filtering by using a single filtering kernel is solved; and finally, performing multi-scale guided filtering on the low-resolution infrared image by taking the visible light image under the same scale as a guide and combining a self-adaptive filtering kernel, wherein before filtering each time, the image to be filtered is up-sampled, the resolution of the image to be filtered is ensured to be consistent with that of the guide image, and after iterative filtering for many times, the infrared super-resolution image can be obtained. According to the infrared image super-resolution method for the multi-scale structure guide image, provided by the invention, the multi-scale structure information is used as a guide, and the self-adaptive filter kernel is introduced, so that the loss of the edge texture and the detail information of the target is reduced, the visual effect of the infrared image is effectively improved, the image definition is enhanced, and the infrared image resolution is remarkably improved.
In addition, the present embodiment also provides an infrared image super-resolution system of a multi-scale structure guide image, which includes a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the steps of the aforementioned infrared image super-resolution method of a multi-scale structure guide image.
In addition, the present embodiment also provides an infrared image super-resolution system of a multi-scale structure guide image, which includes a microprocessor and a memory connected to each other, wherein the memory stores a computer program programmed or configured to execute the aforementioned infrared image super-resolution method of the multi-scale structure guide image.
Further, the present embodiment also provides a computer-readable storage medium having stored therein a computer program programmed or configured to execute the aforementioned infrared image super-resolution method of multi-scale structure guide images.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein. The present application is directed to methods, apparatus (systems), and computer program products according to embodiments of the application wherein instructions, which execute via a flowchart and/or a processor of the computer program product, create means for implementing functions specified in the flowchart and/or block diagram block or blocks. These computer program instructions may also be stored in a computer-readable memory that can direct a graphics computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments, and all technical solutions belonging to the idea of the present invention belong to the protection scope of the present invention. It should be noted that modifications and embellishments within the scope of the invention may occur to those skilled in the art without departing from the principle of the invention, and are considered to be within the scope of the invention.
Claims (10)
1. A super-resolution method for infrared images of multi-scale structure guide images is characterized by comprising the following steps:
1) will infrared imageI nir And visible light images of the same sceneI vis Registering to obtain registered infrared imageI nir-reg ；
2) Imaging visible lightI vis By passingnSub-down sampling to obtain multiple down-sampling scalesiVisible light image ofI i vis The registered infrared image is processedI nir-reg Down-sampling to obtain an imageI nir-ds The imageI nir-ds And minimum down-sampling scalei min Visible light image ofI i-min vis Are equal in size;
3) respectively down-sampling multiple scalesiVisible light image ofI i vis Converting into HSV color space, setting color threshold T and down sampling in different sizesiVisible light image ofI i vis Classifying the target in (1), and storing the targets of different classes respectivelycAt a plurality of down-sampling scalesiVisible light image ofI i vis Index information in (1)d i c Designing a corresponding filtering kernel; image processing methodI nir-ds As the initial current image to be filtered, initializing the sampling scalekIs 1;
4) aiming at the current image to be filtered, selecting a designed filtering kernel according to the target type, and then carrying out down-sampling on multiple scalesiVisible light image ofI i vis As a guide, respectively performing joint bilateral filtering according to the index informationd i c Extracting different guide images to filter the current image to be filteredThe pixel of the later image under the corresponding coordinate replaces the pixel of the corresponding position on the original current image to be filtered before filtering, thereby generating the sampling scalekLower infrared super-resolution imageI i SR ；
5) Judging sampling scalekIs equal to the number of downsampling timesnIf not, sampling scale is determinedkLower infrared super-resolution imageI i SR Up-sampling to a sampling scalek+1Visible light image ofI i vis The size of the image is taken as a new current image to be filtered, and the sampling scale is adjustedkAdding 1, and jumping to execute the step 4); otherwise, the finally obtained sampling scalekLower infrared super-resolution imageI i SR And outputting as a result.
2. The method for super-resolution of infrared images of multi-scale structure-guided images according to claim 1, wherein the infrared images are processed in step 1)I nir And visible light images of the same sceneI vis Registration refers to the registration of infrared imagesI nir Multiplication by the registration matrixHObtaining the registered infrared imageI nir-reg 。
3. The method for super-resolution of infrared images of multi-scale structure-guided images according to claim 2, wherein step 1) is preceded by generating a registration matrixHThe steps of (1): will infrared imageI nir And visible light images of the same sceneI vis Respectively divided into image blocks with specified sizes, and then each image block is subjected to feature detection according to the infrared imageI nir And visible light images of the same sceneI vis Calculating the proportional relation between the characteristic points to obtain a registration matrixH。
4. Infrared image super-resolution of multi-scale structure-guided images of claim 1The method is characterized in that the visible light image is obtained in the step 2)I vis By passingnThe sub-down sampling is performed 2 times, and an image obtained by each down sampling is reduced to 1/2 of the original image.
5. The method for super-resolution of infrared images of multi-scale structure-guided images according to claim 1, wherein the filter kernel in step 3) is a gaussian filter kernel, and the designed parameters of the gaussian filter kernel include the size of the gaussian filter kernel, the spatial standard deviation of the filter kernel to the target, and the standard deviation of the gaussian range.
6. The method for super-resolution of infrared images of multi-scale structure-guided images according to claim 5, wherein when corresponding filter kernels are designed in step 3), the target objects include vegetation and non-vegetation, and the Gaussian filter kernel parameters designed for vegetation are: the size of the filtering kernel is 3 x 3, the spatial standard deviation of the vegetation filtering kernel is 3, and the standard deviation of the Gaussian range is 0.03; the gaussian filter kernel parameters designed for non-vegetation are: the filter kernel size is 3 x 3, the spatial standard deviation of the non-vegetation filter kernel is 10, and the gaussian range standard deviation is 0.03.
7. The method for super-resolution of infrared images of multi-scale structure-guided images according to claim 5, wherein the function expression of the joint bilateral filtering in step 4) is as follows:
in the above formula, the first and second carbon atoms are,J p representing target pixel locationpThe output of the (c) is,W p represents the weight coefficient of the neighborhood pixel, and Ω represents the pixel group around the target pixel，I q Representing the position of a target pixel in a current image to be filteredpSurrounding pixel locationqThe number of pixels of (a) is,f,grespectively, are gaussian weight distribution functions,pin order to be the target pixel position,qis the target pixel positionpOne pixel position of the surroundings is,I guide-p for guiding the position of target pixel in imagepThe number of pixels of (a) is,I guide-q for guiding the position of target pixel in imagepSurrounding pixel locationqThe pixel of (2).
8. An infrared image super-resolution system for multi-scale structure-guided images, comprising a microprocessor and a memory connected to each other, characterized in that the microprocessor is programmed or configured to perform the steps of the infrared image super-resolution method for multi-scale structure-guided images according to any one of claims 1 to 7.
9. An infrared image super-resolution system for multi-scale structure guided images, comprising a microprocessor and a memory connected to each other, wherein the memory stores therein a computer program programmed or configured to perform the infrared image super-resolution method for multi-scale structure guided images according to any one of claims 1 to 7.
10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, which is programmed or configured to perform a method for super-resolution of infrared images of multi-scale structure-guided images according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011226916.0A CN112132753B (en) | 2020-11-06 | 2020-11-06 | Infrared image super-resolution method and system for multi-scale structure guide image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011226916.0A CN112132753B (en) | 2020-11-06 | 2020-11-06 | Infrared image super-resolution method and system for multi-scale structure guide image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112132753A true CN112132753A (en) | 2020-12-25 |
| CN112132753B CN112132753B (en) | 2022-04-05 |
Family
ID=73852507
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011226916.0A Active CN112132753B (en) | 2020-11-06 | 2020-11-06 | Infrared image super-resolution method and system for multi-scale structure guide image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112132753B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112862715A (en) * | 2021-02-08 | 2021-05-28 | 天津大学 | Real-time and controllable scale space filtering method |
| CN116071369A (en) * | 2022-12-13 | 2023-05-05 | 哈尔滨理工大学 | Infrared image processing method and device |
Citations (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103337077A (en) * | 2013-07-01 | 2013-10-02 | 武汉大学 | Registration method for visible light and infrared images based on multi-scale segmentation and SIFT (Scale Invariant Feature Transform) |
| WO2014168880A1 (en) * | 2013-04-12 | 2014-10-16 | Qualcomm Incorporated | Near infrared guided image denoising |
| CN104252704A (en) * | 2014-09-18 | 2014-12-31 | 四川大学 | Total generalized variation-based infrared image multi-sensor super-resolution reconstruction method |
| CN104809734A (en) * | 2015-05-11 | 2015-07-29 | 中国人民解放军总装备部军械技术研究所 | Infrared image and visible image fusion method based on guide filtering |
| CN105761214A (en) * | 2016-01-14 | 2016-07-13 | 西安电子科技大学 | Remote sensing image fusion method based on contourlet transform and guided filter |
| WO2017020595A1 (en) * | 2015-08-05 | 2017-02-09 | 武汉高德红外股份有限公司 | Visible light image and infrared image fusion processing system and fusion method |
| CN106600572A (en) * | 2016-12-12 | 2017-04-26 | 长春理工大学 | Adaptive low-illumination visible image and infrared image fusion method |
| CN107169944A (en) * | 2017-04-21 | 2017-09-15 | 北京理工大学 | A kind of infrared and visible light image fusion method based on multiscale contrast |
| CN107248150A (en) * | 2017-07-31 | 2017-10-13 | 杭州电子科技大学 | A kind of Multiscale image fusion methods extracted based on Steerable filter marking area |
| CN107423709A (en) * | 2017-07-27 | 2017-12-01 | 苏州经贸职业技术学院 | A kind of object detection method for merging visible ray and far infrared |
| CN107578432A (en) * | 2017-08-16 | 2018-01-12 | 南京航空航天大学 | Merge visible ray and the target identification method of infrared two band images target signature |
| WO2018017904A1 (en) * | 2016-07-21 | 2018-01-25 | Flir Systems Ab | Fused image optimization systems and methods |
| WO2018076732A1 (en) * | 2016-10-31 | 2018-05-03 | 广州飒特红外股份有限公司 | Method and apparatus for merging infrared image and visible light image |
| US20180182068A1 (en) * | 2016-12-23 | 2018-06-28 | Signal Processing, Inc. | Method and System for Generating High Resolution Worldview-3 Images |
| WO2018120936A1 (en) * | 2016-12-27 | 2018-07-05 | Zhejiang Dahua Technology Co., Ltd. | Systems and methods for fusing infrared image and visible light image |
| CN108961180A (en) * | 2018-06-22 | 2018-12-07 | 理光软件研究所（北京）有限公司 | infrared image enhancing method and system |
| CN109035189A (en) * | 2018-07-17 | 2018-12-18 | 桂林电子科技大学 | Infrared and weakly visible light image fusion method based on Cauchy's ambiguity function |
| CN109242888A (en) * | 2018-09-03 | 2019-01-18 | 中国科学院光电技术研究所 | A kind of infrared and visible light image fusion method of combination saliency and non-down sampling contourlet transform |
| CN109447909A (en) * | 2018-09-30 | 2019-03-08 | 安徽四创电子股份有限公司 | The infrared and visible light image fusion method and system of view-based access control model conspicuousness |
| CN110111290A (en) * | 2019-05-07 | 2019-08-09 | 电子科技大学 | A kind of infrared and visible light image fusion method based on NSCT and structure tensor |
| CN110148104A (en) * | 2019-05-14 | 2019-08-20 | 西安电子科技大学 | Infrared and visible light image fusion method based on significance analysis and low-rank representation |
| CN110246108A (en) * | 2018-11-21 | 2019-09-17 | 浙江大华技术股份有限公司 | A kind of image processing method, device and computer readable storage medium |
| CN110490914A (en) * | 2019-07-29 | 2019-11-22 | 广东工业大学 | It is a kind of based on brightness adaptively and conspicuousness detect image interfusion method |
| CN110544205A (en) * | 2019-08-06 | 2019-12-06 | 西安电子科技大学 | Image super-resolution reconstruction method based on visible light and infrared cross input |
| CN111080724A (en) * | 2019-12-17 | 2020-04-28 | 大连理工大学 | Infrared and visible light fusion method |
| CN111667520A (en) * | 2020-06-09 | 2020-09-15 | 中国人民解放军63811部队 | Infrared image and visible light image registration method and device and readable storage medium |
| KR102161166B1 (en) * | 2019-03-27 | 2020-09-29 | 한화시스템 주식회사 | Method for image fusion and recording medium |
-
2020
- 2020-11-06 CN CN202011226916.0A patent/CN112132753B/en active Active
Patent Citations (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2014168880A1 (en) * | 2013-04-12 | 2014-10-16 | Qualcomm Incorporated | Near infrared guided image denoising |
| CN103337077A (en) * | 2013-07-01 | 2013-10-02 | 武汉大学 | Registration method for visible light and infrared images based on multi-scale segmentation and SIFT (Scale Invariant Feature Transform) |
| CN104252704A (en) * | 2014-09-18 | 2014-12-31 | 四川大学 | Total generalized variation-based infrared image multi-sensor super-resolution reconstruction method |
| CN104809734A (en) * | 2015-05-11 | 2015-07-29 | 中国人民解放军总装备部军械技术研究所 | Infrared image and visible image fusion method based on guide filtering |
| WO2017020595A1 (en) * | 2015-08-05 | 2017-02-09 | 武汉高德红外股份有限公司 | Visible light image and infrared image fusion processing system and fusion method |
| CN105761214A (en) * | 2016-01-14 | 2016-07-13 | 西安电子科技大学 | Remote sensing image fusion method based on contourlet transform and guided filter |
| WO2018017904A1 (en) * | 2016-07-21 | 2018-01-25 | Flir Systems Ab | Fused image optimization systems and methods |
| WO2018076732A1 (en) * | 2016-10-31 | 2018-05-03 | 广州飒特红外股份有限公司 | Method and apparatus for merging infrared image and visible light image |
| CN106600572A (en) * | 2016-12-12 | 2017-04-26 | 长春理工大学 | Adaptive low-illumination visible image and infrared image fusion method |
| US20180182068A1 (en) * | 2016-12-23 | 2018-06-28 | Signal Processing, Inc. | Method and System for Generating High Resolution Worldview-3 Images |
| WO2018120936A1 (en) * | 2016-12-27 | 2018-07-05 | Zhejiang Dahua Technology Co., Ltd. | Systems and methods for fusing infrared image and visible light image |
| CN107169944A (en) * | 2017-04-21 | 2017-09-15 | 北京理工大学 | A kind of infrared and visible light image fusion method based on multiscale contrast |
| CN107423709A (en) * | 2017-07-27 | 2017-12-01 | 苏州经贸职业技术学院 | A kind of object detection method for merging visible ray and far infrared |
| CN107248150A (en) * | 2017-07-31 | 2017-10-13 | 杭州电子科技大学 | A kind of Multiscale image fusion methods extracted based on Steerable filter marking area |
| CN107578432A (en) * | 2017-08-16 | 2018-01-12 | 南京航空航天大学 | Merge visible ray and the target identification method of infrared two band images target signature |
| CN108961180A (en) * | 2018-06-22 | 2018-12-07 | 理光软件研究所（北京）有限公司 | infrared image enhancing method and system |
| CN109035189A (en) * | 2018-07-17 | 2018-12-18 | 桂林电子科技大学 | Infrared and weakly visible light image fusion method based on Cauchy's ambiguity function |
| CN109242888A (en) * | 2018-09-03 | 2019-01-18 | 中国科学院光电技术研究所 | A kind of infrared and visible light image fusion method of combination saliency and non-down sampling contourlet transform |
| CN109447909A (en) * | 2018-09-30 | 2019-03-08 | 安徽四创电子股份有限公司 | The infrared and visible light image fusion method and system of view-based access control model conspicuousness |
| CN110246108A (en) * | 2018-11-21 | 2019-09-17 | 浙江大华技术股份有限公司 | A kind of image processing method, device and computer readable storage medium |
| KR102161166B1 (en) * | 2019-03-27 | 2020-09-29 | 한화시스템 주식회사 | Method for image fusion and recording medium |
| CN110111290A (en) * | 2019-05-07 | 2019-08-09 | 电子科技大学 | A kind of infrared and visible light image fusion method based on NSCT and structure tensor |
| CN110148104A (en) * | 2019-05-14 | 2019-08-20 | 西安电子科技大学 | Infrared and visible light image fusion method based on significance analysis and low-rank representation |
| CN110490914A (en) * | 2019-07-29 | 2019-11-22 | 广东工业大学 | It is a kind of based on brightness adaptively and conspicuousness detect image interfusion method |
| CN110544205A (en) * | 2019-08-06 | 2019-12-06 | 西安电子科技大学 | Image super-resolution reconstruction method based on visible light and infrared cross input |
| CN111080724A (en) * | 2019-12-17 | 2020-04-28 | 大连理工大学 | Infrared and visible light fusion method |
| CN111667520A (en) * | 2020-06-09 | 2020-09-15 | 中国人民解放军63811部队 | Infrared image and visible light image registration method and device and readable storage medium |
Non-Patent Citations (9)
| Title |
|---|
| 刘刚等: "非下采样轮廓波域红外与可见光图像配准算法", 《计算机科学》 * |
| 刘哲等: "基于引导滤波和多尺度局部自相似单幅红外图像超分辨率方法", 《计算机应用研究》 * |
| 吴一全等: "基于目标提取与引导滤波增强的红外与可见光图像融合", 《光学学报》 * |
| 苏冰山等: "一种基于多传感器的红外图像正则化超分辨率算法", 《光电子·激光》 * |
| 荣传振等: "增强融合图像视觉效果的图像融合方法", 《信号处理》 * |
| 钱钧等: "基于结构特征引导滤波的深度图像增强算法研究", 《应用光学》 * |
| 闫钧华等: "基于多尺度红外与可见光图像配准研究", 《激光与红外》 * |
| 陈峰等: "基于滚动引导滤波的红外与可见光图像融合算法", 《红外技术》 * |
| 陈继光等: "基于多传感器像素分类的红外图像超分辨率算法", 《科学技术创新》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112862715A (en) * | 2021-02-08 | 2021-05-28 | 天津大学 | Real-time and controllable scale space filtering method |
| CN116071369A (en) * | 2022-12-13 | 2023-05-05 | 哈尔滨理工大学 | Infrared image processing method and device |
| CN116071369B (en) * | 2022-12-13 | 2023-07-14 | 哈尔滨理工大学 | Infrared image processing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112132753B (en) | 2022-04-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109584248B (en) | Infrared target instance segmentation method based on feature fusion and dense connection network | |
| CN108229490B (en) | Key point detection method, neural network training method, device and electronic equipment | |
| CN108229468B (en) | Vehicle appearance feature recognition and vehicle retrieval method and device, storage medium and electronic equipment | |
| US10353271B2 (en) | Depth estimation method for monocular image based on multi-scale CNN and continuous CRF | |
| CN111259758B (en) | Two-stage remote sensing image target detection method for dense area | |
| CN109903331B (en) | Convolutional neural network target detection method based on RGB-D camera | |
| CN108427927B (en) | Object re-recognition method and apparatus, electronic device, program, and storage medium | |
| CN112446380A (en) | Image processing method and device | |
| CN111860398B (en) | Remote sensing image target detection method and system and terminal equipment | |
| CN109816694B (en) | Target tracking method and device and electronic equipment | |
| CN112132753B (en) | Infrared image super-resolution method and system for multi-scale structure guide image | |
| CN111383252B (en) | Multi-camera target tracking method, system, device and storage medium | |
| CN112581379A (en) | Image enhancement method and device | |
| CN116188999B (en) | Small target detection method based on visible light and infrared image data fusion | |
| KR102311796B1 (en) | Method and Apparatus for Deblurring of Human Motion using Localized Body Prior | |
| CN111127516A (en) | Target detection and tracking method and system without search box | |
| CN112906794A (en) | Target detection method, device, storage medium and terminal | |
| CN113689578A (en) | Human body data set generation method and device | |
| JP2022522375A (en) | Image collection control methods, devices, electronic devices, storage media and computer programs | |
| WO2023284255A1 (en) | Systems and methods for processing images | |
| CN113744142B (en) | Image restoration method, electronic device and storage medium | |
| CN114708615A (en) | Human body detection method based on image enhancement in low-illumination environment, electronic equipment and storage medium | |
| CN112013820B (en) | Real-time target detection method and device for deployment of airborne platform of unmanned aerial vehicle | |
| CN116958393A (en) | Incremental image rendering method and device | |
| CN117058183A (en) | Image processing method and device based on double cameras, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |