CN108399612B - Three-light image intelligent fusion method based on bilateral filtering pyramid - Google Patents
Three-light image intelligent fusion method based on bilateral filtering pyramid Download PDFInfo
- Publication number
- CN108399612B CN108399612B CN201810118085.1A CN201810118085A CN108399612B CN 108399612 B CN108399612 B CN 108399612B CN 201810118085 A CN201810118085 A CN 201810118085A CN 108399612 B CN108399612 B CN 108399612B
- Authority
- CN
- China
- Prior art keywords
- pyramid
- image
- layer
- bilateral filtering
- bilateral
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000002146 bilateral effect Effects 0.000 title claims abstract description 78
- 238000001914 filtration Methods 0.000 title claims abstract description 58
- 238000007500 overflow downdraw method Methods 0.000 title claims abstract description 18
- 230000004927 fusion Effects 0.000 claims abstract description 37
- 238000001228 spectrum Methods 0.000 claims abstract description 27
- 230000003595 spectral effect Effects 0.000 claims abstract description 18
- 238000004587 chromatography analysis Methods 0.000 claims abstract description 16
- 238000006073 displacement reaction Methods 0.000 claims abstract description 15
- 238000012937 correction Methods 0.000 claims abstract description 12
- 238000012545 processing Methods 0.000 claims abstract description 10
- 238000013461 design Methods 0.000 claims abstract description 7
- 230000003287 optical effect Effects 0.000 claims abstract description 6
- 238000000034 method Methods 0.000 claims description 33
- 230000008569 process Effects 0.000 claims description 21
- 238000004364 calculation method Methods 0.000 claims description 20
- 239000000758 substrate Substances 0.000 claims description 6
- 238000005070 sampling Methods 0.000 claims description 5
- 230000003321 amplification Effects 0.000 claims description 4
- 238000010276 construction Methods 0.000 claims description 4
- 238000000354 decomposition reaction Methods 0.000 claims description 4
- 238000003199 nucleic acid amplification method Methods 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 3
- 238000003325 tomography Methods 0.000 claims description 3
- 238000013519 translation Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 description 9
- 238000007499 fusion processing Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 7
- 238000000605 extraction Methods 0.000 description 6
- 238000003384 imaging method Methods 0.000 description 5
- 238000009434 installation Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000008030 elimination Effects 0.000 description 2
- 238000003379 elimination reaction Methods 0.000 description 2
- 238000003702 image correction Methods 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000002265 prevention Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000010891 electric arc Methods 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000009776 industrial production Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000001931 thermography Methods 0.000 description 1
- 230000004304 visual acuity Effects 0.000 description 1
Images
Classifications
-
- G06T3/147—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/60—Rotation of a whole image or part thereof
- G06T3/608—Skewing or deskewing, e.g. by two-pass or three-pass rotation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G06T5/70—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20024—Filtering details
- G06T2207/20028—Bilateral filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a bilateral filtering pyramid-based three-light image intelligent fusion method, which comprises the following steps of: firstly, erecting a three-optical camera by using a parallel optical axis design, and enabling the three-optical camera to be over against the same scene for video signal acquisition; carrying out displacement deviation and rotational distortion correction on the collected three paths of video signals; and performing pyramid chromatography processing on the bilateral filter, and fusing three pyramid information obtained by the three spectral images after the pyramid chromatography processing is finished. The invention effectively fuses the bilateral filter and the pyramid layering algorithm into a new effective overall algorithm, and transforms the original three-spectrum image fusion into a fused image containing all the characteristics of the three spectrums and outputs the fused image to the outside, so that a user can finish the acquisition of all the target characteristics in a scene by only observing the image, thereby greatly improving the observation efficiency and the use cheapness of the user.
Description
Technical Field
The invention relates to the technical field of intelligent detection imaging and image processing, in particular to a bilateral filtering pyramid-based three-light image intelligent fusion method.
Background
At present, cameras with different wave bands are widely used for monitoring scenes and targets in the scenes in the application fields related to industrial machine vision, such as community security, industrial production, fire safety, forest fire prevention, security inspection explosion prevention and the like, internationally and domestically. Because target characteristics in nature can show different spectral characteristics, the monitoring process of scenes and targets by using a single camera cannot meet the technical requirements of modern military and civil use. The existing solution is to use a plurality of cameras with different wave bands to simultaneously observe a scene, for example, a visible light camera is used to position a target in the scene, and then a thermal infrared imager is used to perform thermal image analysis on the target; for example, in the operation and maintenance inspection process of the power grid, the thermal infrared imager is used for detecting the heat distribution at the electrical connection joints of the cabinet, the switch box and the like, whether abnormal high temperature occurs is judged, and then the ultraviolet camera is used for judging whether high-voltage arc discharge or electric leakage and other phenomena exist at the abnormal high temperature. The missile target tracking method has the advantages that the missile in the long-distance flight process is detected by the thermal infrared imager and the tail flame characteristics of the missile in the flight process are analyzed by the ultraviolet light camera when the missile target is tracked, so that the defect that the identification of the infrared thermal imaging target is influenced by the interference missile sent by the missile is effectively avoided, and meanwhile, the type of the flight target can be effectively distinguished according to the thermal image and the tail flame characteristics. Therefore, in the field of industrial machine vision, the design concept of multi-camera cooperative work has a quite wide application scene and is initially put into use, and the method is an important direction for developing future visual imaging.
At present, the main product solution in the industry is to separately monitor a plurality of cameras installed in the same scene, for example, simultaneously install a visible light camera and an infrared thermal imager, output video signals to a central control host, simultaneously display the two videos in the supporting software of the host, and simultaneously observe the videos by a user. The biggest drawbacks of such systems are the following: 1. the whole system has a dispersed structure, the camera and the central control host are separate devices, and the volume and the cost are too large when the system is erected on site; 2. when two paths of video images are displayed on a display at the same time, due to the fact that focal plane image elements of imaging devices in different wave bands are different in size, even if the same scene is observed, larger parallax exists, namely, the scenes in the two paths of images are only locally the same, and difficulty is brought to a target finding process; 3. because a user needs to observe two videos at the same time, eyes need to switch and observe between two videos continuously, and visual inconvenience is brought to the user; 4. the system is not easy to carry, can only be installed at a fixed position for observation, and cannot be carried about, so that the expansion of the application scene is limited to a great extent. In summary, there is a need for a multispectral fusion product that is small, lightweight, portable, and fixedly installed to fill the market gap.
The optimal scheme is to carry out an image fusion process, and the obstructed image fusion technology is proposed in the industry at present. Among them, the most widespread of military and civil video solutions is the infrared and visible light dual-spectrum fusion technology. With the technological progress and the higher and higher user demands, the conventional dual-spectrum fusion technology cannot completely cope with the complex target characteristic information appearing in the scene, so that the appearance of a high-precision intelligent fusion technology capable of covering three imaging spectrums of visible light, infrared light and ultraviolet light is particularly important.
Disclosure of Invention
The object of the present invention is to solve at least one of the technical drawbacks mentioned.
Therefore, the invention aims to provide a bilateral filtering pyramid-based three-light image intelligent fusion method, which can fuse images acquired by visible light, infrared light and ultraviolet light, and a user can acquire all target characteristics in a scene by observing the image, so that the observation efficiency is greatly improved, and the use cheapness of the user is greatly improved.
In order to achieve the above object, the present invention provides a bilateral filtering pyramid-based three-light image intelligent fusion method, which comprises the following steps:
step S1, firstly, erecting a three-optical camera by using a parallel optical axis design, and enabling the three-optical camera to be over against the same scene for video signal acquisition;
step S2, displacement deviation and rotation distortion correction are carried out on the collected three paths of video signals;
the correction method of displacement deviation and rotational distortion adopts an image registration correction algorithm under a Cartesian coordinate system and a polar coordinate system to calculate target characteristic points in the images, simultaneously calculates pixel positions where the same characteristic points appear in the three spectral images, and adjusts the angular positions and the sizes of the three spectral images to the same dimensionality by utilizing translation scaling calculation of the Cartesian coordinate system and rotation factor calculation under the polar coordinate system so as to perform subsequent fusion calculation;
step S3, carrying out pyramid chromatography processing on the bilateral filter, namely, respectively carrying out pyramid downsampling layering on image results obtained by the bilateral filter, carrying out bilateral filtering operation on layered results, and ensuring that all characteristic information in the image can be completely extracted for fusion calculation in a mode of multi-time chromatography combined operation;
step S4, when the pyramid chromatography is finished, three kinds of pyramid information obtained by the three spectrum images need to be fused; the fusion method is that the pyramid layer substrate and the substrate corresponding to the three spectrum images are subjected to weighted fusion, and the characteristics and the features are subjected to weighted fusion; and carrying out pyramid inverse operation once each weighted fusion from the topmost pyramid layer, expanding the dimension of the high-level pyramid into the next-level pyramid layer, carrying out weighted fusion calculation with the next-level pyramid layer until all pyramid layers are completely calculated, fusing the original three-spectrum image into a fused image containing all the characteristics of the three spectrums, and outputting the fused image outwards.
Further, in step S2, the three spectral images are scaled to the same size and then corrected when the displacement deviation and the rotational distortion are corrected.
Further, in step S3, the main steps of the bilateral filter performing the pyramid tomography include:
starting from the original three spectral images, the following processing is performed for each spectral image: the method comprises the steps that a fundamental frequency layer image and a detail layer image are obtained after a most basal image is subjected to bilateral filtering for the first time, then a pyramid down-sampling process is carried out on the fundamental frequency layer image obtained at the moment to obtain a second layer pyramid image, then bilateral filtering is carried out on the basal pyramid image of the second layer respectively to obtain the fundamental frequency layer image and the detail layer image of the second layer, then bilateral filtering is carried out on the fundamental frequency image of the second layer continuously, and the like until all layers of a pyramid are subjected to bilateral filtering.
Further, in step S4, when the pyramid information of the three spectrum images is fused, the base image is selected as a visible light image, and the characteristics of the infrared light image and the ultraviolet light image need to be superimposed.
Further, the bilateral filter is an effective nonlinear filter for distinguishing features from noise in an image.
Further, the calculation formula of the bilateral filter is as follows:
where k (i, j) represents the normalization coefficient:
here, the notation (i ', j') e Si, j represents that (i ', j') and (i, j) are adjacent elements in the image; typically gs is chosen to be a normalized gaussian kernel, i.e. the sum of all coefficients in the bilateral filter is 1; furthermore, a Gaussian kernel function is also adopted in the intensity domain; the overall weight k (i, j) of this template is obtained by multiplying the results of two gaussian templates of the spatial domain and the intensity domain; which should range between 0 and 1.
Further, in steps S3-S4, assuming that the ith layer image of the pyramid is G1, the construction formula of the bilateral filter pyramid is: gl (i, j) ═ w (2i,2j, σ) × Gl-1(2i,2j), (1 ≦ l ≦ N), (3);
wherein N represents the layer number of the pyramid, and x represents convolution, i/j represents the coordinates of the image, and w (2i,2j, sigma) represents a bilateral filtering kernel with variance sigma;
thus, G0, G1, …, GN constitute the base band pyramid, G0 is the same as the original image; the size of the current layer image is 1/4 of the size of the previous layer image in turn, and the total layer number of the pyramid is N + 1.
Further, interpolating the image Gl to obtain an image which is amplified by four times, wherein the image size of the image is the same as that of Gl-1; then, from equation (3):
in equation (2):
order to
The pyramid formed by LP0, LP1, … and LPN is a bilateral filtering pyramid; each layer of image is the difference between the image of the local layer of the base frequency layer pyramid and the image of the higher layer of the base frequency layer pyramid after interpolation and amplification, and the process is equivalent to band-pass filtering, so that the bilateral filtering pyramid is also called band-pass pyramid decomposition; and finally, restoring the obtained pyramids of all the base frequency layers of each layer and the obtained detail components by a pyramid reconstruction method to obtain the fused image.
Further, the formula of the pyramid reconstruction is as follows:
the bilateral filtering pyramid-based three-light image intelligent fusion method has the following beneficial effects:
1. the invention adopts the parallel optical axis design when the three-light camera is installed, so as to ensure that the scene acquired by the camera has no geometric distortion.
2. Because the three-light camera cannot observe the same scene due to the mechanical error of the structural installation, the invention firstly corrects the displacement deviation and the rotational distortion before the fusion process, eliminates the displacement deviation and the rotational distortion and ensures the high precision of the subsequent fusion process.
3. The bilateral filter is a nonlinear filter which can effectively distinguish the characteristics and the noise in the image, and can arbitrate the image pixel value in the selected filtering window through two indexes of a space domain and an intensity domain to distinguish the characteristics and the noise information due to the nonlinear filtering characteristics, so that more noise is avoided in the fusion process, and the final fusion effect is greatly improved.
4. The invention adopts a pyramid analysis algorithm based on bilateral filtering as a fusion strategy of three light images, namely, the bilateral filtering and the pyramid layering algorithm are effectively fused into a new effective overall algorithm, the original three-spectrum image fusion is changed into a fusion image containing all the characteristics of three spectrums, and the fusion image is output outwards, so that a user can finish the acquisition of all the target characteristics in a scene only by observing the image, and the observation efficiency and the use cheapness of the user are greatly improved.
5. The invention can well perform the processes of image correction and image fusion on the images from different spectral bands, eliminate scene parallax and image size mismatch of the images and videos from different cameras at the same time, and perform real-time fusion on any two or all information in the three spectra through the bilateral filtering pyramid algorithm designed for the invention.
6. According to the invention, because the bilateral filter can well separate the fundamental frequency information (energy information) and the characteristic information (detail information) in the image, high-precision image superposition can be realized during fusion; the method is combined with a pyramid layering technology, and has good screening characteristics and multiple filtering operation characteristics; the traditional bilateral filter only carries out one-time filtering operation on a common single image.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The above and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
FIG. 1 is a flow chart of the main body of the present invention;
FIG. 2 is a schematic diagram of the present invention for offset and rotational distortion correction;
FIG. 3 is a schematic diagram of the calculation process of the bilateral filter of the present invention;
FIG. 4 is a schematic view of the complete pyramid chromatography process of the present invention;
FIG. 5 is a schematic diagram of detail extraction of pyramid components in the pyramid chromatography process of the present invention;
FIG. 6 is a schematic diagram of energy extraction of pyramid components in the pyramid chromatography process of the present invention;
FIG. 7 is a schematic diagram of the pyramid reconstruction algorithm of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative and intended to be illustrative of the invention and are not to be construed as limiting the invention.
The invention provides a bilateral filtering pyramid-based three-light image intelligent fusion method, which is shown by referring to the attached drawings 1-7 and comprises the following steps:
step S1, first, a three-optical camera is erected by using a parallel optical axis design, so that the three-optical camera is directly facing the same scene for video signal acquisition.
Because of the mechanical error of the three-optical camera in the structural installation, the scenes observed by the three cameras cannot be completely the same, therefore, the invention adopts the parallel optical axis design when the three-optical camera is installed, so as to ensure that the scenes obtained by the camera are not subjected to the situation of geometric distortion, and in addition, because of the installation position of the industrial structure of the camera, the three-optical camera is installed in the space and has physical displacement, so that the displacement and the rotation deviation of the scenes can occur when the unified scene is observed, and certain position deviation exists between the scene information obtained by the three-optical camera, so that the subsequent fusion processing needs to be carried out on the part of the same scene in the scene information obtained by the three-optical camera before the fusion process is carried out; because perfect squareness cannot be achieved when the three-optical-camera is structurally installed, and a certain degree of rotational distortion must exist between the three-optical-camera, the rotational distortion needs to be eliminated preferentially, so that the high precision of the subsequent fusion process can be ensured.
Step S2, displacement deviation and rotation distortion correction are carried out on the collected three paths of video signals;
due to the fact that the sizes of the camera pixels and the pixel intervals of the cameras with different spectrums are different, the number of corresponding pixels of the same target focused on a focal plane through a lens is different, and the images obtained by the three corrected spectrums need to be scaled to the same size when displacement and rotation deviation correction is carried out.
Therefore, when performing the subsequent fusion process, the displacement and the rotational deviation need to be corrected first.
The correction method of displacement deviation and rotational distortion adopts an image registration correction algorithm under a Cartesian coordinate system and a polar coordinate system to calculate target characteristic points in the images, simultaneously calculates pixel positions where the same characteristic points appear in the three spectral images, and adjusts the angular positions and the sizes of the three spectral images to the same dimension by utilizing translation scaling calculation of the Cartesian coordinate system and rotation factor calculation under the polar coordinate system so as to perform subsequent fusion calculation.
And step S3, carrying out pyramid chromatography processing on the bilateral filter, namely respectively carrying out pyramid downsampling layering on the image results obtained by the bilateral filter, carrying out bilateral filtering operation on the layered results, ensuring that the obtained three-light image can separate the characteristics and the noise on each pyramid layer, and ensuring that all characteristic information in the image can be completely extracted for fusion calculation through a mode of multi-time chromatography combined operation. As shown in fig. 3-6.
The invention relates to an image fusion technology based on a bilateral filter pyramid algorithm, wherein a bilateral filter and a pyramid layering algorithm are effectively fused into a new effective overall algorithm. Wherein the bilateral filter is an effective nonlinear filter for distinguishing features from noise in the image.
As shown in fig. 3, the calculation formula of the bilateral filter is:
where k (i, j) represents the normalization coefficient:
here, the notation (i ', j') e Si, j represents that (i ', j') and (i, j) are adjacent elements in the image; typically gs is chosen to be a normalized gaussian kernel, i.e. the sum of all coefficients in the bilateral filter is 1; furthermore, a Gaussian kernel function is also adopted in the intensity domain; the overall weight k (i, j) of this template is obtained by multiplying the results of two gaussian templates of the spatial domain and the intensity domain; which should range between 0 and 1.
σ s and σ r represent standard deviation parameters of the two Gaussian kernels, which control the extent of expansion of the two Gaussian kernels. σ s determines the size of the neighborhood and therefore must be proportional to the size of the image, where the invention selects 2.5% of the diagonal size of the image. The choice of σ r is more critical because it represents the amplitude of the so-called detail. If the range of signal fluctuations is less than σ r, then the signal fluctuations are considered to be detail, i.e., smoothed by the bilateral filter, and separated into detail layers. Conversely, if the range of the ripple is greater than σ r, then this detail will be well preserved to the baseband layer due to the nonlinear nature of the bilateral filter. Here the invention selects 20% of the gray levels that the human eye can resolve, i.e. 25 as the value of σ r. The value takes the resolving power of human eyes to gray scale into consideration, and has good adaptability to different scenes.
The bilateral filter can well separate fundamental frequency information (energy information) and characteristic information (detail information) in the image, so that high-precision image superposition can be realized during fusion. The traditional bilateral filter only carries out one-time filtering operation on a common single image. Based on the good screening characteristics of the pyramid hierarchical structure, the pyramid hierarchical structure combines the pyramid hierarchical structure with the pyramid hierarchical structure.
Fig. 4 is a schematic diagram of a complete pyramid chromatography process for analyzing a picture, where the bottom of the pyramid is an original picture, and each layer is above, that is, the content of the current original picture is subjected to both detail extraction and energy extraction, and the image dimension is reduced to one fourth of the original picture, and fig. 5 and 6 are schematic diagrams of pyramid components of detail extraction and energy extraction in the pyramid chromatography process, respectively.
The bilateral filter performs pyramid chromatography processing mainly comprising the following steps:
starting from the original three spectral images, the following processing is performed for each spectral image: the method comprises the steps that a fundamental frequency layer image and a detail layer image are obtained after a most basal image is subjected to bilateral filtering for the first time, then a pyramid down-sampling process is carried out on the fundamental frequency layer image obtained at the moment to obtain a second layer pyramid image, then bilateral filtering is carried out on the basal pyramid image of the second layer respectively to obtain the fundamental frequency layer image and the detail layer image of the second layer, then bilateral filtering is carried out on the fundamental frequency image of the second layer continuously, and the like until all layers of a pyramid are subjected to bilateral filtering.
Since the pyramid downsampling process is a dimensionality reduction operation of reducing four times each time, the number of pyramid layers is determined according to the size of an initial image, for example, the dimension of the initial image is 640 × 480, when downsampling is performed at a speed of reducing 4 times each time, the total number of pixels of an image to the 7 th layer is 75, and the image cannot be continuously divided by 4, and since the total number of pixels to the fifth layer is 4800, the image content cannot be distinguished by naked eyes, the total number of pyramid layers can be 5, and other number of layers can be selected according to the situation. The calculation process is as follows:
setting the original image as G0, taking G0 as the 0 th layer of the pyramid, also called as a basal layer, and carrying out bilateral filtering and alternate row-alternate downsampling on the original image to obtain the first layer of the base frequency layer pyramid; low-pass filtering and down-sampling are carried out on the first layer of image to obtain a second layer of the base frequency layer pyramid; repeating the above processes to form a base frequency layer pyramid. Assuming that the layer 1 image of the pyramid is G1, the construction process of the bilateral filtering pyramid is:
assuming that the ith layer image of the pyramid is G1, the construction formula of the bilateral filtering pyramid is:
Gl(i,j)＝w(2i,2j,σ)*Gl-1(2i,2j)， (1≤l≤N)， (3)；
wherein N represents the layer number of the pyramid, and x represents convolution, i/j represents the coordinates of the image, and w (2i,2j, sigma) represents a bilateral filtering kernel with variance sigma;
thus, G0, G1, …, GN constitute the base band pyramid, G0 is the same as the original image; the size of the current layer image is 1/4 of the size of the previous layer image in turn, and the total layer number of the pyramid is N + 1. Therefore, the pyramid decomposition of the base frequency layer of the image is realized by sequentially carrying out low-pass filtering on the bottom layer image and then carrying out 2-down sampling on the filtering result in an interlaced and alternate mode.
Secondly, interpolating the image Gl to obtain an image which is amplified by four times, wherein the size of the image is the same as that of Gl-1; then, from equation (3):
in equation (2):
order to
The pyramid formed by LP0, LP1, … and LPN is a bilateral filtering pyramid; each layer of image is the difference between the image of the local layer of the base frequency layer pyramid and the image of the higher layer of the base frequency layer pyramid after interpolation and amplification, and the process is equivalent to band-pass filtering, so that the bilateral filtering pyramid is also called band-pass pyramid decomposition; finally, restoring the obtained pyramids of all the fundamental frequency layers of each layer and the obtained detail components by a pyramid reconstruction method to obtain a fused image, as shown in fig. 7.
The formula for pyramid reconstruction is:
step S4, when the pyramid tomography is finished, three kinds of pyramid information obtained from the three spectrum images need to be fused. As shown in fig. 7.
The fusion method is that the pyramid layer substrate and the substrate corresponding to the three spectrum images are subjected to weighted fusion, and the characteristics and the features are subjected to weighted fusion; and carrying out pyramid inverse operation once each weighting fusion from the topmost pyramid layer, expanding the dimension of the high-level pyramid into the next secondary high level, carrying out weighting fusion calculation with the secondary high level until all pyramid layers are completely calculated, fusing the original three-spectrum image into a fused image containing all characteristics of the three spectrums, and outputting the fused image outwards.
It should be noted that the three-optical camera may be a visible light camera, an infrared camera, an ultraviolet camera, or a camera with other light. When the pyramid information of the three spectrum images is fused, the base image is selected as a visible light image, and the characteristics of an infrared light image and an ultraviolet light image are required to be superposed.
The bilateral filtering pyramid-based three-light image intelligent fusion method can be a technology for carrying out real-time high-precision fusion on images and videos obtained under three imaging spectrums of visible light, infrared light and ultraviolet light. The technology can well carry out image correction and image fusion processes on images from different spectral bands. The images and videos from different cameras at the same time are subjected to scene parallax elimination and image size mismatch phenomenon elimination, any two or all information in three spectrums are fused in real time through the bilateral filtering pyramid algorithm designed for the invention, the fusion strategy of the technology is extremely high in precision, and the fusion result is excellent.
Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made in the above embodiments by those of ordinary skill in the art without departing from the principle and spirit of the present invention. The scope of the invention is defined by the appended claims and their full range of equivalents.
Claims (6)
1. A bilateral filtering pyramid-based three-light image intelligent fusion method is characterized by comprising the following steps:
step S1, firstly, erecting a three-optical camera by using a parallel optical axis design, and enabling the three-optical camera to be over against the same scene for video signal acquisition;
step S2, displacement deviation and rotation distortion correction are carried out on the collected three paths of video signals;
the correction method of displacement deviation and rotational distortion adopts an image registration correction algorithm under a Cartesian coordinate system and a polar coordinate system to calculate target characteristic points in the images, simultaneously calculates pixel positions where the same characteristic points appear in the three spectral images, and adjusts the angular positions and the sizes of the three spectral images to the same dimensionality by utilizing translation scaling calculation of the Cartesian coordinate system and rotation factor calculation under the polar coordinate system so as to perform subsequent fusion calculation;
step S3, carrying out pyramid chromatography processing on the bilateral filter, namely, respectively carrying out pyramid downsampling layering on image results obtained by the bilateral filter, carrying out bilateral filtering operation on layered results, and ensuring that all characteristic information in the image can be completely extracted for fusion calculation in a mode of multi-time chromatography combined operation;
the calculation formula of the bilateral filter is as follows:
where k (i, j) represents the normalization coefficient:
here, the symbol (i ', j') ∈ Si,jThe representations (i ', j') and (i, j) are neighboring elements in the image; in general gsIs chosen to be a normalized gaussian kernel, i.e. the sum of all coefficients in the bilateral filter is 1; furthermore, a Gaussian kernel function is also adopted in the intensity domain; the overall weight k (i, j) of this template is obtained by multiplying the results of two gaussian templates of the spatial domain and the intensity domain; it should range between 0 and 1;
step S4, when the pyramid chromatography is finished, three kinds of pyramid information obtained by the three spectrum images need to be fused; the fusion method is that the pyramid layer substrate and the substrate corresponding to the three spectrum images are subjected to weighted fusion, and the characteristics and the features are subjected to weighted fusion; the pyramid inverse operation is carried out once each weighting fusion from the topmost pyramid layer, the dimension of the high-level pyramid is expanded to enter the next-level pyramid layer and the weighting fusion calculation is carried out with the next-level pyramid layer until all pyramid layers are completely calculated, and at the moment, the original three-spectrum image is fused into a fused image containing all the characteristics of the three spectrums and is output outwards, so that a user can finish obtaining all target characteristics in a scene only by observing the image;
wherein, in steps S3-S4, the image of the first layer of the pyramid is assumed to be G1The construction formula of the bilateral filtering pyramid is as follows:
Gl(i,j)＝w(2i,2j,σ)*Gl-1(2i,2j)，(1≤l≤N)， (3)；
wherein N represents the layer number of the pyramid, and x represents convolution, i/j represents the coordinates of the image, and w (2i,2j, sigma) represents a bilateral filtering kernel with variance sigma;
thus, G0、G1、…、GNA base-frequency layer pyramid, G, is formed0Same as the original image; the size of the current layer image is 1/4 of the size of the previous layer image in sequence, and the total layer number of the pyramid is N + 1;
image GlInterpolating to obtain image with four times amplification and image size Gl-1Are the same in size; then, from equation (3):
in equation (2):
order to
From LP0、LP1、…、LPNThe formed pyramid is a bilateral filtering pyramid; each layer of image is the difference between the image of the local layer of the base frequency layer pyramid and the image of the higher layer of the base frequency layer pyramid after interpolation and amplification, and the process is equivalent to band-pass filtering, so that the bilateral filtering pyramid is also called band-pass pyramid decomposition; and finally, restoring the obtained pyramids of all the base frequency layers of each layer and the obtained detail components by a pyramid reconstruction method to obtain the fused image.
2. The bilateral filtering pyramid-based three-light image intelligent fusion method of claim 1, wherein: in step S2, the three spectral images are scaled to the same size and then corrected when the displacement deviation and the rotational distortion are corrected.
3. The bilateral filtering pyramid-based three-light image intelligent fusion method of claim 1, wherein: in step S3, the main steps of the bilateral filter in performing pyramid tomography are:
starting from the original three spectral images, the following processing is performed for each spectral image: the method comprises the steps that a fundamental frequency layer image and a detail layer image are obtained after a most basal image is subjected to bilateral filtering for the first time, then a pyramid down-sampling process is carried out on the fundamental frequency layer image obtained at the moment to obtain a second layer pyramid image, then bilateral filtering is carried out on the basal pyramid image of the second layer respectively to obtain the fundamental frequency layer image and the detail layer image of the second layer, then bilateral filtering is carried out on the fundamental frequency image of the second layer continuously, and the like until all layers of a pyramid are subjected to bilateral filtering.
4. The bilateral filtering pyramid-based three-light image intelligent fusion method of claim 1, wherein: in step S4, when the pyramid information of the three spectrum images is fused, the base image is selected as a visible light image, and the characteristics of the infrared light image and the ultraviolet light image are to be superimposed.
5. The bilateral filtering pyramid-based three-light image intelligent fusion method of claim 1, wherein: the bilateral filter is an effective nonlinear filter that distinguishes features from noise in an image.
6. The bilateral filtering pyramid-based three-light image intelligent fusion method of claim 1, wherein: the formula for pyramid reconstruction is:
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810118085.1A CN108399612B (en) | 2018-02-06 | 2018-02-06 | Three-light image intelligent fusion method based on bilateral filtering pyramid |
| PCT/CN2018/096023 WO2019153651A1 (en) | 2018-02-06 | 2018-07-17 | Bilateral filter pyramid based three-light image intelligent fusion method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810118085.1A CN108399612B (en) | 2018-02-06 | 2018-02-06 | Three-light image intelligent fusion method based on bilateral filtering pyramid |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108399612A CN108399612A (en) | 2018-08-14 |
| CN108399612B true CN108399612B (en) | 2022-04-05 |
Family
ID=63095287
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810118085.1A Active CN108399612B (en) | 2018-02-06 | 2018-02-06 | Three-light image intelligent fusion method based on bilateral filtering pyramid |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN108399612B (en) |
| WO (1) | WO2019153651A1 (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111814511B (en) * | 2019-04-10 | 2021-02-23 | 青岛大学附属医院 | Identification method based on customized data processing |
| CN110400262B (en) * | 2019-04-10 | 2020-11-06 | 诸暨良嘉环保科技咨询有限公司 | Identification device based on customized data processing |
| CN110956592B (en) * | 2019-11-14 | 2023-07-04 | 北京达佳互联信息技术有限公司 | Image processing method, device, electronic equipment and storage medium |
| CN111292284B (en) * | 2020-02-04 | 2024-03-01 | 淮阴师范学院 | Color image fusion method based on dual-tree-quaternion wavelet transformation |
| CN111462025B (en) * | 2020-02-26 | 2023-04-07 | 宁波大学 | Infrared and visible light image fusion method based on multi-scale low-rank matrix decomposition |
| CN111462032B (en) * | 2020-03-31 | 2023-03-31 | 北方夜视技术股份有限公司 | Method for fusing uncooled infrared image and solar blind ultraviolet image and application |
| CN111652818B (en) * | 2020-05-29 | 2023-09-29 | 浙江大华技术股份有限公司 | Pyramid-based image filtering method, pyramid-based image filtering device and storage medium |
| CN112116632B (en) * | 2020-09-21 | 2023-12-05 | 中国科学院长春光学精密机械与物理研究所 | Method, device and medium for tracking target along tail smoke of target |
| CN112135068A (en) * | 2020-09-22 | 2020-12-25 | 视觉感知(北京)科技有限公司 | Method and device for fusion processing of multiple input videos |
| CN112465705B (en) * | 2020-12-08 | 2022-08-19 | 福州大学 | Visual field expanding system and method based on two-aperture rotating biprism |
| CN113643219B (en) * | 2021-08-03 | 2023-11-24 | 武汉三江中电科技有限责任公司 | Image imaging method and device based on three-light fusion |
| CN114815760B (en) * | 2022-06-27 | 2022-09-09 | 天津德通电气股份有限公司 | Tracing disposal method of safety production tracing disposal system |
| CN116681633B (en) * | 2023-06-06 | 2024-04-12 | 国网上海市电力公司 | Multi-band imaging and fusion method |
| CN116996675B (en) * | 2023-09-27 | 2023-12-19 | 河北天英软件科技有限公司 | Instant messaging system and information processing method |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104376546A (en) * | 2014-10-27 | 2015-02-25 | 北京环境特性研究所 | Method for achieving three-path image pyramid fusion algorithm based on DM642 |
| CN107607202A (en) * | 2017-08-31 | 2018-01-19 | 江苏宇特光电科技股份有限公司 | Three light merge intelligent imager and its method |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8928964B1 (en) * | 2012-01-30 | 2015-01-06 | Softronics, Ltd. | Three-dimensional image display |
| CN104182955B (en) * | 2014-09-05 | 2016-09-14 | 西安电子科技大学 | Image interfusion method based on steerable pyramid conversion and device thereof |
| CN105654448B (en) * | 2016-03-29 | 2018-11-27 | 微梦创科网络科技（中国）有限公司 | A kind of image interfusion method and system based on bilateral filtering and weight reconstruction |
-
2018
- 2018-02-06 CN CN201810118085.1A patent/CN108399612B/en active Active
- 2018-07-17 WO PCT/CN2018/096023 patent/WO2019153651A1/en active Application Filing
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104376546A (en) * | 2014-10-27 | 2015-02-25 | 北京环境特性研究所 | Method for achieving three-path image pyramid fusion algorithm based on DM642 |
| CN107607202A (en) * | 2017-08-31 | 2018-01-19 | 江苏宇特光电科技股份有限公司 | Three light merge intelligent imager and its method |
Non-Patent Citations (3)
| Title |
|---|
| 图像拉普拉斯金字塔融合（Laplacian Pyramid Blending）;佚名;《http://www.360doc.com/content/12/0911/09/10724725_235476158.shtml》;20120911;网页正文全文 * |
| 基于双边与高斯滤波混合分解的图像融合方法;刘志强;《系统工程与电子技术》;20150917;第38卷(第01期);第8-13页 * |
| 多传感器图像融合方法研究;刘贵喜;《中国优秀博硕士论文全文数据库（博士） 信息科技辑》;20020615(第01期);第35-61页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108399612A (en) | 2018-08-14 |
| WO2019153651A1 (en) | 2019-08-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108399612B (en) | Three-light image intelligent fusion method based on bilateral filtering pyramid | |
| WO2021120406A1 (en) | Infrared and visible light fusion method based on saliency map enhancement | |
| CN109377469B (en) | Processing method, system and storage medium for fusing thermal imaging with visible light image | |
| EP2779624B1 (en) | Apparatus and method for multispectral imaging with three-dimensional overlaying | |
| CN107976257B (en) | A kind of image display method of infrared thermal imager, device and infrared thermal imager | |
| WO2014185064A1 (en) | Image processing method and system | |
| US11398053B2 (en) | Multispectral camera external parameter self-calibration algorithm based on edge features | |
| CN106657789A (en) | Thread panoramic image synthesis method | |
| CN106952225B (en) | Panoramic splicing method for forest fire prevention | |
| WO2016205419A1 (en) | Contrast-enhanced combined image generation systems and methods | |
| US10726531B2 (en) | Resolution enhancement of color images | |
| CN112184604A (en) | Color image enhancement method based on image fusion | |
| CN112560619A (en) | Multi-focus image fusion-based multi-distance bird accurate identification method | |
| CN110880191B (en) | Infrared stereo camera dynamic external parameter calculation method based on histogram equalization | |
| CN114265427A (en) | Inspection unmanned aerial vehicle auxiliary navigation system and method based on infrared image matching | |
| CN110910457B (en) | Multispectral three-dimensional camera external parameter calculation method based on angular point characteristics | |
| CN110580684A (en) | image enhancement method based on black-white-color binocular camera | |
| CN109029380B (en) | Stereo visual system and its calibration distance measuring method based on film coated type multispectral camera | |
| CN113066011B (en) | Image processing method, device, system, medium and electronic equipment | |
| CN217181103U (en) | Monitoring system of high-voltage switch cabinet | |
| Kim et al. | An efficient correction method of wide-angle lens distortion for surveillance systems | |
| CN115222785A (en) | Infrared and visible light image registration method based on binocular calibration | |
| CN114782502A (en) | Multispectral multi-sensor cooperative processing method and device and storage medium | |
| CN114972625A (en) | Hyperspectral point cloud generation method based on RGB spectrum super-resolution technology | |
| Cho et al. | Improvement on Demosaicking in Plenoptic Cameras by Use of Masking Information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |