CN113077389B - Infrared thermal imaging method based on information distillation structure - Google Patents
Infrared thermal imaging method based on information distillation structure Download PDFInfo
- Publication number
- CN113077389B CN113077389B CN202110467768.XA CN202110467768A CN113077389B CN 113077389 B CN113077389 B CN 113077389B CN 202110467768 A CN202110467768 A CN 202110467768A CN 113077389 B CN113077389 B CN 113077389B
- Authority
- CN
- China
- Prior art keywords
- information
- processed
- distillation
- image
- channel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000004821 distillation Methods 0.000 title claims abstract description 165
- 238000000034 method Methods 0.000 title claims abstract description 61
- 238000001931 thermography Methods 0.000 title claims abstract description 50
- 238000013528 artificial neural network Methods 0.000 claims abstract description 40
- 238000000926 separation method Methods 0.000 claims description 36
- 230000004913 activation Effects 0.000 claims description 14
- 230000003213 activating effect Effects 0.000 claims description 12
- 238000005728 strengthening Methods 0.000 claims description 12
- 230000010354 integration Effects 0.000 claims description 4
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 230000037361 pathway Effects 0.000 claims 1
- 238000005516 engineering process Methods 0.000 abstract description 8
- 238000010586 diagram Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 11
- 230000003287 optical effect Effects 0.000 description 6
- 238000004458 analytical method Methods 0.000 description 4
- 230000001537 neural effect Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000003321 amplification Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000002708 enhancing effect Effects 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000003199 nucleic acid amplification method Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011403 purification operation Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an infrared thermal imaging method based on an information distillation structure, which comprises the following steps: acquiring infrared image information to be processed; inputting the infrared image information to be processed into a distillation neural network trained in advance, and processing the infrared image information to be processed into tensor information of a preset channel number through a convolution kernel in the information distillation network; carrying out information distillation on tensor information through a distillation neural network to obtain target image characteristics after the information distillation and a target channel for storing the target image characteristics; under the condition that the number of the target channels is the preset number of the channels, performing feature enhancement on the target image features to obtain enhanced image features; and performing image reconstruction on the enhanced image characteristics by using a sub-pixel convolution reconstruction method to obtain a target infrared thermal imaging image. By adopting a channel information distillation image processing technology, the problems that the existing infrared thermal imaging method is limited by the resolution of an infrared sensor and is influenced by the external environment are solved.
Description
Technical Field
The invention relates to the technical field of infrared thermal imaging, in particular to an infrared thermal imaging method based on an information distillation structure.
Background
The infrared thermal imaging technology is based on a radio and television technology and a computer vision technology, can realize functions of real-time and visual detection and the like, and has wide application in the aspects of detecting fire, checking faults and the like. The system carried by the infrared thermal imaging method mainly comprises hardware such as an optical lens, an infrared sensor chip, a signal converter, an image acquisition card, an objective table and the like, and image processing software.
The traditional infrared thermal imaging method is easily influenced by the external environment, and the imaging resolution is restricted by the resolution of the infrared sensor, so that the infrared thermal imaging method capable of overcoming the restriction of the resolution of the infrared sensor and avoiding the influence of the external environment has very important significance.
Disclosure of Invention
The invention provides an infrared thermal imaging method based on an information distillation structure, which is used for overcoming the limitations of the infrared sensor resolution ratio and the external environment influence existing in the infrared thermal imaging method at the present stage.
In a first aspect, the present invention provides an infrared thermal imaging method based on an information distillation structure, including:
acquiring infrared image information to be processed;
inputting the infrared image information to be processed into a distillation neural network trained in advance, and processing the infrared image information to be processed into tensor information of a preset channel number through a convolution kernel in the information distillation network;
carrying out information distillation on the tensor information through the distillation neural network to obtain target image characteristics after the information distillation and a target channel for storing the target image characteristics;
under the condition that the number of the target channels is the preset number of the channels, performing feature enhancement on the target image features to obtain enhanced image features;
and carrying out image reconstruction on the enhanced image characteristics by utilizing a sub-pixel convolution reconstruction method to obtain a target infrared thermal imaging image.
Optionally, the acquiring infrared image information to be processed includes:
projecting wave band information of an image to be processed in a preset frequency range to an infrared sensor to obtain an analog signal corresponding to the image to be processed;
converting the analog signals into corresponding digital signals through a signal converter;
and restoring the digital signal to obtain the infrared image information to be processed corresponding to the digital signal.
Optionally, the distilling the information of the tensor information by the distilling neural network to obtain the target image feature after the information distillation and a target channel for storing the target image feature includes:
inputting the tensor information into the distillation neural network for information distillation to obtain image features to be processed and a channel to be processed for storing the image features to be processed;
and integrating the image features to be processed and the channels to be processed for storing the image features to be processed to obtain the target image features and the target channels for storing the target image features.
Optionally, the distillation neural model comprises: a feature enhancement structure and an information distillation structure; inputting the tensor information into the distillation neural network for information distillation to obtain image features to be processed and a channel to be processed for storing the image features to be processed, and the method comprises the following steps:
inputting the strengthened tensor information into the distillation neural network, and performing feature strengthening on the tensor information through a feature strengthening structure to obtain strengthened tensor information;
and carrying out information distillation on the strengthened tensor information through an information distillation structure to obtain the image characteristics to be processed and a channel to be processed for storing the image characteristics to be processed.
Optionally, the image feature to be processed includes: the feature information after the primary separation and the feature information after the secondary analysis; the channel to be processed comprises: a deep characteristic channel; carrying out information distillation on the strengthened tensor information through the information distillation structure to obtain the image features to be processed and a channel to be processed for storing the image features to be processed, and the method comprises the following steps:
separating a characteristic channel for storing low-dimensional characteristics and a characteristic channel for storing high-dimensional characteristics from the strengthened tensor information;
convolving the characteristic channel for storing the high-order characteristics through a plurality of preset convolution layers, and activating by using an lrelu activation function to obtain characteristic information after primary separation; the convolution kernels of the preset convolution layers are different in size;
separating out a characteristic channel and a deep characteristic channel for storing the dimensionalities corresponding to the preset convolution layers from the characteristic information after the primary separation;
and performing convolution on the deep characteristic channel through a plurality of convolution layers, and activating by using an lrelu activation function to obtain characteristic information after secondary separation.
In a second aspect, the present invention also provides an infrared thermal imaging apparatus based on an information distilling structure, comprising:
the acquisition module is used for acquiring infrared image information to be processed;
the input module is used for inputting the infrared image information to be processed into a distillation neural network trained in advance, and processing the infrared image information to be processed into tensor information with the preset channel number through a convolution kernel in the information distillation network;
the distillation module is used for carrying out information distillation on the tensor information through the distillation neural network to obtain target image characteristics after the information distillation and a target channel for storing the target image characteristics;
the characteristic strengthening module is used for strengthening the characteristics of the target image under the condition that the number of the target channels is the preset number of the channels to obtain strengthened image characteristics;
and the reconstruction module is used for carrying out image reconstruction on the enhanced image characteristics by utilizing a sub-pixel convolution reconstruction method to obtain a target infrared thermal imaging image.
Optionally, the obtaining module includes:
the projection submodule is used for projecting the wave band information of the image to be processed in a preset frequency range into the infrared sensor to obtain an analog signal corresponding to the image to be processed;
the conversion submodule is used for converting the analog signals into corresponding digital signals through a signal converter;
and the restoring submodule is used for restoring the digital signal to obtain the infrared image information to be processed corresponding to the digital signal.
Optionally, the distillation module comprises:
the input submodule is used for inputting the tensor information into the distillation neural network for information distillation to obtain image characteristics to be processed and a channel to be processed for storing the image characteristics to be processed;
and the integration submodule is used for integrating the image features to be processed and the channels to be processed for storing the image features to be processed to obtain the target image features and the target channels for storing the target image features.
Optionally, the distillation neural model comprises: a feature enhancement structure and an information distillation structure; the input sub-module includes:
the input unit is used for inputting the strengthened tensor information into the distillation neural network, and performing feature strengthening on the tensor information through a feature strengthening structure to obtain strengthened tensor information;
and the distillation unit is used for carrying out information distillation on the strengthened tensor information through an information distillation structure to obtain the image characteristics to be processed and a channel to be processed for storing the image characteristics to be processed.
Optionally, the image feature to be processed includes: the feature information after the primary separation and the feature information after the secondary analysis; the channel to be processed comprises: a deep characteristic channel; the distillation unit comprises:
the first ion separation unit is used for separating a characteristic channel for storing low-dimensional features and a characteristic channel for storing high-dimensional features from the strengthened tensor information;
the convolution subunit is used for performing convolution on the feature channel for storing the high-order features through a plurality of preset convolution layers and activating the feature channel by using an lrelu activation function to obtain feature information after primary separation; the convolution kernels of the preset convolution layers are different in size;
the second ion separation unit is used for separating a characteristic channel and a deep characteristic channel for storing the dimensionality corresponding to the preset convolution layers from the characteristic information after the primary separation;
and the second convolution subunit is used for performing convolution on the deep characteristic channel through the plurality of convolution layers and activating the deep characteristic channel by using an lrelu activation function to obtain characteristic information after secondary separation.
According to the technical scheme, the invention has the following advantages:
the invention obtains the infrared image information to be processed; inputting the infrared image information to be processed into a distillation neural network trained in advance, and processing the infrared image information to be processed into tensor information with preset channel number through a convolution kernel in the information distillation network; carrying out information distillation on the tensor information through the distillation neural network to obtain target image characteristics after the information distillation and a target channel for storing the target image characteristics; under the condition that the number of the target channels is the preset number of the channels, performing feature enhancement on the target image features to obtain enhanced image features; and carrying out image reconstruction on the enhanced image characteristics by utilizing a sub-pixel convolution reconstruction method to obtain a target infrared thermal imaging image. By adopting a channel information distillation image processing technology, the problems that the existing infrared thermal imaging method is limited by the resolution ratio of an infrared sensor and is influenced by the external environment are solved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts;
FIG. 1 is a flowchart illustrating a first embodiment of a method for infrared thermal imaging based on an information distilling structure according to the present invention;
FIG. 2 is a flow chart of the steps of an embodiment II of the method for infrared thermal imaging based on an information distillation structure according to the present invention;
FIG. 3 is a system diagram of a second embodiment of an infrared thermal imaging method based on an information distilling structure according to the present invention;
FIG. 4 is a schematic diagram of a channel information distillation technique according to a second embodiment of the present invention;
FIG. 5 is a schematic diagram illustrating a sub-pixel convolution reconstruction method according to a second embodiment of an infrared thermal imaging method based on an information distillation structure;
FIG. 6 is a schematic diagram of a second embodiment of an infrared thermal imaging method based on an information distillation structure according to the present invention;
fig. 7 is a block diagram of an embodiment of an infrared thermal imaging apparatus based on an information distilling structure according to the present invention.
Detailed Description
The embodiment of the invention provides an infrared thermal imaging method based on an information distillation structure, which is used for overcoming the limitation of the infrared sensor resolution ratio and the influence of the external environment in the infrared thermal imaging method at the present stage.
In order to make the objects, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the embodiments described below are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
Referring to fig. 1, fig. 1 is a flowchart illustrating a first step of an infrared thermal imaging method based on an information distilling structure according to an embodiment of the present invention, which may specifically include the following steps:
step S101, acquiring infrared image information to be processed;
step S102, inputting the infrared image information to be processed into a distillation neural network trained in advance, and processing the infrared image information to be processed into tensor information of a preset channel number through a convolution kernel in the information distillation network;
step S103, carrying out information distillation on the tensor information through the distillation neural network to obtain target image characteristics after information distillation and a target channel for storing the target image characteristics;
step S104, under the condition that the number of the target channels is the preset number of the channels, performing feature enhancement on the target image features to obtain enhanced image features;
and S105, carrying out image reconstruction on the enhanced image characteristics by using a sub-pixel convolution reconstruction method to obtain a target infrared thermal imaging image.
In the embodiment of the invention, the infrared image information to be processed is acquired; inputting the infrared image information to be processed into a distillation neural network trained in advance, and processing the infrared image information to be processed into tensor information of a preset channel number through a convolution kernel in the information distillation network; carrying out information distillation on the tensor information through the distillation neural network to obtain target image characteristics after the information distillation and a target channel for storing the target image characteristics; performing feature enhancement on the target image features under the condition that the number of the target channels is the preset number of the channels to obtain enhanced image features; and carrying out image reconstruction on the enhanced image characteristics by utilizing a sub-pixel convolution reconstruction method to obtain a target infrared thermal imaging image. By adopting a channel information distillation image processing technology, the problems that the existing infrared thermal imaging method is limited by the resolution ratio of an infrared sensor and is influenced by the external environment are solved.
Referring to fig. 2, a flowchart of a second embodiment of the method for infrared thermal imaging based on an information distilling structure of the present invention includes:
step S201, projecting the wave band information of an image to be processed in a preset frequency range to an infrared sensor to obtain an analog signal corresponding to the image to be processed;
referring to fig. 3, fig. 3 is a system configuration diagram of a second embodiment of an infrared thermal imaging method based on an information distillation structure according to the present invention, including: the system comprises a computer 101, an optical fixed-focus lens 102, an infrared sensing chip 103, a lens fixing frame 104, a signal converter 105, a processing unit 106 and a carrier 107.
In the embodiment of the present invention, under the support of the carrier frame 107 and the lens holder 104, the optical fixed-focus lens 102 fixes an optical lens with a focal length of 4mm, fixes the relative position of the optical lens and the infrared sensing chip 103, and then projects the waveband information with a frequency range of 8-14 μm into the infrared sensing chip 103, so that the infrared sensing chip 103 acquires the analog signal corresponding to the image to be processed.
Step S202, converting the analog signal into a corresponding digital signal through a signal converter;
in the embodiment of the present invention, the analog signal corresponding to the image to be processed obtained in step S201 is converted into a corresponding digital signal by the signal converter 105.
Step S203, restoring the digital signal to obtain infrared image information to be processed corresponding to the digital signal;
in the embodiment of the present invention, the digital signal obtained in step S202 is restored to the to-be-processed infrared image information with the resolution of 320 × 240 and the number of channels of 3 corresponding to the digital signal by the processing unit 106, and the to-be-processed infrared image information is transmitted to the computer 101.
Step S204, inputting the infrared image information to be processed into a distillation neural network trained in advance, and processing the infrared image information to be processed into tensor information of a preset channel number through a convolution kernel in the information distillation network;
referring to fig. 4, fig. 4 is a schematic diagram of a channel information distillation technology of a second embodiment of an infrared thermal imaging method based on an information distillation structure of the invention, wherein 201, 202, and 204 are information distillation modules of three dimensions, 203 is a convolution layer for enhancing channel characteristics, 205, 206, and 207 are collections of distillation channels, and a first distillation module 208, a second distillation module 209, and a third distillation module 210 are composed of three modules 201, 202, and 204, respectively.
In the embodiment of the present invention, a convolution layer 203 with a convolution kernel size of 3 is used to process the to-be-processed infrared image obtained in step S203 into tensor information with a channel number of 64 and a resolution of 320 × 240.
Step S205, inputting the tensor information into the distillation neural network for information distillation to obtain image features to be processed and a channel to be processed for storing the image features to be processed;
in an alternative embodiment, the distillation neural model includes: a feature enhancement structure and an information distillation structure; inputting the tensor information into the distillation neural network for information distillation to obtain image features to be processed and a channel to be processed for storing the image features to be processed, and the method comprises the following steps:
inputting the strengthened tensor information into the distillation neural network, and performing feature strengthening on the tensor information through a feature strengthening structure to obtain strengthened tensor information;
and carrying out information distillation on the strengthened tensor information through an information distillation structure to obtain the image characteristics to be processed and a channel to be processed for storing the image characteristics to be processed.
In the embodiment of the present invention, before processing the tensor information, the feature of the tensor information is enhanced, which is specifically expressed as follows:
C 1 ＝Conv 3 (C input )
wherein, C input The input channel number is 64, and the resolution is 320 multiplied by 240 tensor information, conv 3 Denotes performing a convolution operation with a convolution kernel size of 3, C 1 The output enhanced characteristic information is obtained.
In addition, C 1 Has a size of [16, 64,320,240]The number of channels is 16, 64 is the number of channels, and 320 and 240 are the horizontal and vertical resolutions of the image.
In an alternative embodiment, the image features to be processed include: the feature information after the primary separation and the feature information after the secondary analysis; the channel to be processed comprises: a deep characteristic channel; carrying out information distillation on the strengthened tensor information through the information distillation structure to obtain the image features to be processed and a channel to be processed for storing the image features to be processed, and the method comprises the following steps:
separating a characteristic channel for storing low-dimensional characteristics and a characteristic channel for storing high-dimensional characteristics from the strengthened tensor information;
convolving the feature channel for storing the high-order features through a plurality of preset convolution layers, and activating by using an lrelu activation function to obtain feature information after primary separation; the convolution kernels of the preset convolution layers are different in size;
separating out a characteristic channel and a deep characteristic channel for storing the dimensionality corresponding to the preset convolution layers from the characteristic information after the primary separation;
and performing convolution on the deep characteristic channel through a plurality of convolution layers, and activating by using an lrelu activation function to obtain characteristic information after secondary separation.
In the embodiment of the present invention, after feature enhancement is completed, tensor information is input into the information distillation module 201, the information distillation module 202, and the information distillation module 204 with three dimensions, respectively, to perform feature purification operation, thereby obtaining collections 205, 206, and 207 of distillation channels.
In a specific implementation, the first information distillation is performed in the first distillation module 208 to strip out a part of the channel information, so as to retain the low-dimensional features, and the specific operations are as follows:
[C l_1 ,C s_1 ]＝Split(C 1 )
wherein Split is the separation of 64 channels of tensor information according to 1:8, C l_1 For separating out channels which hold low-dimensional features, at this point C l_1 The number of channels is 8,C s_1 For separating out the channels for further high-dimensional feature extraction, C s_1 The number of channels was 56,C 1 The output enhanced characteristic information is obtained.
Channel C to be treated after the first distillation operation s_1 Will be processed through convolutional layers of three different sizes of convolutional kernels to obtain the channel information required for the next distillation operation. The convolution operation is as follows:
C 2_1 ＝Act(Conv 3 (C s_1 ))
C 2_2 ＝Act(Conv 5 (C s_1 ))
C 2_3 ＝Act(Conv 7 (C s_1 ))
in the above formula C 2_1 ，C 2_2 ，C 2_3 Respectively, the channel information C separated for the first time s_1 The feature information after being processed by convolutional layers with convolutional kernel size 3,5,7 respectively, the number of channels of the information after convolution is 64 at this time, act is an activation function, and an lrelu activation function is used here.
The second information distillation is performed in the second information distillation module 209, and is configured to perform channel separation operation on channel information of each scale, and the separated channels are combined to retain feature information of the dimension, where the specific operations are as follows:
[C l3_2 ,C s3_2 ]＝Split(C 2_1 )
[C l5_2 ,C s5_2 ]＝Split(C 2_2 )
[C l7_2 ,C s7_2 ]＝Split(C 2_3 )
C l_2 ＝Concat(C l3_2 ,C l5_2 ,C l7_2 )
wherein, C l3_2 ，C l5_2 ，C l7_2 Channels which are separated from the distillation operation after the convolution layer treatment with the convolution kernel size of 3,5,7 and are used for storing dimensional characteristics are respectively arranged, and the number of the channels is 8,C s3_2 ，C l5_2 ，C l7_2 Respectively, the number of separated reserved channels for further extracting deep features is 56 l_2 Is the collection 205 of low dimensional characteristic channels left by this distillation run.
The third information distillation is performed in the third distillation module 210, and the specific operations are as follows:
[C l3_3 ,C s3_3 ]＝Split(Act(Conv 3 (C s3_2 )))
[C l5_3 ,C s5_3 ]＝Split(Act(Conv 5 (C s5_2 )))
[C l7_3 ,C s7_3 ]＝Split(Act(Conv 7 (C s7_2 )))
C l_3 ＝Concat(C l3_3 ,C l5_3 ,C l7_3 )
wherein, C l3_3 ，C l5_3 ，C l7_3 Respectively a channel C separated from the distillation operation after the convolution layer with the convolution kernel size of 3,5,7 for storing the dimension characteristic s3_3 ，C l5_3 ，C l7_3 Respectively, the separated reserve channel for further extraction of deep features, C l_3 Is the spliced set of low-dimensional eigen-channels left by the distillation operation 207, the concat operation is the channel splicing of the separated channels for preserving the dimensions,
step S206, integrating the image features to be processed and the channels to be processed for storing the image features to be processed to obtain the target image features and the target channels for storing the target image features;
in a specific implementation, after three information distillation operations are performed, channels separated from each dimension for retaining the dimension characteristics need to be integrated, and the integration operation is as follows:
Out 1 ＝Concat(C l_1 ,C l3_2 ,C l5_2 ,C l7_2 ,C l3_2 ,C l5_2 ,C l7_2 )
Out 2 ＝Concat(Conv 3 (C s3_2 ),Conv 5 (C s5_2 ),Conv 7 (C s7_2 ))
Out_channel＝Conv 1 (CCA(Concat(Out 1 ,Out 2 )))
wherein, out 1 For a spliced set of channel features rejected in each distillation run, out 2 Set of channel splices after convolution operation for the high dimensional features left in the third distillation operation 1 And Out 2 After simple splicing, the CCA channel is processed by using the attention of the CCA channel and then the size of a convolution kernel is 1The convolutional layer enhances the characteristics to finally obtain our information distilled channel Out _ channel.
Step S207, performing feature enhancement on the target image feature under the condition that the number of the target channels is the preset number of the channels to obtain an enhanced image feature;
and S208, carrying out image reconstruction on the enhanced image characteristics by using a sub-pixel convolution reconstruction method to obtain a target infrared thermal imaging image.
Please refer to fig. 5, which is a schematic diagram illustrating a sub-pixel convolution reconstruction method according to a second embodiment of an infrared thermal imaging method based on an information distillation structure of the present invention, wherein 301 is an image feature, 302 is a feature enhancement convolution layer, 303 is a feature map formed after arrangement, 304 is an enhanced feature map, and 305 is a pixel matching rule.
It should be understood that the tensor information is not changed after being processed by the distillation structure, i.e. the number of eigen channels is still 64.
When performing sub-pixel convolution, the number of channels of the feature map needs to satisfy the following condition:
num feature ＝num in_channel ×scale 2
wherein num feature Number of channels 64,num for feature map in_channel For the number of channels input 64, scale is a magnification factor of 2.
In a specific implementation, before the sub-pixel convolution-based picture reconstruction operation 305 is started, the feature map 301 is first subjected to a convolution 302 operation to enhance features, and then a first pixel P is taken out of the enhanced feature map 304 and arranged to form a high-resolution feature map SR303 i Sequentially constructing pixel points and pixels P of the residual high-resolution characteristic diagram SR i The structure of the high-definition graph SR is shown as follows:
wherein, middle FE i For parameter value set matrix, P, at the i-th position in all feature maps i The characteristic data after the i-th position reconstruction, SR i The reconstructed ith feature map.
And repeating the steps to finish the self-defined multiple amplification. The method refers to the step 2 as DL-block operation and the step three as Up-block operation, can adaptively arrange the number of two blocks in the method to finish super-resolution operation with different times, and when 4 times of amplification operation is needed, the number of Up-blocks needed to be set is 2,Q which needs to be an even number. The method adopts 6 DL-blocks and two Up-blocks to complete the quadruple super resolution of the image. Different from the general Up-block, the output channel number of the last Up-block needs to be set as the input channel number of the infrared image, and the output is finished after the enhancement is performed by a convolution layer with the convolution kernel size of 1, and the high-definition image Out is as follows:
Out_final＝Conv 1 (SR i )
the high-definition diagram of the predicted infrared image, namely Out _ final, can be obtained by the above formula, and the output tensor format is 3,1280,960 at this time.
Referring to fig. 6, fig. 6 is a general structural schematic diagram of a second embodiment of an infrared thermal imaging method based on an information distillation structure according to the present invention, in which 401 is a feature enhancement convolution layer, 402 is a jump of a low-dimensional feature, 403 is an operation layer of a channel distillation technique, 404 is a sub-pixel convolution operation layer, and a high-resolution and contrast image corresponding to an infrared image to be processed is obtained by using the feature enhancement convolution layer 401, the jump of the low-dimensional feature 402, and the operation layers 403 and 404 of the channel distillation technique as sub-pixel convolution operation layers.
According to the infrared thermal imaging method based on the information distillation structure, provided by the embodiment of the invention, the information of an infrared image to be processed is obtained; inputting the infrared image information to be processed into a distillation neural network trained in advance, and processing the infrared image information to be processed into tensor information of a preset channel number through a convolution kernel in the information distillation network; carrying out information distillation on the tensor information through the distillation neural network to obtain target image characteristics after the information distillation and a target channel for storing the target image characteristics; performing feature enhancement on the target image features under the condition that the number of the target channels is the preset number of the channels to obtain enhanced image features; and carrying out image reconstruction on the enhanced image characteristics by utilizing a sub-pixel convolution reconstruction method to obtain a target infrared thermal imaging image. By adopting a channel information distillation image processing technology, the problems that the existing infrared thermal imaging method is limited by the resolution of an infrared sensor and is influenced by the external environment are solved.
Referring to fig. 7, a block diagram of an embodiment of an infrared thermal imaging apparatus based on an information distilling structure is shown, which includes the following modules:
the acquisition module 101 is used for acquiring infrared image information to be processed;
the input module 102 is configured to input the to-be-processed infrared image information into a distillation neural network trained in advance, and process the to-be-processed infrared image information into tensor information of a preset number of channels through a convolution kernel in the information distillation network;
the distilling module 103 is configured to perform information distillation on the tensor information through the distilling neural network to obtain target image features after the information distillation and a target channel for storing the target image features;
a feature enhancing module 104, configured to perform feature enhancement on the target image feature to obtain an enhanced image feature when the number of the target channels is the preset number of channels;
and the reconstruction module 105 is configured to perform image reconstruction on the enhanced image features by using a sub-pixel convolution reconstruction method to obtain a target infrared thermal imaging image.
In an optional embodiment, the obtaining module 101 includes:
the projection submodule is used for projecting the wave band information of the image to be processed in a preset frequency range into the infrared sensor to obtain an analog signal corresponding to the image to be processed;
the conversion submodule is used for converting the analog signals into corresponding digital signals through a signal converter;
and the restoring submodule is used for restoring the digital signal to obtain the infrared image information to be processed corresponding to the digital signal.
In an alternative embodiment, the distillation module 103 comprises:
the input submodule is used for inputting the tensor information into the distillation neural network for information distillation to obtain image characteristics to be processed and a channel to be processed for storing the image characteristics to be processed;
and the integration submodule is used for integrating the image features to be processed and the channels to be processed for storing the image features to be processed to obtain the target image features and the target channels for storing the target image features.
In an alternative embodiment, the distillation neural model includes: a feature enhancement structure and an information distillation structure; the input sub-module includes:
the input unit is used for inputting the strengthened tensor information into the distillation neural network, and performing feature strengthening on the tensor information through a feature strengthening structure to obtain strengthened tensor information;
and the distillation unit is used for carrying out information distillation on the strengthened tensor information through an information distillation structure to obtain the image characteristics to be processed and a channel to be processed for storing the image characteristics to be processed.
In an alternative embodiment, the image features to be processed include: the feature information after the primary separation and the feature information after the secondary analysis; the channel to be processed comprises: a deep characteristic channel; the distillation unit comprises:
the first ion separation unit is used for separating a characteristic channel for storing low-dimensional features and a characteristic channel for storing high-dimensional features from the strengthened tensor information;
the convolution subunit is used for performing convolution on the feature channel for storing the high-order features through a plurality of preset convolution layers and activating the feature channel by using an lrelu activation function to obtain feature information after primary separation; the convolution kernels of the preset convolution layers are different in size;
the second ion separation unit is used for separating a characteristic channel and a deep characteristic channel for storing the dimensionalities corresponding to the preset convolution layers from the characteristic information after the primary separation;
and the second convolution subunit is used for performing convolution on the deep characteristic channel through the plurality of convolution layers and activating the deep characteristic channel by using an lrelu activation function to obtain characteristic information after secondary separation.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (6)
1. An infrared thermal imaging method based on an information distilling structure, which is characterized by comprising the following steps:
acquiring infrared image information to be processed;
inputting the infrared image information to be processed into a distillation neural network trained in advance, and processing the infrared image information to be processed into tensor information of a preset channel number through a convolution kernel in the distillation neural network;
information distillation is carried out on the tensor information through the distillation neural network, and target image characteristics after information distillation and a target channel for storing the target image characteristics are obtained;
under the condition that the number of the target channels is the preset number of the channels, performing feature enhancement on the target image features to obtain enhanced image features;
performing image reconstruction on the enhanced image characteristics by using a sub-pixel convolution reconstruction method to obtain a target infrared thermal imaging image;
the distillation neural network comprises: a feature enhancement structure and an information distillation structure; the information distillation is performed on the tensor information through the distillation neural network, so that target image characteristics after the information distillation and a target channel for storing the target image characteristics are obtained, and the method comprises the following steps:
inputting the tensor information into the distillation neural network, and performing feature enhancement on the tensor information through the feature enhancement structure to obtain enhanced tensor information;
performing information distillation on the strengthened tensor information through the information distillation structure to obtain image features to be processed and a channel to be processed for storing the image features to be processed; the image features to be processed include: the method comprises the steps of obtaining feature information after primary separation based on a feature channel for storing high-dimensional features and obtaining feature information after secondary separation based on the feature information after primary separation; the channel to be processed comprises: a deep characteristic channel;
and integrating the image features to be processed and the channels to be processed for storing the image features to be processed to obtain the target image features and the target channels for storing the target image features.
2. The infrared thermal imaging method based on the information distillation structure as claimed in claim 1, wherein the acquiring of the infrared image information to be processed comprises:
projecting wave band information of an image to be processed in a preset frequency range to an infrared sensor to obtain an analog signal corresponding to the image to be processed;
converting the analog signals into corresponding digital signals through a signal converter;
and restoring the digital signal to obtain the infrared image information to be processed corresponding to the digital signal.
3. The infrared thermal imaging method based on the information distilling structure according to claim 1, wherein the information distilling structure performs information distilling on the enhanced tensor information to obtain the image features to be processed and a channel to be processed for storing the image features to be processed, and the method comprises:
separating a characteristic channel for storing low-dimensional characteristics and a characteristic channel for storing high-dimensional characteristics from the strengthened tensor information;
convolving the feature channel for storing the high-order features through a plurality of preset convolution layers, and activating by using an lrelu activation function to obtain feature information after primary separation; the convolution kernels of the preset convolution layers are different in size;
separating out a characteristic channel and a deep characteristic channel for storing the dimensionality corresponding to the preset convolution layers from the characteristic information after the primary separation;
and performing convolution on the deep characteristic channel through a plurality of convolution layers, and activating by using an lrelu activation function to obtain characteristic information after secondary separation.
4. An infrared thermal imaging device based on an information distilling structure, comprising:
the acquisition module is used for acquiring infrared image information to be processed;
the input module is used for inputting the infrared image information to be processed into a distillation neural network trained in advance, and processing the infrared image information to be processed into tensor information of a preset channel number through a convolution kernel in the distillation neural network;
the distillation module is used for carrying out information distillation on the tensor information through the distillation neural network to obtain target image characteristics after the information distillation and a target channel for storing the target image characteristics;
the characteristic strengthening module is used for carrying out characteristic strengthening on the target image characteristics under the condition that the number of the target channels is the preset number of the channels to obtain strengthened image characteristics;
the reconstruction module is used for carrying out image reconstruction on the enhanced image characteristics by utilizing a sub-pixel convolution reconstruction method to obtain a target infrared thermal imaging image;
the distillation module includes:
the input submodule is used for inputting the tensor information into the distillation neural network for information distillation to obtain image characteristics to be processed and a channel to be processed for storing the image characteristics to be processed;
the integration submodule is used for integrating the image features to be processed and a channel to be processed for storing the image features to be processed to obtain the target image features and a target channel for storing the target image features;
the distillation neural network includes: a feature enhancement structure and an information distillation structure; the input sub-module includes:
the input unit is used for inputting the tensor information into the distillation neural network, and performing feature enhancement on the tensor information through the feature enhancement structure to obtain enhanced tensor information;
the distillation unit is used for carrying out information distillation on the strengthened tensor information through the information distillation structure to obtain the image characteristics to be processed and a channel to be processed for storing the image characteristics to be processed; the image features to be processed include: the feature information after the primary separation is obtained based on a feature channel for storing high-dimensional features, and the feature information after the secondary separation is obtained based on the feature information after the primary separation; the channel to be processed comprises: a deep characteristic pathway.
5. The infrared thermal imaging device based on information distillation structure of claim 4, wherein the acquisition module comprises:
the projection submodule is used for projecting the wave band information of the image to be processed in a preset frequency range into the infrared sensor to obtain an analog signal corresponding to the image to be processed;
the conversion submodule is used for converting the analog signals into corresponding digital signals through a signal converter;
and the restoring submodule is used for restoring the digital signal to obtain the infrared image information to be processed corresponding to the digital signal.
6. The infrared thermal imaging apparatus based on information distillation structure according to claim 4, wherein said distillation unit comprises:
the first ion separation unit is used for separating a characteristic channel for storing low-dimensional features and a characteristic channel for storing high-dimensional features from the strengthened tensor information;
the convolution subunit is used for performing convolution on the feature channel used for storing the high-order features through a plurality of preset convolution layers and activating the feature channel by using an lrelu activation function to obtain feature information after primary separation; the convolution kernels of the preset convolution layers are different in size;
the second ion separation unit is used for separating a characteristic channel and a deep characteristic channel for storing the dimensionality corresponding to the preset convolution layers from the characteristic information after the primary separation;
and the second convolution subunit is used for performing convolution on the deep characteristic channel through the plurality of convolution layers and activating the deep characteristic channel by using an lrelu activation function to obtain characteristic information after secondary separation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110467768.XA CN113077389B (en) | 2021-04-28 | 2021-04-28 | Infrared thermal imaging method based on information distillation structure |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110467768.XA CN113077389B (en) | 2021-04-28 | 2021-04-28 | Infrared thermal imaging method based on information distillation structure |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113077389A CN113077389A (en) | 2021-07-06 |
| CN113077389B true CN113077389B (en) | 2023-04-07 |
Family
ID=76619051
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110467768.XA Active CN113077389B (en) | 2021-04-28 | 2021-04-28 | Infrared thermal imaging method based on information distillation structure |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113077389B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113743239A (en) * | 2021-08-12 | 2021-12-03 | 青岛图灵科技有限公司 | Pedestrian re-identification method and device and electronic equipment |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110414368B (en) * | 2019-07-04 | 2021-12-17 | 华中科技大学 | Unsupervised pedestrian re-identification method based on knowledge distillation |
| CN110782406B (en) * | 2019-10-15 | 2022-10-11 | 深圳大学 | Image denoising method and device based on information distillation network |
| CN111144490B (en) * | 2019-12-26 | 2022-09-06 | 南京邮电大学 | Fine granularity identification method based on alternative knowledge distillation strategy |
| CN111626330B (en) * | 2020-04-23 | 2022-07-26 | 南京邮电大学 | Target detection method and system based on multi-scale characteristic diagram reconstruction and knowledge distillation |
| CN112070677B (en) * | 2020-09-18 | 2024-04-02 | 中国科学技术大学 | Video space-time super-resolution enhancement method based on time slicing |
| CN112581409B (en) * | 2021-01-05 | 2024-05-07 | 戚如嬅耳纹科技(深圳)有限公司 | Image defogging method based on end-to-end multiple information distillation network |
-
2021
- 2021-04-28 CN CN202110467768.XA patent/CN113077389B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN113077389A (en) | 2021-07-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111369440B (en) | Model training and image super-resolution processing method, device, terminal and storage medium | |
| CN110717851B (en) | Image processing method and device, training method of neural network and storage medium | |
| CN114972756B (en) | Semantic segmentation method and device for medical image | |
| CN112949636B (en) | License plate super-resolution recognition method, system and computer readable medium | |
| EP2774080A1 (en) | Object detection using extended surf features | |
| WO2020098360A1 (en) | Method, system, and computer-readable medium for processing images using cross-stage skip connections | |
| CN111932480A (en) | Deblurred video recovery method and device, terminal equipment and storage medium | |
| CN113128583B (en) | Medical image fusion method and medium based on multi-scale mechanism and residual attention | |
| US10909353B2 (en) | Information processing apparatus, information processing method, non-transitory computer-readable storage medium | |
| CN112001923B (en) | Retina image segmentation method and device | |
| CN112381711A (en) | Light field image reconstruction model training and rapid super-resolution reconstruction method | |
| CN115953303A (en) | Multi-scale image compressed sensing reconstruction method and system combining channel attention | |
| CN113077389B (en) | Infrared thermal imaging method based on information distillation structure | |
| US20240005649A1 (en) | Poly-scale kernel-wise convolution for high-performance visual recognition applications | |
| US20130243330A1 (en) | Method and apparatus for constructing image blur pyramid, and an image feature extracting circuit | |
| CN114758145B (en) | Image desensitizing method and device, electronic equipment and storage medium | |
| CN114519667A (en) | Image super-resolution reconstruction method and system | |
| CN113674154A (en) | Single image super-resolution reconstruction method and system based on generation countermeasure network | |
| CN113643173A (en) | Watermark removing method, watermark removing device, terminal equipment and readable storage medium | |
| CN117274055A (en) | Polarized image super-resolution reconstruction method and system based on information multiplexing | |
| CN112419146B (en) | Image processing method and device and terminal equipment | |
| CN115496819B (en) | Rapid coding spectral imaging method based on energy concentration characteristic | |
| CN114627293A (en) | Image matting method based on multi-task learning | |
| Chen et al. | Progressive spatial-angular feature enhancement network for light field image super-resolution | |
| CN112991249A (en) | Remote sensing image fusion method based on depth separable CNN model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |