CN114881872A - Athermalization imaging method and device based on deep learning - Google Patents
Athermalization imaging method and device based on deep learning Download PDFInfo
- Publication number
- CN114881872A CN114881872A CN202210411163.3A CN202210411163A CN114881872A CN 114881872 A CN114881872 A CN 114881872A CN 202210411163 A CN202210411163 A CN 202210411163A CN 114881872 A CN114881872 A CN 114881872A
- Authority
- CN
- China
- Prior art keywords
- image
- temperature
- optical system
- deep learning
- imaging method
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 64
- 238000013135 deep learning Methods 0.000 title claims abstract description 33
- 230000003287 optical effect Effects 0.000 claims abstract description 87
- 238000011084 recovery Methods 0.000 claims abstract description 85
- 230000008859 change Effects 0.000 claims abstract description 67
- 238000002076 thermal analysis method Methods 0.000 claims abstract description 7
- 238000000034 method Methods 0.000 claims description 15
- 230000006870 function Effects 0.000 claims description 13
- 230000004927 fusion Effects 0.000 claims description 12
- 230000000694 effects Effects 0.000 claims description 10
- 238000012360 testing method Methods 0.000 claims description 9
- 230000007613 environmental effect Effects 0.000 claims description 7
- 238000012937 correction Methods 0.000 claims description 6
- 238000005457 optimization Methods 0.000 claims description 6
- 230000004075 alteration Effects 0.000 claims description 5
- 239000011521 glass Substances 0.000 claims description 5
- 238000013507 mapping Methods 0.000 claims description 4
- 238000007781 pre-processing Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 claims description 4
- 238000007789 sealing Methods 0.000 claims description 3
- 238000004088 simulation Methods 0.000 claims description 3
- 230000003044 adaptive effect Effects 0.000 abstract description 4
- 238000012545 processing Methods 0.000 abstract description 2
- 238000012549 training Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 239000000463 material Substances 0.000 description 6
- 230000015556 catabolic process Effects 0.000 description 5
- 238000006731 degradation reaction Methods 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000006073 displacement reaction Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000001931 thermography Methods 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 230000000923 atherogenic effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008602 contraction Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000009792 diffusion process Methods 0.000 description 1
- 239000000428 dust Substances 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000017525 heat dissipation Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000010297 mechanical methods and process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012634 optical imaging Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004576 sand Substances 0.000 description 1
- 238000002834 transmittance Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01K—MEASURING TEMPERATURE; MEASURING QUANTITY OF HEAT; THERMALLY-SENSITIVE ELEMENTS NOT OTHERWISE PROVIDED FOR
- G01K13/00—Thermometers specially adapted for specific purposes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Studio Devices (AREA)
Abstract
The invention discloses a deep learning-based athermal imaging method and device, which are used for carrying out thermal analysis on an optical system to divide different temperature change intervals, constructing corresponding sample pairs, constructing image recovery models adaptive to different temperatures based on a generative countermeasure network by using the sample pairs, and carrying out image recovery on scene images acquired by the optical system at different temperatures by using the adaptive image recovery models on the basis of the image recovery models to obtain high-quality images, so that the optical system can obtain high imaging quality at different temperatures without athermal processing.
Description
Technical Field
The invention belongs to the technical field of intersection of computational vision and optical design, and particularly relates to a thermal imaging-free method and device based on deep learning.
Background
In particular fields, such as vehicle cameras, space cameras, military equipment, etc., the optical system is exposed to a severe and complex external environment, especially severe temperature changes, which can affect the performance of the optical material and its support, resulting in degradation of the imaging quality of the system. Taking temperature as an example, image quality degradation due to a wide temperature variation range is affected by three factors together: variation of radius of curvature and thickness of the refractive surface of the optical element, variation of refractive index of the medium, variation of air space, wherein the second point is dominant. In order to take into account the complicated environmental factors, especially the application scenarios in a wide temperature range, such as high temperature exposure in summer and severe cold in northern winter, a technique of keeping the focal length, image plane or image quality of the optical system unchanged or slightly changed in a large temperature range by a special design or compensation method is required, which is called as an athermal technique.
Athermalization techniques are equally important for both infrared and visible light imaging systems. Since thermal changes of infrared optical materials are more remarkable, most of previous research focuses on no thermal changes of an infrared system, but with miniaturization and light weight of a visible light band optical system and wide use of plastic lenses, the imaging quality and thermal stability of the infrared optical materials are far inferior to those of a conventional large and medium optical system, and therefore compensation for temperature changes is also required.
Traditional atherogenic methods can be divided into three categories: electro (mechanical) active, mechanical passive, optical passive. The mechanical active type is that the temperature sensor is used for automatically detecting the temperature, the temperature information is transmitted to the processor, the image plane displacement caused by the temperature change is calculated, and the motor drives the heat dissipation element to reach the correct position. The mechanical passive type, which compensates for the displacement of the image plane by obtaining the amount of expansion and contraction by naturally expanding and contracting in a reliable manner using mechanical elements of two thermal expansion coefficients. These two mechanical methods have common drawbacks: the additional electronic and mechanical parts increase the volume, mass and cost of the instrument, reduce the reliability and make the maintenance more complex; in addition, they do not completely correct for aberration imbalance caused by thermal effects. The optical passive type utilizes the difference of the thermal characteristics of the materials, and the influence of temperature is counteracted by the combination of different materials. The method has high reliability and simple and compact structure, does not need additional parts, and is a method widely applied in the current industry. But the requirements on materials are strict, the design difficulty is high, and the application in small-sized civil optical devices is limited.
Furthermore, the traditional athermal technology usually focuses on preprocessing, does not consider external factors such as sensor noise, and has certain limitations, and the currently vigorously developed deep learning technology provides an end-to-end imaging scheme. Especially, the learning using Convolutional Neural Network (CNN) as the framework is essentially multilayer convolution, which has very strong deblurring capability, and athermalization is essentially against defocus blur generated by temperature change, so CNN has great application potential. In addition, the end-to-end characteristic enables the device to directly learn the mapping relation between the original temperature-change defocused image and the clear image without coding or other intermediate steps through a mask plate, an additional optical structure is not needed, the device can be directly migrated to any existing optical imaging equipment, and the device has high practicability.
Disclosure of Invention
In view of the foregoing, an object of the present invention is to provide a method and an apparatus for athermal imaging based on deep learning, so as to overcome the degradation effect of different temperatures on the imaging quality of an optical system, and achieve that the optical system can acquire high-quality images in a large temperature range without the conventional athermal pre-processing.
To achieve the above object, an embodiment of the present invention provides a deep learning-based athermal imaging method, which includes the following steps:
carrying out thermal analysis on the optical system, carrying out analog simulation on the reference image by using point spread functions at different temperatures to obtain a change curve of a peak signal-to-noise ratio of a first blurred image corresponding to the reference image along with the temperature, and dividing a plurality of temperature change intervals according to the change curve;
constructing and controlling an environment control device to provide different temperatures, carrying out image acquisition on a reference image displayed by a display by an optical system in different temperature change intervals to obtain a second blurred image, and registering the preprocessed second blurred image and the corresponding reference image to form a sample pair;
optimizing parameters of a generative countermeasure network comprising a generator and a discriminator by using sample pairs at different temperatures, and taking the generator after parameter optimization as image recovery models corresponding to different temperature change intervals;
when the optical system collects a scene image, the temperature of the optical system is detected in real time, and the image recovery model corresponding to the real-time temperature is used for recovering the quality of the scene image so as to realize athermalization imaging.
In one embodiment, the dividing the plurality of temperature variation intervals according to the variation curve includes:
and dividing a plurality of temperature change intervals of the change curve according to preset temperature change precision, wherein the temperature change precision refers to the allowable fluctuation range of each peak signal-to-noise ratio value or each preset peak signal-to-noise ratio range, and the peak signal-to-noise ratio range comprises at least 2 peak signal-to-noise ratio values.
In one embodiment, the environment control device has sealing performance and can provide stable temperature for a built-in optical system, a temperature detector and light-transmitting glass are built in the environment control device, the optical system performs real-time image acquisition on a reference image displayed on a display screen through the light-transmitting glass to obtain a second blurred image, and the temperature detector detects the temperature in real time, so that the temperature is associated with the second blurred image to obtain second blurred images corresponding to different temperatures.
In one embodiment, the second blurred image is preprocessed, comprising: denoising, vignetting correction and chromatic aberration correction are carried out on the second blurred image;
the registration process of the preprocessed second blurred image and the corresponding reference image comprises the following steps: and mapping and cutting the second blurred image according to the size of the reference image, and registering the second blurred image and the reference image to form a sample pair.
In one embodiment, the network complexity of the generative countermeasure network varies depending on temperature, and the greater the temperature, the greater the network complexity of the corresponding generative countermeasure network.
In one embodiment, for a plurality of image recovery models corresponding to a plurality of adjacent temperature change intervals, verifying the recovery degree of the plurality of image recovery models by using a test sample pair, and if the recovery degree is within a threshold range, fusing the plurality of image recovery models into 1 fused image recovery model, wherein the fused image recovery model corresponds to a wide temperature change interval formed by splicing the plurality of temperature change intervals;
the restoration degree is the difference between the restored image corresponding to the second blurred image in the test sample pair and the reference image.
In one embodiment, the fusing the plurality of image restoration models is 1 fused image restoration model, including:
taking one of the multiple image recovery models with the best recovery effect for the test samples of the multiple temperature change intervals as a fusion image recovery model;
or carrying out parameter optimization on the generation type countermeasure network corresponding to the selected 1 image recovery model by using the sample pairs corresponding to the adjacent temperature change sections, wherein the generator with the optimized parameters is used as the fusion image recovery model.
In order to achieve the above object, another embodiment of the present invention provides an athermal imaging apparatus based on deep learning, which includes an optical system, a temperature detector, a processor, and an image recovery model or a fusion image recovery model constructed by the above athermal imaging method based on deep learning;
the optical system is used for acquiring a scene image;
the temperature detector is used for detecting the temperature of the optical system when the scene image is collected;
the fusion image recovery model or the image recovery model is used for recovering the scene image;
the processor is used for calling a fused image recovery model or an image recovery model corresponding to the temperature according to the temperature to recover the collected image.
In order to achieve the above object, a further embodiment of the present invention provides a deep learning-based athermal imaging method, which uses the above athermal imaging apparatus, and the athermal imaging method includes the following steps:
acquiring a scene image by using an optical system;
detecting the temperature of the optical system when the scene image is collected by using a temperature detector;
and calling a fused image recovery model or an image recovery model corresponding to the temperature by using the processor according to the temperature to recover the scene image so as to realize athermal imaging.
Compared with the prior art, the invention has the beneficial effects that at least:
the method comprises the steps of carrying out thermal analysis on an optical system to divide different temperature change intervals, constructing corresponding sample pairs, constructing image recovery models adaptive to different temperatures based on a generative countermeasure network by utilizing the sample pairs, and carrying out image recovery on scene images acquired by the optical system at different temperatures by utilizing the adaptive image recovery models to obtain high-quality images, so that the optical system can obtain high imaging quality at different temperatures without athermalization treatment.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a flow chart of a deep learning based athermalization imaging method provided by an embodiment;
fig. 2 is a block flow diagram of an embodiment provided deep learning based athermal imaging method;
FIG. 3 is a schematic structural diagram of a generation countermeasure network provided by an embodiment;
FIG. 4 is a schematic diagram of an environment control apparatus and a captured image according to an embodiment;
fig. 5 is a schematic structural diagram of an athermal imaging apparatus based on deep learning according to an embodiment;
fig. 6 is a flowchart of a deep learning-based athermalization imaging method according to another embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the detailed description and specific examples, while indicating the scope of the invention, are intended for purposes of illustration only and are not intended to limit the scope of the invention.
The environmental factors influencing the imaging quality are many, besides the temperature, the environmental factors also include air humidity, rain and fog weather and sand and dust weather, but the influence of the temperature is more intuitive, and through research, the influence of the environmental temperature change on the optical system is usually reflected in three aspects: firstly, the temperature change causes the refractive index change of the optical lens medium; the temperature change causes the thickness of optical elements in the optical system, the air interval between the elements and the surface type of each refraction surface to change; third, the temperature variation causes the variation of the sensor noise. For the optical lens, the optical path of light passing through the optical lens is changed by the thermal effects, so that the optical lens generates thermal defocusing and aberration, and the imaging quality and the overall stability of the system are reduced; for the sensor, the rise of temperature increases the temperature drift of the sensor, and the noise is enhanced, so that the quality of the received image is reduced. The influence caused by different temperatures is different, and the imaging quality degradation degree is also different. Based on this, there is a need for a athermal imaging method to avoid the effect of temperature variation on image quality.
FIG. 1 is a flow chart of a deep learning based athermalization imaging method provided by an embodiment; fig. 2 is a block flow diagram of a deep learning-based athermalization imaging method provided by an embodiment. As shown in fig. 1 and fig. 2, an embodiment provides a deep learning-based athermal imaging method, including the following steps:
In the embodiments, the optical system refers to an optical system having an imaging function, and may be a vehicle-mounted lens, and it is obvious that the vehicle-mounted lens is not all the optical systems for implementing the present invention, and the description is given only as an embodiment.
In the embodiment, when the optical system is subjected to thermal analysis, a Point Spread Function (PSF) at different temperatures is used to perform analog simulation on a reference image, so as to obtain a change curve of a peak signal to noise ratio (PSNR) of a first blurred image corresponding to the reference image along with the temperature, and a plurality of temperature change sections are divided according to the change curve, that is, a change curve of the peak PSNR at different temperatures along with the temperature change is obtained.
In embodiments where the point spread function describes the response of the imaging system to a point source or point object, the degree of spread of the point object is an indicator of the quality of the imaging system. The scene image captured via the optical system can be considered to result from the convolution of the actual scene object with the optical system point spread function. Therefore, commercial software such as optical studio (zemax) or Code V can be used for performing thermal analysis on the optical lenses at different temperatures, deriving the point spread function of the optical lenses, and convolving the point spread function with the reference image, so that the first blurred image shot by the optical system at different temperatures can be simulated. The point spread function at different temperatures can be obtained by measuring with a real-point spread function device. In the embodiment, the actual-measuring-point diffusion function device mainly comprises a white light point light source and an optical system to be measured. The white light point light source is composed of a white light LED coupling light source and a 50-micrometer optical fiber, and the end face with the smaller optical fiber is used for simulating the point light source. During measurement, the point light source position is moved in the horizontal direction to change the field of view to be measured, the point light source is shot by the optical system to be measured, and the PSF at the field of view can be measured.
After a first blurred image corresponding to a reference image is obtained, the first blurred image at different temperatures is compared with the reference image to obtain a PSNR, a variation curve of a PSNR value along with the temperature is constructed according to the PSNR, and then a plurality of temperature variation intervals are divided for the variation curve. Taking a variation curve corresponding to the temperature as an example, the specific step of dividing the variation curve into a plurality of temperature variation intervals includes: and dividing multiple temperature change intervals of the change curve according to preset temperature change accuracy, wherein the temperature change accuracy refers to an allowable fluctuation range of each peak signal-to-noise ratio value, namely refers to a peak signal-to-noise ratio range of a curve vicinity area corresponding to each peak signal-to-noise ratio value, for example, when the peak signal-to-noise ratio value is selected to be 30, the temperature change accuracy can be 30 +/-0.5, wherein 0.5 is an allowable fluctuation range, and a temperature range corresponding to the peak signal-to-noise ratio value of about 30 {29.5,30.5} ranges is selected to be 1 temperature change interval on the change curve according to the temperature change accuracy {29.5,30.5 }.
In an embodiment, the temperature variation accuracy may be in each preset peak snr range, and the peak snr range includes at least 2 peak snr values. Namely, the temperature range corresponding to the preset peak signal-to-noise ratio range is selected as 1 temperature change interval on the change curve.
When the temperature change interval is divided, the PSNR values of different temperatures have small difference, and the same image recovery model branch can be used for image quality recovery. The temperature change accuracy value varies with the purpose of the experiment and the recovery accuracy. Adobe5K is used as a default for the reference image data set, and obviously, this is not a necessary condition, and only needs to ensure that the images in the data set can be processed into images with consistent size, consistent depth and higher quality.
And 2, constructing sample pairs corresponding to different temperature change intervals.
In the embodiment, the environment control device is constructed and controlled to provide different temperatures, the optical systems in different temperature change intervals perform image acquisition on the reference image displayed by the display to obtain a second blurred image, and the preprocessed second blurred image and the corresponding reference image are registered to form a sample pair.
In order to simulate different temperatures, an environment control device is constructed, as shown in fig. 4, the environment control device has the characteristics of high sealing performance, high temperature stability and wide temperature change range, and can provide stable temperature for a built-in optical system; the device has a larger internal volume and is used for internally arranging a temperature detector and an optical system; the device also has a high transmittance light-transmitting glass.
And a temperature control box is also arranged in the environment control device, the temperature change range in the control device should cover the lowest temperature and the highest temperature in the step 1, and the temperature control precision should be far smaller than the temperature change interval divided in the step 1. The temperature in the environment control device not only affects the optical lens, but also affects the electronic components such as the sensor, which is consistent with the actual use scene. The obtained second blurred image at different temperatures is degraded in imaging quality and is affected by the optical lens and the sensor.
In the embodiment, the reference image is displayed on a corrected high-quality display, the optical system shoots the display to automatically acquire the batch of RGB three-channel second blurred images, and the temperature detector detects the temperature in real time, so that the temperature is associated with the second blurred images to obtain the second blurred images corresponding to different temperatures.
After the second blurred image is obtained, preprocessing such as denoising, vignetting correction and channel-splitting chromatic aberration correction is carried out on the second blurred image, and the preprocessed second blurred image and the corresponding reference image are registered to form a sample pair at different temperatures. Wherein the registration process comprises: and mapping and cutting the second blurred image according to the size of the reference image, registering the second blurred image with the reference image, and forming a sample pair which is used for training and generating a countermeasure network to construct an image restoration model.
In the embodiment, the difference between the imaging quality of the edge of the image field and the imaging quality of the center field is obvious, the imaging quality of the edge of the captured second blurred image field is poor, vignetting is obvious, the edge part is cut, and the edge part is registered and spliced with the reference image to obtain a blurred-real matched sample pair, so that the burden of network training is reduced.
And 3, constructing image recovery models corresponding to different temperature change intervals by using the sample pairs.
In the embodiment, the parameters of the generative confrontation network comprising the generator and the discriminator are optimized by using sample pairs at different temperatures, and the generator after the parameter optimization is used as an image recovery model corresponding to different temperature change intervals.
The network structure used for training includes, but is not limited to, a GAN network structure model, and the network structure model is flexible and changeable according to imaging requirements. When the GAN network is adopted, the parameters of the GAN network are respectively optimized by using sample pairs corresponding to different temperatures, and the generator parameters and the generator structure obtained by optimizing the sample pairs contained in each temperature change range are used as a branch for processing the captured blurred image in the temperature change range, namely the image recovery model.
And respectively inputting the sample pairs corresponding to the temperatures into the initial GAN network for training aiming at different temperatures. A specific GAN network architecture parameter is shown in fig. 3. During the training process, the proportion of each loss function or the sub-components of each loss function, and the Size of the superparameters such as Epoch and Batch Size can be adjusted to adjust the desired result. Meanwhile, high temperature has a large influence on the optical system, low temperature has a small influence on the optical system, and the calculation overhead can be reduced while the same imaging quality is obtained by adjusting the structural layer number and the convolution channel number of the GAN network generator.
It should be noted that the network complexity of the generative countermeasure network varies depending on the temperature, and the greater the temperature, the greater the network complexity of the corresponding generative countermeasure network.
In the embodiment, the image recovery model which is obtained by training and can achieve the best recovery effect at a specific temperature is used as a network branch of the comprehensive image recovery model based on samples with different temperatures. And constructing a comprehensive image recovery model suitable for recovering the captured blurred image of the optical system in different temperature ranges by the branch network with the plurality of temperatures. Since the temperature in the environment control device affects not only the optical lens but also electronic components such as a sensor, the obtained blur images at different temperatures are captured by the entire optical system. Therefore, the restoration model obtained by using the second blur map as a training sample has the restoration effect for the whole optical system and is not limited to the optical lens.
In the embodiment, for a plurality of image recovery models corresponding to a plurality of adjacent temperature change intervals, verifying the recovery degree of the plurality of image recovery models by using a test sample pair, and if the recovery degree is within a threshold range, fusing the plurality of image recovery models into 1 fused image recovery model, wherein the fused image recovery model corresponds to a wide temperature change interval formed by splicing the plurality of temperature change intervals; the restoration degree is the difference between the restored image corresponding to the second blurred image in the test sample pair and the reference image.
In an embodiment, fusing the plurality of image restoration models into 1 fused image restoration model includes: taking one of the multiple image recovery models with the best recovery effect for the test samples of the multiple temperature change intervals as a fusion image recovery model; or carrying out parameter optimization on the generation type countermeasure network corresponding to the selected 1 image recovery model by using the sample pairs corresponding to the adjacent temperature change sections, wherein the generator with the optimized parameters is used as the fusion image recovery model.
And 4, performing quality recovery on the scene image by using the image recovery model corresponding to the real-time temperature.
In the embodiment, the optical system is used for shooting an actual scene, the real-time temperature of the optical system is detected in real time, the temperature label is attached to a shot fuzzy scene image, and an image recovery model corresponding to the real-time temperature label is used for recovering the quality of the scene image, so that athermal imaging is realized, and a high-quality recovery result image is obtained. Experiments prove that the image recovery models corresponding to different temperatures have higher evaluation indexes and better visual effects on the blurred images belonging to the branch.
Fig. 5 is a schematic structural diagram of an athermal imaging device based on deep learning according to an embodiment. As shown in fig. 5, an embodiment provides a deep learning-based athermal imaging apparatus, which includes an optical system, a temperature detector, a processor, and an image restoration model or a fused image restoration model; the optical system is used for acquiring a scene image; the temperature detector is used for detecting the temperature of the optical system when the optical system collects the scene image; the fusion image recovery model or the image recovery model is constructed by the deep learning-based athermal imaging method and is used for recovering the scene image; the processor is used for calling a fused image recovery model or an image recovery model corresponding to the temperature according to the temperature to recover the acquired image.
Fig. 6 is a flowchart of a deep learning-based athermalization imaging method according to another embodiment. As shown in fig. 6, the non-thermal imaging method based on deep learning according to the embodiment adopts the non-thermal imaging apparatus shown in fig. 5, and includes the following steps:
acquiring a scene image by using an optical system;
detecting the temperature of the scene image acquired by the optical system by using a temperature detector;
and calling a fusion image recovery model or an image recovery model corresponding to the temperature by using a processor according to the temperature to recover the scene image so as to realize athermalization imaging.
The non-thermalization imaging method and device based on deep learning provided by the embodiment overcome the degradation influence of different temperatures on the imaging quality of the optical system, and the optical system can acquire high-quality images in a large temperature range without traditional non-thermalization pretreatment.
It should be noted that, for other environmental factors such as humidity that affect the imaging quality, a solution similar to the temperature may also be adopted to solve the effect of other environmental factors on the imaging quality.
The above-mentioned embodiments are intended to illustrate the technical solutions and advantages of the present invention, and it should be understood that the above-mentioned embodiments are only the most preferred embodiments of the present invention, and are not intended to limit the present invention, and any modifications, additions, equivalents, etc. made within the scope of the principles of the present invention should be included in the scope of the present invention.
Claims (9)
1. A deep learning-based athermal imaging method, comprising the steps of:
carrying out thermal analysis on the optical system, carrying out analog simulation on the reference image by using point spread functions at different temperatures to obtain a change curve of a peak signal-to-noise ratio of a first blurred image corresponding to the reference image along with the temperature, and dividing a plurality of temperature change intervals according to the change curve;
constructing and controlling an environment control device to provide different temperatures, carrying out image acquisition on a reference image displayed by a display by an optical system in different temperature change intervals to obtain a second blurred image, and registering the preprocessed second blurred image and the corresponding reference image to form a sample pair;
optimizing parameters of a generative countermeasure network comprising a generator and a discriminator by using sample pairs at different temperatures, and taking the generator after parameter optimization as image recovery models corresponding to different temperature change intervals;
when the optical system collects a scene image, the temperature of the optical system is detected in real time, and the image recovery model corresponding to the real-time temperature is used for recovering the quality of the scene image so as to realize athermalization imaging.
2. The deep learning-based athermal imaging method according to claim 1, wherein the dividing the plurality of temperature variation intervals according to the variation curve comprises:
and dividing a plurality of temperature change intervals of the change curve according to preset temperature change precision, wherein the temperature change precision refers to the allowable fluctuation range of each peak signal-to-noise ratio value or each preset peak signal-to-noise ratio range, and the peak signal-to-noise ratio range comprises at least 2 peak signal-to-noise ratio values.
3. The deep learning-based athermal imaging method according to claim 1, wherein the environmental control device has a sealing property and is capable of providing a stable temperature for the built-in optical system, the device has a built-in temperature detector and a transparent glass, the optical system performs real-time image acquisition of the reference image displayed on the display screen through the transparent glass to obtain the second blurred image, and the temperature detector detects the temperature in real time, so that the temperature is associated with the second blurred image to obtain the second blurred image corresponding to different temperatures.
4. The deep learning-based athermal imaging method according to claim 1, wherein preprocessing the second blurred image comprises: denoising, vignetting correction and chromatic aberration correction are carried out on the second blurred image;
the registration process of the preprocessed second blurred image and the corresponding reference image comprises the following steps: and mapping and cutting the second blurred image according to the size of the reference image, and registering the second blurred image and the reference image to form a sample pair.
5. The deep learning-based athermal imaging method of claim 1, wherein the network complexity of a generative countermeasure network varies with temperature, and the higher the temperature, the higher the network complexity of the corresponding generative countermeasure network.
6. The deep learning-based athermal imaging method according to claim 1, wherein for a plurality of image recovery models corresponding to a plurality of adjacent temperature variation intervals, the degree of recovery of the plurality of image recovery models is verified by using a test sample pair, and if the degree of recovery is within a threshold range, the plurality of image recovery models are fused into 1 fused image recovery model corresponding to a wide temperature variation interval formed by splicing the plurality of temperature variation intervals;
the restoration degree is the difference between the restored image corresponding to the second blurred image in the test sample pair and the reference image.
7. The deep learning-based athermal imaging method according to claim 6, wherein the fusing the plurality of image recovery models is 1 fused image recovery model comprising:
taking one of the multiple image recovery models with the best recovery effect for the test samples of the multiple temperature change intervals as a fusion image recovery model;
or carrying out parameter optimization on the generation type countermeasure network corresponding to the selected 1 image recovery model by using the sample pairs corresponding to the adjacent temperature change sections, wherein the generator with the optimized parameters is used as the fusion image recovery model.
8. An athermal imaging apparatus based on deep learning, comprising an optical system, a temperature detector, a processor, and the image recovery model or the fused image recovery model constructed by the athermal imaging method based on deep learning according to any one of claims 1 to 7;
the optical system is used for acquiring a scene image;
the temperature detector is used for detecting the temperature of the optical system when the scene image is collected;
the fusion image recovery model or the image recovery model is used for recovering the scene image;
the processor is used for calling a fused image recovery model or an image recovery model corresponding to the temperature according to the temperature to recover the collected image.
9. An athermal imaging method based on deep learning, wherein the athermal imaging method employs the athermal imaging apparatus of claim 8, the athermal imaging method comprising the steps of:
acquiring a scene image by using an optical system;
detecting the temperature of the optical system when the scene image is collected by using a temperature detector;
and calling a fusion image recovery model or an image recovery model corresponding to the temperature by using a processor according to the temperature to recover the scene image so as to realize athermalization imaging.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210411163.3A CN114881872A (en) | 2022-04-19 | 2022-04-19 | Athermalization imaging method and device based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210411163.3A CN114881872A (en) | 2022-04-19 | 2022-04-19 | Athermalization imaging method and device based on deep learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114881872A true CN114881872A (en) | 2022-08-09 |
Family
ID=82671592
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210411163.3A Pending CN114881872A (en) | 2022-04-19 | 2022-04-19 | Athermalization imaging method and device based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114881872A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040190762A1 (en) * | 2003-03-31 | 2004-09-30 | Dowski Edward Raymond | Systems and methods for minimizing aberrating effects in imaging systems |
| US20100310165A1 (en) * | 2009-06-09 | 2010-12-09 | Industrial Technology Research Institute | Image restoration method and apparatus |
| US20190266712A1 (en) * | 2018-02-24 | 2019-08-29 | United States Of America As Represented By The Administrator Of The Nasa | System and method for imaging underwater environments using fluid lensing |
| CN110441905A (en) * | 2019-07-29 | 2019-11-12 | 天津大学 | Based on wavefront coded no thermalization medium-wave infrared imaging method and system |
| US20190394437A1 (en) * | 2018-06-20 | 2019-12-26 | Samsung Electronics Co., Ltd. | Method and apparatus for recovering image |
| CN112561828A (en) * | 2020-12-23 | 2021-03-26 | 北京环境特性研究所 | Gas turbulence fuzzy image reconstruction method based on generation countermeasure network |
| CN113129232A (en) * | 2021-04-15 | 2021-07-16 | 中山大学 | Weak light speckle imaging recovery method based on countermeasure network generated by deep convolution |
| CN114298942A (en) * | 2021-12-31 | 2022-04-08 | Oppo广东移动通信有限公司 | Image deblurring method and device, computer readable medium and electronic equipment |
-
2022
- 2022-04-19 CN CN202210411163.3A patent/CN114881872A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040190762A1 (en) * | 2003-03-31 | 2004-09-30 | Dowski Edward Raymond | Systems and methods for minimizing aberrating effects in imaging systems |
| US20100310165A1 (en) * | 2009-06-09 | 2010-12-09 | Industrial Technology Research Institute | Image restoration method and apparatus |
| US20190266712A1 (en) * | 2018-02-24 | 2019-08-29 | United States Of America As Represented By The Administrator Of The Nasa | System and method for imaging underwater environments using fluid lensing |
| US20190394437A1 (en) * | 2018-06-20 | 2019-12-26 | Samsung Electronics Co., Ltd. | Method and apparatus for recovering image |
| CN110441905A (en) * | 2019-07-29 | 2019-11-12 | 天津大学 | Based on wavefront coded no thermalization medium-wave infrared imaging method and system |
| CN112561828A (en) * | 2020-12-23 | 2021-03-26 | 北京环境特性研究所 | Gas turbulence fuzzy image reconstruction method based on generation countermeasure network |
| CN113129232A (en) * | 2021-04-15 | 2021-07-16 | 中山大学 | Weak light speckle imaging recovery method based on countermeasure network generated by deep convolution |
| CN114298942A (en) * | 2021-12-31 | 2022-04-08 | Oppo广东移动通信有限公司 | Image deblurring method and device, computer readable medium and electronic equipment |
Non-Patent Citations (4)
| Title |
|---|
| GANBAYAR BATCHULUUN等: "Thermal Image Reconstruction Using Deep Learning", 《 IEEE ACCESS 》, 8 July 2020 (2020-07-08) * |
| WEI CHEN等: "Temperature-Robust Learned Image Recovery for Shallow-Designed Imaging Systems", 《ADVANCED INTELLIGENT 》, 23 September 2022 (2022-09-23) * |
| 孟颖;裴雪丹;张连波;: "计算成像系统中的低信噪比图像重构研究", 激光杂志, no. 02, 25 February 2020 (2020-02-25) * |
| 漆建军;: "端对端深度学习无损去图像散射研究", 信息技术与网络安全, no. 09, 10 September 2020 (2020-09-10) * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA2435873C (en) | Gaze tracking system, eye-tracking assembly and an associated method of calibration | |
| CN102143305B (en) | Image pickup method and system | |
| CN110207835A (en) | A kind of wave front correction method based on out-of-focus image training | |
| US20120261474A1 (en) | Optical-information-reading apparatus | |
| CN109489620A (en) | A kind of monocular vision distance measuring method | |
| US20160148049A1 (en) | Image collection with increased accuracy | |
| JP2012524467A (en) | Light field camera images, files and configuration data, and methods for using, storing and communicating them | |
| CN114897752B (en) | Single-lens large-depth-of-field computing imaging system and method based on deep learning | |
| CN110889829A (en) | Monocular distance measurement method based on fisheye lens | |
| JP2006309314A (en) | Translation device | |
| CN109677327A (en) | Method for detecting the EMS and the environment for detecting vehicle of the environment of vehicle | |
| CN111595254B (en) | Method and system for measuring axial distance between lens array and LCD display screen | |
| US20110298912A1 (en) | Wavefront sensing for biometric imaging | |
| CN113450418A (en) | Improved method, device and system for underwater calibration based on complex distortion model | |
| CN203011419U (en) | An optical axis parallelism digital calibration instrument for multiple optical sensors | |
| CN110376725B (en) | Fisheye lens system | |
| Wang et al. | High-performance panoramic annular lens design for real-time semantic segmentation on aerial imagery | |
| CN103176349B (en) | Lens detection device and method | |
| CN114881872A (en) | Athermalization imaging method and device based on deep learning | |
| JP2015535091A (en) | A design method for passive single channel imagers capable of estimating depth of field | |
| CN112767267A (en) | Image defogging method based on simulation polarization fog-carrying scene data set | |
| KR20200117507A (en) | Camera module and image operating method performed therein | |
| US11662814B2 (en) | Sight positioning method, head-mounted display device, computer device and computer-readable storage medium | |
| Baudat | Low-cost wavefront sensing using artificial intelligence (AI) with synthetic data | |
| US20190251700A1 (en) | Method and device for depth detection using stereo images |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |