CN111968067B - Short wave infrared image processing method, device and equipment based on silicon sensor camera - Google Patents
Short wave infrared image processing method, device and equipment based on silicon sensor camera Download PDFInfo
- Publication number
- CN111968067B CN111968067B CN201910418875.6A CN201910418875A CN111968067B CN 111968067 B CN111968067 B CN 111968067B CN 201910418875 A CN201910418875 A CN 201910418875A CN 111968067 B CN111968067 B CN 111968067B
- Authority
- CN
- China
- Prior art keywords
- sub
- image
- network model
- conversion
- decomposition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- XUIMIQQOPSSXEZ-UHFFFAOYSA-N Silicon Chemical compound [Si] XUIMIQQOPSSXEZ-UHFFFAOYSA-N 0.000 title claims abstract description 65
- 229910052710 silicon Inorganic materials 0.000 title claims abstract description 65
- 239000010703 silicon Substances 0.000 title claims abstract description 65
- 238000003672 processing method Methods 0.000 title claims abstract description 18
- 238000006243 chemical reaction Methods 0.000 claims abstract description 128
- 238000000354 decomposition reaction Methods 0.000 claims abstract description 95
- 238000000034 method Methods 0.000 claims abstract description 37
- 238000012545 processing Methods 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 8
- 238000013528 artificial neural network Methods 0.000 claims description 7
- 230000002194 synthesizing effect Effects 0.000 claims description 7
- 238000013135 deep learning Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000013507 mapping Methods 0.000 claims description 3
- 230000009466 transformation Effects 0.000 claims 2
- 238000003384 imaging method Methods 0.000 abstract description 17
- 238000012549 training Methods 0.000 description 23
- 238000010586 diagram Methods 0.000 description 17
- 230000008569 process Effects 0.000 description 10
- 238000004088 simulation Methods 0.000 description 8
- 238000003331 infrared imaging Methods 0.000 description 5
- 230000007547 defect Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 229910000530 Gallium indium arsenide Inorganic materials 0.000 description 2
- 230000002238 attenuated effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 238000001308 synthesis method Methods 0.000 description 2
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 210000002615 epidermis Anatomy 0.000 description 1
- 229910052736 halogen Inorganic materials 0.000 description 1
- 150000002367 halogens Chemical class 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000012634 optical imaging Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000013441 quality evaluation Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Studio Devices (AREA)
Abstract
The application provides a short wave infrared image processing method, a device and equipment based on a silicon sensor camera, wherein the method comprises the following steps: acquiring an original image to be processed, wherein the original image is an infrared image comprising at least two wave bands; decomposing the original image by adopting a pre-trained decomposition network model to obtain decomposition sub-images corresponding to each wave band; converting the decomposition sub-images corresponding to each wave band by adopting a pre-trained conversion network model to obtain conversion sub-images corresponding to each decomposition sub-image; and a pre-trained reconstruction network model is adopted to synthesize the converted sub-images, so that the imaging resolution can be improved, and the cost can be saved.
Description
Technical Field
The application relates to the technical field of computer vision and image processing, in particular to a short-wave infrared image processing method, device and equipment based on a silicon sensor camera.
Background
The human eye can only perceive visible light in the wavelength range of 400-700nm, and beyond the visible light range, the human eye cannot perceive, but there are many cases in real life where the human eye cannot perceive, for example. A machine whose surface appears intact, but whose internal defects are not perceptible to the human eye; as another example, the human eye cannot perceive objects hidden in the large fog.
For the above problems, the existing data camera cannot identify, can identify by adopting short-wave infrared imaging, and a special sensor is needed for capturing the short-wave infrared image, for example, an InGaAs sensor is most commonly used, because the sensor can stably work at room temperature, and has the advantages of relatively low power, small volume, high sensitivity and the like. Nevertheless, inGaAs sensors suffer from various drawbacks compared to conventional sensors, such as low spatial resolution, high price and high pixel defect rate, which severely limit the widespread use of InGaAs sensors.
Disclosure of Invention
The application provides a short-wave infrared image processing method, device and equipment based on a silicon sensor camera, which are used for solving the defects of high cost and the like of short-wave infrared imaging equipment in the prior art.
The first aspect of the application provides a short-wave infrared image processing method based on a silicon sensor camera, which comprises the following steps:
acquiring an original image to be processed, wherein the original image is an infrared image comprising at least two wave bands;
decomposing the original image by adopting a pre-trained decomposition network model to obtain decomposition sub-images corresponding to each wave band;
converting the decomposition sub-images corresponding to each wave band by adopting a pre-trained conversion network model to obtain conversion sub-images corresponding to each decomposition sub-image;
and synthesizing each converted sub-image by adopting a pre-trained reconstruction network model to obtain an infrared short-wave image.
A second aspect of the present application provides a short-wave infrared image processing apparatus based on a silicon sensor camera, comprising:
the acquisition module is used for acquiring an original image to be processed, wherein the original image is an infrared image comprising at least two wave bands;
the decomposition module is used for decomposing the original image by adopting a pre-trained decomposition network model to obtain decomposition sub-images corresponding to each wave band;
the conversion module is used for converting the decomposition sub-images corresponding to each wave band by adopting a pre-trained conversion network model to obtain conversion sub-images corresponding to each decomposition sub-image;
and the reconstruction module is used for synthesizing the converted sub-images by adopting a pre-trained reconstruction network model to obtain an infrared short-wave image.
A third aspect of the present application provides a short-wave infrared image processing apparatus based on a silicon sensor camera, comprising: at least one processor and memory;
the memory stores a computer program; the at least one processor executes the computer program stored in the memory to implement the method provided in the first aspect.
A fourth aspect of the application provides a computer readable storage medium having a computer program stored therein, which when executed implements the method provided by the first aspect.
According to the short-wave infrared image processing method, device and equipment based on the silicon sensor camera, the short-wave infrared image is obtained by calculating the decomposition network model, the conversion network model and the reconstruction network model of the acquired original image, so that the imaging resolution can be improved, and the cost can be saved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions of the prior art, the drawings that are needed in the embodiments or the description of the prior art will be briefly described below, it will be obvious that the drawings in the following description are some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort to a person skilled in the art.
Fig. 1 is a flow chart of a short-wave infrared image processing method based on a silicon sensor camera according to an embodiment of the application;
FIG. 2 is a schematic diagram of an imaging system according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a multi-channel imaging system according to an embodiment of the present application;
fig. 4 is a flowchart of a short-wave infrared image processing method based on a silicon sensor camera according to another embodiment of the present application;
FIG. 5a is a schematic diagram of wavelengths corresponding to an original image according to an embodiment of the present application;
FIG. 5b is a schematic diagram of wavelengths corresponding to decomposed sub-images according to an embodiment of the present application;
FIG. 5c is a schematic diagram of wavelengths corresponding to a converted sub-image according to an embodiment of the present application;
FIG. 5d is a schematic diagram of wavelengths corresponding to an infrared short wave image according to an embodiment of the present application;
fig. 6 is a flowchart of a short-wave infrared image processing method based on a silicon sensor camera according to still another embodiment of the present application;
fig. 7 is a flowchart of a short-wave infrared image processing method based on a silicon sensor camera according to another embodiment of the present application;
fig. 8 is a schematic structural diagram of a short-wave infrared image processing device based on a silicon sensor camera according to an embodiment of the present application;
fig. 9 is a schematic structural diagram of a short-wave infrared image processing device based on a silicon sensor camera according to an embodiment of the present application.
Specific embodiments of the present application have been shown by way of the above drawings and will be described in more detail below. These drawings and the written description are not intended to limit the scope of the disclosed concept in any way, but to illustrate the inventive concept to those skilled in the art by reference to specific embodiments.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
An embodiment of the application provides an image processing method which is used for carrying out short wave infrared imaging on the basis of keeping the advantages of low price, high resolution and the like of a silicon sensor camera. The execution subject of the embodiment is a short-wave infrared image processing device based on a silicon sensor camera, which can be arranged on a short-wave infrared image processing device based on the silicon sensor camera, wherein the short-wave infrared image processing device based on the silicon sensor camera can be any computer device, such as a PC, a notebook, a tablet computer, etc.
Fig. 1 is a flow chart of a short-wave infrared image processing method based on a silicon sensor camera according to the present embodiment, as shown in fig. 1, the method includes:
s101, acquiring an original image to be processed, wherein the original image is an infrared image comprising at least two wave bands;
s102, decomposing the original image by adopting a pre-trained decomposition network model to obtain decomposition sub-images corresponding to each wave band;
s103, converting the decomposition sub-images corresponding to each wave band by adopting a pre-trained conversion network model to obtain conversion sub-images corresponding to each decomposition sub-image;
s104, synthesizing the converted sub-images by adopting a pre-trained reconstruction network model to obtain an infrared short-wave image.
Specifically, in step S101, the obtained original image is an infrared image including at least two wavebands, and the quality of the obtained original image is low, for example, the infrared wavelength is generally greater than 950nm, in this embodiment of the present application, a suitable waveband may be arbitrarily selected according to needs, for example, an infrared short wave of 1000nm-1800nm is selected, and the obtained original image includes at least an infrared image of 1000nm-1800 nm.
In step S102, after the original infrared image is acquired, a pre-trained decomposition network model is adopted to decompose the original infrared image to obtain decomposition sub-images corresponding to different wavebands, for example, the wavebands in the acquired original image are 1000nm-1800nm, in this waveband, at least two narrower wavebands, for example, 5 narrower wavebands, such as 1000nm,1050nm,1100nm,1150nm and 1200nm, are selected, that is, the original infrared image is the imaging result of the five wavebands, and in the embodiment of the application, the pre-trained decomposition network model is adopted to decompose the original infrared image into decomposition sub-images corresponding to the five wavebands.
It should be noted that, the above-mentioned narrower band 1000nm is a narrower band with 1000nm as a center point and covering a certain range, for example, 50nm is used as a band range, the coverage range of the narrower band 1000nm should be 975nm-1025nm, and other several similar bands are used.
S103, after the decomposed sub-images corresponding to the wave bands are obtained, converting the obtained decomposed sub-images by adopting a pre-trained conversion network model to obtain converted sub-images corresponding to the decomposed sub-images;
on the basis of the above embodiment, 5 converted sub-images can be obtained and correspond to 5 different bands.
In step S104, after the converted sub-images are obtained, a pre-trained reconstruction network model is adopted to synthesize the obtained 5 converted sub-images, so that a high-quality infrared short-wave image is obtained.
According to the day and night general image processing method based on the silicon sensor camera, the short-wave infrared image is obtained by calculating the decomposition network model, the conversion network model and the reconstruction network model of the acquired original image, so that the imaging resolution can be improved, and the cost can be saved.
A further embodiment of the present application further provides a method according to the above embodiment.
Optionally, the original image is obtained by a silicon sensor camera with an added long pass filter to remove the infrared filter.
On the basis of the above embodiment, fig. 2 is a schematic structural diagram of an imaging system according to an embodiment of the present application, as shown in fig. 2, an infrared filter is removed from a silicon sensor camera, that is, the silicon sensor camera may be used to receive infrared signals or may receive visible light signals.
By way of example, the embodiment of the application adopts a common silicon camera (GS 3-U3-15S 5N-C) provided with a long-pass filter (Thorlabs FELH 0950), and a deep learning neural network is selected as a specific example of a short-wave infrared image synthesis method, so that a virtual short-wave infrared imaging system is constructed, and short-wave infrared imaging can be performed on the basis of keeping the advantages of low price, high resolution and the like of a silicon sensor camera.
Optionally, the decomposition network model, the conversion network model and the reconstruction network model are obtained by deep learning neural network calculation.
Based on the above embodiments, in the embodiments of the present application, the decomposition network model, the conversion network model, and the reconstruction network model may be obtained by training using a neural network, and in the embodiments of the present application, training using a deep learning neural network is preferable.
The decomposition sub-network comprises n branches, consists of n identical modules and is used for simulating the separation effect of different band-pass filters from the long-pass filter signals; the conversion network is similar to the decomposition network and also comprises n branches, each branch converts the image of the current wave band and simulates the image shot by the professional short wave infrared camera in the wave band; and the reconstruction network performs combination processing on the output of the conversion network, so as to generate a final short wave infrared image.
Fig. 3 is a schematic structural diagram of a multi-channel imaging system according to an embodiment of the present application, and as shown in fig. 3, when training a neural network model, a set of multi-channel imaging systems needs to be built, including: the device comprises an additional light source, 1 beam splitter, 1 electric rotating machine, n band-pass filters with different wave bands, 1 long-pass filter, 1 silicon sensor camera and 1 professional short-wave infrared camera; the electric rotating machine is used for ensuring that the switching of the optical filters does not affect the image alignment, for example, the switching of different optical filters under the same scene, and the light splitting sheet is used for ensuring that the silicon sensor camera and the short-wave infrared camera are physically aligned.
Specifically, the hardware of the imaging system comprises: the light source is a halogen lamp, the light splitting sheet is Thorlabs CCN1-BS015, the electric rotating machine is Thorlabs FW102C, the band-pass filters of 5 different wave bands are 1000nm, 1100nm,1150nm and 1200nm CWL,Ednund Hard Coated 0D 4 50nm bandpass Filter, the long-pass filter is Thorlabs FELH0950, the silicon sensor camera GS3-U3-15S5N-C and the short-wave infrared camera is BK-51IGA.
After a multichannel imaging system is built, aligned paired images shot by a silicon sensor camera and a professional short-wave infrared camera under different wave bands are acquired by utilizing a multichannel optical imaging system, a required training data set is provided for the short-wave infrared image synthesis method provided by the embodiment of the application, 2n+2 images are required to be acquired for each scene, and sample data are respectively:
1 image of low-quality infrared band shot by a silicon sensor camera is recorded as A;
1 high-quality short-wave infrared image shot by a professional short-wave infrared camera is marked as B;
n images shot by the silicon sensor camera under n narrower different wave bands are marked as C;
the method comprises the steps that n images shot by a professional short-wave infrared camera under n narrower different wave bands are recorded as D;
fig. 4 is a flowchart of a short-wave infrared image processing method based on a silicon sensor camera according to another embodiment of the present application, as shown in fig. 4, after training sample data is obtained, the sample data is trained,
the training process mainly comprises three steps:
(1) And (3) a decomposition stage: decomposing a low-quality infrared band image shot by the silicon sensor camera, namely the mark A, into n similar images shot by the silicon sensor camera under different narrower bands, namely the mark C, wherein the process can be regarded as simulating to intercept signals of a specific band from a long-pass filter signal physically by using a band-pass filter;
specifically, a low-quality infrared band image shot by a silicon sensor camera with a long-pass filter is decomposed into 5 images, and each image represents an imaging result of the silicon sensor camera in 5 narrower bands (1000 nm,1050nm,1100nm,1150nm and 1200 nm); fig. 5a is a schematic diagram of wavelengths corresponding to an original image according to an embodiment of the present application, as shown in fig. 5a, which is a wavelength of a low-quality infrared band image, and fig. 5b is a schematic diagram of wavelengths corresponding to a decomposed sub-image according to an embodiment of the present application, as shown in fig. 5b, which is a wavelength of a decomposed sub-image obtained by decomposing a low-quality infrared band image.
That is, at this stage, the low-quality infrared band image captured by the silicon sensor camera is input, and the simulation result of the n narrower images captured by the silicon sensor at different bands is output, and ideally, the n images should be infinitely close to the n images captured by the silicon sensor camera labeled C.
(2) Conversion stage: in n narrower wave bands, sequentially processing the images shot by the silicon sensor cameras in n narrower different wave bands obtained in the decomposition stage, and converting the images into images shot by the near-professional short-wave infrared camera in the wave bands;
specifically, the mapping relation between the silicon sensor cameras and the short-wave infrared cameras in 5 different wave bands is learned, and the images shot by the silicon sensor cameras in different wave bands are converted into images shot by the short-wave infrared cameras in corresponding wave bands;
the simulation results of the n decomposed images shot by the silicon sensor under the different narrower wavebands are input, and the simulation results of the n images shot by the professional short-wave infrared camera with the corresponding wavebands are output, in ideal cases, the n images should be infinitely close to the n images shot by the professional short-wave infrared camera marked as D, wherein fig. 5c is a schematic diagram of the wavelength corresponding to the converted sub-image provided by an embodiment of the present application, and as shown in fig. 5c, the wavelength of the converted sub-image is shown.
(3) And (3) a reconstruction stage: and synthesizing the images shot by the professional short-wave infrared cameras under n narrower wave bands obtained in the conversion stage, thereby synthesizing the image of the high-quality short-wave infrared wave band shot by the near-professional short-wave infrared camera.
Specifically, according to 5 different short-wave infrared camera analog images obtained in the conversion stage, high-quality short-wave infrared images which are approximately shot by a professional short-wave infrared camera are synthesized.
The simulation result of the image shot by the short-wave infrared camera with narrower different wave bands of the converted sub-image is input, and the simulation result is output as the simulation result of the image shot by the professional short-wave infrared camera, and ideally, the image should be infinitely close to the image shot by the professional short-wave infrared addition marked as B, wherein fig. 5d is a schematic diagram of wavelengths corresponding to the infrared short-wave image provided by an embodiment of the application, and as shown in fig. 5d, the wavelengths of the finally obtained short-wave infrared image are shown.
As an implementation manner, on the basis of the foregoing embodiment, optionally, the conversion network model includes n conversion sub-network models, where n is an integer greater than 1;
converting the decomposition sub-images corresponding to each wave band by adopting a pre-trained conversion network model to obtain conversion sub-images corresponding to each decomposition sub-image, wherein the method comprises the following steps:
and converting the decomposed sub-images corresponding to the n different wave bands by adopting n pre-trained conversion sub-network models to obtain converted sub-images corresponding to the decomposed sub-images in the n different wave bands.
Specifically, fig. 6 is a flow chart of a short-wave infrared image processing method based on a silicon sensor camera according to still another embodiment of the present application, as shown in fig. 6, specifically:
after the obtained original image is decomposed, 5 decomposed sub-images are obtained, and in the conversion stage, 5 conversion sub-network models are also included, and the training process is as follows:
for the first conversion sub-network model, a first decomposition sub-image and a second decomposition sub-image are input and output as the first conversion sub-image, and the first conversion sub-network model is trained by gradually approaching the image of the professional infrared short wave camera under the wave band through the first conversion sub-image.
For the second conversion sub-network model, a first decomposition sub-image, a second decomposition sub-image, a third decomposition sub-image and the first conversion sub-image are input, the first decomposition sub-image, the second decomposition sub-image, the third decomposition sub-image and the first conversion sub-image are output as a second conversion sub-image, and the second conversion sub-image is gradually approximated to an image of the professional infrared short wave camera under the wave band so as to train the second conversion sub-network model.
And for the third conversion sub-network model, a second decomposition sub-image, a third decomposition sub-image, a fourth decomposition sub-image and a second conversion sub-image are input, the third conversion sub-image is output, and the third conversion sub-network model is trained by gradually approaching the image of the professional infrared short wave camera under the wave band through the third conversion sub-image.
And for the fourth conversion sub-network model, a third decomposition sub-image, a fourth decomposition sub-image, a fifth decomposition sub-image and a third conversion sub-image are input, the third decomposition sub-image, the fifth decomposition sub-image and the third conversion sub-image are output as a fourth conversion sub-image, and the fourth conversion sub-network model is trained by gradually approaching the image of the professional infrared short wave camera under the wave band through the fourth conversion sub-image.
And for the fifth conversion sub-network model, inputting a fourth decomposition sub-image, a fifth decomposition sub-image and a fourth conversion sub-image, outputting the fourth decomposition sub-image, and training the fifth conversion sub-network model by gradually approaching the image of the professional infrared short wave camera under the wave band through the fifth conversion sub-image.
In the training process, adjacent images of the images corresponding to the current wave band and the result after the last training are used as the input of the next training model, the obtained output result has higher precision, and the imaging resolution is improved.
As another implementation manner, on the basis of the foregoing embodiment, optionally, the conversion network model includes n conversion sub-network models and n residual sub-network models, where the conversion sub-network models and the residual sub-network models are in one-to-one correspondence, and n is an integer greater than 1.
Optionally, the converting the decomposed sub-images corresponding to each band by using a pre-trained conversion network model to obtain converted sub-images corresponding to each decomposed sub-image includes:
and converting the decomposed sub-images corresponding to the n different wave bands by adopting n pre-trained conversion sub-network models and n residual sub-network models to obtain converted sub-images corresponding to the decomposed sub-images in the n different wave bands.
Specifically, on the basis of the above embodiment, a residual sub-network model is added to the conversion network model, and fig. 7 is a schematic flow chart of a short-wave infrared image processing method based on a silicon sensor camera according to still another embodiment of the present application, as shown in fig. 7, specifically:
for a first conversion sub-network model, a first decomposition sub-image and a second decomposition sub-image are input, the first decomposition sub-image and the second decomposition sub-image sequentially pass through a residual sub-network model and a conversion sub-network model and are output as the first conversion sub-image, and the first conversion sub-network model is trained by gradually approaching the image of the professional infrared short-wave camera under the wave band through the first conversion sub-image.
And for the second conversion sub-network model, inputting the first decomposition sub-image, the second decomposition sub-image and the third decomposition sub-image into a residual sub-network model, taking the output of the residual sub-network and the obtained first conversion sub-image as the input of the conversion sub-network model, outputting the output as the second conversion sub-image, gradually approaching the image of the professional infrared short wave camera under the wave band through the second conversion sub-image, and training the second conversion sub-network model.
And for a third conversion sub-network model, inputting a second decomposition sub-image, a third decomposition sub-image and a fourth decomposition sub-image into a residual sub-network model, taking the output of the residual sub-network and the obtained second conversion sub-image as the input of the conversion sub-network model, outputting the output as a third conversion sub-image, gradually approaching the image of the professional infrared short wave camera under the wave band through the third conversion sub-image, and training the third conversion sub-network model.
And for the fourth conversion sub-network model, inputting a third decomposition sub-image, a fourth decomposition sub-image and a fifth decomposition sub-image into the residual sub-network model, taking the output of the residual sub-network and the obtained third conversion sub-image as the input of the conversion sub-network model, outputting the output as a fourth conversion sub-image, gradually approaching the image of the professional infrared short wave camera under the wave band through the fourth conversion sub-image, and training the fourth conversion sub-network model.
And for a fifth conversion sub-network model, inputting a fourth decomposition sub-image and a fifth decomposition sub-image into a residual sub-network model, taking the output of the residual sub-network and the obtained fourth conversion sub-image as the input of the conversion sub-network model, outputting the output as a fifth conversion sub-image, gradually approaching the image of the professional infrared short wave camera under the wave band through the fifth conversion sub-image, and training the fifth conversion sub-network model.
In the training process, a residual sub-network model is added in the conversion network model, so that the accuracy of the obtained output result is higher, and the imaging resolution is improved.
Specifically, U-Net (transition subnetwork model) and Res-Net (residual subnetwork) are employed as basic elements of the training network in embodiments of the present application. For the U-Net module, downsampling is performed using a convolution layer with a step size of 2, and upsampling is performed using a linear scaling method. The number of feature map channels of the first layer is set to 32, and subsequently the number of feature maps doubles as the size of the feature maps decreases. The Res-Net module consists of a convolutional layer, three residual blocks, and a stack of convolutional layers.
For each layer of the Res-Net module, the size of the profile remains equal to the input and the profile channel number is set to 32. For all convolutional layers, using ReLU as an activation function, the convolutional kernel size is designed to be 3 x 3, and when residual transfer is performed, stitching along the dimension is used instead of direct addition.
In the embodiment of the application, the image with the W multiplied by H multiplied by 1 of the visible light filter is normalized, the pixel value of the image is scaled from [0,65535] to [0,1], and the pixel value is used as the input of the decomposition sub-network. The decomposition sub-network comprises 5 branches, each branch is composed of 1U-Net module, and the decomposition sub-network is used for simulating the separation effect of different band-pass filters from the long-pass filter signals.
The conversion sub-network also contains five branches, each of which is formed by a Res-Net and U-Net splice, to convert the signal captured by the silicon sensor camera into a signal captured by the short wave infrared camera. In addition, simulation results in shorter wavelength regions are used to guide the simulation of adjacent longer wavelength regions.
And the reconstruction network combines the outputs of the conversion sub-networks to generate a final short wave infrared image. Firstly, each conversion branch is processed by using a U-Net module, then the processing result and the output of a conversion network are sent into a Res-Net module, and finally, the output of the Res-Net module is used as the input of the U-Net module, and the output of the U-Net module is the final short wave infrared image, wherein the number of the characteristic diagrams of the last layer of the U-Net module is set to be 1.
In the embodiment of the application, training is performed on NVIDIA GPU 1080Ti, kears and TensorFlow are adopted as realization frames, in the training process, an input image L is subjected to a decomposition sub-network, a conversion sub-network and a reconstruction sub-network to obtain a short wave infrared image E, and besides the comparison of E with a target result G, the results of the decomposition sub-network and the conversion sub-network are also restrained. The mean square error (NSE) and SSIN quality evaluation criteria are used as the Loss function of the network, then the back propagation algorithm is adopted, the Adan optimization method is adopted for parameter updating and training, the initial learning rate is 0.001, and the number of batch training samples is 30.
Network overfitting is avoided using a method of random clipping and data enhancement, the random clipping being 80 x 1 in size. The training process adopts a learning rate attenuation method, the learning rate is attenuated to be 95% of the current learning rate after one epoch passes, and the learning rate is attenuated to be 50% of the current learning rate when the Loss is not reduced any more. And stopping training when the Loss is lower than a certain threshold value or the iteration number reaches an upper limit (200 in the example), and considering the network to be converged and keeping the current parameters of the network.
It should be noted that, in this embodiment, each of the embodiments may be implemented separately, or may be implemented in any combination without conflict, without limiting the application.
Still another embodiment of the present application provides a short-wave infrared image processing apparatus based on a silicon sensor camera, for performing the method of the above embodiment.
Fig. 8 is a schematic structural diagram of a short-wave infrared image processing device based on a silicon sensor camera according to an embodiment of the present application, and as shown in fig. 8, the image processing device includes an acquisition module 10, a decomposition module 20, a conversion module 30, and a reconstruction module 40;
the acquiring module 10 is configured to acquire an original image to be processed, where the original image is an infrared image including at least two wavebands;
the decomposition module 20 is configured to decompose the original image by using a pre-trained decomposition network model, so as to obtain decomposition sub-images corresponding to each band;
the conversion module 30 is configured to convert the decomposed sub-images corresponding to each band by using a pre-trained conversion network model, so as to obtain converted sub-images corresponding to each decomposed sub-image;
the reconstruction module 40 is configured to synthesize each of the converted sub-images by using a pre-trained reconstruction network model, so as to obtain an infrared short-wave image.
The specific manner in which the individual modules perform the operations of the apparatus of this embodiment has been described in detail in connection with embodiments of the method and will not be described in detail herein.
According to the day and night universal image processing device based on the silicon sensor camera, the short-wave infrared image is obtained by calculating the decomposition network model, the conversion network model and the reconstruction network model of the acquired original image, so that the cost is saved, and the imaging resolution can be improved.
A further embodiment of the present application provides a device for performing the method of the above embodiment, which is further described in the foregoing embodiment.
Optionally, the original image is obtained by a silicon sensor camera with an added long pass filter to remove the infrared filter.
Optionally, the decomposition network model, the conversion network model and the reconstruction network model are obtained by deep learning neural network calculation.
Optionally, the conversion network model includes n conversion sub-network models, where n is an integer greater than 1;
the conversion module is specifically configured to:
and converting the decomposed sub-images corresponding to the n different wave bands by adopting n pre-trained conversion sub-network models to obtain converted sub-images corresponding to the decomposed sub-images in the n different wave bands.
Optionally, the conversion network model includes n conversion sub-network models and n residual sub-network models, where the conversion sub-network models and the residual sub-network models are in one-to-one correspondence, and n is an integer greater than 1.
Optionally, the conversion module is specifically configured to:
and converting the decomposed sub-images corresponding to the n different wave bands by adopting n pre-trained conversion sub-network models and n residual sub-network models to obtain converted sub-images corresponding to the decomposed sub-images in the n different wave bands.
The method or the device provided by the embodiment of the application can be used for detecting agricultural products, and the principle is that the agricultural products are imaged through the pericarp by utilizing different properties of short wave infrared light and visible light, so that possible quality defects under the intact epidermis of the agricultural products are found, and a decision making system is facilitated to make more accurate and correct decisions. The application can also be widely applied to the fields of circuit board detection, solar cell detection, video monitoring under extreme weather conditions, and the like.
Still another embodiment of the present application provides a short-wave infrared image processing apparatus based on a silicon sensor camera, for performing the method provided in the above embodiment.
Fig. 9 is a schematic structural diagram of a short-wave infrared image processing apparatus based on a silicon sensor camera according to the present embodiment, and as shown in fig. 9, the apparatus includes: at least one processor 90 and a memory 91;
the memory stores a computer program; the at least one processor executes the computer program stored in the memory to implement the methods provided by the above embodiments.
According to the short-wave infrared image processing equipment based on the silicon sensor camera, the short-wave infrared image is obtained by calculating the decomposition network model, the conversion network model and the reconstruction network model of the acquired original image, so that the cost is saved, and the imaging resolution can be improved.
A further embodiment of the present application provides a computer readable storage medium having stored therein a computer program which when executed implements the method provided by any of the above embodiments.
According to the computer readable storage medium of the embodiment, the short-wave infrared image is obtained by calculating the decomposition network model, the conversion network model and the reconstruction network model of the acquired original image, so that the cost is saved, and the imaging resolution can be improved.
In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in hardware plus software functional units.
The integrated units implemented in the form of software functional units described above may be stored in a computer readable storage medium. The software functional unit is stored in a storage medium, and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform part of the steps of the methods according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only memory (RON), a random access memory (Randon Access Nenory, RAN), a magnetic disk or an optical disk, or other various media capable of storing program codes.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional modules is illustrated, and in practical application, the above-described functional allocation may be performed by different functional modules according to needs, i.e. the internal structure of the apparatus is divided into different functional modules to perform all or part of the functions described above. The specific working process of the above-described device may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the application.
Claims (8)
1. A short wave infrared image processing method based on a silicon sensor camera is characterized by comprising the following steps:
acquiring an original image to be processed, wherein the original image is an infrared image comprising at least two wave bands, and the original image is obtained by adding a silicon sensor camera with an extended-pass filter to remove the infrared filter;
decomposing the original image by adopting a pre-trained decomposition network model to obtain decomposition sub-images corresponding to each wave band, wherein each decomposition sub-image is used for reflecting the shot image of the silicon sensor camera in each corresponding wave band, and the decomposition network model is obtained by learning the shot image of the silicon sensor camera in each wave band;
converting the decomposition sub-images corresponding to each wave band by adopting a pre-trained conversion network model to obtain conversion sub-images corresponding to each decomposition sub-image, wherein each conversion sub-image is used for reflecting the shot images of the short-wave infrared camera in each corresponding wave band, and the conversion network model is obtained by learning the mapping relation between the shot images of the silicon sensor camera and the shot images of the short-wave infrared camera in each wave band;
and synthesizing each converted sub-image by adopting a pre-trained reconstruction network model to obtain an infrared short-wave image.
2. The method of claim 1, wherein the decomposition network model, the transformation network model, and the reconstruction network model are obtained using deep learning neural network calculations.
3. The method of claim 1, wherein the conversion network model comprises n conversion sub-network models, wherein n is an integer greater than 1;
converting the decomposition sub-images corresponding to each wave band by adopting a pre-trained conversion network model to obtain conversion sub-images corresponding to each decomposition sub-image, wherein the method comprises the following steps:
and converting the decomposed sub-images corresponding to the n different wave bands by adopting n pre-trained conversion sub-network models to obtain converted sub-images corresponding to the decomposed sub-images in the n different wave bands.
4. The method of claim 1, wherein the conversion network model comprises n conversion sub-network models and n residual sub-network models, wherein the conversion sub-network models and the residual sub-network models are in one-to-one correspondence, and n is an integer greater than 1.
5. The method of claim 4, wherein converting the decomposed sub-images corresponding to each band using a pre-trained conversion network model to obtain converted sub-images corresponding to each decomposed sub-image comprises:
and converting the decomposed sub-images corresponding to the n different wave bands by adopting n pre-trained conversion sub-network models and n residual sub-network models to obtain converted sub-images corresponding to the decomposed sub-images in the n different wave bands.
6. A short wave infrared image processing apparatus based on a silicon sensor camera, comprising:
the acquisition module is used for acquiring an original image to be processed, wherein the original image is an infrared image comprising at least two wave bands, and the original image is obtained by adding a silicon sensor camera with an extended-pass filter and removing the infrared filter;
the decomposition module is used for decomposing the original image by adopting a pre-trained decomposition network model to obtain decomposition sub-images corresponding to each wave band, wherein each decomposition sub-image is used for reflecting the shot image of the silicon sensor camera in each corresponding wave band, and the decomposition network model is obtained by learning the shot image of the silicon sensor camera in each wave band;
the conversion module is used for converting the decomposition sub-images corresponding to each wave band by adopting a pre-trained conversion network model to obtain conversion sub-images corresponding to each decomposition sub-image, wherein each conversion sub-image is used for reflecting the shooting images of the short wave infrared camera in each corresponding wave band, and the conversion network model is obtained by learning the mapping relation between the shooting images of the silicon sensor camera and the shooting images of the short wave infrared camera in each wave band;
and the reconstruction module is used for synthesizing the converted sub-images by adopting a pre-trained reconstruction network model to obtain an infrared short-wave image.
7. The apparatus of claim 6, wherein the decomposition network model, the transformation network model, and the reconstruction network model are obtained using deep learning neural network calculations.
8. A short wave infrared image processing apparatus based on a silicon sensor camera, comprising: a memory and at least one processor;
a memory; a memory for storing the processor-executable instructions;
wherein the memory stores a computer program; the at least one processor executes the computer program stored by the memory to implement the method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910418875.6A CN111968067B (en) | 2019-05-20 | 2019-05-20 | Short wave infrared image processing method, device and equipment based on silicon sensor camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910418875.6A CN111968067B (en) | 2019-05-20 | 2019-05-20 | Short wave infrared image processing method, device and equipment based on silicon sensor camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111968067A CN111968067A (en) | 2020-11-20 |
| CN111968067B true CN111968067B (en) | 2023-08-22 |
Family
ID=73358227
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910418875.6A Active CN111968067B (en) | 2019-05-20 | 2019-05-20 | Short wave infrared image processing method, device and equipment based on silicon sensor camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111968067B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106911876A (en) * | 2015-12-22 | 2017-06-30 | 三星电子株式会社 | For the method and apparatus of output image |
| CN108414468A (en) * | 2017-02-09 | 2018-08-17 | 哈尔滨工业大学 | Infrared spectrum wave band feature Enhancement Method based on wavelet transformation and nonlinear transformation |
| CN109410159A (en) * | 2018-09-11 | 2019-03-01 | 上海创客科技有限公司 | Binocular visible light and infrared thermal imaging complex imaging system, method and medium |
| CN109741256A (en) * | 2018-12-13 | 2019-05-10 | 西安电子科技大学 | Image super-resolution rebuilding method based on rarefaction representation and deep learning |
-
2019
- 2019-05-20 CN CN201910418875.6A patent/CN111968067B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106911876A (en) * | 2015-12-22 | 2017-06-30 | 三星电子株式会社 | For the method and apparatus of output image |
| CN108414468A (en) * | 2017-02-09 | 2018-08-17 | 哈尔滨工业大学 | Infrared spectrum wave band feature Enhancement Method based on wavelet transformation and nonlinear transformation |
| CN109410159A (en) * | 2018-09-11 | 2019-03-01 | 上海创客科技有限公司 | Binocular visible light and infrared thermal imaging complex imaging system, method and medium |
| CN109741256A (en) * | 2018-12-13 | 2019-05-10 | 西安电子科技大学 | Image super-resolution rebuilding method based on rarefaction representation and deep learning |
Non-Patent Citations (1)
| Title |
|---|
| 基于小波变换的双色红外图像融合检测方法;孙玉秋等;《红外与激光工程》;20070425(第02期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111968067A (en) | 2020-11-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110991236B (en) | Image classification method and related device | |
| CN110097129B (en) | Remote sensing target detection method based on profile wave grouping characteristic pyramid convolution | |
| CN111062880A (en) | Underwater image real-time enhancement method based on condition generation countermeasure network | |
| CN110675336A (en) | Low-illumination image enhancement method and device | |
| CN111080567A (en) | Remote sensing image fusion method and system based on multi-scale dynamic convolution neural network | |
| CN111079764B (en) | Low-illumination license plate image recognition method and device based on deep learning | |
| CN111695407B (en) | Gender identification method, system, storage medium and terminal based on multispectral fusion | |
| CN113837974B (en) | NSST domain power equipment infrared image enhancement method based on improved BEEPS filtering algorithm | |
| CN112419212A (en) | Infrared and visible light image fusion method based on side window guide filtering | |
| CN109801232A (en) | A kind of single image to the fog method based on deep learning | |
| CN109255758A (en) | Image enchancing method based on full 1*1 convolutional neural networks | |
| CN109753996B (en) | Hyperspectral image classification method based on three-dimensional lightweight depth network | |
| CN115205147A (en) | Multi-scale optimization low-illumination image enhancement method based on Transformer | |
| CN106709967A (en) | Endoscopic imaging algorithm and control system | |
| CN115937794B (en) | Small target object detection method and device, electronic equipment and storage medium | |
| CN110827375B (en) | Infrared image true color coloring method and system based on low-light-level image | |
| CN114419392A (en) | Hyperspectral snapshot image recovery method, device, equipment and medium | |
| CN116757986A (en) | Infrared and visible light image fusion method and device | |
| CN117690161B (en) | Pedestrian detection method, device and medium based on image fusion | |
| Sun et al. | Remote sensing images dehazing algorithm based on cascade generative adversarial networks | |
| CN114998667A (en) | Multispectral target detection method, multispectral target detection system, computer equipment and storage medium | |
| CN114022730A (en) | Point target phase retrieval method based on self-supervision learning neural network | |
| CN114764752B (en) | Night image defogging algorithm based on deep learning | |
| Zhang et al. | Enhanced visual perception for underwater images based on multistage generative adversarial network | |
| CN111968067B (en) | Short wave infrared image processing method, device and equipment based on silicon sensor camera |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |