CN111968067B - Short wave infrared image processing method, device and equipment based on silicon sensor camera - Google Patents

Abstract

The application provides a short wave infrared image processing method, a device and equipment based on a silicon sensor camera, wherein the method comprises the following steps: acquiring an original image to be processed, wherein the original image is an infrared image comprising at least two wave bands; decomposing the original image by adopting a pre-trained decomposition network model to obtain decomposition sub-images corresponding to each wave band; converting the decomposition sub-images corresponding to each wave band by adopting a pre-trained conversion network model to obtain conversion sub-images corresponding to each decomposition sub-image; and a pre-trained reconstruction network model is adopted to synthesize the converted sub-images, so that the imaging resolution can be improved, and the cost can be saved.

Description

Short-wave infrared image processing method, device and equipment based on silicon sensor camera

technical field

The present application relates to the technical field of computer vision and image processing, and in particular to a short-wave infrared image processing method, device and equipment based on a silicon sensor camera.

Background technique

The human eye can only perceive the visible light in the wavelength range of 400-700nm, and the human eye cannot perceive the wavelength beyond the visible light range, but there are many situations in real life that the human eye cannot perceive, for example. A machine that looks intact on the surface, but its internal defects cannot be perceived by the human eye; another example, the human eye cannot perceive objects hidden in thick fog.

For the above problems, the existing data cameras cannot be identified, and short-wave infrared imaging can be used for identification. Capturing short-wave infrared images requires special sensors, such as InGaAs sensors, which are most commonly used because they can work stably at room temperature and have relatively low power. Low, small size, high sensitivity and other advantages. Nevertheless, InGaAs sensors have various disadvantages compared with conventional sensors, such as low spatial resolution, high price, and high pixel defect rate, which severely limit the widespread use of InGaAs sensors.

Contents of the invention

The present application provides a short-wave infrared image processing method, device and equipment based on a silicon sensor camera to solve the defects of high cost of short-wave infrared imaging equipment in the prior art.

The first aspect of the present application provides a short-wave infrared image processing method based on a silicon sensor camera, including:

Acquiring an original image to be processed, wherein the original image is an infrared image including at least two bands;

Decomposing the original image by using a pre-trained decomposition network model to obtain decomposed sub-images corresponding to each band;

Converting the decomposed sub-images corresponding to each band by using a pre-trained conversion network model to obtain converted sub-images corresponding to each decomposed sub-image;

The converted sub-images are synthesized by using a pre-trained reconstructed network model to obtain an infrared short-wave image.

The second aspect of the present application provides a short-wave infrared image processing device based on a silicon sensor camera, including:

An acquisition module, configured to acquire an original image to be processed, wherein the original image is an infrared image including at least two bands;

The decomposition module is used to decompose the original image by using a pre-trained decomposition network model to obtain decomposed sub-images corresponding to each band;

A conversion module, configured to convert the decomposed sub-images corresponding to each band by using a pre-trained conversion network model, to obtain converted sub-images corresponding to each decomposed sub-image;

The reconstruction module is configured to use a pre-trained reconstruction network model to synthesize the converted sub-images to obtain an infrared short-wave image.

The third aspect of the present application provides a short-wave infrared image processing device based on a silicon sensor camera, including: at least one processor and a memory;

The memory stores a computer program; and the at least one processor executes the computer program stored in the memory to implement the method provided by the first aspect.

A fourth aspect of the present application provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed, the method provided by the first aspect is implemented.

The short-wave infrared image processing method, device and equipment based on the silicon sensor camera provided by this application can obtain short-wave infrared images through the calculation of decomposing the network model, converting the network model and reconstructing the network model on the collected original images, which can improve the imaging resolution. rates and save costs.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present application. Those skilled in the art can also obtain other drawings based on these drawings without any creative effort.

Fig. 1 is a schematic flow chart of a short-wave infrared image processing method based on a silicon sensor camera provided by an embodiment of the present application;

FIG. 2 is a schematic structural diagram of an imaging system provided by an embodiment of the present application;

FIG. 3 is a schematic structural diagram of a multi-channel imaging system provided by an embodiment of the present application;

FIG. 4 is a schematic flow diagram of a short-wave infrared image processing method based on a silicon sensor camera provided in another embodiment of the present application;

Fig. 5a is a schematic diagram of the wavelength corresponding to the original image provided by an embodiment of the present application;

Fig. 5b is a schematic diagram of wavelengths corresponding to decomposed sub-images provided by an embodiment of the present application;

FIG. 5c is a schematic diagram of wavelengths corresponding to converted sub-images provided by an embodiment of the present application;

Fig. 5d is a schematic diagram of wavelengths corresponding to infrared short-wave images provided by an embodiment of the present application;

FIG. 6 is a schematic flow diagram of a short-wave infrared image processing method based on a silicon sensor camera provided in another embodiment of the present application;

FIG. 7 is a schematic flow chart of a short-wave infrared image processing method based on a silicon sensor camera provided in another embodiment of the present application;

FIG. 8 is a schematic structural diagram of a short-wave infrared image processing device based on a silicon sensor camera provided by an embodiment of the present application;

FIG. 9 is a schematic structural diagram of a short-wave infrared image processing device based on a silicon sensor camera provided by an embodiment of the present application.

By means of the above drawings, specific embodiments of the present application have been shown, which will be described in more detail hereinafter. These drawings and written description are not intended to limit the scope of the disclosed concept in any way, but to illustrate the concept of the application for those skilled in the art by referring to specific embodiments.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present invention will be described below with reference to the accompanying drawings.

An embodiment of the present application provides an image processing method for short-wave infrared imaging while retaining the advantages of silicon sensor cameras such as low price and high resolution. The executive subject of this embodiment is a short-wave infrared image processing device based on a silicon sensor camera, which can be set on a short-wave infrared image processing device based on a silicon sensor camera, wherein the short-wave infrared image processing device based on a silicon sensor camera can be any computer Devices such as PCs, laptops, tablets, etc.

Fig. 1 is a schematic flow chart of the short-wave infrared image processing method based on the silicon sensor camera provided in this embodiment, as shown in Fig. 1, the method includes:

S101. Acquire an original image to be processed, where the original image is an infrared image including at least two bands;

S102. Using a pre-trained decomposition network model, decompose the original image to obtain decomposed sub-images corresponding to each band;

S103. Using a pre-trained conversion network model, convert the decomposed sub-images corresponding to each band, and obtain converted sub-images corresponding to each decomposed sub-image;

S104. Synthesize each of the converted sub-images by using a pre-trained reconstructed network model to obtain an infrared short-wave image.

Specifically, in step S101, the acquired original image is an infrared image containing at least two wavebands, and the quality of the acquired original image is low, for example, the infrared wavelength is generally greater than 950nm, and in this embodiment of the application, any Select the appropriate wavelength band, such as selecting the infrared short-wave of 1000nm-1800nm, the acquired original image includes at least the infrared image of 1000nm-1800nm.

In step S102, after acquiring the original infrared image, the pre-trained decomposition network model is used to decompose the original infrared image to obtain decomposed sub-images corresponding to different wave bands. For example, the wave bands in the acquired original image are 1000nm-1800nm, in this band, you can choose at least two narrower bands, for example, choose 5 narrower bands, such as 1000nm, 1050nm, 1100nm, 1150nm and 1200nm, that is to say, the original infrared image is the above five For the results of band imaging, in the embodiment of the application, a pre-trained decomposition network model is used to decompose the original infrared image into decomposed sub-images corresponding to the five bands.

It should be noted that the above-mentioned narrow band 1000nm is a narrow band with 1000nm as the center point and covers a certain range before and after. Taking the band range of 50nm as an example, the coverage of the narrow band of 1000nm should be 975nm-1025nm, other Several are similar.

In step S103, after obtaining the decomposed sub-images corresponding to each band, the pre-trained conversion network model is used to convert the obtained decomposed sub-images to obtain converted sub-images corresponding to each decomposed sub-image;

On the basis of the above-mentioned embodiment, 5 converted sub-images can be obtained and correspond to 5 different bands.

In step S104, after obtaining the converted sub-images, the pre-trained reconstruction network model is used to synthesize the obtained 5 converted sub-images to obtain a high-quality infrared short-wave image.

The general day and night image processing method based on the silicon sensor camera provided by this application obtains the short-wave infrared image by decomposing the network model, converting the network model and reconstructing the network model on the original image collected, which can improve the imaging resolution and can also cut costs.

Yet another embodiment of the present application makes further supplementary descriptions to the methods provided in the foregoing embodiments.

Optionally, the original image is obtained by a silicon sensor camera with a long-pass filter removed and an infrared filter removed.

On the basis of the above-mentioned embodiments, Fig. 2 is a schematic structural diagram of an imaging system provided by an embodiment of the present application. As shown in Fig. 2, the infrared filter is removed from the silicon sensor camera, that is, the silicon sensor camera can be used to receive infrared Signals can also receive visible light signals. In the embodiment of this application, visible light signals need to be removed. Therefore, a long-pass filter is added to the silicon sensor camera to filter out visible light signals, thereby obtaining images in the infrared band, but the quality is relatively low. Low.

Exemplarily, in the embodiment of the present application, an ordinary silicon camera (GS3-U3-15S5N-C) equipped with a long-pass filter (Thorlabs FELH0950) is used, and a deep learning neural network is selected as one of the short-wave infrared image synthesis methods As a specific example, a virtual short-wave infrared imaging system is constructed, which can perform short-wave infrared imaging on the basis of retaining the advantages of silicon sensor cameras such as low price and high resolution.

Optionally, the decomposed network model, the converted network model and the reconstructed network model are obtained by using deep learning neural network calculations.

On the basis of the above-mentioned embodiments, in the embodiments of the present application, the decomposition network model, the conversion network model and the reconstruction network model can be obtained by training with a neural network. In the embodiments of the present application, preferably , using a deep learning neural network for training.

The decomposition sub-network consists of n branches, composed of n identical modules, used to simulate the separation effect of different band-pass filters from the long-pass filter signal; the conversion network is similar to the decomposition network, and also contains n Each branch converts the image of the current band to simulate the image taken by a professional short-wave infrared camera in this band; the reconstruction network combines the outputs of the conversion network to generate the final short-wave infrared image.

Exemplarily, FIG. 3 is a schematic structural diagram of a multi-channel imaging system provided by an embodiment of the present application. As shown in FIG. 3 , when training a neural network model, a multi-channel imaging system needs to be established, including: an additional light source, 1 A beam splitter, an electric rotating machine, n band-pass filters of different bands, a long-pass filter, a silicon sensor camera and a professional short-wave infrared camera; among them, the electric rotating machine is used to ensure The switching of the optical filter will not affect the image alignment. For example, the switching of different optical filters in the same scene, the beam splitter is used to ensure the physical alignment of the silicon sensor camera and the SWIR camera.

Specifically, the hardware of this imaging system includes: 1 additional light source is a halogen lamp, 1 beam splitter is ThorlabsCCN1-BS015, 1 electric rotating machine is Thorlabs FW102C, 5 bandpass filters of different wavelengths are 1000nm, 1050nm, 1100nm, 1150nm and 1200nm CWL, Ednund Hard Coated 0D 4 50nm bandpassFilter, 1 long pass filter for Thorlabs FELH0950, 1 silicon sensor camera GS3-U3-15S5N-C and 1 shortwave infrared camera for BK-51IGA .

After building the multi-channel imaging system, use the multi-channel optical imaging system to collect aligned paired images taken by silicon sensor cameras and professional short-wave infrared cameras in different bands, providing the required information for the short-wave infrared image synthesis method proposed in the embodiment of this application. For the training data set, 2n+2 images need to be collected for each scene, which are sample data, respectively:

1 low-quality infrared band image taken by a silicon sensor camera, denoted as A;

1 high-quality short-wave infrared image taken by a professional short-wave infrared camera, denoted as B;

n images taken by the silicon sensor camera in n narrower different bands, denoted as C;

n images taken by a professional shortwave infrared camera in n narrower different wavebands, denoted as D;

Fig. 4 is a schematic flow chart of a short-wave infrared image processing method based on a silicon sensor camera provided by another embodiment of the present application. As shown in Fig. 4, after obtaining the training sample data, the sample data is trained,

There are three main steps in the training process:

(1) Decomposition stage: decompose the low-quality infrared band image captured by the silicon sensor camera, which is the above-mentioned marked A, into n narrower images taken by the silicon sensor camera in different bands, which is the above-mentioned marked C. Treated as analog physically using a bandpass filter to intercept a specific band signal from a longpass filter signal;

Specifically, the low-quality infrared band image captured by a silicon sensor camera with a long-pass filter is decomposed into 5 images, and each image represents the silicon sensor camera in 5 narrower bands (1000nm, 1050nm, 1100nm, 1150nm and 1200nm) imaging results; wherein, Figure 5a is a schematic diagram of the wavelength corresponding to the original image provided by an embodiment of the present application, as shown in Figure 5a, is the wavelength of a low-quality infrared band image, and Figure 5b is an embodiment of the present application The provided schematic diagram of the wavelengths corresponding to the decomposed sub-images, as shown in Figure 5b, is the wavelength of the decomposed sub-images obtained after decomposing the low-quality infrared band images.

That is to say, at this stage, the input is the low-quality infrared band image taken by the silicon sensor camera, and the output is the simulation result of n narrow images taken by the silicon sensor in different bands. Ideally, the n images should be infinitely close to n images taken with the silicon sensor camera marked C above.

(2) Conversion stage: within n narrower bands, sequentially process the images taken by silicon sensor cameras in n narrower different bands obtained in the decomposition stage, and convert them into images close to those of professional short-wave infrared cameras in this band captured images;

Specifically, learn the mapping relationship between silicon sensor cameras and short-wave infrared cameras in five different bands, and convert images captured by silicon sensor cameras in different bands into images captured by short-wave infrared cameras in corresponding bands;

Input the simulation results of n narrower images taken by silicon sensors in different bands after decomposing, and output the simulation results of n images taken by professional short-wave infrared cameras in corresponding bands. Ideally, the n images should be infinitely close to the above marks It is n images taken by D’s professional short-wave infrared camera, wherein FIG. 5c is a schematic diagram of the wavelength corresponding to the converted sub-image provided by an embodiment of the present application. As shown in FIG. 5c, it is the wavelength of the converted sub-image.

(3) Reconstruction stage: Synthesize the images taken by professional shortwave infrared cameras in n narrower bands obtained in the conversion stage, so as to synthesize images in high-quality shortwave infrared bands similar to those taken by professional shortwave infrared cameras.

Specifically, the high-quality short-wave infrared images captured by professional short-wave infrared cameras are synthesized based on the simulated images of five different short-wave infrared cameras obtained in the conversion stage.

The input is the simulation result of a narrower short-wave infrared camera image of different bands of the converted sub-image, and the output is the simulation result of an image captured by a professional short-wave infrared camera. Ideally, the image should be infinitely close to the above-mentioned marked as B The images taken by professional short-wave infrared addition, wherein, Fig. 5d is a schematic diagram of the wavelength corresponding to the infrared short-wave image provided by an embodiment of the present application. As shown in Fig. 5d, it is the wavelength of the finally obtained short-wave infrared image.

As an implementable manner, on the basis of the foregoing embodiments, optionally, the conversion network model includes n conversion sub-network models, where n is an integer greater than 1;

Using the conversion network model trained in advance, the decomposed sub-image corresponding to each band is converted, and the converted sub-image corresponding to each decomposed sub-image is obtained, including:

The decomposed sub-images corresponding to n different bands are converted by using n pre-trained conversion sub-network models to obtain converted sub-images corresponding to each decomposed sub-image under n different bands.

Specifically, FIG. 6 is a schematic flowchart of a short-wave infrared image processing method based on a silicon sensor camera provided in another embodiment of the present application, as shown in FIG. 6 , specifically:

After decomposing the obtained original image, 5 decomposed sub-images are obtained. In the conversion stage, 5 conversion sub-network models are also included. The specific training process is as follows:

For the first converted sub-network model, the input is the first decomposed sub-image and the second decomposed sub-image, and the output is the first converted sub-image, which gradually approaches the professional infrared short-wave camera in this band through the first converted sub-image The image of the first conversion sub-network model is trained.

For the second converted sub-network model, the input is the first decomposed sub-image, the second decomposed sub-image, the third decomposed sub-image and the first converted sub-image, and the output is the second converted sub-image, passed through the second The converted sub-image gradually approaches the image of the professional infrared short-wave camera in this band, and the second converted sub-network model is trained.

For the third conversion sub-network model, the input is the second decomposed sub-image, the third decomposed sub-image, the fourth decomposed sub-image and the second converted sub-image, and the output is the third converted sub-image, passed through the third The converted sub-image gradually approaches the image of the professional infrared short-wave camera in this band, and the third converted sub-network model is trained.

For the fourth conversion sub-network model, the input is the third decomposed sub-image, the fourth decomposed sub-image, the fifth decomposed sub-image and the third converted sub-image, and the output is the fourth converted sub-image, through the fourth The converted sub-image gradually approaches the image of the professional infrared short-wave camera in this band, and the fourth converted sub-network model is trained.

For the fifth conversion sub-network model, the input is the fourth decomposition sub-image, the fifth decomposition sub-image and the fourth conversion sub-image, and the output is the fifth conversion sub-image, gradually approaching the professional infrared through the fifth conversion sub-image The images of the short-wave camera in the band are used to train the fifth conversion sub-network model.

In this training process, the adjacent images of the image corresponding to the current band and the results after the previous training are used as the input of the next training model, and the output results obtained have higher accuracy and improve the imaging resolution.

As another implementable manner, on the basis of the above-mentioned embodiments, optionally, the conversion network model includes n conversion sub-network models and n residual sub-network models, wherein the conversion sub-network model and the residual sub-network model The difference sub-network models are in one-to-one correspondence, and n is an integer greater than 1.

Optionally, using a pre-trained conversion network model to convert the decomposed sub-images corresponding to each band to obtain converted sub-images corresponding to each decomposed sub-image, including:

Using n pre-trained conversion sub-network models and n residual sub-network models, converting the decomposed sub-images corresponding to n different bands to obtain converted sub-images corresponding to each decomposed sub-image under n different bands .

Specifically, on the basis of the above-mentioned embodiments, a residual sub-network model is added to the conversion network model. FIG. 7 is a schematic flow chart of a short-wave infrared image processing method based on a silicon sensor camera provided by another embodiment of the present application, as shown in FIG. 7, specifically:

For the first converted sub-network model, the input is the first decomposed sub-image and the second decomposed sub-image, which are sequentially passed through the residual sub-network model and the converted sub-network model, and the output is the first converted sub-image, passed through the first The converted sub-image gradually approaches the image of the professional infrared short-wave camera in this band, and the first converted sub-network model is trained.

For the second conversion sub-network model, the first decomposed sub-image, the second decomposed sub-image, and the third decomposed sub-image are input to the residual sub-network model, and the output of the residual sub-network is combined with the obtained first The conversion sub-image is used as the input of the conversion sub-network model, and the output is the second conversion sub-image, and the second conversion sub-image is gradually approaching the image of a professional infrared short-wave camera in this band, and the second conversion sub-network model is trained.

For the third conversion sub-network model, the second decomposed sub-image, the third decomposed sub-image, and the fourth decomposed sub-image are input to the residual sub-network model, and the output of the residual sub-network and the obtained second The conversion sub-image is used as the input of the conversion sub-network model, and the output is the third conversion sub-image, and the third conversion sub-image is gradually approaching the image of a professional infrared short-wave camera in this band, and the third conversion sub-network model is trained.

For the fourth conversion sub-network model, the third decomposed sub-image, the fourth decomposed sub-image, and the fifth decomposed sub-image are input to the residual sub-network model, and the output of the residual sub-network is combined with the obtained third The conversion sub-image is used as the input of the conversion sub-network model, and the output is the fourth conversion sub-image, and the fourth conversion sub-image is gradually approaching the image of a professional infrared short-wave camera in this band, and the fourth conversion sub-network model is trained.

For the fifth transformation subnetwork model, the fourth decomposed subimage and the fifth decomposed subimage are input to the residual subnetwork model, and the output of the residual subnetwork and the obtained fourth transformed subimage are used as the transformation subnetwork The input and output of the model are the fifth converted sub-image, and the fifth converted sub-image is gradually approaching the image of a professional infrared short-wave camera in this band, and the fifth converted sub-network model is trained.

During the training process, the residual sub-network model is added to the conversion network model, and the output result obtained has higher accuracy and improves the imaging resolution.

Specifically, U-Net (conversion sub-network model) and Res-Net (residual sub-network) are used as basic elements of the training network in the embodiment of the present application. For the U-Net module, a convolutional layer with a stride size of 2 is used for downsampling and a linear scaling method is used for upsampling. Set the number of feature map channels in the first layer to 32, and then double the number of feature maps as the size of the feature map decreases. The Res-Net module consists of a convolutional layer, three residual blocks, and a stack of convolutional layers.

For each layer of the Res-Net module, the size of the feature map is kept equal to the input, and the number of feature map channels is set to 32. For all convolutional layers, use ReLU as the activation function, design the convolution kernel size to be 3×3, and use splicing along the dimension instead of direct addition when performing residual transfer.

In the embodiment of the present application, the normalization operation is first performed on the W×H×1 image with a visible light filter, and the image pixel value is scaled from [0,65535] to [0,1], and It serves as input to the decomposition sub-network. The decomposition sub-network contains 5 branches, and each branch is composed of 1 U-Net module, which is used to simulate the effect of different band-pass filters separating the long-pass filter signal.

The conversion sub-network also contains five branches, and each branch is composed of Res-Net and U-Net splicing, so as to convert the signal captured by the silicon sensor camera into the signal captured by the short-wave infrared camera. In addition, the simulation results of the shorter wavelength region are used to guide the simulation of the adjacent longer wavelength region.

The reconstruction network combines the outputs of the transformation sub-networks to generate the final SWIR image. First use the U-Net module to process each conversion branch, then send the processing results and the output of the conversion network to the Res-Net module, and finally use the output of the Res-Net module as the input of the U-Net module, U The output of the -Net module is the final SWIR image, where the number of feature maps of the last layer of the U-Net module is set to 1.

In the embodiment of the present application, training is carried out on NVIDIA GPU 1080Ti, using Kears and TensorFlow as the implementation framework. During the training process, the input image L is decomposed sub-network, converted sub-network and reconstructed sub-network to obtain the short-wave infrared image E. In addition to the In addition to comparing E with the target result G, constraints are also imposed on the results of decomposing sub-networks and transforming sub-networks. Use the mean square error (NSE) and SSIN quality evaluation standard as the Loss function of the network, then use the backpropagation algorithm, and use the Adan optimization method for parameter update and training. The initial learning rate is 0.001, and the number of batch training samples is 30.

Use random clipping and data enhancement to avoid network overfitting, and the size of random clipping is 80×80×1. The training process adopts the learning rate attenuation method. After each epoch, the learning rate is attenuated to 95% of the current learning rate. When the Loss no longer declines, the learning rate is attenuated to 50% of the current learning rate. When the Loss is lower than a certain threshold or the number of iterations reaches the upper limit (this example is set to 200), the training is stopped, the network is considered to be convergent, and the current parameters of the network are maintained.

It should be noted that each implementable manner in this embodiment may be implemented independently, or may be combined in any combination under the condition that there is no conflict, and the present application is not limited thereto.

Yet another embodiment of the present application provides a short-wave infrared image processing device based on a silicon sensor camera, which is used to implement the methods of the above-mentioned embodiments.

FIG. 8 is a schematic structural diagram of a short-wave infrared image processing device based on a silicon sensor camera provided in an embodiment of the present application. As shown in FIG. 8 , the image processing device includes an acquisition module 10, a decomposition module 20, a conversion module 30 and a reconstruction module 40 ;

Wherein, the obtaining module 10 is used to obtain the original image to be processed, wherein the original image is an infrared image including at least two bands;

The decomposition module 20 is used to decompose the original image by using a pre-trained decomposition network model to obtain decomposed sub-images corresponding to each band;

The conversion module 30 is used to convert the decomposed sub-images corresponding to each band by using a pre-trained conversion network model to obtain converted sub-images corresponding to each decomposed sub-image;

The reconstruction module 40 is configured to use a pre-trained reconstruction network model to synthesize the converted sub-images to obtain an infrared short-wave image.

Regarding the apparatus in this embodiment, the specific manner in which each module executes operations has been described in detail in the embodiment of the method, and will not be described in detail here.

According to the day and night general-purpose image processing device based on the silicon sensor camera provided in this embodiment, the short-wave infrared image is obtained by decomposing the network model, converting the network model and reconstructing the network model on the collected original image, which not only saves costs, but also can Improve imaging resolution.

Still another embodiment of the present application provides a further supplementary description of the device provided in the above-mentioned embodiment, which is used to execute the method of the above-mentioned embodiment.

Optionally, the original image is obtained by a silicon sensor camera with a long-pass filter removed and an infrared filter removed.

Optionally, the decomposed network model, the converted network model and the reconstructed network model are obtained by using deep learning neural network calculations.

Optionally, the conversion network model includes n conversion sub-network models, where n is an integer greater than 1;

The conversion module is specifically used for:

The decomposed sub-images corresponding to n different bands are converted by using n pre-trained conversion sub-network models to obtain converted sub-images corresponding to each decomposed sub-image under n different bands.

Optionally, the conversion network model includes n conversion sub-network models and n residual sub-network models, wherein the conversion sub-network model and the residual sub-network model correspond one-to-one, and n is an integer greater than 1.

Optionally, the conversion module is specifically used for:

Using n pre-trained conversion sub-network models and n residual sub-network models, converting the decomposed sub-images corresponding to n different bands to obtain converted sub-images corresponding to each decomposed sub-image under n different bands .

The method or device provided in the embodiments of the present application can be used for agricultural product detection. The principle is to use the different properties of short-wave infrared and visible light to image agricultural products through the peel, so as to find possible quality defects under the intact skin of agricultural products, and then It helps the decision-making system to make more accurate and correct decisions. The application can also be widely used in the fields of circuit board detection, solar cell detection, video surveillance under extreme weather conditions, and the like.

Yet another embodiment of the present application provides a short-wave infrared image processing device based on a silicon sensor camera, which is used to execute the method provided in the foregoing embodiments.

FIG. 9 is a schematic structural diagram of a short-wave infrared image processing device based on a silicon sensor camera provided in this embodiment. As shown in FIG. 9 , the device includes: at least one processor 90 and a memory 91;

The memory stores a computer program; and the at least one processor executes the computer program stored in the memory, so as to implement the methods provided in the foregoing embodiments.

The short-wave infrared image processing device based on the silicon sensor camera of the embodiment of the present application obtains short-wave infrared images by decomposing the network model, converting the network model, and reconstructing the network model for the original image collected, which not only saves costs, but also improves Imaging resolution.

Yet another embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed, the method provided by any of the foregoing embodiments is implemented.

According to the computer-readable storage medium of this embodiment, the short-wave infrared image is obtained by decomposing the network model, transforming the network model and reconstructing the network model on the original collected image, which not only saves cost, but also improves the imaging resolution.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or in the form of hardware plus software functional units.

The above-mentioned integrated units implemented in the form of software functional units may be stored in a computer-readable storage medium. The above-mentioned software functional units are stored in a storage medium, and include several instructions to make a computer device (which may be a personal computer, server, or network device, etc.) or a processor (processor) execute the methods described in various embodiments of the present application. partial steps. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Nenory, RON), random access memory (Randon Access Nenory, RAN), magnetic disk or optical disk, and other media that can store program codes. .

Those skilled in the art can clearly understand that for the convenience and brevity of the description, only the division of the above-mentioned functional modules is used as an example for illustration. The internal structure of the system is divided into different functional modules to complete all or part of the functions described above. For the specific working process of the device described above, reference may be made to the corresponding process in the foregoing method embodiments, and details are not repeated here.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limiting them; although the application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: It is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the various embodiments of the present application. scope.

Claims (8)

1. A short-wave infrared image processing method based on a silicon sensor camera, characterized in that, comprising: Obtaining an original image to be processed, wherein the original image is an infrared image including at least two wavebands, and the original image is obtained by adding a long-pass filter to a silicon sensor camera that removes the infrared filter; Using a pre-trained decomposition network model, the original image is decomposed to obtain decomposed sub-images corresponding to each band, wherein each decomposed sub-image is used to reflect the image taken by a silicon sensor camera in each corresponding band, and the The decomposition network model is obtained by learning the images captured by the silicon sensor camera in each band; Using the pre-trained conversion network model, the decomposed sub-image corresponding to each band is converted to obtain the converted sub-image corresponding to each decomposed sub-image, wherein each converted sub-image is used to reflect the short-wave infrared camera in each corresponding band Taking an image, the conversion network model is obtained by learning the mapping relationship between the shooting image of the silicon sensor camera and the shooting image of the short-wave infrared camera in each band; The converted sub-images are synthesized by using a pre-trained reconstructed network model to obtain an infrared short-wave image. 2. The method according to claim 1, wherein the decomposed network model, the converted network model and the reconstructed network model are obtained by using deep learning neural network calculations. 3. The method according to claim 1, wherein the conversion network model comprises n conversion sub-network models, wherein n is an integer greater than 1; Using the conversion network model trained in advance, the decomposed sub-image corresponding to each band is converted, and the converted sub-image corresponding to each decomposed sub-image is obtained, including: The decomposed sub-images corresponding to n different bands are converted by using n pre-trained conversion sub-network models to obtain converted sub-images corresponding to each decomposed sub-image under n different bands. 4. The method according to claim 1, wherein the conversion network model comprises n conversion sub-network models and n residual sub-network models, wherein the conversion sub-network model and the residual sub-network model are one by one Correspondingly, n is an integer greater than 1. 5. The method according to claim 4, wherein the conversion network model trained in advance is used to convert the decomposed sub-images corresponding to each band to obtain the converted sub-images corresponding to each decomposed sub-image, include: Using n pre-trained conversion sub-network models and n residual sub-network models, converting the decomposed sub-images corresponding to n different bands to obtain converted sub-images corresponding to each decomposed sub-image under n different bands . 6. A short-wave infrared image processing device based on a silicon sensor camera, characterized in that it comprises: An acquisition module, configured to acquire an original image to be processed, wherein the original image is an infrared image including at least two bands, and the original image is obtained by adding a long-pass filter to a silicon sensor camera that removes the infrared filter; The decomposition module is used to decompose the original image by using a pre-trained decomposition network model to obtain decomposed sub-images corresponding to each band, wherein each decomposed sub-image is used to reflect the silicon sensor camera in each corresponding band Taking images, the decomposed network model is obtained by learning the images taken by silicon sensor cameras in each band; The conversion module is used to convert the decomposed sub-images corresponding to each band by using a pre-trained conversion network model to obtain converted sub-images corresponding to each decomposed sub-image, wherein each converted sub-image is used to reflect the short-wave infrared camera In the photographed images of each corresponding band, the conversion network model is obtained by learning the mapping relationship between the photographed images of the silicon sensor camera and the photographed images of the short-wave infrared camera in each band; The reconstruction module is configured to use a pre-trained reconstruction network model to synthesize the converted sub-images to obtain an infrared short-wave image. 7. The device according to claim 6, wherein the decomposed network model, the converted network model and the reconstructed network model are obtained by using deep learning neural network calculations. 8. A short-wave infrared image processing device based on a silicon sensor camera, comprising: a memory and at least one processor; memory; memory for storing said processor-executable instructions; Wherein, the memory stores a computer program; and the at least one processor executes the computer program stored in the memory, so as to realize the method according to any one of claims 1-5.