WO2023109719A1

WO2023109719A1 - Terahertz single-pixel super-resolution imaging method and system

Info

Publication number: WO2023109719A1
Application number: PCT/CN2022/138217
Authority: WO
Inventors: 鲁远甫; 祝永乐; 佘荣斌; 李光元
Original assignee: 深圳先进技术研究院
Priority date: 2021-12-15
Filing date: 2022-12-09
Publication date: 2023-06-22
Also published as: CN114387164A

Abstract

The present invention relates to the field of terahertz imaging. Disclosed are a terahertz single-pixel super-resolution imaging method and system. According to the present invention, stripe patterns needing to be projected in a certain type of patterns are learned a priori in a statistical mode, so that undersampled terahertz single-pixel imaging is achieved, and the purpose of improving an imaging speed is achieved; moreover, in order to compensate for information loss caused by an insufficient sampling rate and interference noise caused by other factors, an image super-resolution network is used for realizing later image denoising and pixel reconstruction, and the high-quality imaging effect is achieved.

Description

A terahertz single-pixel super-resolution imaging method and system

technical field

The invention belongs to the field of terahertz imaging, and more specifically relates to a terahertz single-pixel super-resolution imaging method and system.

Background technique

Terahertz waves have the advantages of low photon energy and penetration of non-polar substances, and have great potential in terahertz imaging, spectral analysis and high-speed communication. Among them, terahertz imaging is of great significance in biomedicine, material detection and safety monitoring. However, due to the lack of suitable materials, the development of terahertz pixelated detector arrays has been slow. At present, most multi-pixel terahertz detector arrays are narrow-band, or need to work in a low-temperature cooling environment, which greatly restricts the practical promotion of terahertz imaging technology.

The current new terahertz imaging technology is a terahertz single-pixel imaging system, which not only saves hardware costs compared to an area-array terahertz imaging system, but also brings new possibilities for the miniaturization and commercialization of terahertz. At present, the terahertz single-pixel imaging system is realized by means of compressed sensing, Hadamard base and Fourier base, etc. Among them, She et al. used 220 μm silicon-based graphene modulator and Fourier fringe to realize subwavelength terahertz image reconstruction. Sparse image characteristics, using low-frequency coefficients and inverse Fourier transform to realize image reconstruction under 10% modulation mask; Rayko et al. used silicon total internal reflection prism and Hadamard mask to realize near real-time terahertz single-pixel video, using Hadam Marfield sparse features use undersampling techniques to reduce sampling time.

Due to the particularity of the wavelength of terahertz waves and the limitations of single-pixel imaging, terahertz single-pixel imaging technology has the following problems to be solved: (1) The imaging speed is slow. Since single-pixel imaging sends a single detector through the modulator to receive the light intensity of several mask patterns, the imaging speed depends on the modulation time, the number of projections, the speed of projections, and the response time of the detector; when the image resolution increases In some cases, the increase in the number of samples will also slow down the imaging speed. (2) Poor imaging quality: Terahertz waves are easily interfered by coherent light during transmission, and the detection error caused by hardware thermal noise greatly affects the imaging quality. In addition, although the under-sampling method can greatly shorten the imaging time, it will cause the loss of high-frequency information of the output image and deteriorate the image; and from an observable point of view, when the sampling rate is lower than a certain ratio, the reconstruction pattern will appear Severe distortion or even distortion, so the sampling rate should not be lower than a certain lower limit.

Contents of the invention

Aiming at the problems of slow imaging speed and poor imaging quality in related technologies, the purpose of the present invention is to provide a terahertz single-pixel super-resolution imaging method and system, which can learn a priori the fringe pattern that needs to be projected for a certain type of pattern through a statistical method , realize under-sampled terahertz single-pixel imaging, and achieve the purpose of improving the imaging speed; at the same time, in order to compensate for the information loss caused by insufficient sampling rate and the interference noise caused by other factors, the image super-resolution network is used to realize the later image Denoising and pixel reconstruction to achieve high-quality imaging.

In order to achieve the above object, one aspect of the present invention provides a terahertz single-pixel super-resolution imaging method, comprising the following steps:

S100. Perform prior learning on the encoding positions of similar patterns, extract the sampling strategy saved through prior learning, and generate mask patterns, the number of the mask patterns is equal to the sampling times and sampling rate of the pixel image in single-pixel imaging reconstruction the product of

S200. Based on the sampling strategy, select a data set to perform corresponding processing to generate a training set, and use the training set to initialize the residual dense network;

S300. Build a terahertz single-pixel imaging system, according to the sampling strategy, use a digital micromirror array to load the mask pattern, and reflect it onto the target pattern through laser light; the terahertz wave is irradiated to the terahertz modulator to modulate the target pattern The laser intensity is , and the terahertz wave is periodically received by the terahertz detector, and the transform domain information is restored from the intensity information and decoded to obtain the first image;

S400. Based on the first image, reconstruct a second image through the residual dense network.

Further, the step S100 specifically includes the following steps:

S101. Classify pictures similar to the collected target patterns to form a data set;

S102. Perform full sampling on multiple pictures of the same type and save the transform domain data;

S103. Using a statistical method to count the intensity of modulus coefficients in the transformation domain of this type of image satisfying the corresponding transformation domain position at a specific sampling rate, so as to generate a mask pattern for this type of image.

Further, in the step S102, the Fourier domain inverse transform is adopted in the transform domain.

Further, the step S400 specifically includes the following steps:

S401. The input image extracts the shallow information feature map through the convolution layer, and then enters the residual dense block;

S402. Each residual dense block will generate a corresponding feature map, splicing all the feature maps to form a whole, and performing dense feature fusion;

S403. Use the up-sampling module to up-interpolate the first image to the second image with high pixels and reconstruct it to restore a single-channel high-definition grayscale image or a three-channel high-definition color image.

Further, the first image is an image of 32×32 pixels, and the second image is an image of 64×64 pixels.

Further, the process of dense feature fusion includes global feature fusion and global residual learning. Firstly, the fusion layer is used to combine each feature map into a whole, and further obtain deep information through convolution operation and restore it into a 64-channel feature map. Finally, the shallow information is added to the deep information through residual learning.

Another aspect of the present invention also provides a terahertz single-pixel super-resolution imaging system, including:

The learning unit performs prior learning on the coding positions of similar patterns, extracts the sampling strategy saved by prior learning, and generates mask patterns, the number of the mask patterns is equal to the sampling times and sampling rate of the pixel image in single-pixel imaging reconstruction the product of

The training unit selects a corresponding data set for image scaling, and simultaneously selects corresponding coefficients in the processed image transformation domain for inverse transformation based on the sampling strategy to form a training set, and uses the training set to initialize the residual dense network;

The acquisition unit uses a digital micromirror array to load a priori learned mask pattern, and reflects it onto the target pattern through laser light; the terahertz wave is irradiated to the terahertz modulator to modulate the laser intensity of the target pattern, and is cycled by the terahertz detector Receive the terahertz wave selectively, restore the transform domain information and decode it to obtain the first image;

A reconstruction unit, based on the first image, reconstructs a second image through the residual dense network.

Further, the learning unit includes:

The classification module is used to classify pictures similar to the collected target patterns to form a data set;

The full sampling module performs full sampling on multiple images of the same type and saves the transform domain data;

The generating module adopts a statistical method to count the intensity of the modulus in the transformation domain of this type of image to meet the corresponding transformation domain position at a specific sampling rate, thereby generating the mask pattern of this type of image.

Further, the reconstruction unit includes:

Convolution module, the input image extracts the shallow information feature map through the convolution layer, and then enters the residual dense block;

In the fusion module, each residual dense block will generate a corresponding feature map, and all the feature maps will be stitched together to form a whole for dense feature fusion;

The interpolation module uses the up-sampling module to up-interpolate the first image to the second image with high pixels and reconstruct it to restore a single-channel high-definition grayscale image or a three-channel high-definition color image.

Through the above technical solutions conceived by the present invention, compared with the prior art, the prior optimal sampling scheme and the super-resolution reconstruction of deep learning are combined into the terahertz single-pixel imaging system to achieve high-definition images within the fastest imaging time Image reconstruction. Specifically, the present invention adopts a computer-aided optimized sampler based on prior statistics, which is suitable for imaging and monitoring common similar targets, and is superior to most under-sampling schemes; at the same time, in order to improve the generalization of this class In the process of expanding and collecting the following samples, the sampling plan can be further counted and optimized, so this is a sampling method that can be learned; in addition, the present invention applies the residual dense network to terahertz imaging, not only The image noise caused by the system is eliminated, and the resolution is improved without increasing the number of samples.

Description of drawings

Fig. 1 is a schematic diagram of a terahertz single-pixel imaging system in an embodiment of the present invention;

Figure 2 (a) is a schematic diagram of the RDN network structure in the embodiment of the present invention;

Fig. 2 (b) is the schematic diagram of RDB network structure in the embodiment of the present invention;

Fig. 3 is a working flow chart of the fast terahertz single-pixel super-resolution reconstruction system according to the embodiment of the present invention;

Fig. 4 is an effect diagram of samples in the embodiment of the present invention under different sampling methods at 8% and 10%.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not constitute a conflict with each other.

First, build a terahertz single-pixel imaging device, as shown in Figure 1. The main optical path of the imaging device is composed of a terahertz laser and a detector, and the modulation part is composed of a laser, a digital mirror array (Digital Mirrors Device, DMD), a projection lens and an intrinsic semiconductor. The laser is masked with DMD and projected on the modulator through the lens through indium tin oxide (ITO) glass. ITO can transmit visible light and reflect terahertz light. The terahertz light is collimated by the lens and penetrates the modulator and the target object X is focused on the detector by the lens, the terahertz light is modulated by the mask, and the modulated intensity signal I is output on the detector.

Based on the above device, an embodiment of the present invention provides a terahertz single-pixel super-resolution imaging method, including the following steps:

S100. Perform prior learning on the encoding positions of similar patterns, extract the sampling strategy saved through prior learning, and generate mask patterns, the number of the mask patterns is equal to the sampling times and sampling rate of the pixel image in single-pixel imaging reconstruction product of .

Any two-dimensional image can be regarded as obtained by weighting a set of complete orthogonal mask patterns, each mask pattern corresponds to a frequency point on the transform domain, and the relationship between the target pattern and the transform domain function is given by the formula ( 1) Give.

In the formula, I(x, y) is the object target, M, N are the length and width of the object target, u, v are the point coordinates of the frequency on the transform domain, f is a two-dimensional matrix function, and the size is determined by (x, y, u, v) is determined, and a _uv is the weight size, which is uniquely determined by (u, v). The process of weighting all orthogonal base patterns to obtain the original pattern is called full sampling, where the number of measurements is equal to K=MxN; in order to reduce the number of measurements, some coefficients on the transform domain are collected for weighted inversion, and the information missing The way the pattern is sampled is called undersampling. Among them, prior statistical sampling is given by formula (2).

In the formula, |F(x,y)| represents the modulus of a certain point on the transform domain, p is the sampling rate, and a is the absolute value of the corresponding threshold value of p, indicating that the coefficient at the selected position in the transform domain accounts for the entire transform domain coefficient modulus The earlier distribution is with probability p.

Wherein, the step S100 specifically includes the following steps:

S200. Based on the sampling strategy, select a data set and perform corresponding processing to generate a training set, and use the training set to initialize a residual dense network.

This method is suitable for the imaging and monitoring of commonly used similar targets, and is superior to most under-sampling schemes; at the same time, in order to improve the generalization degree of this class, further statistics and Optimizing the sampling scheme, so it's a learned way of sampling. Therefore, in order to increase the sampling speed, the terahertz single-pixel imaging system adopts the prior statistical undersampling scheme in the present invention, and adopts relatively mature Fourier domain inverse transform in the transform domain.

S400. Based on the 32×32 pixel image, reconstruct a 64×64 pixel image through a residual dense network.

In order to reconstruct high-definition images from under-sampled images with missing information, further processing of imaging results is required. The residual dense network can extract feature information from the shallow and deep parts of the image for adaptive feature fusion, and use the upsampling layer for interpolation to obtain high-definition images. The network structure of RDN is shown in Figure 2(a), and the structure is divided into: two ConV layers for extracting shallow features; Residual Dense Block (RDB) for extracting features of each layer; for splicing Dense Feature Fusion (DFF) of the features of each layer; used to obtain a single-channel or multi-channel upsampling module. The input image passes through two ConV layers with a convolution kernel size of 3x3 and filters of 1 and 64 to extract shallow information feature maps, and then enters RDBs to obtain deep information. RDBs are composed of multiple RDBs. The network structure of RDBs is given in Figure 2(b). Each RDB corresponds to a different receptive field and can extract different local features, which are given by the formula:

From the perspective of network structure, RDB consists of residual blocks and dense blocks, the filter size is 64, and the convolution kernel size is 3x3. RDB has the following characteristics: continuous memory mechanism, local fusion mechanism and local residual mechanism. Continuous memory mechanism, each RDB transmits several previous states to the current layer for processing, and each layer is composed of a ConV layer and a ReLU layer; the later the number of layers, the more memory states are transmitted. The local fusion mechanism connects the transmitted state and the current input in series, which can adaptively extract deep features from multiple features and reduce the number of channels and outputs. Local residual learning superimposes the output of the previous RDB and the feature fusion part to further improve the information flow and improve the expressive ability of the model.

After passing through a series of RDBs, each RDB will generate a corresponding feature map, and all the feature maps are stitched together to form a whole, which is called dense feature fusion. Dense feature fusion is divided into global feature fusion and global residual learning. First, the fusion layer is used to combine each feature map into a whole, and a 1x1 convolution operation and a 3x3 convolution operation are used to further obtain deep information and restore it to 64 channels. The feature map, and finally add the shallow information and deep information through residual learning, and the entire dense feature fusion can be expressed as:

F _DF ＝F ₀ +H _GEF ([F ₀ ,F _d ,...,F _D ]) (4)

Finally, through the upsampling module, the 32x32 pixel image is interpolated to 64x64 pixels and reconstructed, and restored to a single-channel high-definition grayscale image or a three-channel high-definition color image. The entire RDN network can be expressed as:

In order to reconstruct the target image

The loss function of network training adopts L1loss function, which can be expressed as:

The step S400 specifically includes the following steps:

Another aspect of the embodiments of the present invention also provides a terahertz single-pixel super-resolution imaging system, including:

For the functions of the above units, please refer to the corresponding methods, which will not be repeated here.

The flow chart of the imaging method of the present invention is shown in Figure 3. First, the sampling strategy of the data set similar to the target pattern is learned by the computer prior, and a corresponding mask pattern is generated. At the sampling rate (the sampling rate of this experiment is 8%); select the corresponding data set for image scaling, and at the same time, according to the sampling strategy of prior learning, select the corresponding coefficients of the processed image transformation domain for inverse transformation, and form a training set to initialize Residual dense network; build a terahertz single-pixel imaging system, and load a mask pattern on the DMD; secondly, using the principle of near-field imaging, the target image is placed at a certain distance from the DMD, close to the terahertz modulator, and the 808nm laser Reflect the mask pattern on the DMD to the target, and modulate the laser intensity of the target pattern by the terahertz light irradiated on the terahertz modulator; the light intensity value of each sampling count is collected by the detector, and restored and transformed by the computer The domain information is decoded into a 32x32 pixel image, and the image will have low resolution due to scattering. Finally, the low-resolution image and the high-resolution image are mapped through the residual dense network to restore a 64x64 pixel high-definition image, realizing the effect of undersampling high-resolution image reconstruction.

The following experiments are used to compare the effects of samples under different sampling methods. In order to retain most of the information as much as possible and avoid too much loss of details, this experiment samples the 32x32 pixel sample pattern at a sampling rate of 8%, and introduces the effect picture at a sampling rate of 10% and the corresponding optimal sampling method as Compared. In Figure 4 and Table 1, the corresponding sampling methods are circle, square, square, prior statistics and optimal sampling. The present invention adopts structural similarity (Structural SIMilarity, SSIM) and peak signal to noise ratio (Peak Signal to Noise Ratio, PSNR) two evaluation indexes. It can be seen from Table 1 that the prior statistics of the 8% sampling rate are similar to the 10% block mode, better than the other two sampling methods, and slightly weaker than the optimal sampling of the same sampling.

Table 1 Comparison of the effects of different sampling methods for samples at 8% and 10%

It is easy for those skilled in the art to understand that the above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention, All should be included within the protection scope of the present invention.

Claims

A terahertz single-pixel super-resolution imaging method, characterized in that it comprises the following steps:

S100. Perform prior learning on the encoding positions of similar patterns, extract the sampling strategy saved through prior learning, and generate mask patterns, the number of the mask patterns is equal to the sampling times and sampling rate of the pixel image in single-pixel imaging reconstruction the product of

S200. Based on the sampling strategy, select a data set to perform corresponding processing to generate a training set, and use the training set to initialize the residual dense network;

S300. Build a terahertz single-pixel imaging system, according to the sampling strategy, use a digital micromirror array to load the mask pattern, and reflect it onto the target pattern through laser light; the terahertz wave is irradiated to the terahertz modulator to modulate the target pattern The laser intensity is , and the terahertz wave is periodically received by the terahertz detector, and the transform domain information is restored from the intensity information and decoded to obtain the first image;

S400. Based on the first image, reconstruct a second image through the residual dense network.
The terahertz single-pixel super-resolution imaging method according to claim 1, wherein the step S100 specifically comprises the following steps:

S101. Classify pictures similar to the collected target patterns to form a data set;

S102. Perform full sampling on multiple pictures of the same type and save the transform domain data;

S103. Using a statistical method to count the intensity of modulus coefficients in the transformation domain of this type of image satisfying the corresponding transformation domain position at a specific sampling rate, so as to generate a mask pattern for this type of image.
The terahertz single-pixel super-resolution imaging method according to claim 2, characterized in that, in the step S102, inverse Fourier domain transform is used in the transform domain.
The terahertz single-pixel super-resolution imaging method according to claim 1, wherein the step S400 specifically includes the following steps:

S401. The input image extracts the shallow information feature map through the convolution layer, and then enters the residual dense block;

S402. Each residual dense block will generate a corresponding feature map, stitching all the feature maps together to form a whole, and performing dense feature fusion;

S403. Use the up-sampling module to up-interpolate the first image to the second image with high pixels and reconstruct it to restore a single-channel high-definition grayscale image or a three-channel high-definition color image.
The terahertz single-pixel super-resolution imaging method according to claim 4, wherein the first image is an image of 32×32 pixels, and the second image is an image of 64×64 pixels.
The terahertz single-pixel super-resolution imaging method according to claim 5, wherein the process of said dense feature fusion includes global feature fusion and global residual learning, and first uses the fusion layer to combine each feature map into a whole, and The deep information is further obtained through the convolution operation and restored to a 64-channel feature map, and finally the shallow information is added to the deep information through residual learning.
A terahertz single-pixel super-resolution imaging system, characterized in that it includes:

The learning unit performs prior learning on the coding positions of similar patterns, extracts the sampling strategy saved by prior learning, and generates mask patterns, the number of the mask patterns is equal to the sampling times and sampling rate of the pixel image in single-pixel imaging reconstruction the product of

The training unit selects a corresponding data set for image scaling, and simultaneously selects corresponding coefficients in the processed image transformation domain for inverse transformation based on the sampling strategy to form a training set, and uses the training set to initialize the residual dense network;

The acquisition unit uses a digital micromirror array to load a priori learned mask pattern, and reflects it onto the target pattern through laser light; the terahertz wave is irradiated to the terahertz modulator to modulate the laser intensity of the target pattern, and is cycled by the terahertz detector Receive the terahertz wave selectively, restore the transform domain information and decode it to obtain the first image;

A reconstruction unit, based on the first image, reconstructs a second image through the residual dense network.
The terahertz single-pixel super-resolution imaging system according to claim 7, wherein the learning unit comprises:

The classification module is used to classify pictures similar to the collected target patterns to form a data set;

The full sampling module performs full sampling on multiple images of the same type and saves the transform domain data;

The generation module adopts a statistical method to count the intensity of the modulus in the transformation domain of this type of image to meet the corresponding transformation domain position at a specific sampling rate, thereby generating the mask pattern of this type of image.
The terahertz single-pixel super-resolution imaging system according to claim 7, wherein the reconstruction unit comprises:

Convolution module, the input image extracts the shallow information feature map through the convolution layer, and then enters the residual dense block;

In the fusion module, each residual dense block will generate a corresponding feature map, and all the feature maps will be stitched together to form a whole for dense feature fusion;

The interpolation module uses the up-sampling module to up-interpolate the first image to the second image with high pixels and reconstruct it to restore a single-channel high-definition grayscale image or a three-channel high-definition color image.
The terahertz single-pixel super-resolution imaging system according to claim 9, wherein the first image is an image of 32×32 pixels, and the second image is an image of 64×64 pixels.