CN109191376B - High-resolution terahertz image reconstruction method based on improved SRCNN model - Google Patents

Abstract

The embodiment of the present invention discloses a high-resolution terahertz image reconstruction method based on an improved SRCNN model, including: based on the SRCNN model structure, constructing an improved SRCNN model structure for double-layer feature extraction, after double-layer feature extraction Perform interpolation and amplification, and then add pooling; each layer of the improved SRCNN model structure undergoes the third layer of convolution in turn, and the third convolution is regarded as a non-linear mapping of the fully connected layer; after the fourth layer, the final High resolution images. The present invention improves the problem that the existing SRCNN model needs a lot of time when training the model and the image resolution is low after reconstruction, and obtains the final model and optimal parameters by continuously modifying the weight and other parameters inside the model until the insertion loss value reaches a very small value. The improved SRCNN four-layer convolutional neural network model designed by the present invention is trained, and the PSNR amount is increased by 2.4dB compared with the result processed by the SRCNN model of the prior art.

Description

High-resolution terahertz image reconstruction method based on improved SRCNN model

technical field

The invention relates to the technical field of image data processing, in particular to a high-resolution terahertz image reconstruction method based on an improved SRCNN model.

Background technique

In the existing technology, there are two main methods to solve the problem of terahertz image quality: ① start from the source of terahertz imaging, improve the imaging resolution of terahertz imaging equipment, and reduce the noise in the imaging process; ② from the obtained Start to process terahertz images, optimize and improve algorithms to process terahertz images.

The advantage of the first method is that it solves the root of the problem, but the cost of equipment is high and the cost performance is low; the second method uses digital image processing algorithms to realize the related processing and application of terahertz images, but there is a problem of low resolution of terahertz images question.

The SRCNN model is a model designed by Tang Xiaoou's team. This model is based on a convolutional neural network to achieve image reconstruction. This model only uses a three-layer neural network structure, representing the steps of traditional sparse representation of image reconstruction. . But the existing SRCNN model does not deal with the sharpness of the image enough.

Therefore, the prior art needs to be improved.

Contents of the invention

A technical problem to be solved by the embodiments of the present invention is to provide a high-resolution terahertz image reconstruction method based on the improved SRCNN model to solve the problems existing in the prior art. The high-resolution terahertz image reconstruction method based on the improved SRCNN model Hertzian image reconstruction methods include:

Based on the model structure of SRCNN, an improved SRCNN model structure for double-layer feature extraction is constructed, and interpolation and amplification are performed after double-layer feature extraction, and then pooling is performed;

Each layer of the model structure of the improved SRCNN undergoes the third layer of convolution in turn, which is regarded as a non-linear mapping of the fully connected layer;

The final high-resolution image is reconstructed through the fourth layer.

In another embodiment of the above-mentioned high-resolution terahertz image reconstruction method based on the improved SRCNN model of the present invention, the model structure based on the SRCNN is to construct a two-layer feature extraction improved SRCNN model structure, in the two-layer After feature extraction, interpolation and amplification are performed, and then pooling is added to include:

After the target is enlarged, the input image is Y, and X is the original image. The first layer of convolutional network calculates the output image F ₁ as shown in formula (1):

F ₁ (Y)=max(0,W ₁ *Y+B ₁ ) (1)

In the formula, B ₁ represents the weight bias, * is the convolution operation, W ₁ is used to represent n ₁ c×f ₁ ×f ₂ filters, where c represents the number of channels of the image, and the color image is 3, The grayscale image is 1, and B ₁ represents n ₁ offsets;

The second layer of convolutional neural network is mapped to n ₂ feature maps based on the n ₁ feature maps generated by the first layer. The calculation formula is as follows:

F ₂ (Y)＝max(0,W ₂ *F ₁ (Y)+B ₂ ) (2)

In the formula, B ₂ represents the weight bias, * represents the convolution operation, W ₂ is used to represent n ₂ n ₁ ×f ₂ ×f ₂ filters, B ₂ represents n ₂ offsets;

The third layer reconstructs the image, and uses formula (3) to establish the reconstruction layer:

F(Y)＝W ₃ *F ₂ (Y)+B ₃ (3)

In the formula, W ₃ is used to represent c filters of n ₂ ×f ₃ ×f ₃ , and B ₃ represents c offsets.

In another embodiment of the high-resolution terahertz image reconstruction method based on the improved SRCNN model of the present invention, the input image after the set target is enlarged is Y, X is the original image, and the first layer of convolutional network The output image F ₁ is calculated as in formula (1), and the parameters are f ₁ =9, n ₁ =64.

In another embodiment of the high-resolution terahertz image reconstruction method based on the improved SRCNN model of the present invention, the second layer of convolutional neural network is based on the _n1 feature maps generated by the first layer through Mapping into n ₂ feature maps, the calculation formula is as in formula (2), and the parameters are f ₂ =1, n ₂ =32.

In another embodiment of the above-mentioned high-resolution terahertz image reconstruction method based on the improved SRCNN model of the present invention, the third layer performs image reconstruction, and the reconstruction layer is established using formula (3), and the parameter is c= 3, f ₃ =5.

In another embodiment of the above-mentioned high-resolution terahertz image reconstruction method based on the improved SRCNN model of the present invention, the reconstruction of the final high-resolution image through the fourth layer includes:

Convert the original color image into a grayscale image, and make a low-resolution image by reducing and enlarging the original image;

Multiple images in the training set are overlapped and segmented according to the set step size to obtain multiple sub-pictures;

Convert the training set and test set into H5py format for storage;

Import the necessary toolkits and datasets in Tensorflow;

Define the initialization function, randomly create noise for the weights to eliminate complete symmetry, and set the standard deviation;

Define the loss function to end the training results, the optimizer chooses Adam, and gives a very small learning rate;

Start the training process, initialize all parameters, set the batch_size (batch size), the number of iterations, the training output loss per set number of cycles, and evaluate the performance of the model in real time;

Reconstruct the sub-picture into a whole picture as the high-resolution output picture after reconstruction.

In another embodiment of the high-resolution terahertz image reconstruction method based on the improved SRCNN model of the present invention, the value of the set standard deviation is: 0.01.

In another embodiment of the high-resolution terahertz image reconstruction method based on the improved SRCNN model of the present invention, the value of giving a very small learning rate is 0.000001.

In another embodiment of the high-resolution terahertz image reconstruction method based on the above-mentioned SRCNN improved model of the present invention, the set batch_size size, the number of iterations, and the values of the training output loss per set number of cycles are respectively: batch_size The size is 128, the number of iterations is 100000, and the number of cycles is set to 1000.

Compared with the prior art, the present invention has the following advantages:

The present invention proposes a high-resolution terahertz image reconstruction method based on the improved SRCNN model, which improves the problem that the existing SRCNN model requires a lot of time when training the model, and continuously modifies the weight and other parameters inside the model To obtain the final model and optimal parameters until the interpolation loss value reaches a small value. The improved SRCNN model of the four-layer convolutional neural network designed in the present invention is trained, and the PSNR amount of the SRCNN model of the prior art is improved by 2.4dB.

The technical solutions of the present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

Description of drawings

The accompanying drawings, which constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain principles of the invention.

The present invention can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

Fig. 1 is the flowchart of an embodiment of the high-resolution terahertz image reconstruction method based on the SRCNN improved model of the present invention;

Fig. 2 is a flow chart of another embodiment of the high-resolution terahertz image reconstruction method based on the improved SRCNN model of the present invention.

Detailed ways

Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangements of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

At the same time, it should be understood that, for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

The following description of at least one exemplary embodiment is merely illustrative in nature and in no way taken as limiting the invention, its application or uses.

Techniques, methods and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods and devices should be considered part of the description.

As shown in Figure 1, the high-resolution terahertz image reconstruction method based on the improved SRCNN model includes:

10. Based on the model structure of SRCNN, an improved SRCNN model structure for double-layer feature extraction is constructed. After double-layer feature extraction, interpolation and amplification are performed, and then pooling is performed; by increasing the number of layers of the model, it is only Compared with the feature extraction of one layer, the feature extraction layer of one layer will introduce noise and the information extraction is incomplete, so the present invention uses two-layer feature extraction. Compared with the improved SRCNN model, the difference is not amplified, and the pooling is directly performed. Improved image quality and resolution;

20. For the model structure of the improved SRCNN, each layer undergoes the third layer of convolution in turn, which is regarded as a non-linear mapping of the fully connected layer;

30. Reconstruct the final high-resolution image through the fourth layer.

The SRCNN improved model can see the end-to-end mapping more intuitively, and no pooling layer is added to the improved model. The pooling layer reduces the size of the picture, thereby reducing parameters, reducing the amount of calculation, and improving the training speed, but the book The invention is mainly to achieve high-resolution reconstruction, so when the image is reduced after pooling, the main information of the picture may be lost. Therefore, the improved SRCNN model of the present invention performs interpolation and amplification after double-layer feature extraction, and then adds pooling. The pooling added at the time is not to reduce the size of the image in the prior art. On the contrary, it is to enlarge the size of the image, so that the final output image size is as close as possible to the input image size, avoiding the loss of main information of the image.

Described based on the model structure of SRCNN, construct the model structure of the improved SRCNN of double-layer feature extraction, perform interpolation amplification after double-layer feature extraction, add pooling and include:

After the target is enlarged, the input image is Y, and X is the original image. The first layer of convolutional network calculates the output image F ₁ as shown in formula (1):

F ₁ (Y)=max(0,W ₁ *Y+B ₁ ) (1)

In the formula, W ₁ and B ₁ represent the filter and weight bias, * is the convolution operation, W ₁ is used to represent n ₁ c×f ₁ ×f ₂ filters, where c represents the number of channels of the image, The color image is 3, the grayscale image is 1, and B ₁ represents n ₁ offsets;

The second layer of convolutional neural network is mapped to n ₂ feature maps based on the n ₁ feature maps generated by the first layer. The calculation formula is as follows:

F ₂ (Y)＝max(0,W ₂ *F ₁ (Y)+B ₂ ) (2)

In the formula, W ₂ and B ₂ represent the filter and weight bias, * represents the convolution operation, W ₂ is used to represent n ₂ n ₁ × f ₂ × f ₂ filters, B ₂ represents n ₂ offsets ;

The third layer reconstructs the image, and uses formula (3) to establish the reconstruction layer:

F(Y)＝W ₃ *F ₂ (Y)+B ₃ (3)

In the formula, W ₃ is used to represent c filters of n ₂ ×f ₃ ×f ₃ , and B ₃ represents c offsets.

After the set target is enlarged, the input image is Y, and X is the original image. The first layer of convolutional network calculates the output image F ₁ as in formula (1), and the parameters are f ₁ =9, n ₁ =64.

The second layer of convolutional neural network becomes n ₂ feature maps by mapping on the basis of n ₁ feature maps generated by the first layer, the calculation formula is as in formula (2), and the parameters are f ₂ =1, n ₂ =32.

The third layer performs image reconstruction, and the reconstruction layer is established by formula (3), and the parameters are c=3, f ₃ =5.

The relevant initial parameters used in the following embodiments include: the data set contains a total of 91 training set images and 19 test set images, including 5 Set5 and 14 Set14 data sets.

As shown in Figure 2, the reconstruction of the final high-resolution image through the fourth layer includes:

101. Convert the original color image into a grayscale image, and make a low-resolution image by reducing and enlarging the original image. The implementation procedure includes the following:

image=cv2.cvtColor(image,cv2.COLOR_BGR2YCrCb)

image=image[:,:,0:3]

im_label = modcrop(image, scale)

(hei,wid,_)=im_label.shape

im_input=cv2.resize(im_label,(0,0), fx=1.0/scale, fy=1.0/scale, interpolation=cv2.INTER_CUBIC)

im_input=cv2.resize(im_input,(0,0), fx=scale, fy=scale, interpolation=cv2.

INTER_CUBIC)

102. Overlap and segment multiple images in the training set according to the set step size to obtain multiple sub-pictures. For example:

The 91 images in the training set are overlapped and segmented, the step size is 14, and the size of the divided sub-pictures is 33*33, so that 21884 sub-pictures can be obtained. The specific implementation procedure is as follows:

for x in range(0,hei-size_input+1,stride):

for y in range(0,wid-size_input+1,stride):

subim_input=im_input[x:x+size_input,y:y+size_input,0:3]

subim_label＝im_label[x+padding:x+padding+size_label,y+padding:y+padding g+size_label,0:3]

subim_input = subim_input.reshape([size_input, size_input, 3])

subim_label = subim_label.reshape([size_label, size_label, 3])

103. Convert the training set and test set into H5py format for storage to solve the problem of large memory usage and slow reading. The implementation procedure is as follows:

with h5py.File(savepath,'w') as hf:

hf.create_dataset('test_input', data=im_input)

hf.create_dataset('test_label', data=im_label)

with h5py.File(savepath,'w') as hf:

hf.create_dataset('input', data=data)

hf.create_dataset('label', data=label)

104. Import the necessary toolkit and data set in Tensorflow, and its implementation procedure is as follows:

import tensorflow as tf

import h5py

import numpy as np

import matplotlib.pyplot as plt

import string

import cv2

with h5py.File('train_py.h5','r') as hf:

hf_data = hf.get('input')

data = np.array(hf_data)

hf_label = hf.get('label')

label = np.array(hf_label)

with h5py.File('test_py.h5','r') as hf:

hf_test_data = hf.get('test_input')

test_data = np.array(hf_test_data)

hf_test_label = hf.get('test_label')

test_label=np.array(hf_test_label)

105. Define the initialization function, randomly create noise for the weights to eliminate complete symmetry, and set the standard deviation; import data and the end of the toolkit can be realized. This convolutional neural network requires a large number of weights and biases. In a specific embodiment , the value of the set standard deviation is: 0.01. The procedure for setting weights and biases is as follows:

def init_weights(shape):

return tf.Variable(tf.random_normal(shape,stddev=0.01))

W1=init_weights([9,9,3,64])

W2=init_weights([3,3,64,32])

W3=init_weights([1,1,32,16])

W4=init_weights([5,5,16,3])

B1=tf.Variable(tf.zeros([64]),name="Bias1")

B2=tf.Variable(tf.zeros([32]),name="Bias2")

B3=tf.Variable(tf.zeros([16]),name="Bias3")

B4＝tf.Variable(tf.zeros([3]),name="Bias4")

tf.nn.conv2d is the convolution function in TensorFlow. Among the parameters, X represents the input, W represents the weight of the volume parameter, Strides represents the step size of the convolution template, Padding is the boundary processing method, and SAME represents filling the boundary with zeros. Make the convolution output and input have the same size, VALID means not to process the boundary, so that the image will become smaller after convolution. The defined variable before inputting data can use a placeholder to replace the input. The input amount is X, and the added label (lable) is Y. The implementation procedure is as follows:

X＝tf.placeholder("float32",[None,33,33,3])

Y＝tf.placeholder("float32",[None,19,19,3])

L1=tf.nn.relu(tf.nn.conv2d(X,W1,strides=[1,1,1,1],padding='VALID')+B1)

L2=tf.nn.relu(tf.nn.conv2d(L1,W2,strides=[1,1,1,1],padding='VALID')+B2)

L3=tf.nn.relu(tf.nn.conv2d(L2,W3,strides=[1,1,1,1],padding='VALID')+B3)

hypothesis=tf.nn.conv2d(L3, W4, strides=[1,1,1,1], padding='VALID')+B4

106. Define a loss function to end the training result. The optimizer selects Adam and gives a very small learning rate; in a specific embodiment, the value of giving a very small learning rate is 0.000001. The implementation procedure is as follows:

cost＝tf.reduce_mean(tf.reduce_sum(tf.square((Y-subim_input)-hypothesis),

reduction_indices=0))

var_list=[W1,W2,W3,W4,B1,B2,B3,B4]

optimizer = tf.train.AdamOptimizer(learning_rate).minimize(cost, var_list = var_list)

107. Start the training process, initialize all parameters, set the batch_size size, the number of iterations, the training output loss per set number of cycles, and evaluate the performance of the model in real time; in a specific embodiment, the set batch_size size, the number of iterations, The values of the training output loss per set number of cycles are: the batch_size is 128, the number of iterations is 100,000, and the number of set cycles is 1000. Its implementation procedure is as follows:

with tf.Session() as sess:

tf.initialize_all_variables().run()

for i in range(train_num):

batch_data, batch_label = batch.__next__()

sess.run(optimizer, feed_dict={X: batch_data, Y: batch_label})

step+=1

if(epoch_number+step)%1000==0:

print_step=(epoch_number+step)

epoch_cost_string="[epoch]:"+(str)(print_step)+"[cost]:"

current_cost_sum = 0.0

mean_batch_size=(int)((data.shape[0]/128))

for j in range(0, mean_batch_size):

current_cost_sum+=sess.run(cost.feeed_dict={X:data[j](1,33,33,3)}),

Y:lable[j].reshape(1,19,19,3)})

epoch_cost_string+=str(float(current_cost_sum/mean_batch_size))

epoch_cost_string+="\n"

print(epoch_cost_string)

108. Reconstruct the sub-pictures into a whole picture as a reconstructed high-resolution output picture. The implementation procedure is as follows:

with tf.Session() as sess:

if(epoch_number+step)%1000==0:

test_L1=tf.nn.relu(tf.nn.conv2d(test_data,W1,strides=[1,1,1,1],padding='SAM E')+B1)

test_L2=tf.nn.relu(tf.nn.conv2d(test_L1,W2,strides=[1,1,1,1],padding='SAME')+B2)

test_L3=tf.nn.relu(tf.nn.conv2d(test_L1,W2,strides=[1,1,1,1],padding='SAME')+B3)

test_hypothesis=tf.nn.conv2d(test_L2, W3, strides=[1,1,1,1], padding='SAME')+B3)

output_image=sess.run(test_hypothesis)[0,:,:,0:3]

output_image+＝test_data[0,:,:,0:3]

for k in range(0,test_data.shape[1]):

for j in range(0,test_data.shape[2]):

for c in range(0,3):

if(output_image[k,j,c]>1.0): output_image[k,j,c]=1;

elif(output_image[k,j,c]<0): output_image[k,j,c]=0;

temp_image=(output_image*255).astype('uint8')

temp_image=cv2.cvtColor(temp_image,cv2.COLOR_YCrCb2RGB)

plt.imshow(temp_image)

subname="shot/"+str(epoch_number+step)+".jpg"

plt. savefig(subname)

Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same or similar parts of each embodiment can be referred to each other. As for the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and for the related parts, please refer to the part of the description of the method embodiment.

The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and changes will be apparent to those of ordinary skill in the art. The embodiment was chosen and described in order to better explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention and design various embodiments with various modifications as are suited to the particular use.

Claims (8)

1. A high-resolution terahertz image reconstruction method based on an SRCNN improved model is characterized by comprising the following steps:

constructing an improved SRCNN model structure of double-layer feature extraction based on the model structure of the SRCNN, carrying out interpolation amplification after double-layer feature extraction, and then carrying out pooling;

each layer of the model structure of the improved SRCNN is subjected to the convolution of the third layer in sequence and is regarded as the nonlinear mapping of the full connection layer;

reconstructing a final high-resolution image through a fourth layer of convolution;

the reconstructing the final high resolution image through the fourth layer comprises:

converting an original color image into a gray image, and making a low-resolution image by reducing and amplifying an original image;

carrying out overlapping segmentation on a plurality of images in the training set according to a set step length to obtain a plurality of sub-pictures;

converting the training set and the test set into an H5py format for storage;

importing necessary tool packs and data sets in Tensorflow;

defining an initialization function, randomly manufacturing noise for the weight to eliminate complete symmetry, and setting a standard deviation;

defining a loss function to finish a training result, and giving a minimum learning rate by using Adam through an optimizer;

starting a training process, initializing all parameters, setting the size of batch _ size, the number of iterations, the training output loss of each set period number, and evaluating the performance of the model in real time;

and reconstructing the sub-picture into a whole picture as a reconstructed high-resolution output picture.

2. The method for reconstructing the high-resolution terahertz image based on the SRCNN improved model as claimed in claim 1, wherein the SRCNN based model structure is a model structure of an improved SRCNN for double-layer feature extraction, and the interpolation amplification and pooling after the double-layer feature extraction comprises:

setting the input image of the amplified target as Y and X as the original image, and calculating the output image F by the first layer of convolution network ₁ As shown in formula (1):

F ₁ (Y)＝max(0,W ₁ *Y+B ₁ ) (1)

in the formula, B ₁ Representing weight bias, is a convolution operation, W ₁ Is used to represent n ₁ C x f ₁ ×f ₂ A filter, wherein c represents the number of channels of the image, the color image is 3, and the grayscale image is 1,B ₁ Represents n ₁ An offset amount;

n generated at the first layer by the second layer convolutional neural network ₁ Is mapped into n on the basis of the characteristic diagram ₂ The characteristic diagram, the calculation formula is as follows (2):

F ₂ (Y)＝max(0,W ₂ *F ₁ (Y)+B ₂ ) (2)

in the formula, B ₂ Representing weight bias, representing convolution operation, W ₂ Is used to represent n ₂ N is ₁ ×f ₂ ×f ₂ Filter, B ₂ Represents n ₂ An offset amount;

the third layer carries out image reconstruction, and a reconstruction layer is established by using the formula (3):

F(Y)＝W ₃ *F ₂ (Y)+B ₃ (3)

in the formula W ₃ Is used to represent c n ₂ ×f ₃ ×f ₃ Filter of B ₃ C represents the offset;

f ₁ ＝9，f ₂ ＝1，f ₃ ＝5。

3. the SRCNN improved model-based high-resolution terahertz image reconstruction method as claimed in claim 2, wherein the input image of the set target after amplification is Y, X is an original image, and the first layer of convolutional network calculates an output image F ₁ In the formula (1), the parameter is n ₁ ＝64。

4. The SRCNN improved model-based high-resolution terahertz image reconstruction method according to claim 2, wherein the second layer convolutional neural network generates n at the first layer ₁ On the basis of the characteristic diagram, the characteristic diagram is mapped into n ₂ A characteristic diagram, a calculation formula is shown in formula (2), and the parameter is n ₂ ＝32。

5. The method for reconstructing the high-resolution terahertz image based on the SRCNN improved model as claimed in claim 2, wherein the third layer reconstructs the image, and the parameter c =3 is established by using formula (3).

6. The method for reconstructing the high-resolution terahertz image based on the SRCNN improved model as claimed in claim 1, wherein the set standard deviation values are as follows: 0.01.

7. the method for reconstructing the high-resolution terahertz image based on the SRCNN improved model, wherein the value for giving a very small learning rate is 0.000001.

8. The method of reconstructing a terahertz image with high resolution based on an improved SRCNN model according to claim 1, wherein the setting of the batch _ size, the number of iterations, and the loss of training output per set period are respectively as follows: the size of batch _ size is 128, the number of iterations is 100000, and the number of cycles is set to 1000.

Images