CN107833183B - Method for simultaneously super-resolving and coloring satellite image based on multitask deep neural network - Google Patents
Method for simultaneously super-resolving and coloring satellite image based on multitask deep neural network Download PDFInfo
- Publication number
- CN107833183B CN107833183B CN201711224807.3A CN201711224807A CN107833183B CN 107833183 B CN107833183 B CN 107833183B CN 201711224807 A CN201711224807 A CN 201711224807A CN 107833183 B CN107833183 B CN 107833183B
- Authority
- CN
- China
- Prior art keywords
- network
- resolution
- image
- layer
- coloring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 64
- 238000004040 coloring Methods 0.000 title claims abstract description 47
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 29
- 238000012549 training Methods 0.000 claims abstract description 47
- 238000012545 processing Methods 0.000 claims abstract description 4
- 230000006870 function Effects 0.000 claims description 46
- 230000004913 activation Effects 0.000 claims description 33
- 230000008569 process Effects 0.000 claims description 11
- 238000009826 distribution Methods 0.000 claims description 9
- 238000013507 mapping Methods 0.000 claims description 8
- 238000005520 cutting process Methods 0.000 claims description 6
- 238000007781 pre-processing Methods 0.000 claims description 5
- 101001013832 Homo sapiens Mitochondrial peptide methionine sulfoxide reductase Proteins 0.000 claims description 4
- 102100031767 Mitochondrial peptide methionine sulfoxide reductase Human genes 0.000 claims description 4
- 238000010586 diagram Methods 0.000 claims description 4
- 238000013135 deep learning Methods 0.000 claims description 3
- 238000011478 gradient descent method Methods 0.000 claims description 3
- 238000004519 manufacturing process Methods 0.000 claims description 3
- 239000000126 substance Substances 0.000 claims description 3
- 238000013527 convolutional neural network Methods 0.000 description 13
- 230000003993 interaction Effects 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000011524 similarity measure Methods 0.000 description 1
- 238000010186 staining Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/001—Texturing; Colouring; Generation of texture or colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a method for simultaneously super-resolving and coloring a satellite image based on a multitask deep neural network, and belongs to the technical field of image processing. The invention mainly comprises the following steps: 1. making a gray image block training set with high resolution and low resolution; 2. constructing a multi-task deep neural network for model training; 3. training a network model based on the constructed deep network and the manufactured training set; 4. and inputting a low-resolution gray image according to the learned model parameters, and obtaining the output which is the reconstructed high-resolution color image. The invention not only strengthens the detail part of the satellite image, but also can simultaneously color the gray level image to automatically generate a color satellite image which accords with the reality sense, and also reduces the execution steps and time by combining the depth super-resolution network and the coloring network with excellent performance, thereby having wide application prospect in the fields of gray level image coloring, satellite remote sensing and remote measuring and the like.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a method for simultaneously super-resolving and coloring a satellite image based on a multitask deep neural network.
Background
As networks with a single function become more and more capable, there is an increasing demand for networks with the capability of handling complex multitasking. The traditional approach is to take the output of one network as the input of another and then obtain the final result. Because the method not only needs manual interaction, but also wastes much time when being executed one by one, and also considers the problems of compatibility between two networks and the like, people have to seek other methods.
Existing networks need to have two very important functions, namely super-resolution and shading. In the super-resolution aspect, the reconstruction techniques can be divided into different types, mainly into 3 types: interpolation-based methods, reconstruction-based methods, learning-based methods. Among them, the learning-based method generally learns the mapping relationship between the high resolution image and the low resolution image from an external data set, and then reconstructs the high resolution image using the learned mapping relationship, which is the most popular method at present. For example, Dong et al first apply convolutional neural networks to the task of image super-resolution reconstruction, and they generate super-resolution images by constructing a three-layer convolutional neural network; he et al also reconstruct high resolution images by residual cells. In terms of coloring, the method can be based on interactions from the earliest, such as: luan et al propose coloring using a similarity measure of adjacent pixels, and then some scholars propose a semi-automatic method, which is a color statistic of the input of a reference image or images to gray. Later, fully automated methods were proposed, such as Zhang et al, which let web learning combine low and high level cues for staining.
In recent years, because of the powerful learning capability and the end-to-end training mode of the convolutional neural network, computer vision has been improved in many aspects, such as image classification and face recognition, and many networks have good performance, so that some people begin to consider how to make one network implement multiple tasks, for example, Iizuka et al propose a global network to learn the environmental semantics of an image, so that the coloring result is more accurate. However, since the network is still data-driven, if the tested picture type is not included in the training set class, the poor effect is generated, and the training difficulty of the classification network is large, so that it takes a long time to converge, and it may be used to assist another network to color the gray image, which may have a negative effect from the beginning.
Through search, the Chinese patent application number is 201610856231.1, the application date is 2016, 9 and 27, the name of the invention is: the human face attribute analysis method of the convolutional neural network based on the multitask learning comprises the following steps: 1. single task model analysis: 1) carrying out face key point detection on original samples of face images of all ages, carrying out face alignment, and then cutting the face images according to a preset size to generate new samples containing the face images; 2) respectively training three single-task convolutional neural networks of an age estimation network, a gender identification network and a race classification network by using the new sample generated in the step 1), and comparing the convergence rates of the networks to obtain the weight of the single-task convolutional neural network with the slowest convergence rate; cutting a preset size to generate a new sample containing a face image; 2. multi-task model training: 1) constructing a multi-task convolutional neural network, wherein the network has three task outputs which respectively correspond to age estimation, gender identification and ethnicity classification, and the three tasks all adopt a softmax loss function as a target function; the multitask convolution neural network comprises a sharing part used for data sharing and information exchange in multitask learning and an independent part used for calculating the output of the three tasks; initializing a shared part of the multitask convolutional neural network by using the obtained weight of the single-task convolutional neural network to form an initialized multitask convolutional neural network; 2) training a multitask convolutional neural network by using the generated new sample to obtain a trained multitask convolutional neural network model; 3. judging the face attribute: 1) carrying out face detection on the input picture, judging whether the input picture contains a face image, if so, carrying out face key point detection on the input picture, carrying out face alignment, and then cutting the input picture according to a preset size to generate a new picture containing the face image; 2) and inputting the obtained new picture into the obtained multitask convolutional neural network model for age estimation, gender identification and race classification. Although this method implements a multitask network, by inputting a face, various attributes of the face are obtained, such as: age estimation, gender identification and ethnicity classification, but this application has the following disadvantages: 1) although the application can realize multi-task functions, the attributes of the human face are analyzed, so that the three networks are similar, but the problem in reality is definitely complex and variable, and the situation of approach cannot occur; 2) since the problems solved by such networks are similar, the network structure is also similar, and if two distinct tasks need to be handled simultaneously, the integration of the networks and the sharing of features need to be considered on this basis.
Based on the above analysis, there is a need in the art for a method that enables a more multitasking deep neural network to be obtained.
Disclosure of Invention
1. Technical problem to be solved by the invention
In order to overcome the problems that the prior art cannot process the complex and variable problems in reality and the multiple tasks are incompatible possibly; the invention provides a method for simultaneously super-resolving and coloring a satellite image based on a multitask deep neural network; the invention not only carries out multi-task learning on two completely different aspects, but also solves the problem of incompatibility among networks and meets the complex requirements in reality.
2. Technical scheme
In order to achieve the purpose, the technical scheme provided by the invention is as follows:
the invention discloses a method for simultaneously super-resolving and coloring a satellite image based on a multitask deep neural network, which comprises the following steps of:
and 4, taking a low-resolution gray image as the input of the network, and reconstructing a high-resolution color image as the output by using the parameters obtained by learning in the step 3.
Further, the process of creating the high-resolution and low-resolution color image block training sets in step 1 is as follows:
aiming at each color image in a common satellite image processing data set, firstly carrying out bicubic interpolation twice on a high-resolution image to obtain a low-resolution image with the same size corresponding to the high-resolution image; and then cutting each high-resolution image and each low-resolution image into a plurality of image blocks, wherein the overlapped parts exist between the adjacent image blocks, so that a set of high-resolution image blocks and low-resolution image blocks for depth network training is obtained.
Furthermore, a 43-layer deep network model is constructed in the step 2, the three parts are totally divided, firstly, the image is preprocessed, the next 20 layers form a super-resolution network, and the last 23 layers form a coloring network; in the image preprocessing part, converting a color image from RGB to Lab color space, and then dividing the color image into two parts, wherein one part is an L vector and is used as the input of the whole network; the other part is ab vector, as the label of the final coloring network; in the super-resolution network, a total of 9 residual error layers and two convolution layers are included, wherein each residual error unit has two convolution layers; a PReLU active layer is connected behind each convolution layer; the residual unit is shown in equation (1):
wherein x isiCharacteristic input, w, expressed as residual unit of i-th layeriExpressed as the setting of the i-th layer weights and bias terms, R () represents the residual function, Pr () represents the activation function PReLU, he () is the identity map he (x)i)＝xi；
The depth network will learn the mapping relationship between the low-resolution grayscale image blocks and the high-resolution grayscale image blocks, as shown in equation (2):
x＝F(y,Φ) (2)
wherein x and y are a high-resolution gray image block and a low-resolution gray image block respectively, and phi is a model parameter learned by the super-resolution network and used for subsequent high-resolution image reconstruction;
in the coloring network, the 4 th layer from the last is a deconvolution layer, and the rest are convolution layers; a Relu activation layer following each convolutional layer and deconvolution layer; the network input is a high-resolution gray image block output by a super-resolution network, and the network learns the mapping relationship between the high-resolution gray image block and the ab color component image block, as shown in formula (3):
xc＝fc(yc,θ) (3)
wherein x isc,ycThe method comprises the steps that an ab color component image block and a high-resolution gray scale image block of a coloring network are respectively used, theta is a model parameter learned by the coloring network and used for predicting an ab color component corresponding to the brightness L of each pixel in a high-resolution gray scale image later, the prediction result is combined with the high-resolution gray scale image L to obtain a high-resolution image of a Lab color space, and then the high-resolution image is converted into an RGB color space, so that a desired color image can be obtained.
Further, the loss function of the network training in step 2 adopts different methods in the super-resolution network and the coloring network, and in the super-resolution network, the loss function is expressed by the mean square error, as shown in formula (4):
wherein N is the number of samples in the training set obtained in step 1, xi,yiThe image is an ith high-resolution gray image block and a corresponding low-resolution gray image block; phi is the model parameter learned by the super-resolution network.
In a colored network, the loss function is expressed in terms of polynomial cross-entropy loss, as shown in equation (5):
wherein the content of the first and second substances,
representing the predicted probability distribution and Z the true probability distribution, the function v is a rebalancing factor based on statistics of the ab color components of the training set, h and w represent the length and width of the image, respectively, and q is the total number of classes of the ab color components in the training set.
Further, the activation function of the ReLU activation layer or the prilu layer in step 2 can be expressed by equation (6) or equation (7), respectively, as follows:
f(s)＝max(0,s) (6)
f(s)＝max(0,s)+a*min(0,s) (7)
where s is the input to the ReLU or PReLU activation function, f(s) is the output of the ReLU activation function or the output of the PReLU activation function, and a is a learnable parameter.
Further, in step 2, except for the last convolutional layer, the convolutional kernel sizes of all convolutional layers of the constructed depth network are set to 3 × 3, and the convolutional kernel size of the last layer is set to 1 × 1; the convolution kernel size of the deconvolution layer was set to 4 x 4.
Furthermore, in the super-resolution network, the number of feature maps of the first 19 convolutional layers is set to be 64, and the number of feature maps of the last layer is 1; in the colored network, the number of feature maps corresponding to the first 7 convolutional layers is set to 64, 128, 256, respectively, the next 12 convolutional layers are set to 512, the number of feature maps of the next 3 convolutional layers is set to 256, the number of feature maps of the last layer of convolution is set to 244, and the resulting output of each convolutional layer and the deconvolution layer is expressed as formula (8):
qi＝c(Wipi+bi),i＝1,2,…43 (8)
wherein, WiRepresents the weight of the ith layer, biDenotes the bias of the i-th layer, piRepresenting the input of the i-th layer, qiRepresents the output of the ith layer;
after the activation functions ReLU and prilu, respectively, the results are shown in equations (9) and (10):
zi＝max(0,qi) (9)
zj＝max(0,qj)+a*min(0,qj) (10)
wherein q isiAnd q isjOutput of a layer preceding the activation functions ReLU and PreLU, respectively, ziAnd zjThe outputs of the activation functions ReLU and prilu, respectively.
Further, step 3, training the network by using a Caffe deep learning platform, and initializing the weight and bias of the multitask deep neural network constructed in step 2, wherein the specific process is as follows:
1) after the weight W is initialized in the super-resolution network in the MSRA mode, the weight W meets the following Gaussian distribution:
wherein n represents the number of input units of the layer network, namely the quantity of the input characteristic diagram of the convolutional layer;
in a colored network, the weight initialization is all set to 0, i.e., Wi＝0；
2) Throughout the network, the offsets are all initialized to 0, i.e., bi＝0。
Further, step 3 updates the network parameters by using a gradient descent method, which is expressed by formula (12) as follows:
wherein, Vi+1Represents the weight update value of this time, and ViRepresents the last weight update value, and μ is the weight of the last gradient value, α is the learning rate,
is a gradient;
in the training process, network parameter updating is carried out by specifying iteration times.
3. Advantageous effects
Compared with the prior art, the technical scheme provided by the invention has the following remarkable effects:
(1) according to the method for simultaneously super-resolving and coloring the satellite image based on the multitask deep neural network, the problem of complexity and changeability in reality is considered, two completely different directions are selected to process the image, namely the image is subjected to super-resolution and coloring simultaneously, the multitask requirement is met, the super-resolution and coloring can be simultaneously realized on any other kind of images, and the method meets the complex requirement in reality.
(2) According to the method for simultaneously super-resolving and coloring the satellite image based on the multitask deep neural network, two parts of the network are optimized respectively, and the two parts of the network realize cooperative optimization through feature sharing and feature interaction so as to achieve better results.
(3) According to the method for simultaneously super-resolving and coloring the satellite image based on the multitask deep neural network, the super-resolution network and the coloring network with excellent performance are combined, so that the detail part of the satellite image is enhanced, and the gray image can be colored simultaneously to automatically generate a color satellite image which accords with the reality sense; compared with the original single model network method, the method has the advantages that manual interaction is not needed, the execution time is greatly shortened, and the method has wide application prospects in the fields of historical photos, remote sensing images and the like.
Drawings
FIG. 1 is a flow chart of a method for simultaneously super-resolving and coloring a satellite image based on a multitasking deep neural network of the present invention;
FIG. 2 is a flow chart of the creation of a data set in the present invention;
FIG. 3 is a schematic diagram of a network model constructed in accordance with the present invention, and the ReLU active layer and the PReLU active layer after convolution are not shown in FIG. 3;
fig. 4 is a detailed view of the residual unit in the present invention.
Detailed Description
For a further understanding of the invention, reference should be made to the following detailed description taken in conjunction with the accompanying drawings and examples.
Example 1
With reference to fig. 1, the method for simultaneously super-resolving and coloring a satellite image based on a multitasking deep neural network in this embodiment specifically includes the following steps:
for each color image in a common data set (such as an AID satellite image data set), firstly, carrying out bicubic interpolation twice on a high-resolution image (carrying out bicubic downsampling interpolation for the first time and carrying out bicubic upsampling interpolation for the second time) to obtain a low-resolution image with the same size corresponding to the high-resolution image;
each of the high-resolution image and the low-resolution image is cut into a plurality of 93 × 93 image blocks (the image blocks cut into 93 × 93 image blocks contain feature learning more favorable for super-resolution), and the cutting interval is 27 pixels, so that a part of overlapped parts exist between adjacent image blocks, and therefore a set of the high-resolution image blocks and the low-resolution image blocks for depth network training is obtained.
2-1, constructing a 43-layer deep network model, wherein the specific structure is shown in fig. 3, the depth network model is divided into three parts in total, firstly preprocessing an image, forming a super-resolution network by the following 20 layers, and forming a coloring network by the last 23 layers; during preprocessing, converting a color image from RGB to Lab color space, and then dividing the color image into two parts, wherein one part is an L vector and is used as the input of the whole network; the other part is ab vector, as the label of the final coloring network; in the super-resolution network, a total of 9 residual error layers and two convolutional layers are included, wherein the first residual error unit is taken as an example (the remaining residual error units are consistent with the first residual error unit), the specific structure of the residual error unit is shown in fig. 4, and two convolutional layers are included in each residual error unit; a PReLU active layer is connected behind each convolution layer; the residual unit is shown in equation (1):
wherein x isiCharacteristic input, w, expressed as residual unit of i-th layeriExpressed as the setting of the ith layer weight and bias term, R () representsThe residual function, Pr () then represents the activation function PReLU, he () is the identity map he (x)i)＝xi。
In this network, the network will learn the mapping relationship between the low-resolution gray scale image blocks and the high-resolution gray scale image blocks, as shown in equation (2):
x＝F(y,Φ) (2)
and x and y respectively represent a high-resolution image block and a low-resolution image block, and phi is a model parameter learned by the super-resolution network and used for reconstructing a subsequent high-resolution image.
Finally, in the coloring network, the 4 th layer is a deconvolution layer, and the rest layers are convolution layers; a Relu activation layer following each convolutional layer and deconvolution layer; the network input is a high-resolution grayscale image block output by some of the above networks, which will learn the mapping between the high-resolution grayscale image block and the ab-color component image block, as shown in equation (3):
xc＝fc(yc,θ) (3)
wherein x isc,ycThe method comprises the steps that an ab color component image block and a high-resolution gray scale image block of a coloring network are respectively used, theta is a model parameter learned by the coloring network and used for predicting an ab color component corresponding to the brightness L of each pixel in a high-resolution gray scale image later, the prediction result is combined with the high-resolution gray scale image L to obtain a high-resolution image of a Lab color space, and then the high-resolution image is converted into an RGB color space, so that a desired color image can be obtained.
The loss function of network training adopts different methods in two parts of networks, and in a super-resolution network, the loss function adopts mean square error expression, as shown in formula (4):
wherein N is the number of samples in the training set obtained in step 1, xi,yiPhi is a model parameter learned by the super-resolution network, and is the ith high-resolution gray image block and the corresponding low-resolution gray image block.
In the colored network, the trained loss function is expressed by polynomial cross-entropy loss, as shown in formula (5):
wherein the content of the first and second substances,
representing the predicted probability distribution and Z the true probability distribution, the function v is a rebalancing factor based on statistics of the ab color components of the training set, and h and w represent the length and width of the image, respectively, and q is the ab color component class in the training set.
The activation function of a ReLU activation layer or a PReLU layer may be expressed as follows using equation (6) or equation (7), respectively:
f(s)＝max(0,s) (6)
f(s)＝max(0,s)+a*min(0,s) (7)
where s is the input to the ReLU or PReLU activation function, f(s) is the output of the ReLU activation function or the output of the PReLU activation function, and a is a learnable parameter.
2-2, setting the sizes of convolution kernels of all convolution layers of the constructed depth network to be 3 x 3, and setting the size of the convolution kernel of the last layer to be 1 x 1; the convolution kernel size of the deconvolution layer was set to 4 x 4. In the super-resolution network, the number of feature maps of the first 19 convolutional layers is all set to be 64, and the number of feature maps of the last layer is 1; in the shading network, the numbers of feature maps corresponding to the first 7 convolutional layers are set to 64, 128, 256, respectively, the number of feature maps of the immediately following 12 convolutional layers is set to 512, the number of feature maps of the further following 3 convolutional layers is set to 256, and the number of feature maps of the last convolutional layer is set to 244. The configuration of each layer in the network is specifically shown in table 1.
Table 1 network model configuration of the present invention
The resulting output of each convolutional layer and deconvolution layer is represented as equation (8):
qi＝c(Wipi+bi),i＝1,2,…43 (8)
wherein, WiRepresents the weight of the ith layer, biDenotes the bias of the i-th layer, piRepresenting the input of the i-th layer, qiRepresents the output of the ith layer;
after the activation functions ReLU and prilu, respectively, the results are shown in equations (9) and (10):
zi＝max(0,qi) (9)
zj＝max(0,qj)+a*min(0,qj) (10)
wherein q isiAnd q isjOutput of a layer preceding the activation functions ReLU and PReLU, respectively, ziAnd zjThe outputs of the activation functions ReLU and prilu, respectively.
3-1, training the network by utilizing a Caffe deep learning platform, initializing the super-resolution network for the multitask deep neural network constructed in the step 2 by adopting an MSRA mode, initializing the coloring network to be 0, and initializing all biases to be 0. The specific process is as follows:
1) after the MSRA mode is adopted in the super-resolution network to initialize the weight W, the W satisfies the following Gaussian distribution:
wherein n represents the number of input units of the layer network, namely the number of input characteristic diagrams of the convolutional layers.
In a colored network, however, the weight initialization is all set to 0, i.e., Wi＝0。
2) Throughout the network, the offsets are all initialized to 0, i.e., bi＝0。
3-2, updating the network parameters by adopting a gradient descent method, and expressing the network parameters by a formula (12) as follows:
wherein, Vi+1Represents the weight update value of this time, and ViRepresents the last weight update value, and μ is the weight of the last gradient value, α is the learning rate,
is a gradient;
and 3-3, updating network parameters by specifying iteration times in the training process.
And 4, after training is finished, taking a low-resolution gray image as the input of the network, and reconstructing a high-resolution color image as the output by using the parameters obtained by learning in the step 3.
According to the method for simultaneously super-resolving and coloring the satellite image based on the multitask deep neural network, the problem of complexity and changeability in reality is considered, two completely different directions are selected to process the image, the multitask requirement is met, super-resolution and coloring of any other kind of image can be achieved simultaneously, and the method meets the complex requirement in reality. In addition, the two parts of the network are not only optimized individually but they achieve a collaborative optimization through feature sharing and feature interaction to achieve better results. And not only does not need manual interference, but also greatly shortens the execution time, and has wide application prospect in the fields of historical photos, remote sensing images and the like.
The present invention and its embodiments have been described above schematically, without limitation, and what is shown in the drawings is only one of the embodiments of the present invention, and the actual structure is not limited thereto. Therefore, if the person skilled in the art receives the teaching, without departing from the spirit of the invention, the person skilled in the art shall not inventively design the similar structural modes and embodiments to the technical solution, but shall fall within the scope of the invention.
Claims (8)
1. A satellite image simultaneous super-resolution and coloring method based on a multitask deep neural network comprises the following steps:
step 1, utilizing a satellite color image data set to manufacture an image block training set with high resolution and low resolution;
step 2, constructing a multi-task deep neural network for model training; the method specifically comprises the following steps: constructing a 43-layer deep network model, dividing into three parts in total, preprocessing the image, forming a super-resolution network by the next 20 layers, and forming a coloring network by the last 23 layers; in the image preprocessing part, converting a color image from RGB to Lab color space, and then dividing the color image into two parts, wherein one part is an L vector and is used as the input of the whole network; the other part is ab vector, as the label of the final coloring network; in the super-resolution network, a total of 9 residual error layers and two convolution layers are included, wherein each residual error unit has two convolution layers; a PReLU active layer is connected behind each convolution layer; the residual unit is shown in equation (1):
yi＝0.9*he(xi)+0.1*R(xi,wi)
xi+1＝Pr(yi) (1)
wherein x isiCharacteristic input, w, expressed as residual unit of i-th layeriExpressed as the setting of the i-th layer weight and bias term, R () represents the residual function, Pr () represents the activation function PReLU, he () is the identity maphe(xi)＝xi；
The depth network will learn the mapping relationship between the low-resolution grayscale image blocks and the high-resolution grayscale image blocks, as shown in equation (2):
x＝F(y,Φ) (2)
wherein x and y are a high-resolution gray image block and a low-resolution gray image block respectively, and phi is a model parameter learned by the super-resolution network and used for subsequent high-resolution image reconstruction;
in the coloring network, the 4 th layer from the last is a deconvolution layer, and the rest are convolution layers; a Relu activation layer following each convolutional layer and deconvolution layer; the network input is a high-resolution gray image block output by a super-resolution network, and the network learns the mapping relationship between the high-resolution gray image block and the ab color component image block, as shown in formula (3):
xc＝fc(yc,θ) (3)
wherein x isc,ycThe method comprises the steps that an ab color component image block and a high-resolution gray image block of a coloring network are respectively used, and theta is a model parameter learned by the coloring network;
step 3, adjusting network parameters according to the training set manufactured in the step 1 and the network constructed in the step 2, and carrying out network training;
and 4, taking a low-resolution gray image as the input of the network, and reconstructing a high-resolution color image as the output by using the parameters obtained by learning in the step 3.
2. The method for simultaneously super-resolving and coloring the satellite image based on the multitask deep neural network according to the claim 1, wherein the method comprises the following steps: the process of manufacturing the color image block training set with high resolution and low resolution in the step 1 is as follows:
aiming at each color image in a common satellite image processing data set, firstly carrying out bicubic interpolation twice on a high-resolution image to obtain a low-resolution image with the same size corresponding to the high-resolution image; and then cutting each high-resolution image and each low-resolution image into a plurality of image blocks, wherein the overlapped parts exist between the adjacent image blocks, so that a set of high-resolution image blocks and low-resolution image blocks for depth network training is obtained.
3. The method for simultaneously super-resolving and coloring the satellite image based on the multitask deep neural network according to the claim 2, wherein the method comprises the following steps: in the step 2, the loss function of the network training adopts different methods in the super-resolution network and the coloring network, and in the super-resolution network, the loss function is expressed by mean square error, as shown in formula (4):
wherein N is the number of samples in the training set obtained in step 1, xi,yiThe ith high-resolution image block and the corresponding low-resolution image block are taken as the image blocks; phi is a model parameter learned by the super-resolution network;
in a colored network, the loss function is expressed in terms of polynomial cross-entropy loss, as shown in equation (5):
wherein the content of the first and second substances,
representing the predicted probability distribution and Z the true probability distribution, the function v is a rebalancing factor based on statistics of the ab color components of the training set, h and w represent the length and width of the image, respectively, and q is the total number of classes of the ab color components in the training set.
4. The method for simultaneously super-resolving and coloring the satellite image based on the multitask deep neural network according to the claim 3, wherein the method comprises the following steps: the activation function of the ReLU activation layer in step 2 is expressed by equation (6) as follows:
f(s)＝max(0,s) (6)
where s is the input to the ReLU activation function, and f(s) is the output of the ReLU activation function;
the activation function of the PReLU activation layer is expressed by equation (7) as follows:
f(s)＝max(0,s)+a*min(0,s) (7)
where s is the input to the PReLU activation function, f(s) is the output of the PReLU activation function, and a is a learnable parameter.
5. The method for simultaneously super-resolving and coloring the satellite image based on the multitask deep neural network according to the claim 4, wherein the method comprises the following steps: in step 2, except for the last convolutional layer, the sizes of the convolutional kernels of all convolutional layers of the constructed depth network are set to be 3 × 3, and the size of the convolutional kernel of the last layer is set to be 1 × 1; the convolution kernel size of the deconvolution layer was set to 4 x 4.
6. The method for simultaneously super-resolving and coloring the satellite image based on the multitask deep neural network according to the claim 5, wherein the method comprises the following steps: in the super-resolution network, the number of feature maps of the first 19 convolutional layers is all set to be 64, and the number of feature maps of the last layer is 1; in the colored network, the number of feature maps corresponding to the first 7 convolutional layers is set to 64, 128, 256 respectively, the next 12 convolutional layers are set to 512, the number of feature maps of the next 3 convolutional layers is set to 256, the number of feature maps of the last layer of convolution is set to 244, and the output obtained by each convolutional layer or deconvolution layer is expressed as formula (8):
qi＝c(Wipi+bi),i＝1,2,…43 (8)
wherein, WiRepresents the weight of the ith layer, biDenotes the bias of the i-th layer, piRepresenting the input of the i-th layer, qiRepresents the output of the ith layer;
after the activation functions ReLU and prilu, respectively, the results are shown in equations (9) and (10):
zi＝max(0,qi) (9)
zj＝max(0,qj)+a*min(0,qj) (10)
wherein q isiAnd q isjOutput of a layer preceding the activation functions ReLU and PReLU, respectively, ziAnd zjThe outputs of the activation functions ReLU and prilu, respectively.
7. The method for simultaneously super-resolving and coloring the satellite image based on the multitask deep neural network according to the claim 6, wherein the method comprises the following steps: step 3, training the network by using a Caffe deep learning platform, and initializing the weight and bias of the multitask deep neural network constructed in the step 2, wherein the specific process comprises the following steps:
1) after the weight W is initialized in the super-resolution network in the MSRA mode, the weight W meets the following Gaussian distribution:
wherein n represents the number of input units of the current layer network, namely the quantity of the input characteristic diagrams of the convolutional layers;
in a colored network, the weight initialization is all set to 0, i.e., Wi＝0；
2) Throughout the network, the offsets are all initialized to 0, i.e., bi＝0。
8. The method for simultaneously super-resolving and coloring the satellite image based on the multitask deep neural network according to the claim 7, wherein the method comprises the following steps: step 3, updating the network parameters by adopting a gradient descent method, and expressing the network parameters by using a formula (12) as follows:
wherein, Vi+1Represents the weight update value of this time, and ViRepresents the last weight update value, and μ is the weight of the last gradient value, α is the learning rate,
is a gradient;
in the training process, network parameter updating is carried out by specifying iteration times.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711224807.3A CN107833183B (en) | 2017-11-29 | 2017-11-29 | Method for simultaneously super-resolving and coloring satellite image based on multitask deep neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711224807.3A CN107833183B (en) | 2017-11-29 | 2017-11-29 | Method for simultaneously super-resolving and coloring satellite image based on multitask deep neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107833183A CN107833183A (en) | 2018-03-23 |
| CN107833183B true CN107833183B (en) | 2021-05-25 |
Family
ID=61646641
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711224807.3A Active CN107833183B (en) | 2017-11-29 | 2017-11-29 | Method for simultaneously super-resolving and coloring satellite image based on multitask deep neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107833183B (en) |
Families Citing this family (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108648144B (en) * | 2018-04-20 | 2021-12-10 | 南开大学 | FPM high-resolution color image reconstruction method based on deep learning algorithm |
| CN108830912B (en) * | 2018-05-04 | 2021-04-16 | 北京航空航天大学 | Interactive gray image coloring method for depth feature-based antagonistic learning |
| CN108876870B (en) * | 2018-05-30 | 2022-12-13 | 福州大学 | Domain mapping GANs image coloring method considering texture complexity |
| CN108921783B (en) * | 2018-06-01 | 2022-04-15 | 武汉大学 | Satellite image super-resolution reconstruction method based on mixed loss function constraint |
| CN108921932B (en) * | 2018-06-28 | 2022-09-23 | 福州大学 | Method for generating multiple reasonable colorings of black and white figure pictures based on convolutional neural network |
| CN109063565B (en) * | 2018-06-29 | 2021-12-10 | 中国科学院信息工程研究所 | Low-resolution face recognition method and device |
| CN109146788B (en) * | 2018-08-16 | 2023-04-18 | 广州视源电子科技股份有限公司 | Super-resolution image reconstruction method and device based on deep learning |
| CN109191411B (en) * | 2018-08-16 | 2021-05-18 | 广州视源电子科技股份有限公司 | Multitask image reconstruction method, device, equipment and medium |
| CN110880163B (en) * | 2018-09-05 | 2022-08-19 | 南京大学 | Low-light color imaging method based on deep learning |
| CN109360148B (en) * | 2018-09-05 | 2023-11-07 | 北京悦图遥感科技发展有限公司 | Remote sensing image super-resolution reconstruction method and device based on mixed random downsampling |
| CN109410114B (en) * | 2018-09-19 | 2023-08-25 | 湖北工业大学 | Compressed Sensing Image Reconstruction Algorithm Based on Deep Learning |
| CN109961105B (en) * | 2019-04-08 | 2020-03-27 | 上海市测绘院 | High-resolution remote sensing image classification method based on multitask deep learning |
| CN110111289B (en) * | 2019-04-28 | 2021-09-28 | 深圳市商汤科技有限公司 | Image processing method and device |
| CN110163801B (en) * | 2019-05-17 | 2021-07-20 | 深圳先进技术研究院 | Image super-resolution and coloring method, system and electronic equipment |
| CN110288515A (en) * | 2019-05-27 | 2019-09-27 | 宁波大学 | The method and CNN coloring learner intelligently coloured to the microsctructural photograph of electron microscope shooting |
| CN110675462B (en) * | 2019-09-17 | 2023-06-16 | 天津大学 | Gray image colorization method based on convolutional neural network |
| CN111429350B (en) * | 2020-03-24 | 2023-02-24 | 安徽工业大学 | Rapid super-resolution processing method for mobile phone photographing |
| CN111627080B (en) * | 2020-05-20 | 2022-11-18 | 广西师范大学 | Gray level image coloring method based on convolution nerve and condition generation antagonistic network |
| CN111986084B (en) * | 2020-08-03 | 2023-12-12 | 南京大学 | Multi-camera low-illumination image quality enhancement method based on multi-task fusion |
| CN112508786B (en) * | 2020-12-03 | 2022-04-29 | 武汉大学 | Satellite image-oriented arbitrary-scale super-resolution reconstruction method and system |
| CN112489164B (en) * | 2020-12-07 | 2023-07-04 | 南京理工大学 | Image coloring method based on improved depth separable convolutional neural network |
| CN114463175B (en) * | 2022-01-18 | 2022-11-01 | 哈尔滨工业大学 | Mars image super-resolution method based on deep convolutional neural network |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105354273A (en) * | 2015-10-29 | 2016-02-24 | 浙江高速信息工程技术有限公司 | Method for fast retrieving high-similarity image of highway fee evasion vehicle |
| CN106204449A (en) * | 2016-07-06 | 2016-12-07 | 安徽工业大学 | A kind of single image super resolution ratio reconstruction method based on symmetrical degree of depth network |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10147167B2 (en) * | 2015-11-25 | 2018-12-04 | Heptagon Micro Optics Pte. Ltd. | Super-resolution image reconstruction using high-frequency band extraction |
-
2017
- 2017-11-29 CN CN201711224807.3A patent/CN107833183B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105354273A (en) * | 2015-10-29 | 2016-02-24 | 浙江高速信息工程技术有限公司 | Method for fast retrieving high-similarity image of highway fee evasion vehicle |
| CN106204449A (en) * | 2016-07-06 | 2016-12-07 | 安徽工业大学 | A kind of single image super resolution ratio reconstruction method based on symmetrical degree of depth network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107833183A (en) | 2018-03-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107833183B (en) | Method for simultaneously super-resolving and coloring satellite image based on multitask deep neural network | |
| CN109636905B (en) | Environment semantic mapping method based on deep convolutional neural network | |
| CN111275713B (en) | Cross-domain semantic segmentation method based on countermeasure self-integration network | |
| CN107798381B (en) | Image identification method based on convolutional neural network | |
| CN106547880B (en) | Multi-dimensional geographic scene identification method fusing geographic area knowledge | |
| CN110363290B (en) | Image recognition method, device and equipment based on hybrid neural network model | |
| CN108830913B (en) | Semantic level line draft coloring method based on user color guidance | |
| CN111507993A (en) | Image segmentation method and device based on generation countermeasure network and storage medium | |
| CN110851760B (en) | Human-computer interaction system for integrating visual question answering in web3D environment | |
| CN110070107A (en) | Object identification method and device | |
| CN109685743A (en) | Image mixed noise removing method based on noise learning neural network model | |
| CN111242844A (en) | Image processing method, image processing apparatus, server, and storage medium | |
| CN109359527B (en) | Hair region extraction method and system based on neural network | |
| CN110163801A (en) | A kind of Image Super-resolution and color method, system and electronic equipment | |
| CN110717953A (en) | Black-white picture coloring method and system based on CNN-LSTM combined model | |
| CN111832592A (en) | RGBD significance detection method and related device | |
| CN113627093B (en) | Underwater mechanism trans-scale flow field characteristic prediction method based on improved Unet network | |
| CN112215101A (en) | Attention mechanism-based three-dimensional target identification method and system | |
| CN112991371B (en) | Automatic image coloring method and system based on coloring overflow constraint | |
| CN112257759A (en) | Image processing method and device | |
| CN114581552A (en) | Gray level image colorizing method based on generation countermeasure network | |
| CN115018039A (en) | Neural network distillation method, target detection method and device | |
| CN116258990A (en) | Cross-modal affinity-based small sample reference video target segmentation method | |
| CN113449878B (en) | Data distributed incremental learning method, system, equipment and storage medium | |
| CN112767507B (en) | Cartoon sketch coloring method based on dynamic memory module and generation confrontation network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |