CN115797181A - Image super-resolution reconstruction method for mine fuzzy environment - Google Patents
Image super-resolution reconstruction method for mine fuzzy environment Download PDFInfo
- Publication number
- CN115797181A CN115797181A CN202211709385.XA CN202211709385A CN115797181A CN 115797181 A CN115797181 A CN 115797181A CN 202211709385 A CN202211709385 A CN 202211709385A CN 115797181 A CN115797181 A CN 115797181A
- Authority
- CN
- China
- Prior art keywords
- image
- resolution
- deep
- module
- feature
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Image Analysis (AREA)
Abstract
The invention relates to an image super-resolution reconstruction method for a mine fuzzy environment, belongs to the technical field of image reconstruction, and solves the problem of image blurring in the prior art. The method specifically comprises the following steps: acquiring a high-definition image of the environment in a mine, preprocessing the image to obtain a low-resolution image corresponding to the processed high resolution, and forming an image pair; establishing a hyper-resolution reconstruction network model based on the image pairs; the super-resolution reconstruction network model obtains feature differences by extracting deep features and shallow features of the low-resolution images, calibrates and integrates the deep features based on the feature differences, and reconstructs the high-resolution images based on the integrated deep features; and acquiring a fuzzy environment image of the mine, and acquiring a reconstructed high-resolution image based on the established hyper-resolution reconstruction network model. The method and the device realize that the picture under the mine can be seen clearly, improve the human eye visual effect of the image and have more authenticity.
Description
Technical Field
The invention relates to the technical field of image reconstruction, in particular to a super-resolution image reconstruction method for a mine fuzzy environment.
Background
Most of coal mines in China are mine industrial mines, the underground industrial mine conditions of the mine industrial mines are complex, the lighting environment is poor, and dust and water mist exist simultaneously, so that images seen by an underground safety monitoring and monitoring system of the coal mine are generally fuzzy, further intelligent analysis and judgment early warning are difficult to be carried out, and the intelligent construction of the coal mine is severely limited.
Image Super-Resolution Reconstruction (Image Super-Resolution Reconstruction): the method is a technology for obtaining a higher-resolution image by using a low-resolution image and a certain reconstruction method. Obtaining a high resolution image by improving the performance of hardware such as an imaging chip and an optical component requires a huge economic cost, and the cost of obtaining a high resolution image by applying a super resolution reconstruction technique is low. As shown in fig. 1, the super-resolution reconstruction of the image is divided into a spatial domain reconstruction method and a frequency domain reconstruction method, and the frequency domain reconstruction method has limitations in real-time application, so the spatial domain method receives more attention. The super-resolution reconstruction method of the spatial domain image can be divided into methods based on interpolation, reconstruction and learning according to the technical principle. The interpolation-based method comprises a nearest neighbor interpolation method, a bilinear interpolation method, a bicubic interpolation method and the like, is simple and short in running time, and is widely applied to commercial software such as Adobe Photoshop and GIMP, but the images processed by the method have the phenomena of sawtooth and blurring. The reconstruction-based method comprises an iterative back projection method, a convex set projection method, a maximum posterior probability method and the like, the method starts from a degradation model of an image, uses prior knowledge of the image as a constraint condition, and obtains a high-resolution image by extracting key information in a low-resolution image sequence for fusion, so that a good reconstruction effect can be obtained, but the method has low execution efficiency and cannot obtain a good reconstruction effect on large-scale factors (such as x 4 and x 8). The learning-based method comprises a manifold learning method, a sparse representation method, a deep learning method and the like, and the principle is that a high-resolution image is subjected to downsampling operation to obtain a corresponding low-resolution image, the low-resolution image block is used as a training sample set, and the mapping relation between the low-resolution image block and the high-resolution image block is constructed through learning or other modeling methods to reconstruct the high-resolution image. The learning-based method fully utilizes the prior knowledge of the image, can still generate new high-frequency details under the condition of not increasing the number of input image samples, and obtains a better reconstruction result than other methods. Therefore, at present, a learning-based method is mainly adopted to perform hyper-resolution reconstruction on the image.
With the continuous development of artificial intelligence and computer hardware, hinton et al proposed the concept of deep learning in 2006. In 2014, dong et al applied deep learning to the field of image super-resolution reconstruction for the first time, and used a three-layer convolutional neural network to learn the mapping relationship between the low-resolution image and the high-resolution image, so that the experimental effect is very obvious, and the picture quality is greatly improved.
Although deep learning based methods continue to show effectiveness and efficiency that is superior to other methods, deep learning based hyper-segmentation methods still have some problems and challenges, which are also a future research trend:
(1) The features learned in the network training process are of great importance to image reconstruction, but the features learned by the network are not further refined in the existing network, and the effect of the reconstruction is not high when the features learned by the network are simply used, so that the refinement of the learned features of the network is completed by designing an amplification difference module.
(2) Although the existing image hyper-segmentation method based on deep learning has achieved excellent performance, how to construct an efficient hyper-segmentation network by using the existing technologies (such as attention mechanism, multi-layer convolution feature fusion, etc.) is still a problem worthy of exploration, and more researchers try to integrate the image into the neural network structure design a priori.
Disclosure of Invention
In view of the above analysis, the embodiment of the present invention aims to provide an image super-resolution reconstruction method for a mine fuzzy environment, so as to solve the problem that an image seen by an existing underground coal mine safety monitoring and surveillance system is generally fuzzy.
In one aspect, an embodiment of the present invention provides a super-resolution image reconstruction method for a mine fuzzy environment, including:
acquiring a high-definition image of the environment in a mine, preprocessing the image to obtain a low-resolution image corresponding to the processed high resolution, and forming an image pair;
establishing a super-resolution reconstruction network model based on the image pair; the super-resolution reconstruction network model obtains feature differences by extracting deep features and shallow features of the low-resolution images, calibrates and integrates the deep features based on the feature differences, and reconstructs the high-resolution images based on the integrated deep features;
and acquiring a fuzzy environment image of the mine, and acquiring a reconstructed high-resolution image based on the established hyper-resolution reconstruction network model.
Optionally, the super-resolution reconstruction network model includes: the device comprises a shallow layer feature extraction module, a deep layer feature extraction module, a subtraction module, a feature module, a calibration module, an integration module and a reconstruction module;
the shallow feature extraction module is used for extracting shallow features;
the deep feature extraction module is used for extracting a first deep feature and a fused second deep feature;
the subtraction module is used for subtracting the pixel value of the first deep layer feature from the pixel value of the corresponding shallow layer feature to obtain a feature difference;
the characteristic module is used for refining the obtained characteristic difference to obtain a third deep characteristic;
the calibration module is used for multiplying the pixel value of the first deep feature by the pixel value of the third deep feature to calibrate the deep feature, so as to obtain a calibrated fourth deep feature;
the integration module is used for performing additive integration on the calibrated fourth deep features and the second deep features to obtain integrated fifth deep features;
and the reconstruction module is used for finally obtaining a reconstructed high-resolution image through upsampling and convolutional layer based on the integrated fifth deep layer characteristics.
Optionally, the deep feature extraction module comprises: the first cascade residual channel attention module, the feature fusion module and the layer attention module;
each first cascade residual channel attention module is used for extracting first deep features, and the first deep features extracted by the first cascade residual channel attention module are output through a first output end;
the feature fusion module is used for splicing the first deep features output by each cascaded residual channel attention module on the channel dimension and completing feature fusion;
the layer attention module is used for extracting deep layer features in the feature fusion module to obtain second deep layer features, and the second deep layer features are output through a second output end.
Optionally, the reconstruction module comprises: an upper sampling layer and a convolution layer;
the up-sampling layer is used for processing the integrated fifth deep layer characteristics to obtain a high-resolution image;
the convolution layer is used for reconstructing the up-sampled high-resolution image to obtain a reconstructed high-resolution image, namely a super-resolution image.
Optionally, the feature module further comprises a second cascade residual channel attention module and a sigmoid module;
subtracting the pixel value of the first deep layer feature from the pixel value of the corresponding shallow layer feature to obtain a feature difference;
the second cascade residual channel attention module carries out deep layer feature extraction according to the obtained feature difference;
and the sigmoid module is used for mapping the deep features of the extracted feature difference to obtain a third deep feature.
Optionally, the obtaining a high-definition image of an environment in a mine and preprocessing the image to obtain a low-resolution image corresponding to the processed high resolution to form an image pair includes:
cutting the obtained high-definition images of the environment in the mine to obtain a plurality of local high-resolution images;
carrying out bicubic interpolation operation on the plurality of local high-resolution images respectively to obtain corresponding low-resolution images;
and forming an image pair according to the low-resolution image pair corresponding to the high-resolution image.
Optionally, the performing bicubic interpolation operation on the plurality of local high-resolution images respectively to obtain corresponding low-resolution images includes:
acquiring an original high-resolution image with the size of 1960 × 1960, and cutting the original high-resolution image into a plurality of local high-resolution images with the size of 196 × 196; the original high-resolution image is a mine environment image shot by a high-definition camera;
and carrying out bicubic interpolation operation on the obtained multiple local high-resolution images to obtain a reduced low-resolution image with the size of 49 multiplied by 49.
Optionally, the obtaining of the reconstructed high resolution image by upsampling and convolutional layer includes:
performing up-sampling according to the integrated deep features, and amplifying the low-resolution image with the size of 49 multiplied by 49 by 4 times to obtain a high-resolution image with the same size as the original size;
and reconstructing the up-sampled image according to the convolution layer to obtain a reconstructed high-resolution image.
Optionally, the reconstructing the upsampled image according to the convolutional layer to obtain a reconstructed high-resolution image includes:
I SR ＝H REC (F UP )
wherein H REC To reconstruct the module, I SR For the reconstructed high-resolution image, F UP Is the image feature obtained after up-sampling.
Optionally, in building the hyper-resolution reconstruction network model,
and performing loss calculation according to the obtained high-resolution image with the same size as the original high-resolution image and the original high-resolution image, and performing iterative optimization based on a loss calculation result.
Compared with the prior art, the invention can realize at least one of the following beneficial effects:
1. the output of each deep feature module forms a feature group as the input of the layer attention module, the deep features are extracted by combining the channel attention mechanism, and the generated image is more in line with a human eye perception system through the features between the layer attention mechanism and the channel attention mechanism modeling layers and channels, so that the human eye visual effect of the image is improved, and the image is more authentic.
2. The constructed hyper-resolution network can better recover high-frequency details of images and a dense connection mode, solves the problems of gradient disappearance and explosion of the network, and has stronger network learning capacity and higher efficiency.
3. And further refining the characteristics learned by the network, and calibrating deep characteristics of the image by using the refined characteristics so as to better use the deep characteristics in image reconstruction.
In the invention, the technical schemes can be combined with each other to realize more preferable combination schemes. Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and drawings.
Drawings
The drawings are only for purposes of illustrating particular embodiments and are not to be construed as limiting the invention, wherein like reference numerals are used to designate like parts throughout.
FIG. 1 is a classification of a super-resolution reconstruction method of an image in the prior art;
FIG. 2 is a process diagram of an image super-resolution reconstruction method facing a mine fuzzy environment in the embodiment of the invention;
FIG. 3 is a diagram of a super-resolution reconstruction network structure according to an embodiment of the present invention;
FIG. 4 is a flowchart of a fuzzy image super-resolution reconstruction system for mine AI intelligent video analysis in an embodiment of the present invention;
FIG. 5 is a schematic diagram of a residual channel attention module (RCAB) in an embodiment of the present invention;
FIG. 6 is a schematic diagram of a channel attention mechanism in an embodiment of the present invention;
FIG. 7 is a schematic diagram of a layer attention mechanism in an embodiment of the present invention;
Detailed Description
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate preferred embodiments of the invention and together with the description, serve to explain the principles of the invention and not to limit the scope of the invention.
The invention discloses a super-resolution image reconstruction method facing a mine fuzzy environment, which is shown in figure 2. The method comprises the following steps:
step S1: and acquiring a high-definition image of the environment in the mine, preprocessing the image to obtain a low-resolution image corresponding to the processed high resolution, and forming an image pair.
S11: cutting the obtained original high-resolution image to obtain a plurality of local images;
specifically, the obtained original high-resolution image refers to an image of a mine environment shot by a high-definition camera; since the size of the high resolution image is 1960 × 1960, if the image is directly input into the network for training, video memory overflow may be caused, and the training speed is seriously reduced, so that the image needs to be preprocessed before the image training. The specific steps of the preprocessing are to cut the original high-resolution image into 196 × 196 size to obtain a plurality of local high-resolution images, so that the amount of calculation is greatly reduced, and the method is suitable for the final use of 196 × 196 size test images.
S12: obtaining a low-resolution image with reduced multiple by interpolating the plurality of local high-resolution images;
firstly, local images are subjected to interpolation operation by using a bicubic interpolation algorithm to obtain 4-time reduced images with the low resolution of 49 multiplied by 49, low resolution images corresponding to each local high resolution are obtained to form an image pair, and an image database is established. And (3) using the gray values of 16 points around the to-be-sampled point as cubic interpolation by a Bicubic interpolation algorithm (Bicubic interpolation), and obtaining the value of the target pixel point through weighting calculation. The bicubic interpolation algorithm is used for not only considering the gray influence of 4 directly adjacent points, but also considering the influence of the change rate of the gray value among the adjacent points, and the bicubic interpolation algorithm is used for scaling the image in equal proportion, so that the image super-resolution reconstruction is more suitable.
Step S2: establishing a hyper-resolution reconstruction network model based on the image pairs; the super-resolution reconstruction network model obtains feature differences by extracting deep features and shallow features of the low-resolution images, calibrates and integrates the deep features based on the feature differences, and reconstructs the high-resolution images based on the integrated deep features. As shown in fig. 3-4.
S21: establishing a connection relation of a hyper-resolution reconstruction network model;
specifically, the hyper-resolution reconstruction network model comprises a shallow layer feature extraction module, a deep layer feature extraction module, a subtraction module, a feature module, a calibration module, an integration module and a reconstruction module, and the whole network forms a high-efficiency hyper-resolution reconstruction network by applying a residual attention mechanism. As shown in fig. 3: the input of the shallow layer feature extraction module is a low-resolution image, and the output of the shallow layer feature extraction module is connected with the input end of the deep layer feature extraction module; the input of the subtraction module is respectively connected with the first output end of the deep layer feature extraction module and the output end of the shallow layer feature extraction module, and the output end of the subtraction module is connected with the input end of the feature module; the input end of the calibration module is respectively connected with the first output end of the deep layer feature extraction module and the output end of the feature module, the output end of the calibration module is connected with one input end of the integration module, the other input end of the integration module is connected with the second output end of the deep layer feature extraction module, the output end of the integration module is connected with the input end of the reconstruction module, and finally the output end of the reconstruction module is used for generating and storing a corresponding local high-resolution image.
The shallow layer feature extraction module is a 3*3 convolution layer; for extracting shallow features of the low resolution image.
The deep feature extraction module is used for extracting a first deep feature and a fused second deep feature;
the deep layer feature extraction module comprises a first cascade residual channel attention module, a feature fusion module and a layer attention module; each first cascade residual channel attention module is used for extracting first deep features, and the first deep features extracted by the first cascade residual channel attention module are output through a first output end; the feature fusion module is used for splicing the first deep features output by each cascade residual channel attention module in the channel dimension and completing feature fusion, and the layer attention module is used for extracting the deep features in the feature fusion module to obtain second deep features and outputting the second deep features through a second output end.
And the subtraction module is used for subtracting the pixel value of the first deep layer feature output by the first output end of the deep layer feature extraction module from the pixel value in the corresponding shallow layer feature extraction module to obtain the feature difference.
The feature module comprises a second cascade residual channel attention module and a nonlinear function module and is used for refining the obtained feature difference to obtain a third deep feature.
The second cascade residual channel attention module extracts deep features according to the obtained feature difference;
and the sigmoid module is used for mapping the deep features of the extracted feature difference to obtain a third deep feature.
And the calibration module is used for multiplying the pixel value of the first deep feature extracted by the deep feature extraction module by the pixel value of the third deep feature output by the feature module, and calibrating the deep features to obtain a calibrated fourth deep feature.
And the integration module is used for adding and integrating the calibrated fourth deep features and the second deep features output by the second output end of the deep feature extraction module to obtain integrated fifth deep features.
And the reconstruction module comprises an upsampling layer and a convolution layer and is used for finally obtaining a high-resolution image through upsampling and rolling the base layer based on the integrated fifth deep layer characteristics.
Inspired by a Dense connected Convolutional neural Network (Dense connected Convolutional Network), if the Convolutional Network comprises shorter connections between a layer close to an input layer and a layer close to an output layer, the Convolutional Network can be trained more deeply, accurately and effectively, and the generation of a gradient explosion phenomenon can be prevented in the Network training process. Therefore, the deep feature extraction module of the network and the first and second cascade residual channel attention modules in the feature module adopt a dense connection mode, and the input of each residual channel attention module and the output of all the residual channel attention modules are in jump connection.
S22: sequentially carrying out shallow feature extraction and deep feature extraction on the features of the input low-resolution image;
specifically, the input image passes through a 3*3 convolutional layer to complete the extraction of the shallow features of the image, and the shallow features of the image are used as the input of the deep feature extraction module.
When the deep features are extracted, a first cascade Residual Channel Attention Block (RCAB) of a Residual Channel Attention Network (RCAN) model is used to extract first deep features from the image: the RCAB module consists of a residual block and a channel attention mechanism. The RCAB structure comprises 4 convolutional layers, 2 activation layers and 1 global pooling layer, and a Sigmoid (nonlinear action function of a neuron) function is used for nonlinear operation in a channel attention mechanism. The first deep feature extraction is done over N cascaded RCABs, as shown in fig. 4, except that the channel scaling operation in the channel attention mechanism uses convolutional layers of size 1 × 1 × 4, all of which are 3 × 3 × 64.
It should be noted that, in the process of completing the first deep feature extraction through N cascaded RCABs, the more times of convolution, the more useful information is obtained.
In addition, in the process of extracting the first deep features, the output of each residual channel attention module is spliced on the channel dimension through a fusion module to complete feature fusion, and the second deep feature extraction is performed through a layer attention module.
S23: obtaining feature differences of the first deep features and the shallow features by a subtraction module by using element subtraction; as shown in fig. 5.
Based on the features extracted by the shallow feature extraction module and the deep feature extraction module, in order to ensure that the network can learn more information in the training process, the difference between the deep features and the shallow features is further enlarged by adopting an element subtraction mode, a general convolutional neural network only carries out a one-way reasoning process in the training process and does not effectively utilize the learned features of the network, therefore, the learned difference of the network learning, namely the third deep feature, is obtained by carrying out subtraction operation on corresponding pixels in the deep feature extraction module and the shallow feature extraction module of the image. And the element subtraction is to subtract the first deep-layer characteristic pixel value in the deep-layer characteristic extraction module from the corresponding pixel value in the shallow-layer characteristic extraction module to obtain the characteristic difference, and the characteristic difference is further refined.
In addition, the output of each deep layer feature extraction module is subjected to hierarchical modeling through a combined layer attention mechanism, and features are further refined. As shown in fig. 6-7.
S24: based on the feature difference, further refining the feature module to obtain a third deep-layer feature;
specifically, a feature difference is obtained based on element subtraction, the feature difference is already refined, and a second cascade residual error channel attention module extracts deep features in the feature difference; and mapping the sigmoid (nonlinear function of neuron) function module in the feature module to 0-1 according to the function to obtain a third deep feature.
S25: calibrating according to the extracted first deep features and the extracted third deep features to obtain fourth deep features;
specifically, the calibration module multiplies the third deep layer feature obtained by the feature module by the corresponding first deep layer feature, and completes calibration of the deep layer feature according to the multiplied pixel value to obtain a calibrated fourth deep layer feature.
The multiplication means that the pixel value of the first deep layer feature extracted in the deep layer feature extraction module is multiplied by the pixel value of the third deep layer feature output in the feature module correspondingly.
S26: the integration module is used for carrying out addition integration on the calibrated fourth deep features and the second deep features to obtain fifth deep features;
and adding the calibrated fourth deep feature obtained in the step S25 and the second deep feature obtained in the layer attention module in the step S22 to obtain a fifth deep feature.
In the embodiment of the invention, the input image is a low-resolution image after bicubic interpolation processing. The output of each RCAB structure is taken as the input of the following RCAB structure:
wherein H Conv Represents the mapping relationship learned by the convolutional layer, H ReLU For ReLU activation function, H CA For the mapping of the channel attention mechanism,
and
input and output, R, of the ith RCAB structure, respectively i Representing the image residual.
How to generate different attention to each channel characteristic is a key step. Here we focus mainly on two aspects: first, information in LR (low resolution) space has rich low frequency components and valuable high frequency components. The low frequency part seems to be flatter. The high frequency components are typically regions, filled with edges, texture, and other details. On the other hand, each filter in the convolutional layer works with a local receptive field, and therefore the convolved output cannot utilize context information outside the local region.
FIG. 5 is a schematic diagram of a channel attention mechanism with input feature dimensions H × W × C, scaling to 1 × 1 × C by global pooling in spatial dimensions, and then reducing the channel to the original one
Multiplying, then mapping the features between 0 and 1 through a Sigmoid function as weights, and finally multiplying the weights by the original input feature elements. The mathematical expression is as follows:
wherein H GP For global pooling operations, W D And W U For the respective dimension-down and dimension-up operations on the channel side, for the weights, δ and f are the activation functions ReLU and Sigmoid,
the output characteristic of the channel attention mechanism.
The network based on the channel attention mechanism can recover a part of detailed information, but we notice that the method cannot weight the characteristics of the multi-scale layer, particularly weaken the long-term information of the shallow layer, and even treat the shallow layer characteristics and the deep layer characteristics despite using jump connection to recover the shallow layer characteristics, so that the strong characterization capability of the convolutional neural network is influenced. Information texture and true detail recovery are not well preserved because they treat feature maps of different layers equally, resulting in some loss of detail in the reconstructed image.
In order to solve the problems, a layer Attention mechanism module proposed in a Holistic Attention Network (HAN) is utilized to model features in the aspect of layers, and the layer Attention mechanism module can better utilize the correlation of a hierarchical structure, so that the strong expression capability of a neural Network is stimulated. Extracting hierarchical feature F through a set of residual modules i And then, further carrying out hierarchical feature weighting to obtain more sufficient feature weighting: and the layer feature attention is used for the layer features extracted by each residual error module, and the correlation of each layer feature is fully utilized. The proposed layer attention takes full advantage of the output characteristics of each of the foregoing RCABs as Feature Groups, expressed as:
wherein H LA Representing layer attention mechanism mapping, concat operation splices image features in channel dimension,
and the output of the ith deep level feature extraction module is shown.
The specific process of the layer attention mechanism is as follows: firstly, performing dimension transformation on N input feature matrixes with dimensions of H multiplied by W multiplied by C (H is the height of a feature diagram, W is the width of the feature diagram, and C is the number of feature channels), transforming the input feature matrixes with the dimensions of H multiplied by W multiplied by C into 2-dimensional matrixes A with the dimensions of N multiplied by HWC, multiplying the transformed matrixes with a transpose matrix of the transformed matrixes to obtain an intermediate matrix B, performing exponential weighting on the intermediate matrix B through a softmax function to obtain a correlation matrix X, multiplying the obtained correlation matrix X with the 2-dimensional matrix A to obtain a matrix Y with the dimensions of N multiplied by HWC, performing dimension transformation on the matrix Y, transforming the matrix Y into a matrix Y with the dimensions of N multiplied by H multiplied by W multiplied by C, adding the input feature matrix and the matrix Y' through long jump connection, and finally outputting output feature matrixes with the dimensions of H multiplied by W multiplied by NC through the dimension transformation.
The process of construction of the network is formulated below, I LR And I SR The input and the output of the network are respectively represented, and the cascaded RCAB modules are a residual error learning module group RG. As is conventional, the present invention uses a convolutional layer to remove the residual layer from I LR Extracting shallow feature F from input 0 ：
F 0 ＝H Conv (I LR ) (5)
Wherein H Conv Shallow feature F extracted by representing convolution operation 0 As inputs in the residual learning module set:
wherein the content of the first and second substances,
representing a residual learning module set, F DF Indicating deep level features. Then, F is mixed DF Elemental subtraction with low resolution image and calibration is done:
s27: upsampling the integrated deep features;
specifically, the fifth depth layer feature after the addition is obtained according to step S26, and further refined, and the upsampling operation is completed by using a method in the ESPCNN (efficient sub-pixel convolution neural network), where the magnification is 4. The image of 49 × 49 size is converted into an image of 196 × 196 original size and high resolution by the up-sampling layer in the reconstruction module, and loss calculation is performed together with the original high resolution image.
The method comprises the following specific steps: upsampling based on the integrated fifth deep features:
F UP ＝H UP (F DF ) (8)
wherein H UP Mapping function learned for the upsampling layer, F UP Is the image feature obtained after up-sampling.
And finally, reconstructing the up-sampled image by using a convolution layer:
I SR ＝H REC (F UP ) (9)
wherein H REC To reconstruct the module, I SR Is a reconstructed high resolution image.
That is, the image obtained at this time is a reconstructed super-resolution image of high resolution.
And step S3: training according to the established hyper-resolution reconstruction network model to obtain a trained hyper-resolution reconstruction network model;
step S31: pre-training a super-resolution reconstruction network model;
and inputting the bicubic interpolation low-resolution image into the super-resolution reconstruction network constructed by the second part to obtain the generated super-resolution image. Next, the network is optimized using a loss function, using L 1 Loss function:
wherein, I SR (x,y)、I HR (x, y) are the reconstructed image and the original high resolution image, respectively, and W, H is the width and height of the image. L is 1 The loss function ensures that the image reconstruction quality obtains a good numerical value under the evaluation standard of Peak Signal to Noise Ratio (PSNR), and reduces the difference of the super-resolution image on the pixel level relative to the real high-resolution image.
And calculating the error of the hidden layer by adopting a back propagation algorithm, and updating the parameters of the model by utilizing a random gradient descent algorithm. The iteration times of the pre-training are 100 times, the parameters of the hyper-resolution reconstruction network have a good effect on the image through the pre-training, the situation that the model generates high-frequency information randomly is prevented, and meanwhile, the time consumed by formal training is reduced.
And step S4: and acquiring a fuzzy environment image of the mine, and acquiring a reconstructed high-resolution image based on the trained super-resolution reconstruction network model.
Specifically, the model after the pre-training obtained in step S31 is subjected to the secondary training, in the feature extraction process, an edge image loss situation may occur, and in order to avoid this situation, a VGG19 network is used to calculate the perceptual loss in the feature extraction process, so the super-resolution image and the high-resolution image are calculated by the VGG19 network to obtain the perceptual loss function Lp. The VGG19 network used in the invention is an efficient feature map extraction network, the super-resolution image and the original high-resolution image respectively pass through the VGG19 network to extract and obtain a feature map of a 3 rd convolutional layer before a 4 th pooling layer, namely a deep feature map, the feature map has details which can improve the utilization of the super-resolution network on deep feature information, and the extracted feature map is subjected to mean square error calculation to obtain a perception loss function L p . In the present invention, L p The loss function is:
wherein L is p Representing the Euclidean distance, N, between high-dimensional features of two images VGG19 Representing the feature extraction network and l representing the position of the pooling layer in the feature extraction network.
L 1 Loss function and perceptual loss function L p Combining according to a set proportion to obtain a loss function L total . In the present invention, the loss function L total The specific calculation formula of (A) is as follows:
L total ＝L 1 +λL p (12)
where λ represents a scaling factor and is set to 0.8. In an iterative process, L total The value of (A) is gradually reduced, which shows that the quality of the super-resolution image generated by the network is gradually improved, and the generated image is closer to the high-resolution image.
In the embodiment of the invention, the mine fuzzy image is reconstructed at a super-resolution mode, the obtained image becomes clearer, the coal mine production process can be observed more clearly in real time by combining an intelligent coal mine monitoring platform, and the coal mine production safety is guaranteed.
Those skilled in the art will appreciate that all or part of the processes for implementing the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, for instructing the relevant hardware. The computer readable storage medium is a magnetic disk, an optical disk, a read-only memory or a random access memory.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention.
Claims (10)
1. A super-resolution image reconstruction method for a mine fuzzy environment is characterized by comprising the following steps:
acquiring a high-definition image of the environment in a mine, preprocessing the image to obtain a low-resolution image corresponding to the processed high resolution, and forming an image pair;
establishing a super-resolution reconstruction network model based on the image pair; the super-resolution reconstruction network model obtains feature differences by extracting deep features and shallow features of the low-resolution images, calibrates and integrates the deep features based on the feature differences, and reconstructs the high-resolution images based on the integrated deep features;
training according to the established hyper-resolution reconstruction network model to obtain a trained hyper-resolution reconstruction network model;
and acquiring a fuzzy environment image of the mine, and acquiring a reconstructed high-resolution image based on the trained hyper-resolution reconstruction network model.
2. The mine fuzzy environment-oriented image super-resolution reconstruction method of claim 1, wherein the super-resolution reconstruction network model comprises: the device comprises a shallow layer feature extraction module, a deep layer feature extraction module, a subtraction module, a feature module, a calibration module, an integration module and a reconstruction module;
the shallow feature extraction module is used for extracting shallow features;
the deep feature extraction module is used for extracting a first deep feature and a fused second deep feature;
the subtraction module is used for subtracting the pixel value of the first deep layer feature from the pixel value of the corresponding shallow layer feature to obtain a feature difference;
the characteristic module is used for refining the obtained characteristic difference to obtain a third deep characteristic;
the calibration module is used for multiplying the pixel value of the first deep feature by the pixel value of the third deep feature to calibrate the deep feature, so as to obtain a calibrated fourth deep feature;
the integration module is used for performing additive integration on the calibrated fourth deep features and the second deep features to obtain integrated fifth deep features;
and the reconstruction module is used for finally obtaining a reconstructed high-resolution image through upsampling and convolutional layer based on the integrated fifth deep layer characteristics.
3. The mine fuzzy environment-oriented image super-resolution reconstruction method according to claim 2, wherein the deep feature extraction module comprises: the first cascade residual channel attention module, the feature fusion module and the layer attention module;
each residual channel attention module in the first cascade residual channel attention module is used for extracting first deep features, and the first deep features extracted by the first cascade residual channel attention module are output through a first output end;
the feature fusion module is used for splicing the first deep features output by each cascaded residual channel attention module on the channel dimension and completing feature fusion;
the layer attention module is used for extracting deep layer features in the feature fusion module to obtain second deep layer features, and the second deep layer features are output through a second output end.
4. The mine fuzzy environment-oriented image super-resolution reconstruction method according to claim 2, wherein the reconstruction module comprises: an upsampling layer and a convolutional layer;
the up-sampling layer is used for processing the integrated fifth deep layer characteristics to obtain a high-resolution image;
the convolution layer is used for reconstructing the up-sampled high-resolution image to obtain a reconstructed high-resolution image, namely a super-resolution image.
5. The mine fuzzy environment-oriented image super-resolution reconstruction method of claim 2, wherein the feature modules comprise a second cascade residual channel attention module and a sigmoid module;
the second cascade residual channel attention module extracts deep features according to the obtained feature difference;
and the sigmoid module is used for mapping the deep features of the extracted feature difference to obtain a third deep feature.
6. The image super-resolution reconstruction method facing the mine fuzzy environment as claimed in any one of claims 1 to 5, wherein the obtaining of high-definition images of the mine environment and the preprocessing of the images to obtain processed low-resolution images corresponding to high resolution constitute an image pair, comprises:
cutting the obtained high-definition images of the environment in the mine to obtain a plurality of local high-resolution images;
carrying out bicubic interpolation operation on the plurality of local high-resolution images respectively to obtain corresponding low-resolution images;
and forming an image pair according to the low-resolution image pair corresponding to the high-resolution image.
7. The mine fuzzy environment-oriented image super-resolution reconstruction method according to claim 6, wherein the obtaining of the corresponding low-resolution images by performing bicubic interpolation operation on the plurality of local high-resolution images comprises:
acquiring an original high-resolution image with the size of 1960 × 1960, and cutting the original high-resolution image into a plurality of local high-resolution images with the size of 196 × 196; the original high-resolution image is a mine environment image shot by a high-definition camera;
and carrying out bicubic interpolation operation on the obtained multiple local high-resolution images to obtain a reduced low-resolution image with the size of 49 multiplied by 49.
8. The super-resolution image reconstruction method for the mine fuzzy environment as claimed in claim 2, wherein the obtaining of the reconstructed high-resolution image through the upsampling and the convolutional layer comprises:
performing up-sampling according to the integrated deep features, and amplifying the low-resolution image with the size of 49 multiplied by 49 by 4 times to obtain a high-resolution image with the same size as the original size;
and reconstructing the up-sampled image according to the convolution layer to obtain a reconstructed high-resolution image.
9. The mine fuzzy environment-oriented image super-resolution reconstruction method of claim 8, wherein reconstructing the up-sampled image according to the convolutional layer to obtain a reconstructed high-resolution image comprises:
I SR ＝H REC (F UP )
wherein H REC To reconstruct the module, I SR For the reconstructed high-resolution image, F UP Is the image feature obtained after up-sampling.
10. The mine fuzzy environment-oriented image super-resolution reconstruction method according to claim 8, wherein in the building of the super-resolution reconstruction network model,
and performing loss calculation according to the obtained high-resolution image with the same size as the original high-resolution image and the original high-resolution image, and performing iterative optimization based on a loss calculation result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211709385.XA CN115797181A (en) | 2022-12-29 | 2022-12-29 | Image super-resolution reconstruction method for mine fuzzy environment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211709385.XA CN115797181A (en) | 2022-12-29 | 2022-12-29 | Image super-resolution reconstruction method for mine fuzzy environment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115797181A true CN115797181A (en) | 2023-03-14 |
Family
ID=85428133
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211709385.XA Pending CN115797181A (en) | 2022-12-29 | 2022-12-29 | Image super-resolution reconstruction method for mine fuzzy environment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115797181A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117078516A (en) * | 2023-08-11 | 2023-11-17 | 济宁安泰矿山设备制造有限公司 | Mine image super-resolution reconstruction method based on residual mixed attention |
-
2022
- 2022-12-29 CN CN202211709385.XA patent/CN115797181A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117078516A (en) * | 2023-08-11 | 2023-11-17 | 济宁安泰矿山设备制造有限公司 | Mine image super-resolution reconstruction method based on residual mixed attention |
| CN117078516B (en) * | 2023-08-11 | 2024-03-12 | 济宁安泰矿山设备制造有限公司 | Mine image super-resolution reconstruction method based on residual mixed attention |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111833246B (en) | Single-frame image super-resolution method based on attention cascade network | |
| CN112347859B (en) | Method for detecting significance target of optical remote sensing image | |
| CN112507997B (en) | Face super-resolution system based on multi-scale convolution and receptive field feature fusion | |
| CN110599401A (en) | Remote sensing image super-resolution reconstruction method, processing device and readable storage medium | |
| CN111192200A (en) | Image super-resolution reconstruction method based on fusion attention mechanism residual error network | |
| CN110223234A (en) | Depth residual error network image super resolution ratio reconstruction method based on cascade shrinkage expansion | |
| CN111833261A (en) | Image super-resolution restoration method for generating countermeasure network based on attention | |
| CN113222824B (en) | Infrared image super-resolution and small target detection method | |
| CN116580241B (en) | Image processing method and system based on double-branch multi-scale semantic segmentation network | |
| CN111861886A (en) | Image super-resolution reconstruction method based on multi-scale feedback network | |
| CN113313180A (en) | Remote sensing image semantic segmentation method based on deep confrontation learning | |
| CN115731141A (en) | Space-based remote sensing image space-time fusion method for dynamic monitoring of maneuvering target | |
| CN111986085A (en) | Image super-resolution method based on depth feedback attention network system | |
| CN114092824A (en) | Remote sensing image road segmentation method combining intensive attention and parallel up-sampling | |
| CN115797181A (en) | Image super-resolution reconstruction method for mine fuzzy environment | |
| CN109272450B (en) | Image super-resolution method based on convolutional neural network | |
| CN115713462A (en) | Super-resolution model training method, image recognition method, device and equipment | |
| CN113627487B (en) | Super-resolution reconstruction method based on deep attention mechanism | |
| CN111222453A (en) | Remote sensing image change detection method based on dense connection and geometric structure constraint | |
| Deng et al. | Multiple frame splicing and degradation learning for hyperspectral imagery super-resolution | |
| Yang et al. | Variation learning guided convolutional network for image interpolation | |
| CN114022356A (en) | River course flow water level remote sensing image super-resolution method and system based on wavelet domain | |
| Ren et al. | A lightweight object detection network in low-light conditions based on depthwise separable pyramid network and attention mechanism on embedded platforms | |
| CN117408924A (en) | Low-light image enhancement method based on multiple semantic feature fusion network | |
| CN112184552A (en) | Sub-pixel convolution image super-resolution method based on high-frequency feature learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |