CN115170398A - Image super-resolution reconstruction method and device for chrysanthemum storage warehouse - Google Patents
Image super-resolution reconstruction method and device for chrysanthemum storage warehouse Download PDFInfo
- Publication number
- CN115170398A CN115170398A CN202210853635.0A CN202210853635A CN115170398A CN 115170398 A CN115170398 A CN 115170398A CN 202210853635 A CN202210853635 A CN 202210853635A CN 115170398 A CN115170398 A CN 115170398A
- Authority
- CN
- China
- Prior art keywords
- module
- feature
- chrysanthemum
- feature map
- output
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 235000007516 Chrysanthemum Nutrition 0.000 title claims abstract description 56
- 238000000034 method Methods 0.000 title claims abstract description 29
- 244000189548 Chrysanthemum x morifolium Species 0.000 title 1
- 241000723353 Chrysanthemum Species 0.000 claims abstract description 55
- 238000010586 diagram Methods 0.000 claims abstract description 51
- 238000007670 refining Methods 0.000 claims abstract description 32
- 238000011176 pooling Methods 0.000 claims description 36
- 230000006870 function Effects 0.000 claims description 17
- 230000004927 fusion Effects 0.000 claims description 16
- 238000013507 mapping Methods 0.000 claims description 14
- 238000000605 extraction Methods 0.000 claims description 12
- 230000004913 activation Effects 0.000 claims description 11
- 238000012549 training Methods 0.000 claims description 11
- 238000013178 mathematical model Methods 0.000 claims description 10
- 230000008569 process Effects 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 4
- 239000011159 matrix material Substances 0.000 claims description 4
- 238000005065 mining Methods 0.000 claims description 4
- 108700041286 delta Proteins 0.000 claims description 3
- 238000007499 fusion processing Methods 0.000 claims description 3
- 101100074187 Caenorhabditis elegans lag-1 gene Proteins 0.000 claims description 2
- 101100510615 Caenorhabditis elegans lag-2 gene Proteins 0.000 claims description 2
- 101100510617 Caenorhabditis elegans sel-8 gene Proteins 0.000 claims description 2
- 230000010339 dilation Effects 0.000 claims description 2
- 238000012360 testing method Methods 0.000 abstract description 6
- 230000007246 mechanism Effects 0.000 abstract description 4
- 230000000694 effects Effects 0.000 abstract description 3
- 230000000007 visual effect Effects 0.000 description 5
- 230000009286 beneficial effect Effects 0.000 description 3
- 230000000052 comparative effect Effects 0.000 description 3
- 239000013598 vector Substances 0.000 description 3
- OCKGFTQIICXDQW-ZEQRLZLVSA-N 5-[(1r)-1-hydroxy-2-[4-[(2r)-2-hydroxy-2-(4-methyl-1-oxo-3h-2-benzofuran-5-yl)ethyl]piperazin-1-yl]ethyl]-4-methyl-3h-2-benzofuran-1-one Chemical compound C1=C2C(=O)OCC2=C(C)C([C@@H](O)CN2CCN(CC2)C[C@H](O)C2=CC=C3C(=O)OCC3=C2C)=C1 OCKGFTQIICXDQW-ZEQRLZLVSA-N 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000003213 activating effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000002604 ultrasonography Methods 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an image super-resolution reconstruction method and equipment for a chrysanthemum storage warehouse. According to the method, in the advanced information refining module, information of different spatial positions of the characteristic diagram is calibrated by using two spatial attention mechanisms, then information on different channels of the characteristic diagram is secondarily calibrated by using a channel attention mechanism, the characteristic diagram input into the attention module contains a large amount of differentiated characteristic information, and a test result shows that the CNN network provided by the invention has a good reconstruction effect on-site chrysanthemum images.
Description
Technical Field
The invention belongs to the technical field of chrysanthemum and image processing, and particularly relates to an image super-resolution reconstruction method and device for a chrysanthemum storage warehouse.
Background
After being processed, the chrysanthemum is stored in a warehouse to wait for classified packaging. In the storage process of the chrysanthemum, the chrysanthemum is influenced by various environmental factors and may deteriorate. Moreover, once deteriorated, the chrysanthemum flowers will be infected around, resulting in extended loss. The camera can be installed at the top of the warehouse, and the state of the chrysanthemum in the warehouse is monitored by combining the computer vision technology. However, the distance between the camera and the chrysanthemum is far, and the image is limited by resolution, so that the deteriorated chrysanthemum only occupies a small area, and the target detection network has low identification precision.
Disclosure of Invention
The invention provides an image super-resolution reconstruction method and equipment for a chrysanthemum storage warehouse.
In order to solve the technical problems, the invention adopts the following technical scheme: an image super-resolution reconstruction method for a chrysanthemum storage warehouse comprises the following steps:
the method comprises the following steps that P100, a CNN network is built on a computer according to a designed structure, the CNN network is sequentially provided with a low-level information refining module, a high-level information refining module and a super-resolution mapping module along the depth direction, and the high-level information refining modules are sequentially connected;
the mathematical model of the advanced information extraction module is as follows:
N1＝γ1(j1c3(T m ))
N2＝γ2(j1d3(T m ))
N3＝fu(N1,N2)
N4＝jb(N3,S m )*N1
N5＝jd(N3,L m )*N2
C m+1 ＝jc(||N4,N5||,C m )×||N4,N5||
S m+1 ＝γ3(j2c3(C m+1 ))
L m+1 ＝γ4(j2d3(C m+1 ))
T m+1 ＝γ5(j3c1(||S m+1 ,L m+1 ||))
wherein, T m Feature graph representing input of advanced information refining modules from the core input, S m Feature graph, L, representing input of advanced information refinement modules from a first bypass input m A characteristic diagram representing the input of a high-level information refinement module from a second bypass input, C m A characteristic diagram representing the high-level information extraction module input from the middle input end; j1C3 () and j2C3 () each represent a normal convolution operation with a convolution kernel size of 3 x 3, j1d3 () and j2d3 () each represent a dilated convolution operation with a convolution kernel size of 3 x 3 and a dilation rate of 2, j3C1 () represents a normal convolution operation with a convolution kernel size of 1 x 1, γ 1 (), γ 2 (), γ 3 (), γ 4 (), and γ 5 () each represent an activation function ReLU, jb () represents a first bypass spatial attention module, jd () represents a second bypass spatial attention module, and represents a feature map that is de-aligned using a spatial alignment map (i.e., the spatial alignment map is multiplied by the element correspondence of each layer of the feature map), jc represents a middle channel attention module, and x represents a feature map that is de-aligned using a channel alignment map (i.e., the feature map is multiplied by the corresponding value in the channel alignment map in each layer of the feature map), and | represents a concatenation operation with N () of the feature map, and N () each layer of the feature map and N2N m+1 Feature graph representing the output of advanced information refining modules from the intermediate output, S m+1 Feature graph, L, representing the output of advanced information refinement modules from a first bypass output m+1 Feature graph, T, representing the output of a high level information refinement module from a second bypass output m+1 A feature graph representing the high-level information extraction module output from the core output;
p200, acquiring a training data set, and training the CNN network by using the training data set, wherein the training data set comprises high-resolution warehouse chrysanthemum images and low-resolution warehouse chrysanthemum images corresponding to the high-resolution warehouse chrysanthemum images;
p300, acquiring a field chrysanthemum image needing to be reconstructed, inputting the field chrysanthemum image into the CNN network trained in the step P200, wherein the field chrysanthemum image sequentially passes through the low-level information refining module and each high-level information refining module, and then the core output end of the last high-level information refining module outputs a comprehensive characteristic diagram;
and P400, inputting the comprehensive characteristic map into the super-resolution mapping module, performing super-resolution reconstruction on the comprehensive characteristic map by the super-resolution mapping module, and outputting a target chrysanthemum image with the resolution being greater than that of the field chrysanthemum image.
Furthermore, an intermediate information fusion module is arranged in the CNN network, and the intermediate information fusion module fuses the feature information output by the middle output end of each advanced information refining module with the comprehensive feature map and then inputs the feature information into the super-resolution mapping module.
Further, the mathematical model of the intermediate information fusion module is as follows:
CCS＝jr 1 c1(C 1 )+jr 2 c1(C 2 )+...+jr m+1 c1(C m+1 )
CSTL＝||CCS,CVT||
wherein, C 1 、C 2 、…、C m+1 A characteristic diagram, jr, representing the output of the central output of each of said advanced information refining modules 1 c1()、jr 2 c1()、…、jr m+1 c1 The values (c) all represent common convolution operation with convolution kernel size of 1 x 1, the value of | | · | | represents splicing operation of the feature diagram therein, the CVT represents a comprehensive feature diagram, and the CSTL represents the feature diagram output after fusion operation of the intermediate information fusion module.
Further, the fusion process of the feature map N1 and the feature map N2 is expressed as the following mathematical formula:
Nt＝Ta1(j1t(N1))+Ta2(j2t(N2))+||N1,N2||
N3＝Lc(j3t(Nt))
wherein, | | · | represents that the feature maps therein are subjected to splicing operation, j1t (), j2t () and j3t () all represent common convolution operation with a convolution kernel size of 1 × 1, ta1 () and Ta2 () both represent Tanh activation functions, lc () represents logics activation functions, and N3 represents the feature map obtained by fusing the feature map N1 with the feature map N2.
Further, the mathematical model of the first bypass space attention module is:
PV2＝SMe(S m )
PV3＝SMe(N3)
PV4＝SVr(N3)
S1T＝δ1(ja1c(||PV1,PV2,PV3,PV4||))
wherein, the characteristic diagram S m And feature map N3 is the input to the first bypass space attention module, SMa () represents a full channel max pooling operation on the feature map (i.e., a global max pooling operation in the channel direction), SMe () represents a full channel mean pooling operation on the feature map (i.e., a global mean pooling operation in the channel direction), SVr () represents a full channel variance pooling operation on the feature map (i.e., a global variance pooling operation in the channel direction),
the method comprises the steps of representing a product corresponding to elements, | · | | represents splicing operation of feature graphs in the product, ja1c () represents common convolution operation with a convolution kernel of 1 × 1, delta 1 () represents a sigmoid function, and S1T represents a first bypass space calibration graph output by a first bypass space attention module.
Further, the mathematical model of the second bypass spatial attention module is:
ZO2＝SMe(L m )
PV3＝SMe(N3)
PV4＝SVr(N3)
S2T＝δ2(ja2c(||ZO1,ZO2,PV3,PV4||))
wherein, the characteristic diagram L m And feature map N3 is the input to the second bypass spatial attention module, SMa () representing a pairThe feature map is subjected to full-channel maximum pooling, SMe () represents the full-channel average pooling of the feature map, SVr () represents the full-channel variance pooling of the feature map,
the corresponding product of elements is represented, | · | | represents that the feature graph is subjected to splicing operation, ja2c () represents common convolution operation with the convolution kernel size of 1 × 1, δ 2 () represents a sigmoid function, and S2T represents a second bypass space calibration graph output by the second bypass space attention module.
Further, the mathematical model of the middle channel attention module is:
CW1＝CMaK(||N4,N5||)+CMaK(jw1c(C m ))
CTW＝δ3(FC2(γ6(FC1(CW1))))
wherein, the characteristic diagram is N4, N5 and the characteristic diagram C m For the input of the middle channel attention module, | | · | | represents that the feature maps therein are subjected to splicing operation, CMaK () represents that full-space MaK pooling operation is performed on each layer of the feature maps, jw1c () represents general convolution operation with a convolution kernel size of 1 × 1, FC1 () and FC2 () both represent full join operation, γ 6 () represents an activation function ReLU, δ 3 () represents a sigmoid function, and CTW represents a channel calibration map output by the middle channel attention module.
Since the conventional ultrasound image contains considerable noise, the noise will cause a large interference to the center channel attention module, which is likely to cause a wrong calibration. The present invention employs a full-space MaK pooling operation in the middle aisle attention module. Further, the process of the full-space MaK pooling operation includes the following steps:
p1, arranging elements in the matrix according to a sequence from large to small to obtain a sequence U;
p2, obtaining numerical values of the first three elements in the sequence U, and calculating the result according to the following formula:
GK＝0.6Lag1+0.3Lag2+0.1Lag3
wherein, lag1 represents the numerical value of the first element in the sequence U, lag2 represents the numerical value of the second element in the sequence U, lag3 represents the numerical value of the third element in the sequence U, and GK represents the output value after the MaK pooling operation of the full space. Comparative experimental tests show that the middle channel attention module performs best when the three parameters are set to 0.6, 0.3 and 0.1 respectively.
The invention also provides an image super-resolution reconstruction device for the chrysanthemum storage warehouse, which comprises a processor and a memory, wherein the memory stores a computer program, and the processor is used for executing the image super-resolution reconstruction method for the chrysanthemum storage warehouse by loading the computer program.
The invention has the beneficial effects that:
(1) In the advanced information extraction module, firstly, two space attention mechanisms are adopted to calibrate information of different space positions of the characteristic diagram, and then a channel attention mechanism is adopted to calibrate information on different channels of the characteristic diagram for the second time, so that the calibration pertinence is strong, and the quality of characteristic information in a convolution operation output characteristic diagram is greatly improved;
(2) j1c3 () and j1d3 () are arranged in parallel, and feature information of images is extracted on different visual fields, so that key information under a plurality of visual fields is reserved in the fused N3 feature map, partial invalid information is eliminated, the N3 feature map is used as one input of a space attention module, visual information with different visual fields is provided for the space attention module, the visual information is richer, and the generated first bypass space calibration map and the second bypass space calibration map have a more accurate space information calibration effect;
(3) The spatial attention module not only takes the N3 feature map as input, but also takes the feature map output by the upstream advanced information refining module (S) m /L m ) As input, the spatial attention module has different levels of characteristic information, and the differentiated level information improves the quality of information acquired by the spatial attention module, so that the accuracy of the spatial information calibration by the spatial calibration graph is improved; similarly, the middle channel attention module takes the feature map in the current-level advanced information refinement module and the feature map in the previous-level module as input at the same time, compared with the single information inputThe output channel calibration chart can better calibrate the information of different channels;
(4) In the spatial attention module, after the full-channel maximum pooling operation is respectively carried out on the two characteristic diagrams, the two matrixes are subjected to element corresponding product, and compared with the splicing and dimension reduction of the two matrixes, the spatial attention module can strengthen important information to a greater extent.
Drawings
Fig. 1 is a schematic diagram of a CNN network structure according to the present invention;
fig. 2 is a schematic structural diagram of an advanced information refining module in the CNN network shown in fig. 1;
FIG. 3 is a schematic diagram of the fusion process of the feature map N1 and the feature map N2;
FIG. 4 is a block diagram of a first bypass space attention module of the advanced information mining module of FIG. 2;
FIG. 5 is a block diagram of a middle channel attention module of the advanced information mining module of FIG. 2;
fig. 6 is a schematic structural diagram of a super-resolution mapping module in the CNN network shown in fig. 1;
FIG. 7 is a schematic diagram of a modified first bypass spatial attention module of comparative example 1;
FIG. 8 is a schematic structural view of a modified central passage attention module of comparative example 2;
in the drawings:
the method comprises the steps of 1-field chrysanthemum image, 2-low-level information refining module, 3-high-level information refining module, 31-first bypass space attention module, 32-second bypass space attention module, 33-middle channel attention module, 4-super-resolution mapping module, 41-preset convolution layer, 42-Pixelshuffle layer, 43-rear convolution layer, 5-middle information fusion module and 6-target chrysanthemum image.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings.
Example 1:
based on a TensorFlow framework, a CNN network for reconstructing chrysanthemum images in a storage warehouse is realized by programming on a computer, and the specific structure is shown in FIG. 1. The CNN network includes a low-level information extraction module 2, a high-level information extraction module 3, an intermediate information fusion module 5, and a super-resolution mapping module 4.
The low-level information refining module 2 is realized by adopting common convolution operation with the convolution kernel size of 3 x 3, and the number of output characteristic diagram channels is 32 after the on-site chrysanthemum image 1 is input into the low-level information refining module 2 and is subjected to convolution operation.
The number of the high-level information extraction modules 3 is set to 6, and the specific structure of the high-level information extraction modules 3 is shown in fig. 2. Each advanced information refining module 3 receives as input a plurality of profiles, where T m The profile (number of channels 32) is input from its core input, S m The profile (number of channels 32) is input from its first bypass input, L m The characteristic diagram (number of channels 32) is input from its second bypass input, C m The characteristic diagram is input from the middle input end thereof. For the first high-level information extraction module 3, the feature maps input from the four input ends thereof are all the outputs of the low-level information extraction module 2.
Each advanced information refining module 3 will also output a plurality of characteristic diagrams, C m+1 The characteristic diagram (the number of channels is 64) is output from the middle output end thereof, C m+1 The feature map is not only input into the intermediate information fusion module 5, C m+1 The profile is also input to the mid-lane attention module 33 in the downstream high-level information refinement module 3. S m+1 The signature (number of lanes 32) is output from its first bypass output, S m+1 The characteristic diagram is used as the input of the first bypass input end of the downstream advanced information refining module 3, L m+1 The signature (number of channels 32) is output from its second bypass output, L m+1 The feature map is used as an input of a second bypass input end of the downstream high-level information refining module 3, T m+1 The feature map (number of channels 32) is output from its core output, T m+1 The feature map is used as the input of the core input end of the downstream high-level information refining module 3.
T m The dimensions of the feature map, the N1 feature map, the N2 feature map and the N3 feature map are identical. Please refer to fig. 3When the N1 feature map is fused with the N2 feature map, three operations are firstly performed, that is, the N1 feature map is spliced with the N2 feature map (the number of channels of the output feature map is 64), the N1 feature map is subjected to 1 × 1 convolution and Tanh function activation (the number of channels of the output feature map is 64), and the N2 feature map is subjected to 1 × 1 convolution and Tanh function activation (the number of channels of the output feature map is 64). And then adding the three feature maps, activating by a j3t () convolution and a Logitics function, and outputting to obtain an N3 feature map. In the process of fusing the N1 characteristic diagram and the N2 characteristic diagram, the invention simultaneously adopts two modes of splicing and adding, and in the process, the dimension of the N1 characteristic diagram and the dimension of the N2 characteristic diagram are increased and then reduced, thereby well overcoming the respective defects of the two fusion modes of splicing and adding and improving the characteristic fusion effect.
The operation process inside the first bypass space attention module 31 and the second bypass space attention module 32 is almost the same, please refer to fig. 4, which takes the first bypass space attention module 31 as an example. Characteristic diagram S m After the first bypass spatial attention module 31 is inputted, two matrices are obtained through the full pass maximum pooling operation and the full pass average pooling operation. Similarly, after the feature map N3 is inputted into the first bypass space attention module 31, three matrices are obtained through the full-channel maximum pooling operation, the full-channel average pooling operation, and the full-channel variance pooling operation. And then performing element corresponding product operation on the matrix output by the maximum pooling operation of the two full channels to obtain PV1. And finally, splicing PV1, PV2, PV3 and PV4, and sequentially performing ja1c () convolution and delta 1 () activation to obtain a first bypass space calibration graph (a two-dimensional matrix).
Referring to fig. 5, after the N4 feature map and the N5 feature map are spliced and input to the central channel attention module 33, each layer is output to obtain a vector with a length of 64 through a full-space MaK pooling operation. Characteristic diagram C m When the input data is input to the middle channel attention module 33, the jw1c () convolution operation is performed, then the full-space MaK pooling operation is performed on each layer, and a vector with the length of 64 is obtained through the same output. For the first high-level information refining module 3, the number of channels of the characteristic diagram before the convolution operation is 32, and the number of channels after the convolution is 64; for other high-level information refinement modules 3, jw1c () pre-and post-convolution profilesThe number of channels is 64. After the two vectors are added, the channel calibration graph is obtained after sequentially passing through FC1 (), gamma 6 (), FC2 () and delta 3 (). For FC1 (), its input node number is 64, and its output node number is 12; for FC2 (), its input node number is 12 and its output node number is 64.
In order to avoid information loss and gradient disappearance, the CNN network of this embodiment is provided with an intermediate information fusion module 5, and a feature map C output by the middle output end of the advanced information refining module 3 m+1 After 1 × 1 convolution (the number of channels is 64), the sum is added, and then the sum is spliced with the comprehensive characteristic diagram to be used as the input of the super-resolution mapping module 4. It is emphasized that the characteristic diagram of the input intermediate information fusion module 5 in this embodiment comes from the middle output (C) m+1 ) Rather than the core output (T) m+1 ) Therefore, the feature map input into the super-resolution mapping module 4 can contain more differentiated information, and the quality of the reconstructed target chrysanthemum image 6 can be improved.
As shown in fig. 6, the super-resolution mapping module 4 includes a pre-modulation convolution layer 41, a pixelsuffle layer 42, and a post-modulation convolution layer 43 connected in sequence. When the image resolution is increased by a multiple of A, the number of channels of the output feature map of the pre-modulation convolution layer 41 is 32A 2 The Pixelshuffle layer 42 outputs a signature having 32 channels, but has a length-wise and width-wise pixel value a times that of the output signature of the preconditioning convolutional layer 41. Finally, the post-tone convolutional layer 43 outputs the target chrysanthemum image 6 with the number of channels being 3.
568 high-resolution warehouse chrysanthemum images are collected and obtained, 150 images are randomly extracted to serve as a test set, and the rest images serve as a training set. And manufacturing a low-resolution warehouse chrysanthemum image corresponding to the high-resolution warehouse chrysanthemum image by adopting a bicubic downsampling mode. The CNN network, the CS-NL model (from the article Image Super-Resolution with Cross-Scale Non-Local extension and extreme Self-extensions Mining) and the SRFeat (from the article SRFeat: single Image Super-Resolution with Feature discovery) provided by the present invention are respectively trained by using the same training set, and then the results are shown as follows:
the test results on the same test set are as follows:
as can be seen from the measurement results of the two parameters of the peak signal-to-noise ratio and the structural similarity, compared with the existing model, the CNN network provided in embodiment 1 can better reconstruct a high-resolution chrysanthemum image of a warehouse site, which is beneficial to more accurately detect and discover a quality-changed chrysanthemum.
Example 2:
in order to verify the beneficial effects of the spatial attention module and the full-space MaK pooling operation provided by the present invention, the first bypass spatial attention module 31 and the second bypass spatial attention module 32 are both configured as a conventional structure, and the modified spatial attention module structure is shown in fig. 7. The resulting model was named the example 2A network without changing the rest of the CNN network in example 1.
Similarly, also on the basis of embodiment 1, the full-space MaK pooling operation in the middle channel attention module 33 is modified to a conventional full-space max pooling operation, and the modified channel attention module structure is shown in fig. 8. The resulting model was named example 2B network without changing the rest of the CNN network in example 1.
The example 2A network and the example 2B network were trained on the same training set, respectively, and then tested on the test set, respectively, with the following results:
according to the measurement results of two parameters of the peak signal-to-noise ratio and the structural similarity, after the spatial attention module and the full-space MaK pooling operation provided by the invention are replaced by the prior art, the performance of the network reconstruction of the field chrysanthemum image 1 is reduced to different degrees, and the spatial attention module and the full-space MaK pooling operation provided by the invention are well proved to have positive effects on improving the chrysanthemum image reconstruction quality.
The above is only a preferred embodiment of the present invention, and it should be noted that several modifications and improvements made by those skilled in the art without departing from the technical solution should also be considered as falling within the scope of the claims.
Claims (9)
1. An image super-resolution reconstruction method for a chrysanthemum storage warehouse is characterized by comprising the following steps: the method comprises the following steps:
p100, building a CNN network on a computer according to a designed structure, wherein the CNN network is sequentially provided with a low-level information refining module, a high-level information refining module and a super-resolution mapping module along the depth direction, and the high-level information refining modules are sequentially connected;
the mathematical model of the advanced information extraction module is as follows:
N1＝γ1(j1c3(T m ))
N2＝γ2(j1d3(T m ))
N3＝fu(N1,N2)
N4＝jb(N3,S m )*N1
N5＝jd(N3,L m )*N2
C m+1 ＝jc(||N4,N5||,C m )×||N4,N5||
S m+1 ＝γ3(j2c3(C m+1 ))
L m+1 ＝γ4(j2d3(C m+1 ))
T m+1 ＝γ5(j3c1(||S m+1 ,L m+1 ||))
wherein, T m Feature graph representing input of advanced information refinement modules from core inputs，S m Feature graph, L, representing input of advanced information refinement modules from a first bypass input m A characteristic diagram representing the input of a high-level information refinement module from a second bypass input, C m A characteristic diagram representing the high-level information extraction module input from the middle input end; j1C3 () and j2C3 () each represent a normal convolution operation with a convolution kernel size of 3 x 3, j1d3 () and j2d3 () each represent a dilated convolution operation with a convolution kernel size of 3 x 3 and a dilation rate of 2, j3C1 () represents a normal convolution operation with a convolution kernel size of 1 x 1, γ 1 (), γ 2 (), γ 3 (), γ 4 () and γ 5 () each represent an activation function ReLU, jb () represents a first bypass space attention module, jd () represents a second bypass space attention module, which represents a feature map that is de-aligned using a space alignment map, jc () represents a middle channel attention module, x represents a feature map that is de-aligned using a channel alignment map, | | | | represents a feature map in which a splicing operation is performed, fu () represents a feature map N1 fused with a feature map N2, C m+1 Feature graph representing the output of advanced information refining modules from the intermediate output, S m+1 Feature graph, L, representing the output of advanced information refinement modules from a first bypass output m+1 Feature graph, T, representing the output of a high level information refinement module from a second bypass output m+1 A feature graph representing the high-level information refinement module output from the core output;
p200, acquiring a training data set, and training the CNN network by using the training data set, wherein the training data set comprises high-resolution warehouse chrysanthemum images and low-resolution warehouse chrysanthemum images corresponding to the high-resolution warehouse chrysanthemum images;
p300, acquiring a field chrysanthemum image needing to be reconstructed, inputting the field chrysanthemum image into the CNN network trained in the step P200, wherein the field chrysanthemum image sequentially passes through the low-level information refining module and each high-level information refining module, and then the core output end of the last high-level information refining module outputs a comprehensive characteristic diagram;
and P400, inputting the comprehensive characteristic map into the super-resolution mapping module, performing super-resolution reconstruction on the comprehensive characteristic map by the super-resolution mapping module, and outputting a target chrysanthemum image with the resolution being greater than that of the field chrysanthemum image.
2. The image super-resolution reconstruction method for the chrysanthemum storage warehouse according to claim 1, wherein: and the CNN network is provided with an intermediate information fusion module which fuses the feature information output by the middle output end of each advanced information refining module with the comprehensive feature map and inputs the feature information and the comprehensive feature map into the super-resolution mapping module.
3. The super-resolution image reconstruction method for the chrysanthemum storage warehouse according to claim 2, wherein the super-resolution image reconstruction method comprises the following steps: the mathematical model of the intermediate information fusion module is as follows:
CCS＝jr 1 c1(C 1 )+jr 2 c1(C 2 )+...+jr m+1 c1(C m+1 )
CSTL＝||CCS,CVT||
wherein, C 1 、C 2 、…、C m+1 A characteristic diagram, jr, representing the output of the central output of each of said advanced information mining modules 1 c1()、jr 2 c1()、…、jr m+1 c1 The values (c) all represent common convolution operation with convolution kernel size of 1 x 1, the value of | | · | | represents splicing operation of the feature diagram therein, the CVT represents a comprehensive feature diagram, and the CSTL represents the feature diagram output after fusion operation of the intermediate information fusion module.
4. The image super-resolution reconstruction method for the chrysanthemum storage warehouse according to claim 1, wherein: the fusion process of the feature map N1 and the feature map N2 is expressed by the following mathematical formula:
Nt＝Ta1(j1t(N1))+Ta2(j2t(N2))+||N1,N2||
N3＝Lc(j3t(Nt))
wherein, | | · | represents that the feature maps therein are subjected to splicing operation, j1t (), j2t () and j3t () all represent common convolution operation with a convolution kernel size of 1 × 1, ta1 () and Ta2 () both represent Tanh activation functions, lc () represents logics activation functions, and N3 represents the feature map obtained by fusing the feature map N1 with the feature map N2.
5. The image super-resolution reconstruction method for the chrysanthemum storage warehouse according to claim 1, wherein: the mathematical model of the first bypass space attention module is:
PV2＝SMe(S m )
PV3＝SMe(N3)
PV4＝SVr(N3)
S1T＝δ1(ja1c(||PV1,PV2,PV3,PV4||))
wherein, the characteristic diagram S m And a feature map N3 is an input of the first bypass space attention module, SMa () represents a full channel maximum pooling operation on the feature map, SMe () represents a full channel average pooling operation on the feature map, SVr () represents a full channel variance pooling operation on the feature map,
the method comprises the following steps of representing a product corresponding to elements, | · | | | represents splicing operation of feature graphs in the feature graphs, ja1c () represents common convolution operation with a convolution kernel of 1 × 1, delta 1 () represents a sigmoid function, and S1T represents a first bypass space calibration graph output by a first bypass space attention module.
6. The image super-resolution reconstruction method for the chrysanthemum storage warehouse according to claim 5, wherein: the mathematical model of the second bypass spatial attention module is:
ZO2＝SMe(L m )
PV3＝SMe(N3)
PV4＝SVr(N3)
S2T＝δ2(ja2c(||ZO1,ZO2,PV3,PV4||))
whereinCharacteristic diagram L m And a feature map N3 is input to the second bypass spatial attention module, SMa () represents a full channel maximal pooling operation on the feature map, SMe () represents a full channel average pooling operation on the feature map, SVr () represents a full channel variance pooling operation on the feature map,
the corresponding product of elements is represented, | · | | represents that the feature graph is subjected to splicing operation, ja2c () represents common convolution operation with the convolution kernel size of 1 × 1, δ 2 () represents a sigmoid function, and S2T represents a second bypass space calibration graph output by the second bypass space attention module.
7. The image super-resolution reconstruction method for the chrysanthemum storage warehouse according to claim 1, wherein: the mathematical model of the middle channel attention module is as follows:
CW1＝CMaK(||N4,N5||)+CMaK(jw1c(C m ))
CTW＝δ3(FC2(γ6(FC1(CW1))))
wherein, the characteristic diagram is N4, N5 and the characteristic diagram C m For the input of the middle channel attention module, | | · | | represents that the feature maps therein are subjected to splicing operation, CMaK () represents that full-space MaK pooling operation is performed on each layer of the feature maps, jw1c () represents general convolution operation with a convolution kernel size of 1 × 1, FC1 () and FC2 () both represent full join operation, γ 6 () represents an activation function ReLU, δ 3 () represents a sigmoid function, and CTW represents a channel calibration map output by the middle channel attention module.
8. The image super-resolution reconstruction method for the chrysanthemum storage warehouse according to claim 7, wherein: the process of the full-space MaK pooling operation comprises the following steps:
p1, arranging elements in the matrix according to a descending order to obtain a sequence U;
p2, obtaining numerical values of the first three elements in the sequence U, and calculating the result according to the following formula:
GK＝0.6Lag1+0.3Lag2+0.1Lag3
wherein, lag1 represents the numerical value of the first element in the sequence U, lag2 represents the numerical value of the second element in the sequence U, lag3 represents the numerical value of the third element in the sequence U, and GK represents the output value after the MaK pooling operation of the full space.
9. An image super-resolution reconstruction device for a chrysanthemum storage warehouse is characterized in that: comprises a processor and a memory, wherein the memory stores a computer program, and the processor is used for executing the image super-resolution reconstruction method for the chrysanthemum storage warehouse according to any one of claims 1 to 8 by loading the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210853635.0A CN115170398B (en) | 2022-07-11 | 2022-07-11 | Image super-resolution reconstruction method and device for chrysanthemum storage warehouse |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210853635.0A CN115170398B (en) | 2022-07-11 | 2022-07-11 | Image super-resolution reconstruction method and device for chrysanthemum storage warehouse |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115170398A true CN115170398A (en) | 2022-10-11 |
| CN115170398B CN115170398B (en) | 2023-10-13 |
Family
ID=83494606
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210853635.0A Active CN115170398B (en) | 2022-07-11 | 2022-07-11 | Image super-resolution reconstruction method and device for chrysanthemum storage warehouse |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115170398B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115358931A (en) * | 2022-10-20 | 2022-11-18 | 运易通科技有限公司 | Image reconstruction method and device for warehouse logistics system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113362223A (en) * | 2021-05-25 | 2021-09-07 | 重庆邮电大学 | Image super-resolution reconstruction method based on attention mechanism and two-channel network |
| CN114022750A (en) * | 2021-10-28 | 2022-02-08 | 重庆大学 | Welding spot appearance image identification method and system based on aggregation-calibration CNN |
| CN114626984A (en) * | 2022-03-16 | 2022-06-14 | 西安理工大学 | Super-resolution reconstruction method for Chinese text image |
| US20220208355A1 (en) * | 2020-12-30 | 2022-06-30 | London Health Sciences Centre Research Inc. | Contrast-agent-free medical diagnostic imaging |
| US20220207648A1 (en) * | 2020-12-04 | 2022-06-30 | Commonwealth Scientific And Industrial Research Organisation | Creating super-resolution images |
-
2022
- 2022-07-11 CN CN202210853635.0A patent/CN115170398B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220207648A1 (en) * | 2020-12-04 | 2022-06-30 | Commonwealth Scientific And Industrial Research Organisation | Creating super-resolution images |
| US20220208355A1 (en) * | 2020-12-30 | 2022-06-30 | London Health Sciences Centre Research Inc. | Contrast-agent-free medical diagnostic imaging |
| CN113362223A (en) * | 2021-05-25 | 2021-09-07 | 重庆邮电大学 | Image super-resolution reconstruction method based on attention mechanism and two-channel network |
| CN114022750A (en) * | 2021-10-28 | 2022-02-08 | 重庆大学 | Welding spot appearance image identification method and system based on aggregation-calibration CNN |
| CN114626984A (en) * | 2022-03-16 | 2022-06-14 | 西安理工大学 | Super-resolution reconstruction method for Chinese text image |
Non-Patent Citations (1)
| Title |
|---|
| ABDUL MUQEET等: "HRAN: Hybrid residual attention network for single image super-resolution", IEEE ACCESS, vol. 7, pages 137020 - 137029, XP011748410, DOI: 10.1109/ACCESS.2019.2942346 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115358931A (en) * | 2022-10-20 | 2022-11-18 | 运易通科技有限公司 | Image reconstruction method and device for warehouse logistics system |
| CN115358931B (en) * | 2022-10-20 | 2023-01-03 | 运易通科技有限公司 | Image reconstruction method and device for warehouse logistics system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115170398B (en) | 2023-10-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109255755B (en) | Image super-resolution reconstruction method based on multi-column convolutional neural network | |
| Ahn et al. | Image super-resolution via progressive cascading residual network | |
| Hui et al. | Progressive perception-oriented network for single image super-resolution | |
| CN112861729B (en) | Real-time depth completion method based on pseudo-depth map guidance | |
| CN110378398B (en) | Deep learning network improvement method based on multi-scale feature map jump fusion | |
| CN111870245B (en) | Cross-contrast-guided ultra-fast nuclear magnetic resonance imaging deep learning method | |
| CN111862251B (en) | Method, device, storage medium and electronic equipment for medical image reconstruction technology | |
| US11822900B2 (en) | Filter processing device and method of performing convolution operation at filter processing device | |
| CN115170398A (en) | Image super-resolution reconstruction method and device for chrysanthemum storage warehouse | |
| CN111667444A (en) | Image compressed sensing reconstruction method based on multi-channel residual error network | |
| Gendy et al. | Balanced spatial feature distillation and pyramid attention network for lightweight image super-resolution | |
| Wang et al. | STCD: efficient Siamese transformers-based change detection method for remote sensing images | |
| Varghese et al. | Unpaired image-to-image translation of structural damage | |
| Ma et al. | Towards no-reference image quality assessment based on multi-scale convolutional neural network | |
| CN114519718A (en) | Abdomen multi-organ CT image segmentation method and system | |
| He et al. | Mambaad: Exploring state space models for multi-class unsupervised anomaly detection | |
| Su et al. | High-Similarity-Pass Attention for Single Image Super-Resolution | |
| CN114332625A (en) | Remote sensing image colorizing and super-resolution method and system based on neural network | |
| Cai et al. | Refconv: Re-parameterized refocusing convolution for powerful convnets | |
| Chen et al. | Split-guidance network for salient object detection | |
| CN117635444A (en) | Depth completion method, device and equipment based on radiation difference and space distance | |
| CN115760670B (en) | Unsupervised hyperspectral fusion method and device based on network implicit priori | |
| CN115984949A (en) | Low-quality face image recognition method and device with attention mechanism | |
| CN113450364B (en) | Tree-shaped structure center line extraction method based on three-dimensional flux model | |
| Zhou et al. | Hierarchical Unified Spectral-Spatial Aggregated Transformer for Hyperspectral Image Classification |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20240416 Address after: Building 1-1-5, No. 599 Binjiang Road, Qinglong Street, Yunyang County, Chongqing, 404599 Patentee after: Yunyang Yunshan Agricultural Development Co.,Ltd. Country or region after: China Address before: Group 1, Yongfu Village, Hongshi Town, Yunyang County, Chongqing 404511 (self-promised) Patentee before: Chongqing Yunshan Industrial Co.,Ltd. Country or region before: China |