CN113658075A - Model training method, image edge enhancement method, device, medium and terminal - Google Patents
Model training method, image edge enhancement method, device, medium and terminal Download PDFInfo
- Publication number
- CN113658075A CN113658075A CN202110944014.9A CN202110944014A CN113658075A CN 113658075 A CN113658075 A CN 113658075A CN 202110944014 A CN202110944014 A CN 202110944014A CN 113658075 A CN113658075 A CN 113658075A
- Authority
- CN
- China
- Prior art keywords
- image
- module
- edge enhancement
- neural network
- network model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 75
- 238000012549 training Methods 0.000 title claims abstract description 46
- 238000012545 processing Methods 0.000 claims abstract description 65
- 238000009877 rendering Methods 0.000 claims abstract description 62
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 47
- 238000005070 sampling Methods 0.000 claims abstract description 47
- 230000002708 enhancing effect Effects 0.000 claims abstract description 10
- 238000000605 extraction Methods 0.000 claims abstract description 10
- 230000009467 reduction Effects 0.000 claims description 21
- 230000004927 fusion Effects 0.000 claims description 16
- 230000008569 process Effects 0.000 claims description 15
- 230000006870 function Effects 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 12
- 238000003062 neural network model Methods 0.000 claims description 11
- 238000003860 storage Methods 0.000 claims description 8
- 230000007246 mechanism Effects 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 claims description 3
- 238000012163 sequencing technique Methods 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 7
- 230000009286 beneficial effect Effects 0.000 description 6
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000001965 increasing effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
- G06T2207/20192—Edge enhancement; Edge preservation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Processing (AREA)
Abstract
The invention provides a model training method, an image edge enhancement method, a device, a medium and a terminal, wherein the training method comprises the following steps: carrying out down-sampling processing on an original image sample with a first resolution to obtain a reference image sample; interpolating the reference image sample to obtain a rendering image sample; inputting a rendering image sample to a convolutional neural network model to be trained; performing feature extraction on the rendering image sample to obtain rendering features; enhancing the rendering characteristics by using a channel weighting and fusing module of the convolutional neural network model to obtain an enhanced image; calculating a loss value between the original image sample and the enhanced image by using a loss function; adjusting parameters in the convolutional neural network model according to the loss value; and generating an image edge enhancement model comprising the adjusted convolutional neural network model parameters, wherein the image edge enhancement model obtained by the training method can be used for optimizing the image quality after interpolation processing and improving the conditions of edge blurring and sawtooth.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a model training method, an image edge enhancement device, a medium and a terminal.
Background
Due to excellent robustness, the traditional image interpolation technology is widely applied to the fields of military affairs, aviation, medicine, communication, meteorology, remote sensing, animation production, film synthesis and the like. However, the conventional interpolation method focuses on smoothing of an image, thereby achieving a better visual effect. Such methods, however, often result in blurred edges and jaggies in the image while maintaining the image smooth. The edge information of the image is an important factor of the visual effect of the image and is also a key factor of image processing problems such as target identification and tracking, image matching, image registration and the like, so that an image processing method is urgently needed to optimize the imaging quality of the image processed by the conventional interpolation technology.
Disclosure of Invention
The invention aims to provide a model training method, an image edge enhancement method, a device, a medium and a terminal.
In a first aspect, to achieve the above object, the model training method of the present invention includes:
carrying out down-sampling processing on an original image sample with a first resolution to obtain a reference image sample with a second resolution, wherein the second resolution is smaller than the first resolution;
interpolating the reference image sample with the second resolution to obtain a rendered image sample with the first resolution;
inputting the rendering image sample to a convolutional neural network model to be trained;
performing feature extraction on the rendered image sample by using a cavity convolution pyramid sampling module and a residual error module of the convolution neural network model to obtain rendering features;
enhancing the rendering characteristics by utilizing a channel weighting and fusing module of the convolutional neural network model to obtain an enhanced image;
calculating a loss value between the original image sample and the enhanced image using a loss function;
adjusting parameters in the convolutional neural network model using the loss values;
and generating an image edge enhancement model comprising the adjusted convolutional neural network model parameters.
The model training method has the advantages that: the method comprises the steps of obtaining a reference image sample with a second resolution ratio by carrying out down-sampling processing on an original image sample with the first resolution ratio, obtaining a rendering image sample with the first resolution ratio by carrying out interpolation on the reference image with the second resolution ratio, then carrying out enhancement processing on rendering characteristics of the rendering image sample by a cavity convolution pyramid sampling module, a residual error module and a channel weighting fusion module of a convolution neural network model to obtain a processed enhanced image, then calculating a loss value between the original image sample and the enhanced image according to a loss function, adjusting parameters in the convolution neural network model according to the loss value, and then generating an image edge enhancement model according to the adjusted parameters of the convolution neural network model, thereby training the convolution neural network model into an image edge enhancement model so as to carry out enhancement processing on the image subjected to interpolation by the image edge enhancement model, the situation of image edge blurring is improved.
In some possible embodiments, the performing feature extraction on the rendered image sample by using a cavity convolution pyramid sampling module and a residual error module of the convolution neural network model to obtain a rendered feature includes:
extracting domain features of different scales of points to be interpolated from the rendered image sample by using the cavity convolution with different cavity rates in the cavity convolution pyramid sampling module, and performing weighted fusion on the domain features to obtain an initial rendered feature map;
and raising the channel dimension of the initial rendering feature map by utilizing the conventional convolution in the residual error module, performing separable convolution processing on the initial rendering feature map after the channel dimension is raised, and recovering the channel dimension of the initial rendering feature map to obtain the rendering feature. The beneficial effects are that: by performing weighted fusion on the domain features of different scales of the difference points to be obtained, the accuracy of the obtained feature map is ensured, so that the accuracy of the subsequent rendering features is improved.
In some possible embodiments, the enhancing the rendered feature by using a channel weighted fusion module of the convolutional neural network model to obtain an enhanced image includes:
and determining the attention weight of the rendering feature according to a channel attention mechanism, weighting the rendering feature in the channel direction according to the attention weight, and performing channel fusion to obtain the enhanced image. The beneficial effects are that: attention weights for the rendered features are determined according to an attention mechanism in order to improve the quality of the fused enhanced image.
In some possible embodiments, the sampling rate of the down-sampling process is adjusted within a preset sampling rate. The beneficial effects are that: the sampling multiplying power of the downsampling process is adjusted within a preset sampling multiplying power range, so that the generalization capability of the image edge enhancement model to images with different sampling multiplying powers is improved.
In some possible embodiments, the generating the image edge enhancement model including the adjusted network model parameters includes:
and when the iteration times of the convolutional neural network model to be trained reach a set value or the loss value of the loss function reaches a target value, generating an image edge enhancement model comprising the adjusted network model parameters.
In a second aspect, the present invention further provides an image edge enhancement method, which is applied to the image edge enhancement model described above, where the image edge enhancement method includes:
acquiring a to-be-processed image with a third resolution, wherein the to-be-processed image is an image subjected to interpolation processing;
performing dimensionality reduction on the image to be processed to obtain a spliced image with a fourth resolution;
inputting the stitched image to the image edge enhancement model;
utilizing the image edge enhancement model to carry out image enhancement on the edges of the spliced image, and outputting the enhanced spliced image;
and splicing the enhanced spliced images sequentially output along the direction of the image channel in space to obtain an edge enhanced image.
The image edge enhancement method has the beneficial effects that: the dimension reduction processing is carried out on the image to be processed after the interpolation processing through the image edge enhancement model to obtain a spliced image, when the resolution ratio of the image to be processed is larger, the resolution ratio of the image to be processed input by the model can be reduced through the dimension reduction processing, further, the model training and reasoning are accelerated, meanwhile, the characteristic information of the input image to be processed is stored to the maximum extent, then, the image enhancement is carried out on the spliced image through the image edge enhancement model, then, the enhanced spliced image is output, and the enhanced spliced image which is output in sequence is spliced in space along the image channel direction to obtain a final edge enhanced image, so that the image enhancement is completed, the image after the processing can effectively reduce edge blurring, the anti-aliasing effect of the image is improved, and the image quality is improved.
In some possible embodiments, performing dimension reduction on the image to be processed to obtain a stitched image at a fourth resolution includes:
dividing the image to be processed into N sub-images in equal area, wherein N is a positive integer;
and sequencing the N sub-images, and splicing the N sub-images according to the arrangement sequence of the N sub-images in the direction of an image channel to obtain a spliced image with the fourth resolution.
In a third aspect, the present invention further provides an image edge enhancement model training apparatus, including:
the down-sampling module is used for carrying out down-sampling processing on an original image sample with a first resolution ratio to obtain a reference image sample with a second resolution ratio, wherein the second resolution ratio is smaller than the first resolution ratio;
the interpolation module is used for interpolating the reference image sample with the second resolution ratio to obtain a rendering image sample with the first resolution ratio;
the transmission module is used for inputting the rendering image sample to a convolutional neural network model to be trained;
the extraction module is used for extracting the characteristics of the rendered image sample by utilizing the cavity convolution pyramid sampling module and the residual error module of the convolution neural network model to obtain rendering characteristics;
the rendering enhancement module is used for enhancing the rendering characteristics by utilizing a channel weighting fusion module of the convolutional neural network model to obtain an enhanced image;
a calculation module for calculating a loss value between the original image sample and the enhanced image using a loss function;
the parameter adjusting module is used for adjusting parameters in the convolutional neural network model according to the loss value;
and the generating module is used for generating an image edge enhancement model comprising the adjusted convolutional neural network model parameters.
The image edge enhancement model training device has the beneficial effects that: the image edge enhancement model generated after training by the image edge enhancement model training device is convenient for enhancing the image after interpolation by the image edge enhancement model, and the condition of image edge blurring is improved.
In a fourth aspect, the present invention further discloses an image edge enhancement apparatus, including:
the acquisition module is used for acquiring an image to be processed with a third resolution, wherein the image to be processed is an image subjected to interpolation processing;
the dimensionality reduction module is used for carrying out dimensionality reduction on the image to be processed to obtain a spliced image with a fourth resolution;
an input module, configured to input the stitched image to the image edge enhancement model;
the enhancement processing module is used for utilizing the image edge enhancement model to carry out image enhancement on the edges of the spliced image and outputting the enhanced spliced image;
and the splicing module is used for splicing the enhanced spliced images sequentially output along the image channel direction in space to obtain an edge enhanced image.
The image edge enhancement device has the advantages that: the image to be processed after interpolation processing is obtained through the obtaining module, the image to be processed after interpolation processing is subjected to dimensionality reduction processing through the dimensionality reduction module to obtain a spliced image, image enhancement is carried out on the spliced image through the enhancement processing module, the enhanced spliced image is output, the spliced image which is sequentially output is spliced in space along the image channel direction by the splicing module to obtain a final edge enhanced image, and therefore image enhancement is completed, edge blurring of the processed image can be effectively reduced, the anti-aliasing effect of the image is improved, and image quality is improved.
In a fifth aspect, the present invention further discloses a computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the above-mentioned training method or the above-mentioned image edge enhancement method.
In a sixth aspect, the present invention further provides a terminal, including: a processor and a memory;
the memory is used for storing a computer program;
the processor is configured to execute the computer program stored in the memory, so as to enable the terminal to execute the training method or the image edge enhancement method.
In a seventh aspect, an embodiment of the present application further provides a computer program product, which, when running on a terminal device, causes the terminal device to execute a method for implementing the above-mentioned training method or the above-mentioned image edge enhancement method.
For specific reference to the description of the beneficial effects of the first aspect, the second aspect, the third aspect and the fourth aspect, no further description is provided herein.
Drawings
FIG. 1 is a flow chart of a model training method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a training process of another model training method according to an embodiment of the present invention;
fig. 3 is a structural block diagram of an image edge enhancement model obtained after training is completed by a model training method according to an embodiment of the present invention;
FIG. 4 is a flowchart of an image edge enhancement method according to an embodiment of the present invention;
fig. 5 is a schematic processing diagram of an image edge enhancement method according to an embodiment of the present invention;
fig. 6 is a schematic processing process diagram when an image to be processed is evenly divided into four blocks by the image edge enhancement method according to the embodiment of the present invention;
fig. 7 is a block diagram of an image edge enhancement model training apparatus according to an embodiment of the present invention;
FIG. 8 is a block diagram of an image edge enhancement apparatus according to an embodiment of the present invention;
fig. 9 is a block diagram of an apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings of the present invention, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention. Unless defined otherwise, technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention belongs. As used herein, the word "comprising" and similar words are intended to mean that the element or item listed before the word covers the element or item listed after the word and its equivalents, but does not exclude other elements or items.
To solve the problems in the prior art, an embodiment of the present invention provides an image edge enhancement model training method, as shown in fig. 1, including:
s101, performing down-sampling processing on an original image sample with a first resolution to obtain a reference image sample with a second resolution, wherein the second resolution is smaller than the first resolution;
s102, interpolating the reference image sample with the second resolution ratio to obtain a rendering image sample with the first resolution ratio;
s103, inputting the rendering image sample to a convolutional neural network model to be trained;
s104, extracting the features of the rendered image sample by using a cavity convolution pyramid sampling module and a residual error module of the convolution neural network model to obtain rendering features;
s105, enhancing the rendering characteristics by using a channel weighting and fusing module of the convolutional neural network model to obtain an enhanced image;
s106, calculating a loss value between the original image sample and the enhanced image by using a loss function;
s107, adjusting parameters in the convolutional neural network model by using the loss value;
and S108, generating an image edge enhancement model comprising the adjusted convolutional neural network model parameters.
In some specific embodiments, as shown in fig. 2, a picture with a first resolution is selected as an original image sample as a reference value of training data of a convolutional neural network model, a distorted image with a reduced resolution is obtained by performing down-sampling processing on the original image sample, then the distorted image is interpolated in a conventional interpolation manner to obtain a low-quality rendered image with the same resolution as the original image sample, then the low-quality rendered image is input into the convolutional neural network model for Artificial Intelligence (AI) enhancement to obtain a high-quality rendered image, then a loss value between the original image sample and the high-quality rendered image is calculated through a loss function, parameters of the convolutional neural network model are adjusted according to the loss value, and the above processes are repeated continuously until parameters of the convolutional neural network model meeting requirements are obtained, and generating a final image edge enhancement model.
It should be noted that, in the above training data generation process, different sampling magnifications may be sampled when different rendered image samples are generated, so that, in the model generation process, the generalization capability of the model to different sampling magnifications may be improved by combining training data of different sampling magnifications. Exemplarily, after the convolutional neural network model to be trained is trained by the above training method, the image edge enhancement model shown in fig. 3 may be obtained, and includes a hole space Pyramid hole convolutional Pyramid sampling module (ASPPBlock), a plurality of Residual modules (Residual blocks, ResBlock), and a Channel Weighted Block (CWBlock), where the hole convolution in the hole convolutional Pyramid sampling module generally includes convolution holes whose Receptive Fields (RF) are 3x3, 5x5, and 7x7, respectively. The conventional convolution in the residual block and the hole convolution pyramid sampling block generally adopts the conventional convolution with a convolution kernel of 1x1, but the scheme is not limited to the conventional convolution with a convolution kernel of 1x1, and any convolution capable of satisfying the scheme of the present application can be used for the conventional convolution.
In some embodiments, the step S104 of extracting features of the rendered image sample by using the cavity convolution pyramid sampling module and the residual error module of the convolution neural network model to obtain a rendered feature specifically includes:
extracting domain features of different scales of points to be interpolated from the rendering image sample by using the cavity convolution with different cavity rates in the cavity convolution pyramid sampling module, and performing weighted fusion on the domain features to obtain an initial rendering feature map; and raising the channel dimension of the initial rendering feature map by utilizing the conventional convolution in the residual error module, performing separable convolution processing on the initial rendering feature map after the channel dimension is raised, and recovering the channel dimension of the initial rendering feature map to obtain the rendering feature.
Specifically, the domain features of the points to be interpolated in different scales are extracted through the cavity convolutions with different cavity rates in the cavity convolution pyramid sampling module, the domain features of different scales are subjected to feature weighting through conventional convolution after each cavity convolution, then the domain features of different scales are spliced in the channel direction, and finally the initial rendering feature map with a certain number of channels is obtained through the fusion of the conventional convolution.
In this embodiment, after the initial rendering feature map is obtained, the channel dimension of the initial rendering feature map is increased through the conventional Convolution in the residual module, then Separable Convolution processing (DSC) is performed on the initial rendering feature map with a high dimension to reduce the size of the model parameter, and then the channel dimension of the initial rendering feature map subjected to Separable Convolution processing is restored to the original level through the conventional Convolution, so as to obtain the rendering feature.
Since the separable convolution processing DSC is the content of the prior art, the scheme of the present application does not relate to the improvement of the separable convolution processing DSC itself, and is not described herein again.
In some other embodiments, the step S105 of enhancing the rendering feature by using a channel weighted fusion module of the convolutional neural network model to obtain an enhanced image includes:
and determining the attention weight of the rendering feature according to a channel attention mechanism, weighting the rendering feature in the channel direction according to the attention weight, and performing channel fusion to obtain the enhanced image.
In the above process, according to the channel attention mechanism, the attention weight of the rendering feature is determined so as to perform weighted fusion on the rendering feature in the image channel direction.
Specifically, after determining the weight in the channel direction according to the channel attention mechanism, the rendering features are weighted in the image channel direction, and then the rendering features are channel-fused by the conventional convolution with a convolution kernel of 1 × 1, so as to obtain a high-quality enhanced image.
In some embodiments, the sampling rate of the downsampling process is adjusted within a preset sampling rate.
In the process of obtaining a reference image sample with a second resolution by carrying out downsampling processing on an original image sample, considering the influence of different sampling magnifications on a trained image edge enhancement model, and adjusting the downsampling sampling magnifications in a preset sampling magnification, so that the subsequently obtained image edge enhancement model has good generalization capability on images with different sampling magnifications, and the enhancement requirements of different images are met.
In some embodiments, the generating of the image edge enhancement model including the adjusted network model parameters in step S108 specifically includes:
and when the iteration times of the convolutional neural network model to be trained reach a set value or the loss value of the loss function reaches a target value, generating an image edge enhancement model comprising the adjusted network model parameters.
After the parameters in the convolutional neural network model are adjusted by using the loss value, the parameters of the image edge enhancement model can be correspondingly obtained, and in order to further perfect the image edge enhancement model, when the iteration number of the convolutional neural network model to be trained reaches a set value or the loss value reaches a target value, the training degree of the convolutional neural network model is judged to have reached the requirement, so that the image edge enhancement model can be generated.
In the method, a reference image sample with reduced resolution is obtained by performing down-sampling processing on an original image sample, the reference image sample is interpolated to restore the resolution to obtain a rendered image sample, then the rendered image sample and the original image sample are used as training data, the rendered image sample is input into a convolutional neural network model to perform feature extraction and enhancement to obtain an enhanced image, then parameters in the convolutional neural network model are adjusted according to loss values between the enhanced image and the original image sample, an image edge enhancement model is generated according to the adjusted parameters of the convolutional neural network model, so that the training process of the image edge enhancement model is completed, the image edge enhancement model obtained by adopting the training method is used for processing the image after the traditional interpolation processing, and the image quality is effectively improved, the situation of image edge blurring is improved.
The invention also provides an image edge enhancement method, which is applied to the image edge enhancement model, as shown in fig. 4, and comprises the following steps:
s401, acquiring a to-be-processed image with a third resolution, wherein the to-be-processed image is an image subjected to interpolation processing;
s402, performing dimensionality reduction on the image to be processed to obtain a spliced image with a fourth resolution;
s403, inputting the spliced image into the image edge enhancement model;
s404, performing image enhancement on the edge of the spliced image by using the image edge enhancement model, and outputting the enhanced spliced image;
s405, splicing the enhanced spliced images sequentially output along the image channel direction in space to obtain an edge enhanced image.
In the method, the image to be processed is input to the image edge enhancement model for processing after dimension reduction, so that the problem of very low processing speed caused by the fact that the resolution ratio of the image is high after bilinear interpolation of the original image and the image is directly input to the model for processing can be effectively solved.
When the resolution ratio of the image to be processed is larger, the resolution ratio of the image to be processed input by the model can be reduced through dimension reduction processing, so that the model training and reasoning are accelerated, and meanwhile, the characteristic information of the input image to be processed is stored to the maximum extent.
Specifically, as shown in fig. 5, after obtaining the trained image edge enhancement model, first obtaining the to-be-processed image with the third resolution subjected to interpolation processing, first performing Space To Depth (STD) dimension reduction processing on the to-be-processed image to obtain the stitched image with the fourth resolution, which not only speeds up model training and reasoning efficiency in large resolution, but also effectively stores the feature information of the input to-be-processed image, then inputting the stitched image into the image edge enhancement model to perform saw tooth elimination processing, outputting the enhanced stitched image after the image edge enhancement model processing, then spatially stitching the enhanced stitched image sequentially output along the image channel direction, that is, performing Space To Depth (STD) dimension increasing processing, thereby obtaining the edge enhancement image, and finishing the processing process of the image to be processed, and performing anti-aliasing treatment on the image to be processed.
In some embodiments, performing dimension reduction on the image to be processed to obtain a stitched image of a fourth resolution includes:
dividing the image to be processed into N sub-images in equal area, wherein N is a positive integer; and sequencing the N sub-images, and splicing the N sub-images according to the arrangement sequence of the N sub-images according to the image channel direction to obtain a spliced image with the fourth resolution. Specifically, as shown in fig. 6, the image to be processed is uniformly divided into four equal parts, which are respectively marked with serial numbers 1, 2, 3, and 4, and after the dimension reduction processing, the image to be processed is spliced together along the channel direction of the image according to the sequence of the marked serial numbers, so as to obtain a spliced image.
Further, in step S405, the enhanced stitched images sequentially output in the image channel direction are spatially stitched to obtain an edge-enhanced image, which specifically includes:
and splitting the output spliced images according to the equal division sequence of the previous dimension reduction treatment, splicing the output spliced images together according to the sequence, and finally obtaining a complete edge enhanced image so as to finish the anti-aliasing treatment of the image.
The invention also discloses an image edge enhancement model training device, as shown in fig. 7, comprising:
a down-sampling module 701, configured to down-sample an original image sample with a first resolution to obtain a reference image sample with a second resolution, where the second resolution is smaller than the first resolution;
an interpolation module 702, configured to interpolate the reference image sample with the second resolution to obtain a rendered image sample with the first resolution;
a transmission module 703, configured to input the rendered image sample to a convolutional neural network model to be trained;
an extraction module 704, configured to perform feature extraction on the rendered image sample by using a cavity convolution pyramid sampling module and a residual error module of the convolutional neural network model to obtain a rendered feature;
a rendering enhancement module 705, configured to enhance the rendering feature by using a channel weighting fusion module of the convolutional neural network model to obtain an enhanced image;
a calculating module 706 for calculating a loss value between the original image sample and the enhanced image using a loss function;
a parameter adjusting module 707, configured to adjust a parameter in the convolutional neural network model according to the loss value;
a generating module 708 configured to generate an image edge enhancement model including the adjusted convolutional neural network model parameters.
It should be noted that the structure and principle of the image edge enhancement apparatus correspond to the steps in the image edge enhancement method one to one, and thus are not described herein again.
The invention further discloses an image edge enhancement device, as shown in fig. 8, comprising:
an obtaining module 801, configured to obtain an image to be processed at a third resolution, where the image to be processed is an image after interpolation processing;
a dimension reduction module 802, configured to perform dimension reduction processing on the image to be processed to obtain a stitched image with a fourth resolution;
an input module 803, configured to input the stitched image to the image edge enhancement model;
the enhancement processing module 804 is configured to perform image enhancement on the edge of the stitched image by using the image edge enhancement model, and output an enhanced stitched image;
and a stitching module 805, configured to spatially stitch the enhanced stitched images sequentially output in the image channel direction to obtain an edge-enhanced image.
It should be noted that the structure and principle of the image edge enhancement apparatus correspond to the steps in the image edge enhancement method one to one, and thus are not described herein again.
It should be noted that the division of the modules of the above apparatus is only a logical division, and the actual implementation may be wholly or partially integrated into one physical entity, or may be physically separated. And these modules can be realized in the form of software called by processing element; or may be implemented entirely in hardware; and part of the modules can be realized in the form of calling software by the processing element, and part of the modules can be realized in the form of hardware. For example, the selection module may be a processing element that is set up separately, or may be implemented by being integrated in a chip of the system, or may be stored in a memory of the system in the form of program code, and the function of the above x module may be called and executed by a processing element of the system. Other modules are implemented similarly. In addition, all or part of the modules can be integrated together or can be independently realized. The processing element here may be an integrated circuit with signal processing capabilities. In implementation, each step of the above method or each module above may be implemented by an integrated logic circuit of hardware in a processor element or an instruction in the form of software.
For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more Application Specific Integrated Circuits (ASICs), or one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs), etc. For another example, when one of the above modules is implemented in the form of a Processing element scheduler code, the Processing element may be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. For another example, these modules may be integrated together and implemented in the form of a System-On-a-Chip (SOC).
In other embodiments of the present application, embodiments of the present application disclose an apparatus, as shown in fig. 9, the apparatus 900 may include: one or more processors 901; a memory 902; a display 903; one or more application programs (not shown); and one or more computer programs 904, which may be connected via one or more communication buses 905. Wherein the one or more computer programs 904 are stored in the memory 902 described above and configured to be executed by the one or more processors 901, the one or more computer programs 904 comprising instructions.
The invention also discloses a computer readable storage medium, on which a computer program is stored, which, when being executed by a processor, performs the above-mentioned training method or the above-mentioned image edge enhancement method.
The storage medium of the invention has stored thereon a computer program which, when being executed by a processor, carries out the above-mentioned method. The storage medium includes: a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, a usb disk, a Memory card, or an optical disk, which can store program codes.
In another embodiment of the disclosure, the present invention further provides a chip system, which is coupled to the memory and configured to read and execute the program instructions stored in the memory to perform the steps of the training method or the image edge enhancement method.
Through the above description of the embodiments, it is clear to those skilled in the art that, for convenience and simplicity of description, the foregoing division of the functional modules is merely used as an example, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device may be divided into different functional modules to complete all or part of the above described functions. For the specific working processes of the system, the apparatus and the unit described above, reference may be made to the corresponding processes in the foregoing method embodiments, and details are not described here again.
Each functional unit in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially implemented or make a contribution to the prior art, or all or part of the technical solutions may be implemented in the form of a software product stored in a storage medium and including several instructions for causing a computer device (which may be a personal computer, a server, or a network device) or a processor to execute all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: flash memory, removable hard drive, read only memory, random access memory, magnetic or optical disk, and the like.
The above description is only a specific implementation of the embodiments of the present application, but the scope of the embodiments of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed in the embodiments of the present application should be covered by the scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application shall be subject to the protection scope of the claims.
Although the embodiments of the present invention have been described in detail hereinabove, it is apparent to those skilled in the art that various modifications and variations can be made to these embodiments. However, it is to be understood that such modifications and variations are within the scope and spirit of the present invention as set forth in the following claims. Moreover, the invention as described herein is capable of other embodiments and of being practiced or of being carried out in various ways.
Claims (11)
1. An image edge enhancement model training method is characterized by comprising the following steps:
carrying out down-sampling processing on an original image sample with a first resolution to obtain a reference image sample with a second resolution, wherein the second resolution is smaller than the first resolution;
interpolating the reference image sample with the second resolution to obtain a rendered image sample with the first resolution;
inputting the rendering image sample to a convolutional neural network model to be trained;
performing feature extraction on the rendered image sample by using a cavity convolution pyramid sampling module and a residual error module of the convolution neural network model to obtain rendering features;
enhancing the rendering characteristics by utilizing a channel weighting and fusing module of the convolutional neural network model to obtain an enhanced image;
calculating a loss value between the original image sample and the enhanced image using a loss function;
adjusting parameters in the convolutional neural network model according to the loss values;
and generating an image edge enhancement model comprising the adjusted convolutional neural network model parameters.
2. The training method of claim 1, wherein performing feature extraction on the rendered image sample by using a cavity convolution pyramid sampling module and a residual error module of the convolutional neural network model to obtain a rendered feature comprises:
extracting domain features of different scales of points to be interpolated from the rendered image sample by using the cavity convolution with different cavity rates in the cavity convolution pyramid sampling module, and performing weighted fusion on the domain features to obtain an initial rendered feature map;
and raising the channel dimension of the initial rendering feature map by utilizing the conventional convolution in the residual error module, performing separable convolution processing on the initial rendering feature map after the channel dimension is raised, and recovering the channel dimension of the initial rendering feature map to obtain the rendering feature.
3. The training method according to claim 1 or 2, wherein the enhancing the rendered features by using the channel weighted fusion module of the convolutional neural network model to obtain an enhanced image comprises:
and determining the attention weight of the rendering feature according to a channel attention mechanism, weighting the rendering feature in the channel direction according to the attention weight, and performing channel fusion to obtain the enhanced image.
4. Training method according to claim 1 or 2, wherein the sampling magnification of the down-sampling process is adjusted within a preset sampling magnification.
5. Training method according to any of claims 1 to 4, wherein the generating of the image edge enhancement model comprising the adjusted network model parameters comprises:
and when the iteration times of the convolutional neural network model to be trained reach a set value or the loss value of the loss function reaches a target value, generating an image edge enhancement model comprising the adjusted network model parameters.
6. An image edge enhancement method applied to the image edge enhancement model of any one of claims 1 to 5, comprising:
acquiring a to-be-processed image with a third resolution, wherein the to-be-processed image is an image subjected to interpolation processing;
performing dimensionality reduction on the image to be processed to obtain a spliced image with a fourth resolution;
inputting the stitched image to the image edge enhancement model;
utilizing the image edge enhancement model to carry out image enhancement on the edges of the spliced image, and outputting the enhanced spliced image;
and splicing the enhanced spliced images sequentially output along the direction of the image channel in space to obtain an edge enhanced image.
7. The method according to claim 6, wherein performing dimensionality reduction on the image to be processed to obtain a stitched image of a fourth resolution comprises:
dividing the image to be processed into N sub-images in equal area, wherein N is a positive integer;
and sequencing the N sub-images, and splicing the N sub-images according to the arrangement sequence of the N sub-images in the direction of an image channel to obtain a spliced image with the fourth resolution.
8. An image edge enhancement model training device is characterized by comprising:
the down-sampling module is used for carrying out down-sampling processing on an original image sample with a first resolution ratio to obtain a reference image sample with a second resolution ratio, wherein the second resolution ratio is smaller than the first resolution ratio;
the interpolation module is used for interpolating the reference image sample with the second resolution ratio to obtain a rendering image sample with the first resolution ratio;
the transmission module is used for inputting the rendering image sample to a convolutional neural network model to be trained;
the extraction module is used for extracting the characteristics of the rendered image sample by utilizing the cavity convolution pyramid sampling module and the residual error module of the convolution neural network model to obtain rendering characteristics;
the rendering enhancement module is used for enhancing the rendering characteristics by utilizing a channel weighting fusion module of the convolutional neural network model to obtain an enhanced image;
a calculation module for calculating a loss value between the original image sample and the enhanced image using a loss function;
the parameter adjusting module is used for adjusting parameters in the convolutional neural network model according to the loss value;
and the generating module is used for generating an image edge enhancement model comprising the adjusted convolutional neural network model parameters.
9. An image edge enhancement apparatus, comprising:
the acquisition module is used for acquiring an image to be processed with a third resolution, wherein the image to be processed is an image subjected to interpolation processing;
the dimensionality reduction module is used for carrying out dimensionality reduction on the image to be processed to obtain a spliced image with a fourth resolution;
an input module, configured to input the stitched image to the image edge enhancement model;
the enhancement processing module is used for utilizing the image edge enhancement model to carry out image enhancement on the edges of the spliced image and outputting the enhanced spliced image;
and the splicing module is used for splicing the enhanced spliced images sequentially output along the image channel direction in space to obtain an edge enhanced image.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the training method of any one of claims 1 to 5 or the image edge enhancement method of claim 6 or 7.
11. A terminal, comprising: a processor and a memory;
the memory is used for storing a computer program;
the processor is configured to execute the computer program stored in the memory to cause the terminal to perform the training method of any one of claims 1 to 5 or the image edge enhancement method of claim 6 or 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110944014.9A CN113658075A (en) | 2021-08-17 | 2021-08-17 | Model training method, image edge enhancement method, device, medium and terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110944014.9A CN113658075A (en) | 2021-08-17 | 2021-08-17 | Model training method, image edge enhancement method, device, medium and terminal |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113658075A true CN113658075A (en) | 2021-11-16 |
Family
ID=78480469
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110944014.9A Pending CN113658075A (en) | 2021-08-17 | 2021-08-17 | Model training method, image edge enhancement method, device, medium and terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113658075A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114581806A (en) * | 2022-03-18 | 2022-06-03 | 重庆科技学院 | Industrial part no-load rate calculation method based on trunk edge feature extraction |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106228512A (en) * | 2016-07-19 | 2016-12-14 | 北京工业大学 | Based on learning rate adaptive convolutional neural networks image super-resolution rebuilding method |
| CN108764133A (en) * | 2018-05-25 | 2018-11-06 | 北京旷视科技有限公司 | Image-recognizing method, apparatus and system |
| CN112801918A (en) * | 2021-03-11 | 2021-05-14 | 苏州科达科技股份有限公司 | Training method of image enhancement model, image enhancement method and electronic equipment |
-
2021
- 2021-08-17 CN CN202110944014.9A patent/CN113658075A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106228512A (en) * | 2016-07-19 | 2016-12-14 | 北京工业大学 | Based on learning rate adaptive convolutional neural networks image super-resolution rebuilding method |
| CN108764133A (en) * | 2018-05-25 | 2018-11-06 | 北京旷视科技有限公司 | Image-recognizing method, apparatus and system |
| CN112801918A (en) * | 2021-03-11 | 2021-05-14 | 苏州科达科技股份有限公司 | Training method of image enhancement model, image enhancement method and electronic equipment |
Non-Patent Citations (1)
| Title |
|---|
| 周杨浩: ""基于深度学习的图像超分辨率重建算法研究"", 《中国优秀硕士学位论文全文数据库（电子期刊）》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114581806A (en) * | 2022-03-18 | 2022-06-03 | 重庆科技学院 | Industrial part no-load rate calculation method based on trunk edge feature extraction |
| CN114581806B (en) * | 2022-03-18 | 2024-03-19 | 重庆科技学院 | Industrial part empty rate calculation method based on trunk edge feature extraction |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zamir et al. | Learning enriched features for fast image restoration and enhancement | |
| Wang et al. | Real-esrgan: Training real-world blind super-resolution with pure synthetic data | |
| Ma et al. | Learning deep context-sensitive decomposition for low-light image enhancement | |
| Chang et al. | Spatial-adaptive network for single image denoising | |
| CN111507994B (en) | Portrait extraction method, portrait extraction device and mobile terminal | |
| Yan et al. | Multi-scale dense networks for deep high dynamic range imaging | |
| CN111476719B (en) | Image processing method, device, computer equipment and storage medium | |
| Gondal et al. | The unreasonable effectiveness of texture transfer for single image super-resolution | |
| Li et al. | FilterNet: Adaptive information filtering network for accurate and fast image super-resolution | |
| Song et al. | Fast image super-resolution via local adaptive gradient field sharpening transform | |
| CN116071243B (en) | Infrared image super-resolution reconstruction method based on edge enhancement | |
| CN112308866A (en) | Image processing method, image processing device, electronic equipment and storage medium | |
| Zwart et al. | Segment adaptive gradient angle interpolation | |
| Liu et al. | Multi-scale grid network for image deblurring with high-frequency guidance | |
| Zheng et al. | Constrained predictive filters for single image bokeh rendering | |
| Zhao et al. | Deep pyramid generative adversarial network with local and nonlocal similarity features for natural motion image deblurring | |
| Zhong et al. | Deep attentional guided image filtering | |
| CN113658075A (en) | Model training method, image edge enhancement method, device, medium and terminal | |
| Chang et al. | Beyond camera motion blur removing: How to handle outliers in deblurring | |
| Esmaeilzehi et al. | MuRNet: A deep recursive network for super resolution of bicubically interpolated images | |
| Rashid et al. | Single MR image super-resolution using generative adversarial network | |
| CN116363561A (en) | Time sequence action positioning method, device, equipment and storage medium | |
| Zhang et al. | A Wavelet-based asymmetric convolution network for single image super-resolution | |
| Lee et al. | Locally Adaptive Channel Attention-Based Spatial–Spectral Neural Network for Image Deblurring | |
| CN113538525A (en) | Optical flow estimation method, model training method and corresponding device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20211116 |