CN111105357A

CN111105357A - Distortion removing method and device for distorted image and electronic equipment

Info

Publication number: CN111105357A
Application number: CN201811252155.9A
Authority: CN
Inventors: 姚佳宝
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2018-10-25
Filing date: 2018-10-25
Publication date: 2020-05-05
Anticipated expiration: 2038-10-25
Also published as: CN111105357B

Abstract

The embodiment of the invention provides a distortion removing method and device for a distorted image and electronic equipment, wherein the method comprises the following steps: generating a side information component corresponding to a distorted image, wherein the distorted image is obtained by processing an original image, and the side information component represents the distortion characteristic of the distorted image relative to the original image; acquiring a characteristic component of the distorted image, wherein the characteristic component is obtained by performing characteristic analysis on the distorted image; inputting the characteristic component and the side information component into a pre-established distortion-removing convolutional neural network model for convolutional filtering processing to obtain a distortion-removing component; and generating a distortion-removed image corresponding to the distortion image according to the distortion-removed component. Thereby de-distorting the distorted image using a de-distorting convolutional neural network model that is different from the prior art.

Description

Distortion removing method and device for distorted image and electronic equipment

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to a distortion removing method for a distorted image, a related apparatus, and an electronic device.

Background

During processing in a digital image system, filtering, data rounding, quantization, or other processing may cause pixel values to shift in intensity, resulting in visual impairment or artifacts. To avoid or mitigate the potential problems described above, it is contemplated that the color components of the image may be post-processed using an image de-distortion filter to recover the pixel intensity shift and mitigate visual loss.

Conventional image de-distortion filters usually artificially reduce the characteristics of distorted images, artificially design Filter structures, and configure Filter coefficients based on experiments and experiences, such as a Deblocking Filter (DBK) used in an image/video coding standard; there is also a class of image de-distortion filters, which adaptively configure Filter structures, Filter coefficients, etc. based on some statistical information of local image regions, such as Sample Adaptive compensation (SAO) used in h.265 standard, Adaptive Loop Filter (ALF) used in video coding and decoding, etc., such Adaptive filters have high complexity, and when used in the field of coding and decoding, Filter related parameters depending on local statistical information need to be written into a code stream to ensure the consistency of coding and decoding ends, which undoubtedly increases the number of coding bits.

In recent years, a deep learning theory is rapidly developed, and different from a traditional feature extraction algorithm depending on prior knowledge, a deep neural network can adaptively construct feature description under the drive of training data, so that the deep neural network has higher flexibility and universality.

Deep learning has been highly successful in many high-level computer vision problems, such as image classification, object detection, and object segmentation. For example, in 2015, an end-to-end image Super-resolution reconstruction algorithm (SRCNN) based on a Convolutional Neural Network has been proposed, and the Network has obvious advantages over the most advanced technology in the Super-resolution reconstruction field at that time, and deep learning begins to show great potential in low-level computer vision problems; for another example, inspired by the application of the convolutional neural network in super-resolution restoration, the industry proposes that the color components of the image after JPEG encoding are post-processed and filtered by using the multi-layer convolutional neural network (arcnn) (artifacts Reduction probabilistic neural network) obtained by training to remove image distortion, and the method obtains obvious main and objective quality improvement compared with the traditional filtering method.

At present, it is also proposed in the industry to use a Convolutional Neural network vrcnn (variable Filter size reliable Neural network) obtained by training to replace the conventional deblocking filtering technology and adaptive sampling point compensation technology, and perform post-processing filtering on color components of an image after intra-coding, and experimental results show that compared with the conventional in-loop filtering method, the Convolutional Neural network greatly improves the main and objective quality of a reconstructed image.

The end-to-end image distortion removal algorithm based on the convolutional neural network avoids the process of manually designing filter coefficients, automatically learns distortion characteristics and a compensation method through data driving, is simpler and more convenient to use, has better generalization and more obvious performance improvement, and is particularly suitable for the field of image/video compression in which various distortions are mixed.

Disclosure of Invention

The embodiment of the invention aims to provide a distortion image distortion removing method to realize distortion removing processing on a distortion image by using a convolution neural network model different from the prior art.

The embodiment of the invention provides a distortion removing method for a distorted image, which comprises the following steps:

generating a side information component corresponding to a distorted image, wherein the distorted image is obtained by processing an original image, and the side information component represents the distortion characteristic of the distorted image relative to the original image;

obtaining a feature component of the distorted image, where the feature component is obtained by performing feature analysis on the distorted image, and for example, the feature component is obtained by performing feature analysis on a color component of the distorted image;

inputting the characteristic component and the side information component into a pre-established distortion-removing convolutional neural network model for convolutional filtering processing to obtain a distortion-removing component;

and generating a distortion-removed image corresponding to the distortion image according to the distortion-removed component.

Further, obtaining the characteristic component of the distorted image includes:

and acquiring the spatial domain characteristic component and/or the transformation domain characteristic component of the distorted image.

Further, the following steps are adopted to perform feature analysis on the distorted image to obtain the spatial domain feature component of the distorted image:

and extracting the spatial domain features of the distorted image to obtain spatial domain feature components of the distorted image.

Further, the extracting spatial domain features of the distorted image to obtain spatial domain feature components of the distorted image includes:

singular value decomposition is carried out on the distorted image to obtain singular values of the distorted image, and the singular values are determined as spatial domain characteristic components of the distorted image; or

And inputting the distorted image into a pre-established feature extraction convolutional neural network model for convolutional filtering processing to obtain the filtering feature of the distorted image, and determining the filtering feature as a spatial domain feature component of the distorted image.

Further, the following steps are adopted to perform feature analysis on the distorted image to obtain a transform domain feature component of the distorted image:

carrying out Fourier transform on the distorted image to obtain a frequency domain characteristic component of the distorted image, and determining the frequency domain characteristic component as a transform domain characteristic component of the distorted image; or

And performing wavelet transformation on the distorted image to obtain a wavelet domain characteristic component of the distorted image, and determining the wavelet domain characteristic component as a transformation domain characteristic component of the distorted image.

Further, the distortion removal component is a distortion removal characteristic component; generating a distortion-removed image corresponding to the distorted image according to the distortion-removed component, including:

carrying out inverse transformation processing on the distortion removal characteristic component to obtain a distortion removal color component;

and generating a distortion-removed image corresponding to the distortion image according to the distortion-removed color component.

Further, the distortion removal component is a distortion removal color component; generating a distortion-removed image corresponding to the distorted image according to the distortion-removed component, including:

An embodiment of the present invention further provides a distortion removing device for a distorted image, including:

the generating module is used for generating a side information component corresponding to a distorted image, wherein the distorted image is obtained by processing an original image, and the side information component represents the distortion characteristic of the distorted image relative to the original image;

the acquisition module is used for acquiring a characteristic component of the distorted image, wherein the characteristic component is obtained by performing characteristic analysis on the distorted image;

the filtering processing module is used for inputting the characteristic component and the side information component into a pre-established distortion-removing convolutional neural network model for convolutional filtering processing to obtain a distortion-removing component;

and the image generation module is used for generating a distortion removal image corresponding to the distortion removal image according to the distortion removal component.

Further, the obtaining module is specifically configured to:

Further, the distortion removal component is a distortion removal characteristic component;

the image generation module is specifically configured to perform inverse transformation processing on the distortion removal characteristic component to obtain a distortion removal color component; and generating a distortion-removed image corresponding to the distortion image according to the distortion-removed color component.

Further, the distortion removal component is a distortion removal color component;

the image generating module is specifically configured to generate a distortion-removed image corresponding to the distorted image according to the distortion-removed color component.

The embodiment of the invention also provides electronic equipment which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;

a memory for storing a computer program;

and the processor is used for realizing the steps of the distortion image distortion removing method when the program stored in the memory is executed.

The embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the method for removing distortion of a distorted image is implemented.

In the distortion removing method for the distorted images, the used distortion removing convolutional neural network model is obtained by training based on a preset distortion removing training set, the preset distortion removing training set comprises a first original sample image, distorted image characteristic components of a plurality of distorted images corresponding to the first original sample image and side information components corresponding to each distorted image, and the side information components can represent the distortion characteristics of the distorted images relative to the original sample image; in the process of carrying out distortion removal filtering, aiming at a distorted image, firstly generating a side information component corresponding to the distorted image, and then inputting a characteristic component and the side information component of the distorted image into a pre-established convolution neural network model to carry out convolution filtering processing to obtain a distortion removal image. Therefore, the distortion image is subjected to distortion removal processing by using the convolutional neural network model.

Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a system architecture diagram of a technical solution provided by an embodiment of the present invention;

fig. 2 is a schematic data flow diagram of a technical solution provided by an embodiment of the present invention;

FIG. 3A is a diagram illustrating one of the side information components according to an embodiment of the present invention;

FIG. 3B is a second diagram illustrating a side information component according to an embodiment of the present invention;

FIG. 4 is a flowchart of a method for removing distortion of a distorted image according to an embodiment of the present invention;

FIG. 5 is a flow chart of another method for removing distortion from a distorted image according to an embodiment of the present invention;

FIG. 6A is a flowchart of obtaining a undistorted color component from the output of a convolutional neural network model according to an embodiment of the present invention;

FIG. 6B is a flow chart of another method for deriving a undistorted color component from the output of the convolutional neural network model according to an embodiment of the present invention;

FIG. 7 is a flowchart of a convolutional neural network model training method according to an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of an apparatus for removing distortion in a distorted image according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of an electronic device for implementing a method for removing distortion of a distorted image according to an embodiment of the present invention;

FIG. 10 is a schematic structural diagram of convolutional neural network model training according to an embodiment of the present invention;

fig. 11 is a schematic structural diagram of an electronic device for implementing a convolutional neural network model training method according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In the processing of the digital image system, filtering, data rounding, quantization or other processing on the original image may cause intensity shift of pixel values of the original image, resulting in visual obstruction or artifacts, that is, the original image may generate distortion after being subjected to image processing, in which case, the original image after being subjected to image processing may be referred to as a distorted image.

In order to solve the problem of how to perform distortion removal processing on a distorted image by using a convolutional neural network, an embodiment of the present invention provides a solution, where for a distorted image obtained by performing image processing on an original image, a corresponding side information component capable of representing a distortion characteristic of the distorted image with respect to the original image is generated, a characteristic component of the distorted image is obtained, the generated side information component and the obtained characteristic component are used together as an input of a pre-established de-distortion convolutional neural network model, and after performing convolutional filtering processing by using the convolutional neural network model, a de-distortion component is obtained and used for generating the de-distorted image.

In the solution, the distortion-removed convolutional neural network model is obtained by training based on a preset training set, and the preset training set includes a first original sample image, feature components of a plurality of distorted images corresponding to the first original sample image, and side information components corresponding to each distorted image, and is obtained by performing iterative training for a plurality of times according to the network structure based on initialized network parameters.

Further, the preset distortion removal training set may include a first original sample image, and the image processing is performed on the first original sample image to obtain a plurality of distorted images with different distortion characteristics, so as to obtain characteristic components of the plurality of distorted images and side information components corresponding to each distorted image;

the preset distortion removal training set may also include a plurality of first original sample images, and the image processing is performed on each first original sample image to obtain a plurality of distorted images with different distortion characteristics, so as to obtain characteristic components of the plurality of distorted images and side information components corresponding to each distorted image.

When the training of a distortion removing convolution neural network model is carried out and the distortion removing convolution neural network model is used for carrying out distortion removing filtering on a distorted image, side information components representing distortion characteristics of the distorted image relative to an original image are used as input of the model, namely, information capable of representing the distortion characteristics of the distorted image is introduced, and through model training and practical application tests aiming at various side information components, the fact that when some specific side information components are used is determined, the convolution neural network model with stronger generalization capability can be trained, namely, on average, the distortion removing capability of a large number of distorted images with difference is stronger, further, better distortion removing processing can be carried out on the distorted image, and the distortion removing color component closer to the color component of the original image is obtained.

Although there are a plurality of methods for performing image distortion removal processing based on a convolutional neural network model at present, in most of the image distortion removal filtering techniques based on a convolutional neural network disclosed currently, if a single pre-training network is used to perform post-processing filtering on images with different distortion degrees (for example, images coded by using different quantization parameters in the coding and decoding field, different upsampling multiples in the super-resolution problem, etc.), there are problems of limited quality improvement or quality loss on images with certain distortion degrees, which indicates that there is a certain problem in the generalization capability of the network.

In order to solve the problem, the prior art proposes to respectively train a set of network parameters for images with different distortion degrees, and to load a corresponding set of network parameters according to the distortion degree of the image in the current application during actual use. However, a large number of parameters waste storage space, and frequent loading of parameters in an application is an unnecessary waste of computing resources.

Therefore, under the processing process of a definite digital image system, the invention of a single convolution neural network which can adapt to post-processing filtering of images with different distortion degrees has important value.

Aiming at the problem that the convolutional neural network with only one set of network parameters in the prior art cannot cope with the distortion removal filtering problem of distorted images with different distortion degrees, further, in the solution provided by the embodiment of the invention, a side information component may be generated that represents a degree of distortion of the distorted image relative to the original image, the pre-distortion training set may include a plurality of distorted images of different degrees of distortion, then, when training the convolutional neural network model, and performing the distortion removal filtering on the distorted image by using the trained distortion removal convolutional neural network model, namely, information which can accurately represent the distortion degree of the distorted image is introduced, so that a convolutional neural network model which is suitable for the distorted images with different distortion degrees can be trained, and then, aiming at the distorted images with different distortion degrees, a better distortion removing effect can be obtained by only using one set of network parameters.

Fig. 1 is a system architecture diagram for implementing the solution, comprising: a side information component generating module 11, a characteristic component acquiring module 12, a convolutional neural network 13 and a network training module 14;

the feature component obtaining module 12 may obtain the distorted image first, and then, according to needs, obtain the feature component of the distorted image by performing feature analysis on the distorted image, for example, may perform spatial domain feature extraction and/or transform domain feature extraction on a color component of the distorted image, where the color component includes a chrominance component and/or a luminance component, or may perform filtering processing again on the feature component of the distorted image, and so on;

or, for the distorted image obtained by encoding and decoding, the characteristic component obtaining module 12 may also directly obtain the characteristic component of the distorted image from the encoding and decoding code stream of the original image.

The convolutional neural network 13 may include the following three layers:

an input layer processing unit 131, configured to receive an input of a convolutional neural network, where the input includes a feature component of a distorted image and a side information component of the distorted image; performing a first layer of convolution filtering processing on the input data;

hidden layer processing section 132 for performing convolution filter processing of at least one layer on the output data from input layer processing section 131;

an output layer processing unit 133, which performs convolution filtering processing of the last layer on the output data of the hidden layer processing unit 132 to obtain a distortion removal component for generating a distortion removal image;

after the output layer processing unit 133 performs the convolution filtering processing of the last layer on the output data of the hidden layer processing unit 132, the distortion color component can be output as a distortion removal component according to the difference of the pre-established distortion removal convolution neural network models; the distortion characteristic component may also be output as a distortion removal component, which is not limited specifically.

After obtaining the distortion removal component, the output layer processing unit 133 may generate a distortion removal image corresponding to the distortion image according to the distortion removal component. For example, under the condition that the undistorted color component is obtained after the convolution filtering processing, a undistorted image corresponding to the distorted image can be generated directly according to the undistorted color component; alternatively, in the case where the distortion-removed feature component is obtained after the convolution filtering process, the distortion-removed feature component may be subjected to an inverse transform process to generate a distortion-removed image.

Fig. 2 is a schematic diagram of a data flow for implementing the solution, where feature components of a distorted image and side information components of the distorted image are input as input data into a pre-trained de-distortion convolutional neural network model, the de-distortion convolutional neural network model may be represented by a convolutional neural network of a preset structure and a configured network parameter set, and the input data is subjected to convolution filtering processing of an input layer, a hidden layer and an output to obtain a de-distortion component, so as to generate a de-distortion image according to the de-distortion component.

In the above solution provided by the embodiment of the present invention, the input data of the undistorted convolutional neural network model may include one or more side information components according to actual needs, and may also include one or more distorted image feature components, for example, spatial domain feature components and/or transform domain feature components of a distorted image. The resulting distortion-removed component may include at least one color component in a certain color space, for example, in an RGB (Red-Green-Blue) color space, at least one of an R color component, a G color component, and a B color component; in the YUV color space, at least one of a Y color component, a U color component, and a V color component may be included, where "Y" represents brightness, "U" represents chroma, and "V" represents concentration; alternatively, the distortion-removed color component may be represented in another color space, and the distortion-removed component may also be a transform domain feature component in a frequency domain or a wavelet domain, which is not limited specifically.

The spatial domain of the image refers to a two-dimensional plane where the image is located, the spatial domain feature extraction processing refers to feature extraction based on image pixel values, and spatial domain feature components can represent information such as color features, texture features or geometric shape features of the image in the spatial domain; the transform domain feature extraction is to perform some conversion processing on an image defined in a spatial domain, convert the image from an original spatial domain into a transform domain, and express the converted image, so as to obtain transform domain feature components of the image, where the transform domain feature components can express feature information of the image in each transform domain.

For example, in some image processing, some spatial domain feature extraction processing may be performed on the distorted image to obtain a spatial domain feature component of the distorted image, and the spatial domain feature component of the distorted image may be used as input data in the distortion removal processing.

For example, the method for extracting the spatial domain features of the distorted image may include the following steps:

singular value decomposition can be carried out on the distorted image to obtain singular values of the distorted image, wherein the singular values can represent energy information of the distorted image;

the distorted image can also be input into a pre-established feature extraction convolutional neural network model for convolutional filtering processing, wherein the feature extraction convolutional neural network model is obtained by training based on a preset feature training set, and the preset feature training set comprises a second original sample image and a spatial domain feature component of the second original sample image;

alternatively, an image feature analysis method based on a gray level co-occurrence matrix, an image feature analysis method based on a direction gradient histogram, or the like may also be used.

For example, in some image processing, some transformation processing may be performed on the distorted image to obtain a transform domain feature component of the distorted image, and in this case, the transform domain feature component of the distorted image may be used as input data when the distortion removal processing is performed.

For example, the method for performing transform domain feature extraction on the distorted image may include the following steps:

fourier transformation can be carried out on the distorted image, and the distorted image is converted from a space domain to a frequency domain to obtain a frequency domain characteristic component of the distorted image;

performing wavelet transformation on the distorted image, and transforming the distorted image from a spatial domain to a wavelet transformation domain to obtain wavelet domain characteristic components of the distorted image;

alternatively, the distorted image may be subjected to discrete cosine transform, PCA (Principal component analysis) transform, or the like.

For the side information component, which represents the distortion characteristics of the distorted image relative to the original image, is an expression of the distortion characteristics determined by the image processing process.

In practical applications, the distortion characteristics may include at least one of the following distortion characteristics:

distortion degree, distortion position, distortion type:

first, the side information component may represent a degree of distortion of the distorted image with respect to the original image.

For example, in a mainstream video coding and decoding application, an image is generally divided into a plurality of non-overlapping and non-fixed-size coding units, the coding units respectively perform predictive coding and quantization processing of different degrees, distortion between the coding units is generally not consistent, and sudden pixel change generally occurs at the boundary of the coding units, so that the boundary coordinates of the coding units can be used as side information for representing the distortion position a priori.

The side information component may also indicate a distortion type of the distorted image relative to the original image, for example, in a video coding and decoding application, different prediction modes may be adopted by different coding units in the image, and the different prediction modes may affect distribution of residual data, thereby affecting characteristics of the distorted image, and therefore, the prediction mode of the coding unit may be used as side information for characterizing the distortion type.

In the above solution provided by the embodiment of the present invention, the side information component may be a combination of one or more of the above, or may be multiple side information components of one of the above, for example, after image processing, the distortion degree of the distorted image may be represented by a parameter with one physical meaning, or the distortion degree of the distorted image may be represented by two parameters with different physical meanings, and accordingly, one or more side information components each representing the distortion degree may be used as input data according to actual needs.

As shown in fig. 3A, the matrix structure of the side information component is the same as the matrix structure of the distorted image, where coordinates [0,0], [0,1] represent distortion positions, and an element value 1 of the matrix represents a distortion degree, i.e., the side information component can represent both the distortion degree and the distortion positions.

As shown in fig. 3B, the coordinates [0,0], [0,1], [2,0], [2,4] represent the distortion position, and the values of the

elements

1 and 2 of the matrix represent the distortion type, i.e., the side information component can represent both the distortion degree and the distortion position.

Moreover, the above solution provided by the embodiment of the present invention may simultaneously include two side information components respectively illustrated in fig. 3A and fig. 3B.

Further, when the distorted image feature components include a plurality of types, the side information component may include side information components corresponding to each type of the distorted image feature components, respectively, according to practical application and requirements of the scheme.

The solution provided by the embodiment of the present invention can be applied to various currently known practical application scenarios, for example, an application scenario in which super-resolution processing is performed on an image, and the present invention is not limited herein.

In the above solution provided by the embodiment of the present invention, a distortion removing method for a distorted image is provided, as shown in fig. 4, specifically including the following processing steps:

and 41, generating a side information component corresponding to the distorted image, wherein the distorted image is obtained by processing the original image, and the side information component represents the distortion characteristic of the distorted image relative to the original image.

The distorted image may be any one of distorted images distorted with respect to the input original image. For example, the distorted image is processed data in an encoding and decoding system, the processed data may be residual coefficients obtained after code stream analysis, may be reconstructed video data, or may be video data obtained by filtering the reconstructed video data by using other filtering techniques, and the video data has distortion with respect to an original video that is not subjected to encoding compression; in some systems, the reconstructed video data or the reconstructed video data after certain processing may be transformed to be used as data to be filtered.

The side information component, representing the distortion characteristics of the distorted image relative to the original image, is a representation of the distortion characteristics determined by the image processing process.

And 42, acquiring a characteristic component of the distorted image, wherein the characteristic component is obtained by performing characteristic analysis on the distorted image.

The characteristic component of the distorted image can be a spatial domain characteristic component of the distorted image, such as a color characteristic, a texture characteristic or a geometric shape characteristic of the distorted image, and the like; and may also be a transform domain feature component of the distorted image, such as frequency domain information, wavelet transform domain information, PCA domain information, and the like of the distorted image.

And 43, inputting the characteristic component and the side information component into a pre-established distortion-removing convolutional neural network model for convolutional filtering processing to obtain a distortion-removing component.

The predistortion convolutional neural network model is obtained by training based on a preset predistortion training set, wherein the preset predistortion training set comprises a first original sample image, a plurality of characteristic components of a distorted image corresponding to the first original sample image and a side information component corresponding to each distorted image.

According to the difference of the pre-established distortion removal convolution neural network models, when the output of the distortion removal convolution neural network model is the characteristic component, the distortion removal component can be the distortion removal characteristic component, and when the output of the distortion removal convolution neural network model is the color component, the distortion removal component can be the distortion removal color component.

And step 44, generating a distortion-removed image corresponding to the distortion image according to the distortion-removed component.

In one case, the distortion removal component is a distortion removal characteristic component, and in this case, the distortion removal characteristic component may be subjected to inverse transformation to obtain a distortion removal color component, and then a distortion removal image corresponding to the distortion removal image may be generated according to the distortion removal color component.

In another case, the distortion component is a distortion-removed color component, and in this case, a distortion-removed image corresponding to the distorted image may be generated directly according to the distortion-removed color component.

The following describes the above-mentioned method for removing distortion of a distorted image in detail by taking the side information component as an example to indicate the distortion degree of the distorted image relative to the original image, with reference to the drawings.

Fig. 5 is a flowchart of another method for removing distortion of a distorted image according to an embodiment of the present invention, which specifically includes the following processing steps:

and 51, determining a distortion degree value of each pixel point of the distorted image aiming at the distorted image to be processed.

In practical application, after the original image is subjected to image processing in different manners, the physical parameters representing the distortion degree may also be different, and therefore, in this step, the corresponding distortion degree value capable of accurately representing the distortion degree of the pixel point may be determined based on different image processing manners, and specifically may be as follows:

the first mode is as follows: for a distorted image obtained by encoding and decoding, the quantization parameter of each coding region is known, that is, the quantization parameter of each coding region can be obtained, and the quantization parameter of the coding region where each pixel point of the distorted image is located is determined as the distortion degree value of each pixel point of the distorted image;

the second mode is as follows: for a distorted image obtained through super-resolution processing, the up-sampling multiple of the image is known, namely the up-sampling multiple during super-resolution processing can be determined as a distortion degree value of each pixel point of the distorted image; for example, in the super-resolution processing of an image, if an image with a resolution of 100 × 100 is processed into an image with a resolution of 200 × 200, the up-sampling multiple is 2.

Both of the above two methods are adopted when the distortion degree of the image is known, and as long as the distortion degree of the distorted image obtained by other image processing methods is known, a similar method can be adopted, and the parameter capable of representing the distortion degree used in the image processing can be directly determined as the distortion degree value representing the distortion degree of the pixel point.

For a distorted image with unknown distortion degree, the following third method can be adopted:

the third mode is as follows: and evaluating the distorted image by using a no-reference image quality evaluation method to obtain a distortion degree value of each pixel point of the distorted image.

For example, the no-reference image quality evaluation method may be an image subjective quality evaluation method. And the testee scores the quality of the current distorted image according to the subjective viewing experience, and the score can be determined as the distortion degree value of each pixel point of the distorted image.

Alternatively, the non-reference image quality evaluation method may be a non-supervised learning method. An evaluation function of the image quality itself may be established in advance to score the quality of the current distorted image, where the evaluation function of the image quality itself may be a sharpness function or a gradient function of the image, and the like, and is not limited specifically.

And step 52, based on the position of each pixel point of the distorted image, generating a side information component corresponding to the distorted image by using the acquired distortion degree value of each pixel point, wherein each component value included in the side information component corresponds to the pixel point at the same position on the distorted image.

Because each component value included in the side information component corresponds to a pixel point at the same position on the distorted image, the side information component and the distorted image have the same structure, namely, the matrix representing the side information component and the matrix representing the color component of the distorted image are of the same type.

In this step, the obtained distortion degree value of each pixel point may be determined as a component value of the same position of the pixel point in the side information component corresponding to the distorted image based on the position of each pixel point of the distorted image, that is, the distortion degree value of each pixel point is directly determined as the component value corresponding to the pixel point.

When the pixel value range of the distorted image is different from the value range of the distortion degree value of the pixel point, the obtained distortion degree value of each pixel point can be standardized based on the pixel value range of the distorted image to obtain a processed distortion degree value, and the value range of the processed distortion degree value is the same as the pixel value range, for example, if the pixel value range of the distorted image is 0 to 255 and the value range of the distortion degree value of the pixel point is 0 to 100, the value range of the processed distortion degree value is 0 to 255 after the distortion degree value of each pixel point is standardized;

and then determining the processed distortion degree value of each pixel point as a component value of the same position of the pixel point in the side information component corresponding to the distorted image based on the position of each pixel point of the distorted image.

In this step, the distortion degree value of the pixel point may be normalized by the following formula:

wherein norm (x) is a processed distortion degree value obtained after standardization, x is a distortion degree value of a pixel point, and the pixel value range of a distorted image is [ PIEXL_MIN,PIXEL_MAX]The value range of the distortion degree value of the pixel point is [ QP ]_MIN,QP_MAX]。

In the above-described steps 61 and 62, that is, the process of generating the side information component of the distorted image and generating the side information component, the side information guide map corresponding to the distorted image may be generated, the side information guide map indicating the degree of distortion of the distorted image by the side information component, and the side information guide map and the distorted image may have the same height and the same width.

In the embodiment of the invention, a scheme is described by taking the structure of a convolutional neural network model comprising an input layer, a hidden layer and an output layer as an example.

And step 53, acquiring characteristic components of the distorted image.

According to the difference of the original images, the manner of obtaining the feature component of the distorted image may also be different, and therefore, in this step, the corresponding manner of obtaining the feature component of the distorted image may be determined based on different original images, which may specifically be as follows:

the first mode is as follows: for a distorted image obtained by encoding and decoding, the characteristic components of the distorted image can be known, and in this case, the characteristic components of the distorted image can be directly obtained from the encoding and decoding code stream of the original image;

the second mode is as follows: firstly, a distorted image can be obtained, and then, according to the requirement, the characteristic component of the distorted image is obtained by carrying out characteristic analysis on the distorted image; for example, the feature analysis of the distorted image includes performing a spatial domain feature extraction process on the distorted image, such as a singular value decomposition, a feature extraction convolution neural network model performing a convolution filtering process, and the like, and also includes performing a transformation process on the distorted image, such as a fourier transform, a wavelet transform, and the like.

In the case of obtaining the feature component of the distorted image according to the second method, taking an example of fourier transform of the color component of the distorted image:

in this step, the fourier transform may be performed on the color components of the distorted image using the following formula:

the matrix size corresponding to the color component of the distorted image is M multiplied by N, F (x, y) is the color component of the distorted image, x is in a value range of [0, M-1], y is in a value range of [0, N-1], F (u, v) is the frequency domain characteristic component of the distorted image, u is in a value range of [0, M-1], v is in a value range of [0, N-1], e is a constant, the value is approximately equal to 2.71828, and j is an imaginary number unit. It can be seen that the matrix size corresponding to the frequency domain characteristic component of the distorted image is the same as the matrix size corresponding to the color component of the distorted image, wherein each point in the matrix corresponding to the frequency domain characteristic component of the distorted image represents a function with a frequency u, v.

Step 54, taking the feature component of the distorted image and the generated side information component as input data of a pre-established distortion-removing convolutional neural network model, and performing a first-layer convolutional filtering process by an input layer, which may specifically be as follows:

in the convolutional neural network model, the input data may be input into the network through respective channels, and in this step, c may be input_yDistorted image feature components Y and c of the channel_mThe side information components M of the channels are combined in the dimension of the channels to form c_y+c_mInput data I of the channel, and carrying out multidimensional convolution filtering and nonlinear mapping on the input data I by adopting the following formula to generate n₁Image blocks represented in sparse form:

F₁(I)＝g(W₁*I+B₁)；

wherein, F₁(I) Is the output of the input layer, I is the input of the convolution layer in the input layer, W is the convolution operation₁Weight coefficients of a convolutional layer filter bank for an input layer, B₁For the offset coefficients of the convolutional layer filter bank of the input layer, g () is a nonlinear mapping function.

Wherein, W₁Corresponds to n₁A convolution filter, i.e. having n₁The convolution filter acts on the input of the convolution layer of the input layer to output n₁Each image block; the size of the convolution kernel of each convolution filter is c₁×f₁×f₁Wherein c is₁For the number of input channels, f₁The size in space for each convolution kernel.

In a toolIn an embodiment of the present invention, the parameters of the input layer may be: c. C₁＝2，f₁＝5，n₁As g () the function of relu (rectified linear unit) is used, which is expressed as:

g(x)＝max(0,x)；

the input layer convolution processing expression in this embodiment is:

F₁(I)＝max(0,W₁*I+B₁)；

step 55 image block F of sparse representation of input layer output by hidden layer₁(I) Further high dimensional mapping is performed.

In the embodiment of the present invention, the number of convolutional layers, the connection manner of the convolutional layers, the attribute of the convolutional layers, and the like included in the hidden layer are not limited, and various structures known at present may be adopted, but the hidden layer includes at least 1 convolutional layer.

For example, the hidden layer comprises N-1(N ≧ 2) convolutional layers, and the hidden layer process is represented by the following formula:

F_i(I)＝g(W_i*F_i-1(I)+B_i),i∈{2,3,…,N}；

wherein, F_i(I) Representing the output of the i-th convolutional layer in a convolutional neural network, W, a convolution operation_iIs the weight coefficient of the i-th convolutional layer filter bank, B_iFor the convolution layer filter bank offset coefficients, g () is the nonlinear mapping function.

Wherein, W_iCorresponds to n_iA convolution filter, i.e. having n_iA convolution filter acting on the input of the i-th convolution layer and outputting n_iEach image block; the size of the convolution kernel of each convolution filter is c_i×f_i×f_iWherein c is_iFor the number of input channels, f_iThe size in space for each convolution kernel.

In one specific embodiment, the hidden layer may include 1 convolutional layer whose convolutional filter parameters are: c. C₂＝64，f₂＝1，n₂Using the relu (rectified linear unit) function as g () 32, the convolution process of the hidden layer in this embodimentThe expression is as follows:

F₂(I)＝max(0,W₂*F₁(I)+B₂)；

step 56, the output layer outputs the high-dimensional image block F output by the hidden layer_N(I) And carrying out aggregation to obtain a distortion removal component for generating a distortion removal image.

In the embodiment of the present invention, as shown in fig. 6A, the output layer may directly output the distortion-removed color component, and then generate a distortion-removed image according to the distortion-removed color component;

or, as shown in fig. 6B, the output layer may also output the distortion removal characteristic component, then perform inverse transformation on the distortion removal characteristic component to obtain a distortion removal color component, and further generate a distortion removal image according to the distortion removal color component; the inverse transformation process corresponds to a transformation process performed when obtaining the transform domain feature component of the distorted image.

For example, continuing the above example, in the case of obtaining the feature component of the distorted image by fourier transform, the inverse transform processing may be performed on the distortion-removed feature component using the following formula:

the matrix size corresponding to the frequency domain characteristic component of the distorted image is M multiplied by N, F (u, v) is the frequency domain characteristic component of the distorted image, u is in a value range of [0, M-1], v is in a value range of [0, N-1], e is a constant, the value is approximately equal to 2.71828, j is an imaginary number unit, F ' (x ', y ') is a distortion-removed color component, x ' is in a value range of [0, M-1], y ' is in a value range of [0, N-1 ]. It can be seen that the frequency domain characteristic components of the distorted image are the same size as the matrix corresponding to the undistorted color components.

In the embodiment of the present invention, the structure of the output layer is not limited, and the output layer may be a Residual Learning structure, a Direct Learning structure, or another structure.

The process using the Residual Learning structure is as follows:

and performing convolution operation on the output of the hidden layer to obtain a compensation residual error, and adding the compensation residual error and the characteristic component of the input distorted image to obtain a distortion removal component and further obtain a distortion removal image. The output layer processing can be represented by the following equation:

F(I)＝W_N+1*F_N(I)+B_N+1+Y；

wherein F (I) is the output of the output layer, F_N(I) As output of the hidden layer, as convolution operation, W_N+1Weight coefficients of the convolutional layer filter bank as output layer, B_N+1Y is a characteristic component of a distorted image to be subjected to a distortion removal process without being subjected to a convolution filter process.

Wherein, W_N+1Corresponds to n_N+1A convolution filter, i.e. having n_N+1A convolution filter for outputting N by acting on the input of the (N + 1) th convolution layer_N+1Image block, n_N+1The number of output distortion removal components is generally equal to the number of input distortion image characteristic components, and if only one type of distortion removal component is output, n is_N+1The value is generally 1; the size of the convolution kernel of each convolution filter is c_N+1×f_N+1×f_N+1Wherein c is_N+1For the number of input channels, f_N+1The size in space for each convolution kernel.

The process using the Direct Learning structure is as follows:

and carrying out convolution operation on the output of the hidden layer and then directly outputting distortion components to obtain a distortion-removed image. The output layer processing can be represented by the following equation:

F(I)＝W_N+1*F_N(I)+B_N+1；

wherein F (I) is the output of the output layer, F_N(I) As output of the hidden layer, as convolution operation, W_N+1Weight coefficients of the convolutional layer filter bank as output layer, B_N+1Is the offset coefficient of the convolutional layer filter bank of the output layer.

Wherein, W_N+1Corresponds to n_N+1A convolution filter, i.e. having n_N+1A convolution filter acting on the input and output of the (N + 1) th convolution layern_N+1Image block, n_N+1The number of output distortion removal components is generally equal to the number of input distortion image characteristic components, and if only one type of distortion removal component is output, n is_N+1The value is generally 1; the size of the convolution kernel of each convolution filter is c_N+1×f_N+1×f_N+1Wherein c is_N+1For the number of input channels, f_N+1The size in space for each convolution kernel.

In a specific embodiment, the output layer adopts a Residual Learning structure, the output layer includes 1 convolution layer, and the convolution filter parameters of the output layer are: c. C₃＝32，f₃＝3，n ₃1, the convolution processing expression of the output layer in this embodiment is:

F(I)＝W₃*F₃(I)+B₃+Y。

in the solution provided by the embodiment of the present invention, a method for training a distortion-removed convolutional neural network model is further provided, as shown in fig. 7, the method specifically includes the following processing steps:

step 71, obtaining a preset distortion removal training set, where the preset distortion removal training set includes a first original sample image, distortion image feature components of a plurality of distortion images corresponding to the first original sample image, and side information components corresponding to each distortion image, where the side information components corresponding to the distortion images represent distortion features of the distortion images relative to the original sample image. The plurality of distorted images differ in distortion characteristics.

In this step, one image processing with different distortion degrees may be performed on the first original sample image (i.e., an undistorted natural image) in advance to obtain respective corresponding distorted images, and according to the steps in the above-mentioned distortion removal method, a corresponding side information component is generated for each distorted image, so that each original sample image, the corresponding distorted image, and the corresponding side information component form an image pair, and the image pair forms a preset training set Ω.

Further, the preset distortion removal training set may include a first original sample image, and the image processing is performed on the first original sample image to obtain a plurality of distorted images with different distortion characteristics and a side information component corresponding to each distorted image;

the preset distortion removal training set may also include a plurality of first original sample images, and the image processing is performed on each first original sample image to obtain a plurality of distorted images with different distortion characteristics and a side information component corresponding to each distorted image.

Step 72, initializing parameters in a network parameter set of the predistortion convolutional neural network CNN with respect to the predistortion convolutional neural network CNN with a preset structure, where the initialized parameter set may be represented by Θ₁It is shown that the initialized parameters can be set according to actual needs and experience.

In this step, the training-related high-level parameters, such as the learning rate and the gradient descent algorithm, may also be set reasonably, and specifically, various manners in the prior art may be adopted, which are not described in detail herein.

Step 73, forward calculation is performed, specifically as follows:

inputting the distorted image characteristic component and the corresponding side information component of each distorted image in the preset distortion removal training set into a distortion removal convolution neural network with a preset structure for convolution filtering processing to obtain a distortion removal component corresponding to the distorted image, wherein the distortion removal component can be a distortion removal color component and/or a distortion removal characteristic component.

In this step, the parameter set may be Θ specifically performed on the preset distortion removal training set Ω_iThe forward calculation of the convolutional neural network CNN obtains the output f (y) of the convolutional neural network, i.e., the distortion-removed component corresponding to each distorted image.

When the processing of this step is entered for the first time, the current parameter set is Θ₁When the processing of this step is subsequently performed again, the current parameter set Θ is used_iFor the last used parameter set Θ_i-1Obtained after adjustment, see the following description.

And step 74, determining a loss value of the undistorted image based on the original image characteristic components of the plurality of first original sample images and the obtained undistorted components.

Specifically, taking the obtained distortion removal component as an example, a Mean Square Error (MSE) formula may be used as a loss function to obtain a loss value L (Θ)_i) See the following formula for details:

wherein H represents the number of image pairs selected from a predetermined training set in a single training, I_hF (I) represents the input data corresponding to the h-th distorted image and obtained by combining the side information component and the characteristic component of the distorted image_h|Θ_i) Indicating that the convolutional neural network CNN is at the parameter set theta for the h-th distorted image_iUndistorted color component, X, from a downward forward calculation_hAnd i is the count of the number of times forward calculation is currently carried out.

And step 75, determining whether the convolutional neural network adopting the preset structure of the current parameter set is converged or not based on the loss value, if not, entering step 76, and if so, entering step 77.

Specifically, convergence may be determined when the loss value is less than a preset loss value threshold; or when the difference between the loss value obtained by the current calculation and the loss value obtained by the previous calculation is smaller than a preset change threshold, determining convergence, which is not limited herein.

Step 76, adjust the parameters in the current parameter set to obtain the adjusted parameter set, and then go to step 73 for the next forward calculation.

The parameters in the current parameter set may be specifically adjusted by using a back propagation algorithm.

Step 77, the final parameter set Θ with the current parameter set as output_finalAnd will use the final parameter set Θ_finalThe predistortion convolutional neural network of the preset structure is used as a training-finished predistortion convolutional neural network model.

Based on the same inventive concept, according to the method for removing distortion of the distorted image provided by the above embodiment of the present invention, correspondingly, another embodiment of the present invention further provides a device for removing distortion of the distorted image, a schematic structural diagram of which is shown in fig. 8, specifically including:

the generating module 81 is configured to generate a side information component corresponding to a distorted image, where the distorted image is obtained by performing image processing on an original image, and the side information component represents a distortion characteristic of the distorted image relative to the original image;

an obtaining module 82, configured to obtain a feature component of the distorted image, where the feature component is obtained by performing feature analysis on the distorted image;

the filtering processing module 83 is configured to input the feature component and the side information component into a pre-established distortion-removed convolutional neural network model for convolutional filtering processing, so as to obtain a distortion-removed component;

and an image generating module 84, configured to generate a de-distorted image corresponding to the distorted image according to the de-distortion component.

The predistortion convolutional neural network model is obtained by training based on a preset predistortion training set, wherein the preset predistortion training set comprises a first original sample image, characteristic components of a plurality of distorted images corresponding to the first original sample image and side information components corresponding to each distorted image.

Further, the obtaining module 82 is specifically configured to obtain a spatial domain feature component and/or a transform domain feature component of the distorted image.

Further, the obtaining module 82 is specifically configured to perform spatial domain feature extraction on the distorted image to obtain a spatial domain feature component of the distorted image.

Further, the obtaining module 82 is specifically configured to perform singular value decomposition on the distorted image to obtain a singular value of the distorted image, and determine the singular value as a spatial domain feature component of the distorted image; or inputting the distorted image into a pre-established feature extraction convolutional neural network model for convolutional filtering processing to obtain the filtering feature of the distorted image, and determining the filtering feature as the spatial domain feature component of the distorted image.

The feature extraction convolutional neural network model is obtained by training based on a preset feature training set, wherein the preset feature training set comprises a second original sample image and a spatial domain feature component of the second original sample image.

Further, the obtaining module 82 is specifically configured to perform fourier transform on the distorted image to obtain a frequency domain feature component of the distorted image, and determine the frequency domain feature component as a transform domain feature component of the distorted image; or performing wavelet transformation on the distorted image to obtain a wavelet domain characteristic component of the distorted image, and determining the wavelet domain characteristic component as a transformation domain characteristic component of the distorted image.

Further, the obtaining module 82 is specifically configured to, for a distorted image obtained through encoding and decoding, obtain a feature component of the distorted image from an encoding and decoding code stream of the original image.

Further, the distortion removal component is a distortion removal characteristic component; the image generating module 84 is specifically configured to perform inverse transformation processing on the distortion-removed feature component to obtain a distortion-removed color component; and generating a distortion-removed image corresponding to the distortion image according to the distortion-removed color component.

Further, the distortion removal component is a distortion removal color component; the image generating module 84 is specifically configured to generate a de-distorted image corresponding to the distorted image according to the de-distorted color component.

Further, the side information component represents at least one of the following distortion characteristics:

representing a degree of distortion of the distorted image relative to the original sample image;

representing a distortion position of the distorted image relative to the original sample image;

representing a type of distortion of the distorted image relative to the original sample image.

Further, the generating module 81 is specifically configured to determine a distortion degree value of each pixel point of the distorted image; and based on the position of each pixel point of the distorted image, generating a side information component corresponding to the distorted image by using the acquired distortion degree value of each pixel point, wherein each component value included in the side information component corresponds to the pixel point at the same position on the distorted image.

Further, the generating module 81 is specifically configured to obtain a quantization parameter of each coding region for a distorted image obtained through encoding and decoding, and determine the quantization parameter of the coding region in which each pixel point of the distorted image is located as a distortion degree value of each pixel point of the distorted image; or, aiming at the distorted image obtained by the super-resolution processing, determining the up-sampling multiple during the super-resolution processing as the distortion degree value of each pixel point of the distorted image; or, evaluating the distorted image by using a no-reference image quality evaluation method to obtain a distortion degree value of each pixel point of the distorted image.

Further, the generating module 81 is specifically configured to determine, based on the position of each pixel point of the distorted image, the obtained distortion degree value of each pixel point as a component value of the same position of the pixel point in the side information component corresponding to the distorted image; or based on the pixel value range of the distorted image, standardizing the acquired distortion degree value of each pixel point to obtain a processed distortion degree value, wherein the value range of the processed distortion degree value is the same as the pixel value range; and determining the processed distortion degree value of each pixel point as a component value of the same position of the pixel point in the side information component corresponding to the distorted image based on the position of each pixel point of the distorted image.

Based on the same inventive concept, according to the method for removing distortion of a distorted image provided in the above embodiment of the present invention, correspondingly, another embodiment of the present invention further provides an electronic device, a schematic structural diagram of which is shown in fig. 9, and the method specifically includes:

the system comprises a processor 91, a communication interface 92, a memory 93 and a communication bus 94, wherein the processor 91, the communication interface 92 and the memory 93 are communicated with each other through the communication bus 94;

a memory 93 for storing a computer program;

the processor 91 is configured to implement any of the above-described steps of the method for removing distortion of a distorted image when executing the program stored in the memory.

The embodiment of the invention also provides a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and when being executed by a processor, the computer program realizes the steps of the distortion removing method of any distorted image.

Based on the same inventive concept, according to the convolutional neural network model training method provided in the above embodiment of the present invention, correspondingly, another embodiment of the present invention further provides a convolutional neural network model training device, a schematic structural diagram of which is shown in fig. 10, and specifically includes:

an obtaining module 101, configured to obtain a preset distortion removal training set, where the preset distortion removal training set includes a first original sample image, distortion image feature components of multiple distortion images corresponding to the first original sample image, and a side information component corresponding to each distortion image, where the side information component corresponding to the distortion image represents a distortion feature of the distortion image relative to the original sample image;

a calculating module 102, configured to input a distorted image feature component and a corresponding side information component of each distorted image in the preset undistorted training set into a convolutional neural network of a preset structure to perform convolutional filtering processing, so as to obtain a undistorted component corresponding to the distorted image, where the undistorted component includes: a de-distorted color component and/or a de-distorted feature component;

a loss value determining module 103, configured to determine a loss value of the de-distorted image based on the obtained original image feature component of the first original sample image and the obtained de-distorted component;

and the model determining module 104 is configured to complete training to obtain a distortion-removed convolutional neural network model when determining that the convolutional neural network of the preset structure converges based on the loss value.

Further, a feature component of a distorted image comprises: spatial domain feature components and/or transform domain feature components of the distorted image.

Further, the obtaining module 101 is specifically configured to perform spatial domain feature extraction on the distorted image to obtain a spatial domain feature component of the distorted image.

Further, the obtaining module 101 is specifically configured to perform singular value decomposition on the distorted image; or inputting the distorted image into a pre-established feature extraction convolution neural network model for convolution filtering processing.

Further, the obtaining module 101 is specifically configured to perform transformation processing on the distorted image to obtain a transform domain feature component of the distorted image.

Further, the obtaining module 101 is specifically configured to perform fourier transform on the distorted image; or performing wavelet transform on the distorted image.

Further, the obtaining module 101 is specifically configured to, for a distorted image obtained through encoding and decoding, obtain a feature component of the distorted image from an encoding and decoding code stream of the original image.

Further, the method also comprises the following steps:

a generating module 105, configured to generate a side information component corresponding to a distorted image by using the following steps:

determining a distortion degree value of each pixel point of the distorted image;

and based on the position of each pixel point of the distorted image, generating a side information component corresponding to the distorted image by using the acquired distortion degree value of each pixel point, wherein each component value included in the side information component corresponds to the pixel point at the same position on the distorted image.

Further, the generating module 105 is specifically configured to obtain a quantization parameter of each coding region for a distorted image obtained through encoding and decoding, and determine the quantization parameter of the coding region in which each pixel point of the distorted image is located as a distortion degree value of each pixel point of the distorted image; or, aiming at the distorted image obtained by the super-resolution processing, determining the up-sampling multiple during the super-resolution processing as the distortion degree value of each pixel point of the distorted image; or, evaluating the distorted image by using a no-reference image quality evaluation method to obtain a distortion degree value of each pixel point of the distorted image.

Further, the generating module 105 is specifically configured to determine, based on the position of each pixel point of the distorted image, the obtained distortion degree value of each pixel point as a component value of the same position of the pixel point in the side information component corresponding to the distorted image; or based on the pixel value range of the distorted image, standardizing the acquired distortion degree value of each pixel point to obtain a processed distortion degree value, wherein the value range of the processed distortion degree value is the same as the pixel value range; and determining the processed distortion degree value of each pixel point as a component value of the same position of the pixel point in the side information component corresponding to the distorted image based on the position of each pixel point of the distorted image.

Based on the same inventive concept, according to the convolutional neural network model training method provided in the above embodiment of the present invention, correspondingly, another embodiment of the present invention further provides an electronic device, a schematic structural diagram of which is shown in fig. 11, specifically including:

the system comprises a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112 and the memory 113 complete mutual communication through the communication bus 114;

a memory 113 for storing a computer program;

the processor 111 is configured to implement any of the above-mentioned convolutional neural network model training method steps when executing the program stored in the memory.

The embodiment of the invention also provides a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and when being executed by a processor, the computer program realizes the steps of any one of the convolutional neural network model training methods.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the device embodiment, the electronic device embodiment and the storage medium embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method of de-distorting a distorted image, comprising:

acquiring a characteristic component of the distorted image, wherein the characteristic component is obtained by performing characteristic analysis on the distorted image;

2. The method of claim 1, wherein obtaining the feature components of the distorted image comprises:

3. The method of claim 2, wherein the distorted image is subjected to feature analysis to obtain spatial domain feature components of the distorted image by the following steps:

4. The method of claim 3, wherein the performing spatial domain feature extraction on the distorted image to obtain spatial domain feature components of the distorted image comprises:

5. The method of claim 2, wherein the distorted image is subjected to feature analysis to obtain transform domain feature components of the distorted image by the steps of:

6. The method of any one of claims 1 to 5, wherein the undistorted component is a undistorted feature component; generating a distortion-removed image corresponding to the distorted image according to the distortion-removed component, including:

7. The method of any of claims 1 to 5, wherein the undistorted component is a undistorted color component; generating a distortion-removed image corresponding to the distorted image according to the distortion-removed component, including:

8. An apparatus for removing distortion from a distorted image, comprising:

9. The apparatus of claim 8, wherein the obtaining module is specifically configured to:

10. The apparatus of claim 9, wherein the obtaining module is specifically configured to:

11. The apparatus of claim 10, wherein the acquisition module is specifically configured to:

12. The apparatus of claim 9, wherein the obtaining module is specifically configured to:

13. The apparatus according to any one of claims 8 to 12, wherein the distortion removal component is a distortion removal characteristic component;

14. The apparatus of any of claims 8 to 12, wherein the undistorted component is a undistorted color component;

15. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;

a memory for storing a computer program;

a processor for implementing the method steps of any of claims 1 to 7 when executing a program stored in the memory.

16. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of claims 1 to 7.