CN112119408A - Method for acquiring image quality enhancement network, image quality enhancement method, image quality enhancement device, movable platform, camera and storage medium - Google Patents
Method for acquiring image quality enhancement network, image quality enhancement method, image quality enhancement device, movable platform, camera and storage medium Download PDFInfo
- Publication number
- CN112119408A CN112119408A CN201980031413.7A CN201980031413A CN112119408A CN 112119408 A CN112119408 A CN 112119408A CN 201980031413 A CN201980031413 A CN 201980031413A CN 112119408 A CN112119408 A CN 112119408A
- Authority
- CN
- China
- Prior art keywords
- image
- image quality
- network
- quality enhancement
- function
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 86
- 238000000605 extraction Methods 0.000 claims abstract description 76
- 238000012549 training Methods 0.000 claims abstract description 62
- 238000012545 processing Methods 0.000 claims abstract description 15
- 230000006870 function Effects 0.000 claims description 151
- 230000004913 activation Effects 0.000 claims description 40
- 238000004422 calculation algorithm Methods 0.000 claims description 27
- 238000013441 quality evaluation Methods 0.000 claims description 10
- 238000004891 communication Methods 0.000 claims description 6
- 239000013598 vector Substances 0.000 claims description 6
- 238000012546 transfer Methods 0.000 claims description 3
- 230000002708 enhancing effect Effects 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 16
- 238000010586 diagram Methods 0.000 description 13
- 230000009467 reduction Effects 0.000 description 6
- 238000007906 compression Methods 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 238000010606 normalization Methods 0.000 description 4
- 230000006978 adaptation Effects 0.000 description 3
- 230000006835 compression Effects 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N3/061—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using biological neurons, e.g. biological neurons connected to an integrated circuit
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/50—Constructional details
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- General Physics & Mathematics (AREA)
- Molecular Biology (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Neurology (AREA)
- Microelectronics & Electronic Packaging (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the application provides a method for acquiring an image quality enhancement network, an image quality enhancement method, an image quality enhancement device, a movable platform, a camera and a computer-readable storage medium. The method for acquiring the image quality enhancement network comprises the following steps: training a preset generative countermeasure network according to a plurality of preset training images; wherein the generative confrontation network comprises a generator; after the training of the generative confrontation network is finished, taking the generator as an image quality enhancement network; the generator or the image quality enhancement network comprises N scale extraction networks for extracting features of different scales, wherein N is an integer greater than 0, and the image quality enhancement network is used for carrying out image quality enhancement processing on any input image to be enhanced. According to the embodiment, the image quality enhancement network capable of enhancing the image quality is obtained through training of the image quality enhancement network, and the user experience is improved.
Description
Technical Field
The embodiment of the application relates to the technical field of image processing, in particular to a method for acquiring an image quality enhancement network, an image quality enhancement method, an image quality enhancement device, a movable platform, a camera and a computer-readable storage medium.
Background
With the development of the technology, the image quality of the image is required to be higher and higher, but due to the limitation of network bandwidth or storage space, the image quality of the image has to be reduced in the image transmission or storage process to reduce the size of the image, so that the user experience is not good. However, the lossy compression method reduces the image quality of an image.
In the related art, image quality enhancement algorithms, such as histogram equalization or gamma correction, mainly perform image enhancement through human summary experience and human eye characteristics, and the image quality enhancement capability is limited and is easily limited by the image scene.
Disclosure of Invention
The embodiment of the application provides a method for acquiring an image quality enhancement network, an image quality enhancement method, an image quality enhancement device, a movable platform, a camera and a computer-readable storage medium.
In a first aspect, an embodiment of the present application provides a method for obtaining an image quality enhancement network, including:
training a preset generative countermeasure network according to a plurality of preset training images; wherein the generative confrontation network comprises a generator;
after the training of the generative confrontation network is finished, taking the generator as an image quality enhancement network; the generator or the image quality enhancement network comprises N scale extraction networks for extracting features of different scales, wherein N is an integer greater than 0, and the image quality enhancement network is used for carrying out image quality enhancement processing on any input image to be enhanced.
In a second aspect, an embodiment of the present application provides an image quality enhancement method, including:
inputting an image to be promoted into a preset image quality enhancement network, and carrying out image quality enhancement on the image to be promoted by the image quality enhancement network to obtain an image with image quality higher than that of the image to be promoted; the image quality enhancement network comprises N scale extraction networks for extracting different scale features, and is obtained by training as a generator of a generating countermeasure network; n is an integer greater than 0.
In a third aspect, an embodiment of the present application provides an apparatus for acquiring an image quality enhancement network, including:
a memory and a processor; the memory is connected with the processor through a communication bus and is used for storing computer instructions executable by the processor; the processor is configured to read the computer instructions from the memory to obtain the method of the first aspect.
In a fourth aspect, an embodiment of the present application provides an image quality enhancement apparatus, including a memory and a processor; the memory is connected with the processor through a communication bus and is used for storing computer instructions executable by the processor; the processor is configured to read a computer instruction from the memory to implement the image quality enhancement method according to any one of the second aspect.
In a fifth aspect, an embodiment of the present application provides a movable platform, including:
a body;
the power system is arranged in the machine body and used for providing power for the movable platform; and the number of the first and second groups,
the image quality enhancement apparatus according to the fourth aspect.
In a sixth aspect, an embodiment of the present application provides a camera, including:
a housing;
the lens assembly is arranged inside the shell;
the sensor assembly is arranged in the shell and used for sensing the light passing through the lens assembly and generating an electric signal; and the number of the first and second groups,
the image quality enhancement apparatus according to the fourth aspect.
In a seventh aspect, this application provides a computer-readable storage medium, on which computer instructions are stored, and when executed, the computer instructions implement the steps of the method according to any one of the first and second aspects.
In this embodiment, training is performed based on the architecture of the generative countermeasure network, so that the generator in the generative countermeasure network includes N scale extraction networks for extracting features of different scales, and thus the generator is used as an image quality enhancement network to implement image quality enhancement processing on any input image to be enhanced.
In this embodiment, an image to be enhanced is input to a preset image quality enhancement network, and the image quality enhancement network performs image quality enhancement on the image to be enhanced to obtain a target image with a higher image quality than the image quality of the image to be enhanced, so as to implement an image quality enhancement process.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive labor.
Fig. 1 is a flowchart of a method for acquiring an image quality enhancement network according to an embodiment of the present disclosure.
Fig. 2 is a structural diagram of an image quality enhancement network according to an embodiment of the present application.
Fig. 3 is a structural diagram of a scale extraction network according to an embodiment of the present application.
Fig. 4 is a flowchart of another method for acquiring an image quality enhancement network according to an embodiment of the present disclosure.
Fig. 5 is a training structure diagram of a generative confrontation network according to an embodiment of the present disclosure.
Fig. 6 is a structural diagram of an image quality evaluation network according to an embodiment of the present application.
Fig. 7 is a flowchart of an image quality enhancement method according to an embodiment of the present disclosure.
Fig. 8 is a scene diagram of an image quality enhancement network application according to an embodiment of the present application.
Fig. 9 is a block diagram of an apparatus for acquiring an image quality enhancement network according to an embodiment of the present application.
Fig. 10 is a block diagram of an image quality enhancement apparatus according to an embodiment of the present application.
Fig. 11 is a structural diagram of a movable platform according to an embodiment of the present application.
Fig. 12 is a structural diagram of a camera according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context. The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Image quality enhancement algorithms in the related art, such as histogram equalization or gamma correction, mainly perform image enhancement through human summary experience and human eye characteristics, have limited image quality improvement capability, are easily limited by image scenes, and can only process the same type of images in specific scenes; or some professional image quality optimization applications such as Photoshop applications require users to have higher software use skills and experiences, and meanwhile, the processing time is longer, which depends on a manual operation process, and the image quality improvement efficiency is not high.
Based on this, please refer to fig. 1, which is a flowchart illustrating a method for acquiring an image quality enhancement network according to an exemplary embodiment of the present application, in the embodiment illustrated in fig. 1, the method may be executed by an electronic device, where the electronic device may be a computing device such as a server or a terminal, and the method includes:
in step S101, a preset generative countermeasure network is trained according to a plurality of preset training images; wherein the generative countermeasure network comprises a generator.
In step S102, after the training of the generative confrontation network is completed, the generator is used as the image quality enhancement network; the generator or the image quality enhancement network comprises N scale extraction networks for extracting features of different scales, wherein N is an integer greater than 0, and the image quality enhancement network is used for carrying out image quality enhancement processing on any input image to be enhanced.
It should be noted that the picture quality is picture quality, which is related to the degree of lossy compression of the image, and if image information is lost in the compression process of the image, the picture quality is also reduced correspondingly. In one example, for example, an image contains high frequency information and low frequency information, the frequency of the high frequency information changes rapidly, the frequency of the low frequency information changes slowly, and a part of the high frequency information may be lost during the compression process of the image, so that the image quality of the compressed and decompressed image is reduced compared with the uncompressed original image.
In one example, the original image is encoded by an encoder to obtain a lossy encoded binary file, and then the binary file is decoded by a decoder to obtain the training image.
It is to be understood that, the specific types of the encoder and the decoder are not limited in any way in the embodiments of the present application, and may be specifically selected according to actual situations; in a possible implementation manner, a plurality of encoders of different types may be selected to encode an original image, so as to obtain a compressed image encoded based on different encoders, and then the compressed image is decoded by a decoder to obtain a training image. In the training process, according to different types of encoders, a plurality of training images corresponding to the types of the encoders are adopted to respectively train the generative networks so as to obtain the image quality enhancement networks meeting the encoding modes of different types, realize the image quality enhancement in different directions and improve the image quality enhancement capability.
In one possible implementation, to improve the efficiency of network training, the original image is an image that is pre-cropped to a specified size, in one example, the original image may be an image block cropped to a size (length, width, and depth) of 384 × 3.
In this embodiment, the Generative Adaptation Networks (GAN) is a deep learning model, and the Generative adaptation Networks include a generator and a discriminator, the electronic device takes the training image as an input of the generator, trains the generator, and takes the generator as an image quality enhancement network after the training of the generator, where the image quality enhancement network includes N scale extraction Networks for extracting features of different scales, and N is an integer greater than 0.
In addition, in order to improve the training stability and convergence of the network, the training image and the original image may be normalized, and for example, the electronic device may divide each channel pixel of the training image and the original image by 255.0, so that each channel pixel of the training image and the original image is represented between [0,1], thereby implementing the normalization process of the training image and the original image.
Referring to fig. 2, it is a structural diagram of the image quality enhancement network, where the image quality enhancement network further includes a first convolution layer 21, where the first convolution layer 21 is used to perform a convolution operation on the training image to realize a feature extraction process, so as to obtain image data after the convolution operation; the first convolution layer 21 includes a convolution kernel and an activation function, and the activation function may be any one of a PReLU function, a ReLU function, a tanh function, a Sigmod function, and an ELU function, and may be, for example, a PReLU function with a faster calculation speed; it is understood that, in the embodiment of the present application, there is no limitation to the size of the convolution kernel, and the size may be specifically set according to actual situations.
In addition, since the scale of the output image is reduced after the convolution operation, based on this, the first convolution layer 21 is also used to perform an expansion (padding) operation on the input training image to ensure that the size (length and width) of the output image is the same as that of the input image.
Referring to fig. 2, the first convolution layer 21 is connected to the scale extraction network 22, N scale extraction networks 22 are connected in sequence, and fig. 2 illustrates an example of 4 scale extraction networks 22, where an input image of each scale extraction network 22 is an output image of the first convolution layer 21 or an output image of a previous scale extraction network 22, that is, if the scale extraction network 22 is connected to the first convolution layer 21, an input image thereof is an output image of the first convolution layer 21, and if the scale extraction network 22 is not connected to the first convolution layer 21, an input image thereof is an output image of the previous scale extraction network 22.
In an embodiment, the scale extraction networks 22 are respectively configured to extract different scale features of an input image, please refer to fig. 2, the image quality enhancement network further includes at least one third convolution layer 23, which is exemplarily illustrated in fig. 2 by taking 3 third convolution layers 23 as an example, and the at least one third convolution layer 23 is configured to perform dimension reduction on the scale features output by the scale extraction networks 22 to obtain a low-dimensional feature vector to generate an image quality enhanced target image; it is understood that the number of the third convolutional layers 23 can be specifically set according to practical situations, and the embodiment of the present application does not limit this.
In one implementation, the plurality of scale extraction networks 22 extract different scale features of an input image to generate an image including the different scale features, since the dimensions (depths) of the output image including the different scale features are increased after being processed by the first convolution layer 21 and the scale extraction network 22, and in order to ensure that the dimensions (depths) of the finally output image and the training image are the same, dimension reduction processing needs to be performed on the image including the different scale features, at least one third convolution layer 23 sequentially connected is arranged in the image quality enhancement network, the third convolution layer 23 performs dimension reduction on the image including the different scale features to obtain a low-dimensional image including the different scale features, and generates an image quality enhanced target image, wherein the number of the third convolution layers can be adaptively adjusted according to the compression degree of the training data, to output an image with a good quality.
As an example, for example, the size of the training image is 384 × 3, the dimension thereof is 3, the size of the image including the features of different scales generated by the scale extraction network 22 is 384 × 9, the dimension (depth) thereof is 9, and a third convolution layer 23 is required to perform dimension reduction processing on the image, for example, the image quality enhancement network includes 2 third convolution layers 23, the two third convolution layers 23 sequentially perform dimension reduction on the image including the features of different scales, for example, the first third convolution layer 23 reduces the image into an image with a size of 384 × 6, and the second third convolution layer 23 reduces the image into an image with a size of 384 × 3, so that the dimension of the finally obtained image including the features of different scales is the same as the dimension of the training image; it is understood that the dimension of each dimension reduction can be specifically set according to actual situations, and the embodiment of the present application does not set any limitation.
Wherein the third convolutional layer 23 includes a convolutional kernel and an activation function, the activation function of the third convolutional layer 23 may be any one of a prellu function, a ReLU function, a tanh function, a Sigmod function, and an ELU function, as an example, the activation function of the third convolutional layer which is not the last layer may be a prellu function with a faster calculation speed, and the activation function of the third convolutional layer which is the last layer may be a Sigmod function, so as to implement the normalization process; it is understood that, in the embodiment of the present application, there is no limitation to the size of the convolution kernel, and the size may be specifically set according to actual situations.
Referring to fig. 3, a structure diagram of the scale extraction network is shown, the scale extraction network includes a plurality of filters 221, a splicing layer 222, a second convolution layer 223, and an addition layer 224, which are connected in sequence:
the plurality of filters 221 are used for extracting scale features of the input image to obtain a plurality of different scale features, the scale features extracted by the different filters 221 are different, and the plurality of filters 221 contribute to effective extraction of the scale features. For example, the filter includes convolution kernels, different scale features of the input image can be obtained through different convolution kernels, a large convolution kernel can obtain contour information of the input image, and a small convolution kernel can obtain detail information of the input image. It should be noted that, in the embodiment of the present application, there is no limitation on the arrangement manner of the plurality of filters 221, for example, the filters 221 may be arranged in parallel, and as an example, 2 filters 221 listed in fig. 3 are used for exemplary illustration: the 2 filters 221 form a parallel relationship, and the electronic device inputs the input image into the first filter 221 and the second filter 221 respectively to perform scale feature extraction, so as to obtain 2 different scale features; it should be noted that the "first" and "second" are only used to distinguish the filter 221, and do not have any meaning per se.
Wherein the filter 221 includes a convolution kernel and an activation function, and the activation function may be any one of a prilu function, a ReLU function, a tanh function, a Sigmod function, and an ELU function, and may be, for example, a prilu function with a faster computation speed; it can be understood that, in the embodiment of the present application, no limitation is imposed on the size of the convolution kernel, the sizes of the convolution kernels between different filters 221 may be the same or different, and may be specifically set according to actual situations.
In addition, considering that the image size is reduced after the convolution operation by the convolution kernel, in order to ensure that the input image and the output image have the same size for the convenience of subsequent splicing, the filter 221 is also used for performing an expansion (padding) operation on the input image to ensure that the output image and the input image have the same size (length and width).
Then, the outputs of the filters 221 are used as the inputs of the splicing layer 222, and the splicing layer 222 is used for splicing a plurality of different scale features output by the filters 221 to obtain spliced image data; the description is given by taking fig. 3 as an example: for example, 2 images (including length, width, and dimension) of size 254 × 128 that are finally output by the plurality of filters 221, the size of the stitched image is 254 × 256(128 × 2 — 256).
As can be seen, the dimension of the stitched image data is increased, and the stitched image data needs to be processed with the input image input to the scale extraction network subsequently, the scale extraction network sets a second convolution layer 223, and the second convolution layer 223 performs convolution operation on the stitched image data to obtain image data with the same dimension as the input image; finally, the addition layer 224 performs point-by-point summation on the image data output by the second convolution layer 223 and the input image to obtain image data containing different scale features, that is, the addition layer 224 fuses the extracted different scale features with the original scale features of the input image to further obtain image data containing different scale features.
Referring to fig. 4, which is a flowchart illustrating another method for obtaining an image quality enhancement network according to an exemplary embodiment of the present application, in the embodiment illustrated in fig. 4, the method may be executed by an electronic device, where the electronic device may be a computing device such as a server or a terminal, and the method includes:
in step S201, a preset generative countermeasure network is trained according to a plurality of preset training images and original images; wherein the generative confrontation network comprises a generator and an arbiter.
In step S202, after the training of the generative confrontation network is completed, the generator is used as an image quality enhancement network, and the discriminator is used as an image quality evaluation network; the generator or the image quality enhancement network comprises N scale extraction networks for extracting features of different scales, wherein N is an integer greater than 0, and the image quality enhancement network is used for carrying out image quality enhancement processing on any input image to be enhanced so as to obtain a target image; the image quality evaluation network is used for evaluating the probability that the target image is the original image.
In an embodiment, the input data of the image quality evaluation network is a target image and the original image output by the image quality enhancement network, and the output data is a probability that the target image is the original image.
Referring to fig. 5, in the training structure diagram of the generative countermeasure network, after an original image is decoded by an encoder 40 and a decoder 10, a training image is obtained, the training image is input into a generator 20, the generator 20 outputs an image quality enhanced target image, the target image and the original image are input into a discriminator 30, then the generative countermeasure network is trained according to the probability that the generated target image is the original image, and parameters of the generator and the discriminator in the generative countermeasure network are updated.
Referring to fig. 6, it is a structural diagram of the image quality evaluation network, the image quality evaluation network includes a fifth convolutional layer 31, a plurality of sixth convolutional layers 32 (4 sixth convolutional layers 32 are exemplified in fig. 6), a first fully-connected layer 33, and a second fully-connected layer 34, which are connected in sequence:
the fifth convolution layer 31 is used for performing convolution operation on the input target image and the input original image to realize feature extraction; the sixth convolutional layers 32 are sequentially connected, and an input image of the sixth convolutional layer 32 is an output image of the fifth convolutional layer 31 or an output image of the previous sixth convolutional layer 32, that is, if the sixth convolutional layer 32 is connected to the fifth convolutional layer 31, an input image thereof is an output image of the fifth convolutional layer 31, and if the sixth convolutional layer 32 is not connected to the fifth convolutional layer 31, an input image thereof is an output image of the previous sixth convolutional layer 32; the sixth convolutional layer 32 is used for extracting features of the input image to obtain high-dimensional features, taking fig. 6 as an example for explanation, assuming that the dimension of the image data output from the fifth convolutional layer 31 is 64, the 1 st sixth convolutional layer 32 performs high-dimension feature extraction on the output image of the fifth convolutional layer 31, performs convolution operation, so that the dimension of the output image is 128 dimensions, the 2 nd sixth convolutional layer 32 performs high-dimensional feature extraction on the last sixth convolutional layer 32, and after the convolution operation, so that the dimension of the output image is 256 dimensions, the 3 rd sixth convolutional layer 32 performs high-dimensional feature extraction on the last sixth convolutional layer 32, and after the convolution operation, the dimensionality of the output image is 512 dimensions, the 4 th sixth convolutional layer 32 performs high-dimensional feature extraction on the last sixth convolutional layer 32, and the dimensionality of the output image is 1024 dimensions through convolution operation.
Wherein the fifth convolutional layer 31 includes a convolutional kernel and an activation function, and the activation function may be any one of a PReLU function, a ReLU function, a tanh function, a Sigmod function, and an ELU function, and for example, the activation function may be a PReLU function with a faster calculation speed; it is understood that, in the embodiment of the present application, there is no limitation to the size of the convolution kernel, and the size may be specifically set according to actual situations.
The sixth convolution layer 32 includes a convolution kernel, an activation function, and a BN (Batch Normalization) layer, the activation function may be any one of a prellu function, a ReLU function, a tanh function, a Sigmod function, and an ELU function, and the activation function may be, for example, a ReLU function with a faster computation speed; it can be understood that, in the embodiment of the present application, no limitation is imposed on the size of the convolution kernel, and the size may be specifically set according to actual situations; the BN layer can accelerate the network convergence speed.
Next, the first fully-connected layer 33 classifies the high-dimensional features output by the sixth convolutional layer 32; finally, the second fully-connected layer 34 classifies the result output by the first fully-connected layer 33 to obtain the probability that the target image is the original image, and trains the entire generative countermeasure network based on the probability and a preset loss function.
In one embodiment, the training of the predetermined generative confrontation network includes: training the generative countermeasure network based on a preset loss function; wherein the loss function comprises a countering loss function and a picture quality loss function; the resistance loss function is used for evaluating the generation capability of the image quality enhancement network for generating a target image and the discrimination capability of the image quality evaluation network for discriminating the target image from an original image; the image quality loss function is used for evaluating the generation capability of the image quality enhancement network, namely, evaluating whether the image quality enhancement network converges, whether the target image meets a preset condition and the like.
The countermeasure loss function is realized by Wasserstein loss, so that the problem of gradient disappearance can be solved, and the network training process is simplified; the image quality loss function in the generative countermeasure network may be any one of: the Multi-level structure Similarity algorithm (MS-SSIM, Multi-Scale-Structure Similarity Index), the structure Similarity algorithm (SSIM, Structure Similarity Index), the Mean Absolute Difference algorithm (MAD, Mean Absolute Difference), the Sum of Absolute differences algorithm (SAD, Sum of Absolute Difference), the Hadamard transform algorithm (SATD, Sum of Absolute Transform Difference), the Sum of Squared differences algorithm (SSD, Sum of Squared Difference) and the Mean Squared error algorithm (MSD, Mean Squared Difference).
For example, the loss function is related to a VGG-19 pre-training model, and the loss function is set to lVGGThen, thenWherein I ═ G (D (E (I))0) E) represents an encoder, D represents a decoder, G represents an image quality enhancement network, and the embodiment compares the image quality enhancement network output in a high-dimensional feature space by using a VGG-19 pre-training model as an image quality loss function through the VGG-19 pre-training modelAnd (4) information loss between the target image and the original image.
It can be understood that the ratio between the penalty function and the loss of image quality function can be specifically set according to actual situations, and the embodiment of the present application does not limit the ratio; by way of example, let the penalty function be ladverLet the loss function of the generative countermeasure network be lossallThen the penalty function can be set to: lossall=lVGG+0.001*ladver。
In one embodiment, the optimizer in the generative countermeasure network employs any one of: root Mean Square transfer algorithm (RMSProp), Adaptive Moment Estimation algorithm (Adam), adagard algorithm, and adalta algorithm.
In an embodiment, the parameters of the image quality enhancement network may also be adjusted based on different training image formats (such as RGB format, YUV format, etc.) to adapt to images of different formats.
Accordingly, referring to fig. 7, an embodiment of the present application further provides an image quality enhancement method, which can be executed by an electronic device, where the electronic device may be a server or a terminal, and the method includes:
step 301, inputting an image to be promoted into a preset image quality enhancement network, and performing image quality enhancement on the image to be promoted by the image quality enhancement network to obtain a target image with image quality higher than that of the image to be promoted; the image quality enhancement network comprises N scale extraction networks for extracting different scale features, and is obtained by training as a generator of a generating countermeasure network; n is an integer greater than 0.
In this embodiment, the image to be promoted is an image that is output after being encoded by an encoder and decoded by a decoder and loses image information, and it can be understood that the format of the image to be promoted is not limited in this embodiment, and the image to be promoted may be an RGB image or a YUV image, for example.
Since the image quality enhancement network has a convolutional layer structure, an image with an arbitrary resolution can be processed.
Referring to fig. 8, in order to apply the scene graph to the image quality enhancement network, the image to be enhanced decoded by the decoder 10 is input into the image quality enhancement network 20 for operation, so as to obtain the target image with the image quality higher than that of the image to be enhanced.
In an implementation manner, the image quality enhancement network includes a first convolution layer, the first convolution layer is connected to a first one of the N scale extraction networks, and is configured to perform convolution operation on the image to be enhanced to obtain image data after the convolution operation, the N scale extraction networks are sequentially connected, an input image of each scale extraction network is an output image of the first convolution layer or an output image of a previous scale extraction network, and the scale extraction networks are used for extracting different scale features of the input image; wherein the first convolution layer includes a convolution kernel and an activation function, for example, the activation function is a PreLU function.
Then, the image quality enhancement network further comprises at least one third convolution layer, wherein the at least one third convolution layer is used for reducing the dimension of the scale features output by the scale extraction structure to obtain low-dimension feature vectors so as to generate an image with enhanced image quality; the activation function of the third convolutional layer 23 may be any one of a prellu function, a ReLU function, a tanh function, a Sigmod function, and an ELU function, for example, the activation function of the third convolutional layer that is not the last layer may be a prellu function with a faster calculation speed, and the activation function of the third convolutional layer that is the last layer may be a Sigmod function, so as to implement the normalization process.
In one embodiment, the scale extraction network comprises: a plurality of filters, a splice layer, a second convolution layer and an addition layer connected in sequence:
the filters are used for respectively extracting different scale features of the input image to obtain a plurality of different scale features, and the scale features extracted by the different filters are different; the splicing layer is used for splicing a plurality of different scale characteristics to obtain spliced image data; the second convolution layer performs convolution operation on the spliced image data to obtain image data with the same dimensionality as the input image; the addition layer carries out point-by-point summation on the image data output by the second convolution layer and the input image to obtain image data containing different scale characteristics; wherein the filter includes a convolution kernel and an activation function, which is, for example, a PreLU function.
In an example, for example, if a to-be-promoted image loses part of image information (e.g., high-frequency information or other image information) during an encoding process, in the process of enhancing the image quality of the to-be-promoted image, the multiple filters in the scale extraction network may extract different scale features of the to-be-promoted image, so as to restore the image information.
Accordingly, referring to fig. 9, an embodiment of the present application further provides an apparatus 100 for acquiring an image quality enhancement network, including:
a memory 101 and a processor 102; the memory 101 is connected to the processor 102 through a communication bus 103 for storing computer instructions executable by the processor 102; the processor 102 is configured to read computer instructions from the memory 101, and when executed, is configured to:
training a preset generative countermeasure network according to a plurality of preset training images; wherein the generative countermeasure network comprises a generator.
After the training of the generative confrontation network is finished, taking the generator as an image quality enhancement network; the generator or the image quality enhancement network comprises N scale extraction networks for extracting features of different scales, wherein N is an integer greater than 0, and the image quality enhancement network is used for carrying out image quality enhancement processing on any input image to be enhanced.
The Processor 102 executes the program codes included in the memory 101, and the Processor 102 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, a discrete hardware component, or the like. The general purpose processor 102 may be a microprocessor or the processor 102 may be any conventional processor or the like.
The memory 101 stores a program code of the method for acquiring the picture quality enhancement network, and the memory 101 may include at least one type of storage medium, which includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. Further, the apparatus 100 for acquiring the image quality enhancement network may cooperate with a network storage apparatus that performs a storage function of the memory 101 through network connection. The storage 101 may be an internal storage unit of the apparatus 100 for acquiring the image quality enhancement network, for example, a hard disk or a memory of the apparatus 100 for acquiring the image quality enhancement network. The memory 101 may also be an external storage device of the apparatus 100 for obtaining the image quality enhancement network, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the apparatus 100 for obtaining the image quality enhancement network. Further, the memory 101 may include both an internal storage unit and an external storage device of the apparatus 100 for acquiring the image quality enhancement network. The memory 101 is used to store computer program code and other programs and data needed to acquire the device 100 of the network for quality enhancement. The memory 101 may also be used to temporarily store data that has been output or is to be output.
The various embodiments described herein may be implemented using a computer-readable medium such as computer software, hardware, or any combination thereof. For a hardware implementation, the embodiments described herein may be implemented using at least one of an Application Specific Integrated Circuit (ASIC), a digital signal processor 102(DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a processor 102, a controller, a microcontroller, a microprocessor 102, an electronic unit designed to perform the functions described herein. For a software implementation, the implementation such as a process or a function may be implemented with a separate software module that allows performing at least one function or operation. The software code may be implemented by a software application (or program) written in any suitable programming language, which may be stored in memory 101 and executed by the controller.
The apparatus 100 for acquiring an image quality enhancement network may include, but is not limited to, a memory 101 and a processor 102. It will be understood by those skilled in the art that fig. 9 is only an example of the apparatus 100 for acquiring the image quality enhancement network, and does not constitute a limitation to the apparatus 100 for acquiring the image quality enhancement network, and may include more or less components than those shown in the drawings, or may combine some components, or different components, for example, the apparatus may further include an input/output device, a network access device, and the like.
As an example, the training image is generated from the original image by encoding and decoding.
As an example, N of the scale extraction networks are connected in sequence.
The image quality enhancement network further comprises a first convolution layer; the first convolution layer is connected with a first one of the N scale extraction networks and is used for performing convolution operation on the training image to obtain image data after the convolution operation.
The input image of each scale extraction network is the output image of the first convolution layer or the output image of the last scale extraction network.
As an example, the scale extraction network comprises a plurality of filters; the filters are used for extracting the scale features of the input image to obtain a plurality of different scale features.
By way of example, the filter includes a convolution kernel and an activation function.
As an example, the activation function includes at least one of: a PReLU function, a ReLU function, a tanh function, a Sigmod function, and an ELU function.
As an example, the activation function is a PreLU function.
As an example, the scale extraction network further includes a splicing layer, where the splicing layer is configured to splice a plurality of different scale features output by the plurality of filters to obtain spliced image data.
As an example, the scale extraction network further comprises a second convolutional layer; and the second convolution layer performs convolution operation on the spliced image data to obtain image data with the same dimensionality as the input image.
As an example, the scale extraction network further comprises an addition layer; and the addition layer performs point-by-point summation on the image data output by the second convolution layer and the input image to obtain image data containing high-frequency characteristics and low-frequency characteristics.
As an example, the image quality enhancement network further includes at least one third convolution layer; and the at least one third convolution layer is used for reducing the dimension of the scale features output by the scale extraction network to obtain low-dimension feature vectors and generating a target image with enhanced image quality.
As an example, the third convolutional layer includes a convolution kernel and an activation function.
As an example, the activation function is a PreLU function or a Sigmod function.
As an example, the generative confrontation network includes an arbiter.
The processor 102 is further configured to perform the following operations:
and after the training of the generative confrontation network is finished, taking the discriminator as an image quality evaluation network.
As an example, the input data of the image quality evaluation network is a target image and the original image output by the image quality enhancement network, and the output data is a probability that the target image is the original image.
As an example, the processor 102 is further configured to perform the following operations:
training the generative countermeasure network based on a preset loss function; wherein the loss function includes a penalty loss function and a loss of image quality function.
As an example, the countering loss function is implemented using Wasserstein loss.
As an example, the loss of image function is associated with a VGG-19 pre-training model.
As an example, the optimizer in the generative countermeasure network is implemented using a root mean square transfer algorithm.
As an example, the picture quality loss function in the generative countermeasure network includes at least one of: a multi-hierarchy structural similarity algorithm, a mean absolute difference algorithm, a sum of absolute errors algorithm, a hadamard transform algorithm, a sum of squared differences algorithm, and a mean squared errors algorithm.
As an example, the original image is an image cropped to a designated size.
Accordingly, referring to fig. 10, an embodiment of the present invention further provides an image quality enhancement apparatus 200, including:
a memory 201 and a processor 202; the memory 201 is connected to the processor 202 via a communication bus 203 for storing computer instructions executable by the processor 202; the processor 202 is configured to read computer instructions from the memory 201, and when executed, is configured to:
inputting an image to be promoted into a preset image quality enhancement network, and carrying out image quality enhancement on the image to be promoted by the image quality enhancement network to obtain a target image with the image quality higher than that of the image to be promoted; the image quality enhancement network comprises N scale extraction networks for extracting different scale features, and is obtained by training as a generator of a generating countermeasure network; n is an integer greater than 0.
The Processor 202 executes the program code included in the memory 201, and the Processor 202 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, a discrete hardware component, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 201 stores a program code of the image quality enhancement method, and the memory 201 may include at least one type of storage medium, which includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. Further, the image quality enhancement apparatus 200 may cooperate with a network storage apparatus that performs a storage function of the memory 201 through a network connection. The memory 201 may be an internal storage unit of the image quality enhancement apparatus 200, such as a hard disk or a memory of the image quality enhancement apparatus 200. The memory 201 may also be an external storage device of the image quality enhancement apparatus 200, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the image quality enhancement apparatus 200. Further, the memory 201 may include both an internal storage unit of the image quality enhancement apparatus 200 and an external storage device. The memory 201 is used for storing the computer program code 23 and other programs and data required by the image quality enhancement apparatus 200. The memory 201 may also be used to temporarily store data that has been output or is to be output.
The various embodiments described herein may be implemented using a computer-readable medium such as computer software, hardware, or any combination thereof. For a hardware implementation, the embodiments described herein may be implemented using at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a processor, a controller, a microcontroller, a microprocessor, and an electronic unit designed to perform the functions described herein. For a software implementation, the implementation such as a process or a function may be implemented with a separate software module that allows performing at least one function or operation. The software codes may be implemented by software applications (or programs) written in any suitable programming language, which may be stored in memory and executed by the controller.
The image quality enhancement apparatus 200 may include, but is not limited to, a memory 201 and a processor 202. Those skilled in the art will appreciate that fig. 10 is only an example of the image quality enhancement apparatus 200, and does not constitute a limitation to the image quality enhancement apparatus 200, and may include more or less components than those shown in the drawings, or combine some components, or different components, for example, the device may further include an input/output device, a network access device, and the like.
As an example, the image to be promoted is an image output by a decoder.
As an example, N of the scale extraction networks are connected in sequence.
The image quality enhancement network further comprises a first convolution layer; the first convolution layer is connected with a first one of the N scale extraction networks and is used for performing convolution operation on the image to be promoted to obtain image data after the convolution operation.
The input image of each scale extraction network is the output image of the first convolution layer or the output image of the last scale extraction network.
As an example, the scale extraction network comprises: the filter, the splicing layer and the second convolution layer are connected in sequence.
The filters are used for extracting different scale features of the input image to obtain a plurality of different scale features.
The splicing layer is used for splicing a plurality of different scale characteristics under each template to obtain spliced image data.
And the second convolution layer performs convolution operation on the spliced image data to obtain image data with the same dimensionality as the input image.
As an example, the scale extraction network further comprises an addition layer.
And the addition layer performs point-by-point summation on the image data output by the second convolution layer with the same dimensionality as the input image and the input image to obtain image data with different scales containing high-frequency characteristics and low-frequency characteristics.
By way of example, the filter includes a convolution kernel and an activation function.
As an example, the activation function includes at least one of: a PReLU function, a ReLU function, a tanh function, a Sigmod function, and an ELU function.
As an example, the activation function is a PreLU function.
As an example, the image quality enhancement network further includes at least one third convolution layer; and the at least one third convolution layer is used for reducing the dimension of the scale features output by the scale extraction structure to obtain low-dimension feature vectors and generating an image with enhanced image quality.
As an example, the third convolutional layer includes a convolution kernel and an activation function.
As an example, the activation function is a PreLU function or a Sigmod function.
Accordingly, referring to fig. 11, an embodiment of the present invention further provides a movable platform 001, including:
a machine body 02.
And the power system 03 is arranged in the machine body 02 and used for providing power for the movable platform 001.
And the image quality enhancing apparatus 200 described above.
Those skilled in the art will appreciate that fig. 11 is merely an example of a movable platform and is not intended to be limiting and may include more or fewer components than shown, or some components in combination, or different components, e.g., the movable platform may also include input-output devices, network access devices, etc.
Accordingly, referring to fig. 12, an embodiment of the present application further provides a camera 002, including:
a housing 04.
And the lens assembly 05 is arranged inside the shell 04.
And the sensor component 06 is arranged inside the shell 04 and used for sensing the light passing through the lens component and generating an electric signal.
And the image quality enhancing apparatus 200.
Those skilled in the art will appreciate that fig. 12 is merely an example of a camera and is not meant to be limiting and may include more or fewer components than those shown, or some components may be combined, or different components, e.g., the camera may also include a network access device, etc.
Accordingly, the present embodiment also provides a computer-readable storage medium, on which computer instructions are stored, and when executed, the computer instructions implement the steps of any one of the above methods.
Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It will be understood that the present application is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.
The present invention is not intended to be limited to the particular embodiments shown and described, but is to be accorded the widest scope consistent with the principles and novel features herein disclosed.
Claims (37)
1. A method for obtaining an image quality enhancement network is characterized by comprising the following steps:
training a preset generative countermeasure network according to a plurality of preset training images; wherein the generative confrontation network comprises a generator;
after the training of the generative confrontation network is finished, taking the generator as an image quality enhancement network; the generator or the image quality enhancement network comprises N scale extraction networks for extracting features of different scales, wherein N is an integer greater than 0, and the image quality enhancement network is used for carrying out image quality enhancement processing on any input image to be enhanced.
2. The method of claim 1, wherein the training image is generated from an original image by encoding and decoding.
3. The method of claim 2, wherein N of said scaling networks are connected in series;
the image quality enhancement network further comprises a first convolution layer; the first convolution layer is connected with a first one of the N scale extraction networks and is used for performing convolution operation on the training image to obtain image data after the convolution operation;
the input image of each scale extraction network is the output image of the first convolution layer or the output image of the last scale extraction network.
4. The method of claim 3, wherein the scale extraction network comprises a plurality of filters; the filters are used for extracting the scale features of the input image to obtain a plurality of different scale features.
5. The method of claim 4, wherein the filter comprises a convolution kernel and an activation function.
6. The method of claim 5, wherein the activation function comprises at least one of: a PReLU function, a ReLU function, a tanh function, a Sigmod function, and an ELU function.
7. The method of claim 6, wherein the activation function is a PreLU function.
8. The method of claim 4, wherein the scale extraction network further comprises a stitching layer for stitching a plurality of different scale features output by the plurality of filters to obtain stitched image data.
9. The method of claim 8, wherein the scale extraction network further comprises a second convolutional layer; and the second convolution layer performs convolution operation on the spliced image data to obtain image data with the same dimensionality as the input image.
10. The method of claim 9, wherein the scale extraction network further comprises an addition layer; and the addition layer performs point-by-point summation on the image data output by the second convolution layer and the input image to obtain image data containing different scale characteristics.
11. The method of claim 1, wherein the network further comprises at least one third convolutional layer; and the at least one third convolution layer is used for reducing the dimension of the scale features output by the scale extraction network to obtain low-dimension feature vectors and generating a target image with enhanced image quality.
12. The method of claim 11, wherein the third convolutional layer comprises a convolutional kernel and an activation function.
13. The method of claim 12, wherein the activation function is a PreLU function or a Sigmod function.
14. The method of claim 2, wherein the generative confrontation network comprises an arbiter;
the method further comprises the following steps:
and after the training of the generative confrontation network is finished, taking the discriminator as an image quality evaluation network.
15. The method of claim 14, wherein the input data of the quality estimation network is a target image and the original image output by the quality enhancement network, and the output data is a probability that the target image is the original image.
16. The method of claim 1, wherein training the pre-established generative countermeasure network comprises:
training the generative countermeasure network based on a preset loss function; wherein the loss function includes a penalty loss function and a loss of image quality function.
17. The method of claim 16,
the penalty function is implemented using Wasserstein loss.
18. The method of claim 16,
the loss of picture quality function is related to the VGG-19 pre-training model.
19. The method of claim 1, wherein the optimizer in the generative countermeasure network is implemented using a root mean square transfer algorithm.
20. The method of claim 1, wherein the generated picture quality loss function in the countermeasure network comprises at least one of: a multi-hierarchy structural similarity algorithm, a mean absolute difference algorithm, a sum of absolute errors algorithm, a hadamard transform algorithm, a sum of squared differences algorithm, and a mean squared errors algorithm.
21. The method of claim 1, wherein the original image is an image cropped to a specified size.
22. An image quality enhancement method, comprising:
inputting an image to be promoted into a preset image quality enhancement network, and carrying out image quality enhancement on the image to be promoted by the image quality enhancement network to obtain a target image with the image quality higher than that of the image to be promoted; the image quality enhancement network comprises N scale extraction networks for extracting different scale features, and is obtained by training as a generator of a generating countermeasure network; n is an integer greater than 0.
23. The method of claim 22, wherein the image to be promoted is an image output by a decoder.
24. The method of claim 22, wherein N of said scaling networks are connected in series;
the image quality enhancement network further comprises a first convolution layer; the first convolution layer is connected with a first one of the N scale extraction networks and is used for performing convolution operation on the image to be promoted to obtain image data after the convolution operation;
the input image of each scale extraction network is the output image of the first convolution layer or the output image of the last scale extraction network.
25. The method of claim 22, wherein the scale extraction network comprises: the filter, the splicing layer and the second convolution layer are connected in sequence;
the filters are used for extracting different scale features of the input image to obtain a plurality of different scale features;
the splicing layer is used for splicing a plurality of different scale characteristics to obtain spliced image data;
and the second convolution layer performs convolution operation on the spliced image data to obtain image data with the same dimensionality as the input image.
26. The method of claim 25, wherein the scale extraction network further comprises an addition layer;
and the addition layer performs point-by-point summation on the image data output by the second convolution layer and the input image to obtain image data containing different scale characteristics.
27. The method of claim 25, wherein the filter comprises a convolution kernel and an activation function.
28. The method of claim 27, wherein the activation function comprises at least one of: a PReLU function, a ReLU function, a tanh function, a Sigmod function, and an ELU function.
29. The method of claim 28, wherein the activation function is a PreLU function.
30. The method of claim 22, wherein the network further comprises at least one third convolutional layer connected in sequence; and the at least one third convolution layer is used for reducing the dimension of the scale features output by the scale extraction structure to obtain low-dimension feature vectors and generating a target image with enhanced image quality.
31. The method of claim 30, wherein the third convolutional layer comprises a convolutional kernel and an activation function.
32. The method of claim 31, wherein the activation function is a PreLU function or a Sigmod function.
33. An apparatus for acquiring an image quality enhancement network, comprising:
a memory and a processor; the memory is connected with the processor through a communication bus and is used for storing computer instructions executable by the processor; the processor is used for reading computer instructions from the memory to realize the method for acquiring the picture quality enhancement network according to any one of claims 1 to 21.
34. An image quality enhancement device is characterized by comprising a memory and a processor; the memory is connected with the processor through a communication bus and is used for storing computer instructions executable by the processor; the processor is used for reading computer instructions from the memory to realize the image quality enhancement method of any one of claims 22 to 32.
35. A movable platform, comprising:
a body;
the power system is arranged in the machine body and used for providing power for the movable platform; and the image quality enhancement apparatus according to claim 34.
36. A camera, comprising:
a housing;
the lens assembly is arranged inside the shell;
the sensor assembly is arranged in the shell and used for sensing the light passing through the lens assembly and generating an electric signal; and the number of the first and second groups,
the image quality enhancement apparatus according to claim 34.
37. A computer-readable storage medium having stored thereon computer instructions which, when executed, implement the steps of the method of any one of claims 1 to 32.
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/CN2019/103356 WO2021035629A1 (en) | 2019-08-29 | 2019-08-29 | Method for acquiring image quality enhancement network, image quality enhancement method and apparatus, mobile platform, camera, and storage medium |
Publications (1)
Publication Number | Publication Date |
---|---|
CN112119408A true CN112119408A (en) | 2020-12-22 |
Family
ID=73799219
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201980031413.7A Pending CN112119408A (en) | 2019-08-29 | 2019-08-29 | Method for acquiring image quality enhancement network, image quality enhancement method, image quality enhancement device, movable platform, camera and storage medium |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN112119408A (en) |
WO (1) | WO2021035629A1 (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112862073A (en) * | 2021-02-03 | 2021-05-28 | 北京大学 | Compressed data analysis method and device, storage medium and terminal |
CN116458894A (en) * | 2023-04-21 | 2023-07-21 | 山东省人工智能研究院 | Electrocardiosignal enhancement and classification method based on composite generation countermeasure network |
WO2024104000A1 (en) * | 2022-11-17 | 2024-05-23 | 京东方科技集团股份有限公司 | Terminal image quality enhancement method and device, and computer readable storage medium |
Families Citing this family (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113538230A (en) * | 2021-06-09 | 2021-10-22 | 安徽七天教育科技有限公司 | Image recovery system based on scanning test paper and use method |
CN114549334A (en) * | 2021-07-22 | 2022-05-27 | 中山大学 | Image rain removing method and system of multidimensional aggregation network based on rain line characteristic reconstruction |
CN113989152A (en) * | 2021-10-29 | 2022-01-28 | 北京百度网讯科技有限公司 | Image enhancement method, device, equipment and storage medium |
CN114827723B (en) * | 2022-04-25 | 2024-04-09 | 阿里巴巴(中国)有限公司 | Video processing method, device, electronic equipment and storage medium |
US20230368336A1 (en) * | 2022-05-12 | 2023-11-16 | Honeywell International Inc. | Using integrated system to enhance a reference image |
CN114972107B (en) * | 2022-06-14 | 2024-07-26 | 福州大学 | Low-illumination image enhancement method based on multi-scale stacked attention network |
CN116188346B (en) * | 2023-05-04 | 2023-07-11 | 安翰科技(武汉)股份有限公司 | Image quality enhancement method and device for endoscope image |
CN117152025B (en) * | 2023-10-30 | 2024-03-01 | 硕橙(厦门)科技有限公司 | Method, device and equipment for enhancing over-bright image based on composite scale features |
CN118172285B (en) * | 2024-05-14 | 2024-09-20 | 浙江芯劢微电子股份有限公司 | Image quality optimization method and device |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108765319A (en) * | 2018-05-09 | 2018-11-06 | 大连理工大学 | A kind of image de-noising method based on generation confrontation network |
CN110060216A (en) * | 2019-04-17 | 2019-07-26 | 广东工业大学 | A kind of image repair method, device and equipment based on generation confrontation network |
CN110084757A (en) * | 2019-04-15 | 2019-08-02 | 南京信息工程大学 | A kind of infrared depth image enhancement method based on generation confrontation network |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2018053340A1 (en) * | 2016-09-15 | 2018-03-22 | Twitter, Inc. | Super resolution using a generative adversarial network |
CN107481209B (en) * | 2017-08-21 | 2020-04-21 | 北京航空航天大学 | Image or video quality enhancement method based on convolutional neural network |
CN107590786A (en) * | 2017-09-08 | 2018-01-16 | 深圳市唯特视科技有限公司 | A kind of image enchancing method based on confrontation learning network |
CN110163809A (en) * | 2019-03-31 | 2019-08-23 | 东南大学 | Confrontation network DSA imaging method and device are generated based on U-net |
-
2019
- 2019-08-29 WO PCT/CN2019/103356 patent/WO2021035629A1/en active Application Filing
- 2019-08-29 CN CN201980031413.7A patent/CN112119408A/en active Pending
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108765319A (en) * | 2018-05-09 | 2018-11-06 | 大连理工大学 | A kind of image de-noising method based on generation confrontation network |
CN110084757A (en) * | 2019-04-15 | 2019-08-02 | 南京信息工程大学 | A kind of infrared depth image enhancement method based on generation confrontation network |
CN110060216A (en) * | 2019-04-17 | 2019-07-26 | 广东工业大学 | A kind of image repair method, device and equipment based on generation confrontation network |
Non-Patent Citations (1)
Title |
---|
张元祺: "基于生成对抗网络的彩色图像去噪方法", 《中国优秀硕士学位论文全文数据库》 * |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112862073A (en) * | 2021-02-03 | 2021-05-28 | 北京大学 | Compressed data analysis method and device, storage medium and terminal |
WO2024104000A1 (en) * | 2022-11-17 | 2024-05-23 | 京东方科技集团股份有限公司 | Terminal image quality enhancement method and device, and computer readable storage medium |
CN116458894A (en) * | 2023-04-21 | 2023-07-21 | 山东省人工智能研究院 | Electrocardiosignal enhancement and classification method based on composite generation countermeasure network |
CN116458894B (en) * | 2023-04-21 | 2024-01-26 | 山东省人工智能研究院 | Electrocardiosignal enhancement and classification method based on composite generation countermeasure network |
Also Published As
Publication number | Publication date |
---|---|
WO2021035629A1 (en) | 2021-03-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN112119408A (en) | Method for acquiring image quality enhancement network, image quality enhancement method, image quality enhancement device, movable platform, camera and storage medium | |
CN110473137B (en) | Image processing method and device | |
US11481613B2 (en) | Execution method, execution device, learning method, learning device, and recording medium for deep neural network | |
US9349072B2 (en) | Local feature based image compression | |
CN112396645B (en) | Monocular image depth estimation method and system based on convolution residual learning | |
CN109816615A (en) | Image processing method, device, equipment and storage medium | |
CN107871306B (en) | Method and device for denoising picture | |
US11983906B2 (en) | Systems and methods for image compression at multiple, different bitrates | |
US11062210B2 (en) | Method and apparatus for training a neural network used for denoising | |
CN110677651A (en) | Video compression method | |
CN112598579A (en) | Image super-resolution method and device for monitoring scene and storage medium | |
WO2016050729A1 (en) | Face inpainting using piece-wise affine warping and sparse coding | |
US20220414838A1 (en) | Image dehazing method and system based on cyclegan | |
CN110753225A (en) | Video compression method and device and terminal equipment | |
CN108734653A (en) | Image style conversion method and device | |
CN114402596A (en) | Neural network model compression | |
CN113450290A (en) | Low-illumination image enhancement method and system based on image inpainting technology | |
CN111698508B (en) | Super-resolution-based image compression method, device and storage medium | |
CN114978189A (en) | Data coding method and related equipment | |
Uddin et al. | A perceptually inspired new blind image denoising method using $ L_ {1} $ and perceptual loss | |
CN113256744B (en) | Image coding and decoding method and system | |
CN114612316A (en) | Method and device for removing rain from nuclear prediction network image | |
CN113592009A (en) | Image semantic segmentation method and device, storage medium and electronic equipment | |
CN113382244B (en) | Coding and decoding network structure, image compression method, device and storage medium | |
CN111107377A (en) | Depth image compression method, device, equipment and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
RJ01 | Rejection of invention patent application after publication | ||
RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201222 |