CN112365551A - Image quality processing system, method, device and medium - Google Patents
Image quality processing system, method, device and medium Download PDFInfo
- Publication number
- CN112365551A CN112365551A CN202011104004.6A CN202011104004A CN112365551A CN 112365551 A CN112365551 A CN 112365551A CN 202011104004 A CN202011104004 A CN 202011104004A CN 112365551 A CN112365551 A CN 112365551A
- Authority
- CN
- China
- Prior art keywords
- quality
- discriminator
- image data
- data set
- low
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 29
- 238000000034 method Methods 0.000 title claims abstract description 23
- 238000012549 training Methods 0.000 claims abstract description 38
- 238000009826 distribution Methods 0.000 claims abstract description 37
- 238000004088 simulation Methods 0.000 claims abstract description 7
- 238000004519 manufacturing process Methods 0.000 claims abstract description 4
- 230000006870 function Effects 0.000 claims description 66
- 230000004913 activation Effects 0.000 claims description 18
- 239000013598 vector Substances 0.000 claims description 18
- 238000010606 normalization Methods 0.000 claims description 16
- 238000013527 convolutional neural network Methods 0.000 claims description 15
- 238000003672 processing method Methods 0.000 claims description 15
- 238000011176 pooling Methods 0.000 claims description 11
- 238000003860 storage Methods 0.000 claims description 8
- 230000015556 catabolic process Effects 0.000 claims description 5
- 238000006731 degradation reaction Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 claims description 4
- 230000006872 improvement Effects 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 238000013441 quality evaluation Methods 0.000 claims description 3
- 230000009467 reduction Effects 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 238000011144 upstream manufacturing Methods 0.000 claims description 3
- 238000013528 artificial neural network Methods 0.000 abstract description 2
- 230000001133 acceleration Effects 0.000 abstract 1
- 238000003908 quality control method Methods 0.000 description 19
- 230000008569 process Effects 0.000 description 7
- 238000010801 machine learning Methods 0.000 description 5
- 238000005457 optimization Methods 0.000 description 5
- 238000013135 deep learning Methods 0.000 description 4
- 230000003044 adaptive effect Effects 0.000 description 3
- 210000004556 brain Anatomy 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000011478 gradient descent method Methods 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000036039 immunity Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000011158 quantitative evaluation Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
- G06T9/002—Image coding using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Image Processing (AREA)
Abstract
The invention discloses an image quality processing system, method, device and medium, the system comprises a high quality data supply unit, a generator and a discriminator, wherein the generator is designed based on a production countermeasure network framework and is embedded into a convolution neural network; the generator reduces the high-quality image data set to a low-dimensional space, adds noise and degrades to generate a simulated low-quality image data set, and transmits the generated simulated low-quality image data set to the discriminator; and the discriminator which carries out model training through the resistance loss function carries out discrimination comparison on the simulation low-quality image data set generated by the generator and the high-quality image data set provided by the data supply unit, and outputs a high-low mixed quality image data set. According to the scheme, the variational automatic encoder is used for modeling the distribution of high-quality image data, the problem of unbalanced training samples is solved, the acceleration and real-time control of image quality processing are realized, the cost is low, and the universality is high.
Description
Technical Field
The present invention relates to an image data processing technology, and more particularly, to an image quality processing system, method, device, and medium.
Background
In an image data processing system, the performance optimization is carried out on a quality control link in the image data processing process by utilizing the technology in the field of computer vision, and the targeted objects include but are not limited to medical images, remote sensing images, microscope slice images, natural images and the like. The quality control of the image, including the inspection of whether the image quality meets the requirements or not, the inspection of the preprocessing quality and the like, plays an important role in the subsequent batch analysis of the image data.
Because of the lack of uniform quantitative evaluation indexes for the high quality of images in practical application, the quality control and screening of the images at present mainly depend on manual completion or on specific indexes defined under specific tasks; the former brings higher time cost and labor cost, and the latter 'specific index' is a task-related indirect indication of image quality and fails to provide a guidance method with better generalization performance for image quality control.
In recent years, with the development of machine learning/deep learning technology, some researches have proposed an image quality control method based on an adaptive learning algorithm, and such an algorithm is driven by data and relies on iterative training of a large amount of data to obtain a certain quality control effect. At present, in the field of machine learning/deep learning, solutions for unbalanced samples are all self-adaptive weighting, and the actual effect is usually unstable, so that whether a model achieves the optimum or not cannot be determined.
Taking the data of the magnetic resonance structural image of the brain as an example, the degradation of the quality of the magnetic resonance image comes from the multi-factor influence of a space domain and a transformation domain, and the explicit modeling of the degradation is difficult to perform; compared with the amount of high-quality data, the amount of data in the actually obtained low-quality image data set is small, which is insufficient to provide the classifier with the distribution information of the real low-quality image data, so if the supervised learning method in the conventional machine learning is adopted to train the classification model for distinguishing the high-quality image from the low-quality image, the performance of the model is limited due to the limitation of the training data.
Therefore, in the field of medical images, an image quality control system which is high in speed, good in universality and independent of manually defined features is urgently needed to be applied to clinical and scientific research.
Disclosure of Invention
To overcome the disadvantages of the prior art, it is an object of the present invention to provide an image quality processing system, method, apparatus and medium, which can solve the above problems.
The purpose of the invention is realized by adopting the following technical scheme:
an image quality processing system includes a high quality data supply unit, a generator, and a discriminator, wherein the high quality data supply unit is disposed logically upstream of the generator and the discriminator, and the discriminator is disposed logically downstream of the generator; the high-quality data supply unit supplies a high-quality image data set to the generator and the discriminator; the generator is designed based on a production countermeasure network framework and is embedded into a convolutional neural network; the generator reduces the high-quality image data set to a low-dimensional space, adds noise and degrades to generate a simulated low-quality image data set, and transmits the generated simulated low-quality image data set to the discriminator; and the discriminator which carries out model training through the resistance loss function carries out discrimination comparison on the simulation low-quality image data set generated by the generator and the high-quality image data set provided by the data supply unit, and outputs a high-low mixed quality image data set.
Preferably, the generator adopts a variational self-encoder, and the variational self-encoder comprises an encoder and a decoder; the encoder performs encoding dimension reduction on the high-quality image data set, and sets difference constraint on the encoder to avoid the result approaching zero; the decoder collects samples in the encoder, and the samples are subjected to characteristic graph size doubling and channel number halving to obtain a picture which has the same size as an original input high-quality image and is subjected to noise processing and is used as a produced simulation low-quality image.
Preferably, the encoder and the decoder both adopt a deep convolutional neural network, wherein the network structure of the encoder comprises a plurality of convolutional layers, a batch normalization layer, an activation function layer and a pooling layer, and each time the processing of one group of convolutional layer, batch normalization layer, activation function layer and pooling layer is performed, the size of a feature map is reduced by half, the number of channels is doubled, n-dimensional vectors are output, and n is a positive integer greater than 2; processing by a full connection layer and outputting 2-dimensional vector distribution; sampling n numerical values from n-dimensional vector distribution to form an n-dimensional vector as the input of a decoder; the network structure of the decoder comprises a plurality of layers of deconvolution layers, a batch normalization layer, a ReLU activation function layer and convolution layers, wherein each time the decoder is processed by one group of deconvolution layers, batch normalization layers, activation function layers and convolution layers, the size of a characteristic image is doubled, the number of channels is halved, and a noise-processed image which has the same size as an original input high-quality image is obtained and is used as a simulated low-quality image.
Preferably, the difference constraint adopts a JS divergence function; the activation function layer adopts a ReLU activation function layer.
Preferably, the network structure of the encoder includes 5 convolutional layers with convolutional kernel size of 3 × 3 × 3 and step size of 1, each convolutional layer includes a batch normalization layer, a ReLU activation function layer, and a maximum pooling layer with pooling kernel size of 2 × 2 × 2 and step size of 2, and n is a 128-dimensional vector after processing; the network structure of the decoder comprises 5 deconvolution layers with convolution kernel size of 2 multiplied by 2 and step length of 2, and each deconvolution layer comprises a batch normalization layer, a ReLU activation function layer and a convolution layer with convolution kernel size of 3 multiplied by 3 and step length of 1.
Preferably, the 2-dimensional vector distribution output by the full-connected layer processing is a normal distribution with the mean value mu and the equation sigma
Preferably, the structure of the discriminator adopts a VGG-16 network model and comprises a JS divergence loss function LJSAnd generating a penalty function LGANAnd the JS divergence loss function is as follows:
in the formula, P is the distribution of the encoder, U is the standard normal distribution, and KL (· | ·) is KL divergence;
for generating the penalty function LGANThen the resulting penalty function is:
LGAN-log (Dis (x)) -log (1-Dis (x')) … … … … … … … … … … … … … … formula-2,
where x is a sample extracted from the real high-quality image data set, x' is a noisy sample generated by the decoder, and Dis (-) is an output between 0 and 1 of the output of the discriminator, and a value closer to 1 indicates that the discriminator is more likely to determine the input image as high-quality, and a value closer to 0 indicates that the discriminator is more likely to determine the input image as low-quality.
An image quality processing method according to the system, the method comprising:
s1, constructing a generator and a discriminator, wherein the generator adopts a variational self-encoder comprising an encoder and a decoder; the discriminator adopts a network model which is subjected to cross training through a generated countermeasure function, the improvement of the discrimination performance of the discriminator on the high/low quality of the image data is realized through the countermeasure training, and the obtained discriminator is used as an image data quality discriminator after the model is converged;
s2, providing a high-quality image data set, wherein a high-quality data supply unit provides the high-quality image data set to an encoder and a discriminator;
s3, generating a simulated low-quality image data set, wherein the generator reduces the high-quality image data set to a low-dimensional space, adds noise and degrades to generate the simulated low-quality image data set, and the method specifically comprises the following steps:
s31, reducing and adding noise of the high-quality image, reducing the dimension of the high-quality data to a low-dimensional space by an encoder, outputting n-dimensional vector distribution with half of the size of a characteristic diagram and double of the number of channels, and adding noise to the high-quality data in the low-dimensional space;
s32, generating a simulated low-quality image data set through degradation, mapping the noise-added image back to an original image space through a decoder, doubling the size of a characteristic image, and halving the number of channels to obtain a degraded simulated low-quality image;
and S4, outputting a high-low mixed quality image data set, transmitting the generated simulated low quality image data set to a discriminator by the generator, discriminating and comparing the simulated low quality image data set transmitted by the generator with the high quality image data set provided by the data supply unit by the image quality discriminator, and outputting the high-low mixed quality image data set.
Preferably, in step S1, the cross-training strategy of the network model of the arbiter comprises:
when training the variational self-encoder, the parameters of the discriminator are fixed, the parameters of the variational self-encoder are adjusted, and when training the variational self-encoder, the integral loss function of the model is
LVAE＝LJS-LGAN… … … … … … … … … … … … … … formula-3;
when training the discriminator, the parameters of the variational self-encoder are fixed, the parameters of the discriminator are adjusted, and when training the discriminator, the integral loss function of the model is
LVAE＝LJS+LGAN… … … … … … … … … … … … … … formula-4;
in the formula, LJSIs the JS divergence loss function, LGANGenerating a pair loss-tolerance function;
and through cross training, the noise addition generation capability of the variational self-encoder and the high-low quality sample discrimination capability of the discriminator are synchronously enhanced, and finally, the image quality discriminator for image quality evaluation is obtained after the model is converged.
Preferably, the model optimization mode adopts an adaptive moment estimation algorithm, the learning rate is set to be 1e-4, and the model parameters are updated through a gradient descent method.
The invention also provides an apparatus comprising:
one or more processors;
a memory for storing one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned image quality processing method.
The present invention also provides a computer-readable storage medium on which a computer program is stored, which program, when executed by a processor, implements the aforementioned image quality processing method.
Compared with the prior art, the invention has the beneficial effects that:
1. the processing speed of image quality control is accelerated, the real-time image quality control is realized, and the labor cost of operation is greatly reduced;
2. the image quality control strategy with strong universality is provided, and the method has good mobility for different application scenes.
3. The method utilizes a variational automatic encoder to model the distribution of high-quality image data, and solves the problem of unbalanced training samples faced by the quality control method based on machine learning/deep learning in practical application.
4. The invention combines the variational automatic encoder with the generation countermeasure network model for the first time, is applied to the field of image quality control, and provides a new solution for the image quality control.
Drawings
FIG. 1 is a schematic diagram of a training process of an image data quality control model of a variational-based autocoder-depth generation countermeasure network according to the present invention;
FIG. 2 is a flow chart of an image quality processing method;
fig. 3 is a schematic view of the apparatus of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
Technical terms used in the present invention are explained as follows.
VAE: a Variational auto-encoder (Variational auto-encoder);
enc: an Encoder (Encoder);
and Dec: a Decoder (Decoder);
dis: a Discriminator (Discriminator);
and (3) GAN: generating a countermeasure network (generated adaptive network);
CNN: convolutional neural networks (convolutional neural networks);
gen: a Generator (Generator).
Example one
An image quality processing system, see fig. 1, includes a high quality data supply unit, a generator Gen, and a discriminator Dis.
The logical connection relation is as follows: a high quality data supply unit disposed logically upstream of the generator and the discriminator, supplying a high quality image data set to the generator and the discriminator; the discriminator is arranged at the logic downstream of a generator, and the generator is designed based on a production countermeasure network framework and is embedded into a convolutional neural network; the generator reduces the high-quality image data set to a low-dimensional space, adds noise and degrades to generate a simulated low-quality image data set, and transmits the generated simulated low-quality image data set to the discriminator.
Further, a discriminator for model training through the immunity loss function discriminates and compares the simulation low-quality image data set generated by the generator with the high-quality image data set provided by the data supply unit, and outputs a high-quality and low-quality mixed-quality image data set.
The invention is based on a priori information: the distribution of the high quality image data set follows a normal distribution, specifically as follows.
The high quality image data set that meets the requirements should be represented as a normal distribution in the spatial domain, the transform domain, and the low dimensional manifold of the space. Therefore, the distribution of high quality data can be modeled using a Variational auto-encoder (VAE) with a large amount of high quality data.
Meanwhile, an Encoder (Encoder) in the VAE can reduce the dimension of the high-quality data to a low-dimensional space, noise is added to the high-quality data in the low-dimensional space, and the noise-added image is mapped back to an original image space by a Decoder (Decoder) to obtain a degraded simulated low-quality image.
In order to make the simulated low-quality image as realistic as possible and obtain a classifier for realizing high/low-quality image discrimination by using the simulated low-quality image and the real high-quality image, a Convolutional Neural Network (CNN) classifier is embedded into the VAE by using the thought of zero sum game and referring to the design of a generation countermeasure network (GAN) framework, namely the VAE is regarded as a Generator (Generator) in the GAN, and the CNN is regarded as a Discriminator in the GAN.
In the optimization process of the model, a countermeasure loss function of the GAN replaces a reconstruction error loss function in the original VAE, the improvement of the image data high/low quality discrimination performance of the discriminator is realized through countermeasure training, and after the model is converged, the obtained discriminator is used as an image data quality discriminator and can be used in an image quality control process.
The training process of the model is shown in fig. 1. The high-quality image is firstly coded (dimensionality reduction) by a coder (Encoder) of VAE, the Encoder is a deep convolutional neural network, the network structure of the high-quality image mainly comprises 5 convolutional layers with convolutional kernel size of 3 multiplied by 3 and step length of 1, each convolutional layer comprises a batch normalization layer, a ReLU activation function layer and a maximum pooling layer with pooling kernel size of 2 multiplied by 2 and step length of 2, each convolutional layer, batch normalization layer, activation function layer and pooling layer are processed, the size of a characteristic graph is reduced by half, the number of channels is doubled, finally 128-dimensional vectors are output, and the 2-dimensional vectors are processed by a full connection layer and are respectively made to be the mean value mu and the variance sigma of normal distribution. According to the foregoing assumptions, the distribution of high quality image data sets follows a normal distribution that can theoretically be mapped to a standard normal distribution using an Encoder; in addition, in order to make the image generated by the final VAE be a noisy image, it is necessary to avoid that the variance σ calculated by the Encoder approaches 0 during the training process. Therefore, we use JS divergence between the normal distribution N (μ, σ) obtained by Encoder and the standard normal distribution N (0,1) to constrain the difference between the two normal distributions. 128 values are sampled from the normal distribution N (μ, σ) to form a 128-dimensional vector as an input to a Decoder (Decoder) of the VAE.
The Decoder is a deep convolution neural network, the network structure of which mainly comprises 5 deconvolution layers with convolution kernel size of 2 multiplied by 2 and step length of 2, each deconvolution layer comprises a batch normalization layer, a ReLU activation function layer and a convolution layer with convolution kernel size of 3 multiplied by 3 and step length of 1, each convolution layer is processed by a group of deconvolution layer, batch normalization layer, activation function layer and convolution layer, the size of a characteristic image is doubled and the number of channels is halved, and finally, a picture which has the same size as an original input high-quality image and is subjected to noise processing is obtained and is used as a simulation low-quality image.
In order to make the simulated low-quality image as realistic as possible and simultaneously obtain a classifier for realizing high/low-quality image discrimination by using the simulated low-quality image and the real high-quality image, a Convolutional Neural Network (CNN) classifier is embedded into the VAE by using the thought of zero sum game and referring to the design of a GAN framework, namely the VAE is regarded as a Generator (Generator) in the GAN, and the CNN is regarded as a Discriminator in the GAN.
The structure of the Discrimatoror adopts a VGG-16 network model. In the optimization process of the model, a countermeasure loss function of the GAN replaces a reconstruction error loss function in the original VAE, and the enhancement of the high/low quality discrimination performance of the image data by the discriminator is realized through countermeasure training.
According to the above design, the loss function of the model includes two parts: JS divergence loss function comprising JS divergence loss function LJSAnd generating a penalty function LGAN. Specifically, if the distribution obtained by the Encoder is P and the standard normal distribution is U, the JS divergence loss function is:
in the formula, P is the distribution of the encoder, U is the standard normal distribution, and KL (· | ·) is KL divergence;
for generating the penalty function LGANThen the resulting penalty function is:
LGAN-log (Dis (x)) -log (1-Dis (x')) … … … … … … … … … … … … … … formula-2,
where x is a sample extracted from the real high-quality image data set, x' is a noisy sample generated by the decoder, and Dis (-) is an output between 0 and 1 of the output of the discriminator, and a value closer to 1 indicates that the discriminator is more likely to determine the input image as high-quality, and a value closer to 0 indicates that the discriminator is more likely to determine the input image as low-quality.
The cross training strategy of the network model of the discriminator comprises the following steps:
when training the variational self-encoder, the parameters of the discriminator are fixed, the parameters of the variational self-encoder are adjusted, and when training the variational self-encoder, the integral loss function of the model is
LVAE＝LJS-LGAN… … … … … … … … … … … … … … formula-3;
when training the discriminator, the parameters of the variational self-encoder are fixed, the parameters of the discriminator are adjusted, and when training the discriminator, the integral loss function of the model is
LVAE＝LJS+LGAN… … … … … … … … … … … … … … formula-4;
in the formula, LJSIs the JS divergence loss function, LGANGenerating a pair loss-tolerance function;
and through cross training, the noise addition generation capability of the variational self-encoder and the high-low quality sample discrimination capability of the discriminator are synchronously enhanced, and finally, the image quality discriminator for image quality evaluation is obtained after the model is converged.
Furthermore, the optimization mode of the model adopts a self-adaptive moment estimation algorithm, the learning rate is set to be 1e-4, and the model parameters are updated by a gradient descent method.
Example two
The image quality processing method according to the aforementioned system, see fig. 2, comprises the following steps.
S1, constructing a generator and a discriminator, wherein the generator adopts a variational self-encoder comprising an encoder and a decoder; the discriminator adopts a network model which is cross-trained by a generated countermeasure function, the improvement of the discrimination performance of the discriminator on the high/low quality of the image data is realized through the countermeasure training, and the obtained discriminator is used as the image data quality discriminator after the model is converged.
S2, providing a high-quality image data set, and providing the high-quality image data set to the encoder and the discriminator by the high-quality data supply unit.
S3, generating a simulated low-quality image data set, wherein the generator reduces the high-quality image data set to a low-dimensional space, adds noise and degrades to generate the simulated low-quality image data set, and the method specifically comprises the following steps:
s31, reducing and adding noise of the high-quality image, reducing the dimension of the high-quality data to a low-dimensional space by an encoder, outputting n-dimensional vector distribution with half of the size of a characteristic diagram and double of the number of channels, and adding noise to the high-quality data in the low-dimensional space;
and S32, generating a simulated low-quality image data set through degradation, mapping the noise-added image back to an original image space by a decoder, doubling the size of the characteristic image, and halving the number of channels to obtain the degraded simulated low-quality image.
And S4, outputting a high-low mixed quality image data set, transmitting the generated simulated low quality image data set to a discriminator by the generator, discriminating and comparing the simulated low quality image data set transmitted by the generator with the high quality image data set provided by the data supply unit by the image quality discriminator, and outputting the high-low mixed quality image data set.
In general, the present invention is an image quality control method based on a variational auto-coder-depth generation countermeasure network (VAE-GAN).
EXAMPLE III
An apparatus, see fig. 3, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned image quality processing method.
Specifically, the apparatus includes a processor 10, a memory 11, a communication module 12, an input device 13, and an output device 14; the number of processors 10 in the device may be one or more, and one processor 10 is taken as an example in fig. 3; the processor 10, the memory 11, the communication module 12, the input device 13 and the output device 14 in the apparatus may be connected by a bus or other means, which is exemplified in fig. 3.
The memory 11, which is a computer-readable storage medium, may be used to store software programs, computer-executable programs, and modules, such as a module corresponding to an image quality processing method in the present embodiment. The processor 10 executes various functional applications of the apparatus and data processing by executing software programs, instructions, and modules stored in the memory 11, that is, implements one of the image quality processing methods described above.
The memory 11 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the device, and the like. Further, the memory 11 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, memory 11 may further include memory located remotely from processor 10, which may be connected to devices through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
And the communication module 12 is used for establishing connection with the display screen and realizing data interaction with the display screen. The input device 13 may be used to receive input numeric or character information and to generate key signal inputs relating to user settings and function controls of the apparatus.
The apparatus provided in this embodiment may perform the image quality processing method provided in any embodiment of the present invention, and its corresponding functions and advantages are described in detail.
Example four
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, implements the aforementioned image quality processing method. The computer-executable instructions, when executed by a computer processor, are for performing an image quality processing method, and are not limited to the method operations described above, but may also perform related operations in an image quality processing method provided by any of the embodiments of the present invention.
Alternative examples
a. In terms of network structure, the network structure of Encoder and Decoder in the Discriminator and VAE is not limited to the aforementioned described structure, and may include but is not limited to the down-sampling branch of VGG, ResNet, DenseNet, VNet, etc.;
in VAE, the noise adding mode of the image obtained by the Encoder is not limited to the normal distribution, and the noise and the high-quality image can be correspondingly modified according to the definition of the noise and the high-quality image in a specific scene;
c. part of the loss function: the difference constraint between the distributions can adopt other differentiable indexes except the JS divergence; the logarithmic loss function in the loss function of GAN can be derived from the remaining classification loss functions, such as cross-entropy loss function, Focal loss, etc.
d. The invention is explained by taking a brain magnetic resonance structure image as an example, but the application objects of the invention include, but are not limited to, medical images, remote sensing images, microscope slice images, natural images and the like of other modalities.
e. The invention utilizes the variational automatic encoder to model the high-quality image data distribution, and solves the problem of unbalanced training samples faced by the quality control method based on machine learning/deep learning in practical application.
f. The invention combines the variational automatic encoder with the generation countermeasure network model for the first time, is applied to the field of image quality control, and provides a new solution for the image quality control.
System authentication
The image quality discriminator obtained by training is used for judging the quality condition of a single image within 0.1 second by utilizing a Pythroch building model and testing on an Intel (R) Xeon (R) CPU E5-2670 v22.50GHz processor with a CentOs 6.5 system, and is tested on 3200 cases of brain magnetic resonance structural image data (3000 cases of high-quality data and 200 cases of low-quality data) in a hospital, so that the sensitivity (the proportion of correctly judging the high-quality data into the high-quality data) is about 99 percent, and the specificity (the proportion of correctly judging the low-quality data into the low-quality data) is about 95 percent.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. An image quality processing system characterized by: the system includes a high quality data supply unit, a generator, and a discriminator, wherein the high quality data supply unit is disposed logically upstream of the generator and the discriminator, and the discriminator is disposed logically downstream of the generator;
the high-quality data supply unit supplies a high-quality image data set to the generator and the discriminator;
the generator is designed based on a production countermeasure network framework and is embedded into a convolutional neural network; the generator reduces the high-quality image data set to a low-dimensional space, adds noise and degrades to generate a simulated low-quality image data set, and transmits the generated simulated low-quality image data set to the discriminator;
and the discriminator which carries out model training through the resistance loss function carries out discrimination comparison on the simulation low-quality image data set generated by the generator and the high-quality image data set provided by the data supply unit, and outputs a high-low mixed quality image data set.
2. The image quality processing system according to claim 1, characterized in that: the generator adopts a variational self-encoder, and the variational self-encoder comprises an encoder and a decoder; the encoder performs encoding dimension reduction on the high-quality image data set, and sets difference constraint on the encoder to avoid the result approaching zero; the decoder collects samples in the encoder, and the samples are subjected to characteristic graph size doubling and channel number halving to obtain a picture which has the same size as an original input high-quality image and is subjected to noise processing and is used as a produced simulation low-quality image.
3. The image quality processing system according to claim 2, characterized in that: the encoder and the decoder both adopt a deep convolutional neural network, wherein the network structure of the encoder comprises a plurality of convolutional layers, a batch normalization layer, an activation function layer and a pooling layer, the size of a characteristic graph is reduced by half, the number of channels is doubled after each group of convolutional layer, batch normalization layer, activation function layer and pooling layer is processed, n-dimensional vectors are output, and n is a positive integer greater than 2; processing by a full connection layer and outputting 2-dimensional vector distribution;
sampling n numerical values from n-dimensional vector distribution to form an n-dimensional vector as the input of a decoder;
the network structure of the decoder comprises a plurality of layers of deconvolution layers, a batch normalization layer, a ReLU activation function layer and convolution layers, wherein each time the decoder is processed by one group of deconvolution layers, batch normalization layers, activation function layers and convolution layers, the size of a characteristic image is doubled, the number of channels is halved, and a noise-processed image which has the same size as an original input high-quality image is obtained and is used as a simulated low-quality image.
4. The image quality processing system according to claim 3, characterized in that: the network structure of the encoder comprises 5 convolutional layers with the convolutional kernel size of 3 multiplied by 3 and the step length of 1, each convolutional layer comprises a batch normalization layer, a ReLU activation function layer and a maximum pooling layer with the pooling kernel size of 2 multiplied by 2 and the step length of 2, and n is output as a 128-dimensional vector after processing; the network structure of the decoder comprises 5 deconvolution layers with convolution kernel size of 2 multiplied by 2 and step length of 2, and each deconvolution layer comprises a batch normalization layer, a ReLU activation function layer and a convolution layer with convolution kernel size of 3 multiplied by 3 and step length of 1.
5. The image quality processing system according to claim 3, characterized in that: the 2-dimensional vector distribution output by the processing of the full connection layer is normal distribution with the mean value of mu and the equation of sigma
6. The image quality processing system according to claim 5, wherein: the discriminator structure adopts a VGG-16 network model and comprises a JS divergence loss function LJSAnd generating a penalty function LGANAnd the JS divergence loss function is as follows:
in the formula, P is the distribution of the encoder, U is the standard normal distribution, and KL (· | ·) is KL divergence;
for generating the penalty function LGANThen the resulting penalty function is:
LGAN-log (Dis (x)) -log (1-Dis (x')) … … … … … … … … … … … … … … formula-2,
where x is a sample extracted from the real high-quality image data set, x' is a noisy sample generated by the decoder, and Dis (-) is an output between 0 and 1 of the output of the discriminator, and a value closer to 1 indicates that the discriminator is more likely to determine the input image as high-quality, and a value closer to 0 indicates that the discriminator is more likely to determine the input image as low-quality.
7. The image quality processing method according to any one of claims 1 to 6, characterized in that the method comprises:
s1, constructing a generator and a discriminator, wherein the generator adopts a variational self-encoder comprising an encoder and a decoder; the discriminator adopts a network model which is subjected to cross training through a generated countermeasure function, the improvement of the discrimination performance of the discriminator on the high/low quality of the image data is realized through the countermeasure training, and the obtained discriminator is used as an image data quality discriminator after the model is converged;
s2, providing a high-quality image data set, wherein a high-quality data supply unit provides the high-quality image data set to an encoder and a discriminator;
s3, generating a simulated low-quality image data set, wherein the generator reduces the high-quality image data set to a low-dimensional space, adds noise and degrades to generate the simulated low-quality image data set, and the method specifically comprises the following steps:
s31, reducing and adding noise of the high-quality image, reducing the dimension of the high-quality data to a low-dimensional space by an encoder, outputting n-dimensional vector distribution with half of the size of a characteristic diagram and double of the number of channels, and adding noise to the high-quality data in the low-dimensional space;
s32, generating a simulated low-quality image data set through degradation, mapping the noise-added image back to an original image space through a decoder, doubling the size of a characteristic image, and halving the number of channels to obtain a degraded simulated low-quality image;
and S4, outputting a high-low mixed quality image data set, transmitting the generated simulated low quality image data set to a discriminator by the generator, discriminating and comparing the simulated low quality image data set transmitted by the generator with the high quality image data set provided by the data supply unit by the image quality discriminator, and outputting the high-low mixed quality image data set.
8. The method of claim 7, wherein in step S1, the cross-training strategy of the network model of the arbiter comprises:
when training the variational self-encoder, the parameters of the discriminator are fixed, the parameters of the variational self-encoder are adjusted, and when training the variational self-encoder, the integral loss function of the model is
LVAE＝LJS-LGAN… … … … … … … … … … … … … … formula (I) is represented by formula-3,
when training the discriminator, the parameters of the variational self-encoder are fixed, the parameters of the discriminator are adjusted, and when training the discriminator, the integral loss function of the model is
LVAE＝LJS+LGAN… … … … … … … … … … … … … … formula (I) is represented by formula-4,
in the formula, LJSIs the JS divergence loss function, LGANGenerating a pair loss-tolerance function;
and through cross training, the noise addition generation capability of the variational self-encoder and the high-low quality sample discrimination capability of the discriminator are synchronously enhanced, and finally, the image quality discriminator for image quality evaluation is obtained after the model is converged.
9. An apparatus, characterized in that the apparatus comprises:
one or more processors;
a memory for storing one or more programs;
when executed by one or more processors, cause the one or more processors to implement the image quality processing method of claim 7 or 8.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the image quality processing method according to claim 7 or 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011104004.6A CN112365551A (en) | 2020-10-15 | 2020-10-15 | Image quality processing system, method, device and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011104004.6A CN112365551A (en) | 2020-10-15 | 2020-10-15 | Image quality processing system, method, device and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112365551A true CN112365551A (en) | 2021-02-12 |
Family
ID=74507693
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011104004.6A Pending CN112365551A (en) | 2020-10-15 | 2020-10-15 | Image quality processing system, method, device and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112365551A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113781343A (en) * | 2021-09-13 | 2021-12-10 | 叠境数字科技（上海）有限公司 | Super-resolution image quality improvement method |
| CN114584675A (en) * | 2022-05-06 | 2022-06-03 | 中国科学院深圳先进技术研究院 | Self-adaptive video enhancement method and device |
-
2020
- 2020-10-15 CN CN202011104004.6A patent/CN112365551A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113781343A (en) * | 2021-09-13 | 2021-12-10 | 叠境数字科技（上海）有限公司 | Super-resolution image quality improvement method |
| CN114584675A (en) * | 2022-05-06 | 2022-06-03 | 中国科学院深圳先进技术研究院 | Self-adaptive video enhancement method and device |
| CN114584675B (en) * | 2022-05-06 | 2022-08-02 | 中国科学院深圳先进技术研究院 | Self-adaptive video enhancement method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111754596B (en) | Editing model generation method, device, equipment and medium for editing face image | |
| CN109919204B (en) | Noise image-oriented deep learning clustering method | |
| CN110992252B (en) | Image multi-grid conversion method based on latent variable feature generation | |
| CN112132959B (en) | Digital rock core image processing method and device, computer equipment and storage medium | |
| CN111738363B (en) | Alzheimer disease classification method based on improved 3D CNN network | |
| CN110490239B (en) | Training method, quality classification method, device and equipment of image quality control network | |
| CN111932529B (en) | Image classification and segmentation method, device and system | |
| CN114936979B (en) | Model training method, image denoising method, device, equipment and storage medium | |
| CN112365551A (en) | Image quality processing system, method, device and medium | |
| CN110930378A (en) | Emphysema image processing method and system based on low data demand | |
| WO2021184195A1 (en) | Medical image reconstruction method, and medical image reconstruction network training method and apparatus | |
| CN116051382A (en) | Data enhancement method based on deep reinforcement learning generation type antagonistic neural network and super-resolution reconstruction | |
| CN112541566B (en) | Image translation method based on reconstruction loss | |
| CN113850796A (en) | Lung disease identification method and device based on CT data, medium and electronic equipment | |
| CN117934824A (en) | Target region segmentation method and system for ultrasonic image and electronic equipment | |
| WO2021179198A1 (en) | Image feature visualization method, image feature visualization apparatus, and electronic device | |
| CN116030077B (en) | Video salient region detection method based on multi-dataset collaborative learning | |
| CN117097876A (en) | Event camera image reconstruction method based on neural network | |
| CN117079339A (en) | Animal iris recognition method, prediction model training method, electronic equipment and medium | |
| Wang et al. | Retracted: Complex image denoising framework with CNN‐wavelet under concurrency scenarios for informatics systems | |
| CN114283301A (en) | Self-adaptive medical image classification method and system based on Transformer | |
| CN112446345A (en) | Low-quality three-dimensional face recognition method, system, equipment and storage medium | |
| CN114998990B (en) | Method and device for identifying safety behaviors of personnel on construction site | |
| Xiang et al. | Quality-distinguishing and patch-comparing no-reference image quality assessment | |
| Saaim et al. | Generative Models for Data Synthesis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |