CN111815529A - Low-quality image classification enhancement method based on model fusion and data enhancement - Google Patents

Low-quality image classification enhancement method based on model fusion and data enhancement Download PDF

Info

Publication number
CN111815529A
CN111815529A CN202010607913.5A CN202010607913A CN111815529A CN 111815529 A CN111815529 A CN 111815529A CN 202010607913 A CN202010607913 A CN 202010607913A CN 111815529 A CN111815529 A CN 111815529A
Authority
CN
China
Prior art keywords
image
classified
neural network
probability vector
enhancement
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010607913.5A
Other languages
Chinese (zh)
Other versions
CN111815529B (en
Inventor
王道累
张天宇
朱瑞
孙嘉珺
李明山
李超
韩清鹏
袁斌霞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Electric Power University
Original Assignee
Shanghai Electric Power University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Electric Power University filed Critical Shanghai Electric Power University
Priority to CN202010607913.5A priority Critical patent/CN111815529B/en
Publication of CN111815529A publication Critical patent/CN111815529A/en
Application granted granted Critical
Publication of CN111815529B publication Critical patent/CN111815529B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/254—Fusion techniques of classification results, e.g. of results related to same input data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Biophysics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention relates to a low-quality image classification enhancement method based on model fusion and data enhancement, which comprises the following steps of: s1: establishing an image set; s2: performing data enhancement on the image set; s3: constructing and training a VGG16 convolutional neural network model; s4: constructing and training a ResNet convolution neural network model; s5: inputting an image to be classified; s6: acquiring a first probability vector and a second probability vector of an image to be classified; s7: acquiring a fusion probability vector, acquiring the image type of the image to be classified, if the image type is clear, entering a step S9, and if not, entering a step S8; s8: enhancing the image to be classified to obtain an enhanced image, and inputting the enhanced image as the image to be classified into step S6; s9: and outputting the image. Compared with the prior art, the method and the device have the advantages that two modes of data enhancement and model fusion are adopted to improve the accuracy of classification, the images can be effectively classified, and the enhancement effect is good.

Description

Low-quality image classification enhancement method based on model fusion and data enhancement
Technical Field
The invention relates to a low-quality image classification enhancement method, in particular to a low-quality image classification enhancement method based on model fusion and data enhancement.
Background
When a camera shoots a picture, a low-quality image is usually shot due to poor lighting environment or unstable self-stability, and the like, so that the image details are lost due to the fact that shadow, low brightness or blurred pictures exist on the low-quality image, and subsequent operations such as identification and analysis are difficult to perform on the image.
In a low-quality image enhancement method under extreme weather conditions disclosed in chinese patent No. cn201610079472.x, etc., a classification enhancement method for distinguishing a haze image and a rain and snow image according to chromaticity component values is proposed. The method adopts a pure physics algorithm, can distinguish a few image types, has poor robustness, and is very easy to have misjudgment and the like. In an imaging identification method and system in severe weather disclosed in chinese patent CN201811484514.3, a method for enhancing and then identifying collected low-quality images in severe weather is proposed, an electronic image stabilization algorithm is used to eliminate motion blur of the images or an adaptive image defogging algorithm is used to eliminate cloud and fog interference of the images, and then a VGG16 convolutional neural network is used to identify and classify the images. However, the invention does not mention how to distinguish whether the acquired image is a low-quality image, and only one enhancement algorithm is adopted to perform enhancement processing on the low-quality images of all categories, so that the enhancement effect is general.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provide a low-quality image classification enhancement method based on model fusion and data enhancement.
The purpose of the invention can be realized by the following technical scheme:
a low-quality image classification enhancement method based on model fusion and data enhancement comprises the following steps:
s1: establishing an image set, wherein the image set comprises a clear image, a fuzzy image and a low-brightness image;
s2: performing data enhancement on the images in the image set;
s3: constructing a VGG16 convolutional neural network model, and training the VGG16 convolutional neural network model through an image set;
s4: constructing a ResNet convolutional neural network model, and training the ResNet convolutional neural network model through an image set;
s5: inputting an image to be classified;
s6: inputting an image to be classified into a VGG16 convolutional neural network model to obtain a first probability vector of the image to be classified, inputting the image to be classified into a ResNet convolutional neural network model to obtain a second probability vector of the image to be classified;
s7: fusing the first probability vector and the second probability vector to obtain a fused probability vector, and acquiring the image type of the image to be classified, if the image type is clear, entering step S9, otherwise entering step S8;
s8: selecting an image enhancement algorithm corresponding to the image type to enhance the image to be classified to obtain an enhanced image, and inputting the enhanced image serving as the image to be classified into the step S6;
s9: and outputting the image.
Preferably, the first probability vector is:
[λa1，λa2，λa3]
wherein λ isa1，λa2，λa3Probability that the images acquired in the VGG16 convolutional neural network models respectively are sharp images, blurred images and low-brightness images respectively,
the second probability vector is:
[λb1，λb2，λb3]
wherein λ isb1，λb2，λb3Respectively the probability that the images acquired in the ResNet convolutional neural network model are sharp images, blurred images and low-brightness images,
the fusion probability vector is as follows:
[λa1+λb1，λa2+λb2，λa3+λb3]
and the image type represented by the maximum value in the fusion probability vector is the image type of the image to be classified.
Preferably, the classifier of the VGG16 convolutional neural network model comprises two fully-connected layers, wherein the second fully-connected layer adopts a Softmax activation function, and the second fully-connected layer maps the input values of the first fully-connected layer to the interval (0, 1), so as to obtain the first probability vector.
Preferably, the classifier of the ResNet convolutional neural network model includes two fully-connected layers, wherein the second fully-connected layer uses a Softmax activation function, and the second fully-connected layer maps the input values of the first fully-connected layer to the interval (0, 1) to obtain the second probability vector.
Preferably, the data enhancement is one or more of flipping, rotating, scaling, cropping, shifting, adding noise, and modifying contrast, at random, of the image data.
Preferably, the data enhancement in step S2 is to perform horizontal rotation and vertical rotation on the image data, and the step S2 specifically includes:
s21: taking the vertex of the lower left corner of the image as an origin O, establishing an xyz space coordinate system, wherein the size of the image is a multiplied by b, and the image is superposed with an xy plane;
s22: randomly judging whether the image is horizontally rotated or not, wherein the probability of horizontal rotation or not is 0.5, if the image is horizontally rotated, the image is rotated by 180 degrees by taking (x is a/2, and z is 0) as a rotating shaft, and if not, the step S22 is carried out;
s23: and randomly judging whether the image is vertically rotated or not, wherein the probability of vertical rotation or not is 0.5, if the image is vertically rotated, rotating the image 180 degrees by taking (y is b/2 and z is 0) as a rotating shaft, finishing data enhancement, and if the image is not vertically rotated, finishing data enhancement.
Preferably, when the number of times of enhancement of the image is greater than the maximum number of times of enhancement, the image is output:
s5: inputting an image to be classified, and setting the enhancing times p to be 0;
s6: inputting an image to be classified into a VGG16 convolutional neural network model to obtain a first probability vector of the image to be classified, inputting the image to be classified into a ResNet convolutional neural network model to obtain a second probability vector of the image to be classified;
s7: fusing the first probability vector and the second probability vector to obtain a fused probability vector, and acquiring the image type of the image to be classified, if the image type is clear, entering step S9, otherwise entering step S8;
s8: selecting an image enhancement algorithm corresponding to the image type to enhance the image to be classified, acquiring an enhanced image, setting the enhancement times p to be p +1, if p is more than or equal to the maximum enhancement times, entering the step S9, and otherwise, inputting the enhanced image serving as the image to be classified into the step S6;
s9: and outputting the image.
Preferably, the VGG16 convolutional neural network model comprises 13 convolutional layers, 3 fully-connected layers and 5 pooling layers.
Preferably, in S8, the GAN-based blind motion blur removal algorithm is used to perform enhancement processing on the image with the blurred image as the image type.
Preferably, in S8, the image with the image type of low-brightness image is enhanced by using a low-brightness image enhancement algorithm based on the corresponding camera model.
Compared with the prior art, the invention has the following advantages:
(1) compared with the prior art, the image classification method based on the convolutional neural network has the advantages that the convolutional neural network is adopted for classifying the image, the accuracy rate is higher compared with that of a traditional classifier based on physics, and meanwhile, the classification accuracy rate is improved by adopting two modes of data enhancement and model fusion;
(2) the invention integrates two algorithms special for enhancing two types of images, namely fuzzy images and low-brightness images, and can effectively classify and restore the detailed information of the images;
(3) in order to prevent two types of conditions on one image, the method sets the maximum enhancement times, and performs classification enhancement on the image for multiple times to realize the maximum enhancement effect;
(4) the random image data enhancement mode is utilized to perform data enhancement of horizontal rotation and vertical rotation on the image data, and the classification accuracy is effectively improved.
Drawings
FIG. 1 is a flow chart of the present invention;
FIG. 2 is a flow chart of an embodiment of the present invention;
FIG. 3 is a block diagram of the VGG16 model;
FIG. 4 is a block diagram of the VGG16 model;
FIG. 5 is a block diagram of a residual unit;
FIG. 6 is a block diagram of the ResNet model;
fig. 7 is a structural diagram of the DeblurGAN model.
Detailed Description
The invention is described in detail below with reference to the figures and specific embodiments. Note that the following description of the embodiments is merely a substantial example, and the present invention is not intended to be limited to the application or the use thereof, and is not limited to the following embodiments.
Examples
A low-quality image classification enhancement method based on model fusion and data enhancement is disclosed, as shown in FIG. 1, and comprises the following steps:
s1: an image set is established.
In the image set of the present embodiment, the image set includes three types of images, i.e., a sharp image, a blurred image, and a low-brightness image, and includes a training set, a test set, and a verification set, where 300 images of each type are included in the training set, 50 images of each type are included in the test set, and 50 images of each type are included in the verification set.
S2: data enhancement is performed on the images in the image set.
Data enhancement is one or more of flipping, rotating, scaling, cropping, shifting, adding noise, and modifying contrast on the image data at random.
In this embodiment, in step S2, the data enhancement is to perform horizontal rotation and vertical rotation on the image data, where the step S2 specifically includes:
s21: taking the vertex of the lower left corner of the image as an origin O, establishing an xyz space coordinate system, wherein the size of the image is a multiplied by b, and the image is superposed with an xy plane;
s22: randomly judging whether the image is horizontally rotated or not, wherein the probability of horizontal rotation or not is 0.5, if the image is horizontally rotated, the image is rotated by 180 degrees by taking (x is a/2, and z is 0) as a rotating shaft, and if not, the step S22 is carried out;
s23: and randomly judging whether the image is vertically rotated or not, wherein the probability of vertical rotation or not is 0.5, if the image is vertically rotated, rotating the image 180 degrees by taking (y is b/2 and z is 0) as a rotating shaft, finishing data enhancement, and if the image is not vertically rotated, finishing data enhancement.
S3: and constructing a VGG16 convolutional neural network model, and training the VGG16 convolutional neural network model through the image set.
As shown in fig. 3 and 4, the model structure of VGG16 includes 13 Convolutional layers (volumetric Layer), 3 Fully connected layers (full connected Layer), and 5 pooling layers (Pool Layer). The convolutional layers and the pooling layers can be divided into different blocks (blocks), which are numbered from front to back as blocks 1-Block5, and each Block comprises a plurality of convolutional layers and pooling layers, for example, Block4 comprises 3 convolutional layers (Conv3-512) and 1 pooling layer (Maxpool); and within the same block, the number of channels of the convolutional layers is the same, for example: the Block2 comprises 2 convolutional layers, each convolutional layer is represented by Conv2-128, namely the convolutional core is 3 × 3 × 3, and the number of channels is 128; block3 contains 3 convolutional layers, each denoted by Conv3-256, i.e., the convolutional kernel is 3 × 3 × 3, and the number of channels is 256. The input structure of the model VGG16 is 224 × 224 × 3, where the number of channels is doubled, from 64 to 128 in sequence, to 256 until 512 remains the same and is no longer doubled, and the height and width of the image is halved, from 224 → 112 → 56 → 28 → 14 → 7.
S4: and constructing a ResNet convolutional neural network model, and training the ResNet convolutional neural network model through the image set.
The ResNet convolution neural network solves the degradation problem of the deep network through residual learning: as network depth increases, network accuracy saturates and even decreases. As shown in fig. 5 and 6, the structure of the ResNet model refers to the VGG19 structure, on the basis of which the residual unit is modified and added through a short circuit mechanism.
The residual unit has two layers, the first layer:
F(x)＝W2σ(W1x)
where σ represents the nonlinear activation function ReLu, W1And W2Two weights in the first layer are represented.
Then the first layer is connected with the second layer through a short circuit, the ReLu function of the second layer obtains an output y,
y＝F(x)+x
wherein F (x) is the output of the first layer.
ResNet is mainly characterized in that the convolution with the convolution kernel size of 2 is directly used for down-sampling, and a global average pooling layer is adopted to replace a full-connection layer. Compared with the common network, the ResNet network imposes a short circuit mechanism between every two layers, so that residual error learning is formed.
S5: inputting an image to be classified;
s6: and inputting the image to be classified into a VGG16 convolutional neural network model to obtain a first probability vector of the image to be classified, inputting the image to be classified into a ResNet convolutional neural network model to obtain a second probability vector of the image to be classified.
S7: and fusing the first probability vector and the second probability vector to obtain a fused probability vector, and acquiring the image type of the image to be classified, if the image type is clear, entering the step S9, otherwise, entering the step S8.
The invention adopts a VGG16 convolutional neural network model and a ResNet convolutional neural network model for model fusion. Model fusion is to directly add probability vectors predicted by two or more basic models, and take the class with the highest probability after addition as the prediction class of the image.
In this embodiment, the probability vector is represented in a Onehot encoding form. The Onehot coding is a common class label coding form in a neural network, and converts a class label with an integer value into a binary value, wherein the index of the label value is marked as 1, and the other indexes are 0. In this embodiment, there are 3 category labels for the sharp image, the blurred image, and the low-luminance image, which correspond to label values 0,1, and 2, respectively. The tag value 0 is converted into [1,0,0] after One-Hot encoding, the tag value 1 is converted into [0,1,0], and the tag value 2 is converted into [0,0,1 ].
In this embodiment, the two model classifiers each include two fully-connected layers, and the second fully-connected layer employs a Softmax activation function, and the second fully-connected layer maps the input values from the first fully-connected layer to the intervals (0, 1), and obtains a first probability vector and a second probability vector. Specifically, the Softmax function maps the input values of the first fully-connected layer between intervals (0, 1), forming, for example, [ lambda ]1，λ2，λ3]Of the probability vector ofiFor each class of probability, the sum of which is 1, usually taken as λiThe largest median is used as the predicted class.
The formula of the Softmax function is:
Figure BDA0002561455890000071
wherein Z isiAnd C is the output value of the ith node, and the number of output nodes, namely the number of classified categories.
In this embodiment, the pre-trained VGG16 convolutional neural network model and ResNet convolutional neural network model are adopted to output the first probability vector [ λ [ lambda ] ], respectivelya1，λa2，λa3]And a second probability vector [ lambda ]b1，λb2，λb3]。
Wherein, λa1，λa2，λa3Probability, λ, of the images acquired in the VGG16 convolutional neural network model to be sharp, blurred and low-luminance images, respectivelyb1，λb2，λb3Respectively, the probability that the image obtained in the ResNet convolutional neural network model is a sharp image, a blurred image, and a low-brightness image, respectively.
Adding the two probability vectors to obtain a fusion probability vector
[λa1+λb1，λa2+λb2，λa3+λb3]
And the image type represented by the maximum value in the fusion probability vector is the image type of the image to be classified.
S8: and (4) selecting an image enhancement algorithm corresponding to the image type to enhance the image to be classified to obtain an enhanced image, and inputting the enhanced image serving as the image to be classified into the step S6.
Specifically, a GAN-based blind motion blur removal algorithm is adopted to enhance an image with a blurred image as an image type, and a corresponding camera model-based low-brightness image enhancement algorithm is adopted to enhance an image with a low-brightness image as an image type.
Further, a GAN-based blind motion blur removal algorithm:
the algorithm treats image blurring removal as an end-to-end task through a GAN network, and a clear image is automatically generated according to a blurred image by using a generator through learning the blurred image and the clear image. The blind deblurring in the algorithm aims to restore a clear image Is by giving a blurred image IB under the condition of no blur kernel, deblurring Is carried out by adopting a generator, and a discrimination network Is introduced in the training process to carry out training and learning in a countermeasure mode.
As shown IN fig. 7, the generator contains two downsampling convolution modules, 9 residual modules (containing one convolution, IN and ReLU), and two upsampling transposed convolution modules, while also introducing global residual concatenation. This structure may allow faster training and at the same time better performance. Besides the generator, the algorithm also defines a discriminator in the training process, and adopts WGAN (Wasserstein GAN) with a penalty term to carry out countertraining.
The loss function of the algorithm consists of two parts: content loss and antagonism loss, the formula is:
L＝LCAN+λLx
wherein L isGANTo combat losses, LxFor content loss, λ is the weight.
In countering loss, using WGAN-GP as a penalty function, the loss is calculated as:
Figure BDA0002561455890000081
wherein, IBIn order to input a blurred image, the image is displayed,
Figure BDA0002561455890000085
in order to be a generator,
Figure BDA0002561455890000086
to discriminate the network.
The content loss adopts the perception loss, and is based on the difference between the generated and target image CNN feature mapping, and the formula is as follows:
Figure BDA0002561455890000082
wherein, IsFor the generated image, IBIn order to input a blurred image, the image is displayed,
Figure BDA0002561455890000083
to be a generator, phii，jIs a feature map obtained by the jth convolution (after activation) before the ith max pooling layer in a VGG19 network, Wi，jAnd Hi，jIs a dimension function graph.
Further, a low brightness picture enhancement algorithm based on the corresponding model of the camera:
for a picture taken by a camera, the pixel value is not proportional to the brightness value reflected by the object. The nonlinear transformation Function from the luminance value sensed by the Camera photosensitive element to the actual pixel value of the image is called a Camera Response Function (CRF). The algorithm firstly waits for a camera response model by analyzing the relation between pictures with different exposure degrees, then obtains an exposure contrast image of the image by using a picture brightness component estimation method, and finally enhances the low-brightness picture by using the camera response model and the exposure contrast image.
The camera response model is defined as:
P＝f(E)
where E is the irradiance of the picture, P is the pixel value of the picture, and f should satisfy:
Figure BDA0002561455890000084
the algorithm uses a luminance Transform Function (BTF) to estimate f.
BTF is a mapping function between two graphs of the same scene but different exposure:
R＝g(P0，k)
wherein P is1And P0Is an image with different exposure degrees in the same scene, and k is an exposure ratio.
CRM can therefore be calculated from g (f (e), k) ═ f (ke).
S9: and outputting the image.
The experimental environment in this example is: the accuracy of the system Win10, the display card Tesla P100, the deep learning framework Tensorflow2.0, the VGG16, the ResNet model and the fused model is shown in the following table, and the accuracy of the fused model is higher than that of any single model, which is shown in the following table.
Figure BDA0002561455890000091
In an embodiment of the present invention, in order to prevent the image processed by the two enhancement algorithms from being blurred or with low brightness, as shown in fig. 2, a maximum enhancement number is set, and when the enhancement number of the image is greater than or equal to the maximum enhancement number, the image is output:
s1: establishing an image set, wherein the image set comprises a clear image, a fuzzy image and a low-brightness image;
s2: performing data enhancement on the images in the image set;
s3: constructing a VGG16 convolutional neural network model, and training the VGG16 convolutional neural network model through an image set;
s4: constructing a ResNet convolutional neural network model, and training the ResNet convolutional neural network model through an image set;
s5: inputting an image to be classified, and setting the enhancing times p to be 0;
s6: inputting an image to be classified into a VGG16 convolutional neural network model to obtain a first probability vector of the image to be classified, inputting the image to be classified into a ResNet convolutional neural network model to obtain a second probability vector of the image to be classified;
s7: fusing the first probability vector and the second probability vector to obtain a fused probability vector, and acquiring the image type of the image to be classified, if the image type is clear, entering step S9, otherwise entering step S8;
s8: selecting an image enhancement algorithm corresponding to the image type to enhance the image to be classified, acquiring an enhanced image, setting the enhancement times p to be p +1, if p is more than or equal to the maximum enhancement times, entering the step S9, and otherwise, inputting the enhanced image serving as the image to be classified into the step S6;
s9: and outputting the image.
The above embodiments are merely examples and do not limit the scope of the present invention. These embodiments may be implemented in other various manners, and various omissions, substitutions, and changes may be made without departing from the technical spirit of the present invention.

Claims (10)

1. A low-quality image classification enhancement method based on model fusion and data enhancement is characterized by comprising the following steps:
s1: establishing an image set, wherein the image set comprises a clear image, a fuzzy image and a low-brightness image;
s2: performing data enhancement on the images in the image set;
s3: constructing a VGG16 convolutional neural network model, and training the VGG16 convolutional neural network model through an image set;
s4: constructing a ResNet convolutional neural network model, and training the ResNet convolutional neural network model through an image set;
s5: inputting an image to be classified;
s6: inputting an image to be classified into a VGG16 convolutional neural network model to obtain a first probability vector of the image to be classified, inputting the image to be classified into a ResNet convolutional neural network model to obtain a second probability vector of the image to be classified;
s7: fusing the first probability vector and the second probability vector to obtain a fused probability vector, and acquiring the image type of the image to be classified, if the image type is clear, entering step S9, otherwise entering step S8;
s8: selecting an image enhancement algorithm corresponding to the image type to enhance the image to be classified to obtain an enhanced image, and inputting the enhanced image serving as the image to be classified into the step S6;
s9: and outputting the image.
2. The method of claim 1, wherein the first probability vector is:
[λa1，λa2，λa3]
wherein λ isa1，λa2，λa3Probability that the images acquired in the VGG16 convolutional neural network models respectively are sharp images, blurred images and low-brightness images respectively,
the second probability vector is:
[λb1，λb2，λb3]
wherein λ isb1，λb2，λb3Respectively the probability that the images acquired in the ResNet convolutional neural network model are sharp images, blurred images and low-brightness images,
the fusion probability vector is as follows:
[λa1+λb1，λa2+λb2，λa3+λb3]
and the image type represented by the maximum value in the fusion probability vector is the image type of the image to be classified.
3. The method of claim 1, wherein the classifier of the VGG16 convolutional neural network model comprises two fully-connected layers, wherein the second fully-connected layer uses Softmax activation function, and the second fully-connected layer maps the input values from the first fully-connected layer to the interval (0, 1) to obtain the first probability vector.
4. The method as claimed in claim 1, wherein the classifier of the ResNet convolutional neural network model comprises two fully-connected layers, wherein the second fully-connected layer uses Softmax activation function, and the second fully-connected layer maps the input values from the first fully-connected layer to the interval (0, 1) to obtain the second probability vector.
5. The method of claim 1, wherein the data enhancement is one or more of flipping, rotating, scaling, cropping, shifting, adding noise, and modifying contrast of the image data.
6. The method as claimed in claim 5, wherein the data enhancement in step S2 is performed by rotating the image data horizontally and vertically, and the step S2 specifically comprises:
s21: taking the vertex of the lower left corner of the image as an origin O, establishing an xyz space coordinate system, wherein the size of the image is a multiplied by b, and the image is superposed with an xy plane;
s22: randomly judging whether the image is horizontally rotated or not, wherein the probability of horizontal rotation or not is 0.5, if the image is horizontally rotated, the image is rotated by 180 degrees by taking (x is a/2, and z is 0) as a rotating shaft, and if not, the step S22 is carried out;
s23: and randomly judging whether the image is vertically rotated or not, wherein the probability of vertical rotation or not is 0.5, if the image is vertically rotated, rotating the image 180 degrees by taking (y is b/2 and z is 0) as a rotating shaft, finishing data enhancement, and if the image is not vertically rotated, finishing data enhancement.
7. The method for classifying and enhancing the low-quality image based on the model fusion and the data enhancement as claimed in claim 1, wherein when the enhancing times of the image is greater than or equal to the maximum enhancing times, the image is output:
s5: inputting an image to be classified, and setting the enhancing times p to be 0;
s6: inputting an image to be classified into a VGG16 convolutional neural network model to obtain a first probability vector of the image to be classified, inputting the image to be classified into a ResNet convolutional neural network model to obtain a second probability vector of the image to be classified;
s7: fusing the first probability vector and the second probability vector to obtain a fused probability vector, and acquiring the image type of the image to be classified, if the image type is clear, entering step S9, otherwise entering step S8;
s8: selecting an image enhancement algorithm corresponding to the image type to enhance the image to be classified, acquiring an enhanced image, setting the enhancement times p to be p +1, if p is more than or equal to the maximum enhancement times, entering the step S9, and otherwise, inputting the enhanced image serving as the image to be classified into the step S6;
s9: and outputting the image.
8. The method of claim 1, wherein the VGG16 convolutional neural network model comprises 13 convolutional layers, 3 fully-connected layers and 5 pooling layers.
9. The method according to claim 1, wherein in S8, the blind motion blur removal algorithm based on GAN is used to enhance the image with the blurred image as the image type.
10. The method according to claim 1, wherein in S8, the image with the image type of low-brightness image is enhanced by a low-brightness image enhancement algorithm based on a corresponding camera model.
CN202010607913.5A 2020-06-30 2020-06-30 Low-quality image classification enhancement method based on model fusion and data enhancement Active CN111815529B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010607913.5A CN111815529B (en) 2020-06-30 2020-06-30 Low-quality image classification enhancement method based on model fusion and data enhancement

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010607913.5A CN111815529B (en) 2020-06-30 2020-06-30 Low-quality image classification enhancement method based on model fusion and data enhancement

Publications (2)

Publication Number Publication Date
CN111815529A true CN111815529A (en) 2020-10-23
CN111815529B CN111815529B (en) 2023-02-07

Family

ID=72856666

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010607913.5A Active CN111815529B (en) 2020-06-30 2020-06-30 Low-quality image classification enhancement method based on model fusion and data enhancement

Country Status (1)

Country Link
CN (1) CN111815529B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112541877A (en) * 2020-12-24 2021-03-23 广东宜教通教育有限公司 Condition-based generation of deblurring method, system, device and medium for countermeasure network
CN113469083A (en) * 2021-07-08 2021-10-01 西安电子科技大学 SAR image target classification method and system based on anti-sawtooth convolution neural network
CN113469083B (en) * 2021-07-08 2024-05-31 西安电子科技大学 SAR image target classification method and system based on antialiasing convolutional neural network

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106650721A (en) * 2016-12-28 2017-05-10 吴晓军 Industrial character identification method based on convolution neural network
CN106875352A (en) * 2017-01-17 2017-06-20 北京大学深圳研究生院 A kind of enhancement method of low-illumination image
CN107169450A (en) * 2017-05-15 2017-09-15 中国科学院遥感与数字地球研究所 The scene classification method and system of a kind of high-resolution remote sensing image
CN108734667A (en) * 2017-04-14 2018-11-02 Tcl集团股份有限公司 A kind of image processing method and system
CN109522945A (en) * 2018-10-31 2019-03-26 中国科学院深圳先进技术研究院 One kind of groups emotion identification method, device, smart machine and storage medium
CN109801224A (en) * 2018-12-04 2019-05-24 北京奇艺世纪科技有限公司 A kind of image processing method, device, server and storage medium
CN109934293A (en) * 2019-03-15 2019-06-25 苏州大学 Image-recognizing method, device, medium and obscure perception convolutional neural networks
CN110264424A (en) * 2019-06-20 2019-09-20 北京理工大学 A kind of fuzzy retinal fundus images Enhancement Method based on generation confrontation network
CN110428011A (en) * 2019-08-06 2019-11-08 华南理工大学 A kind of deep learning image fault classification method towards video transmission quality
CN110956201A (en) * 2019-11-07 2020-04-03 江南大学 Image distortion type classification method based on convolutional neural network
CN111127435A (en) * 2019-12-25 2020-05-08 福州大学 No-reference image quality evaluation method based on double-current convolutional neural network

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106650721A (en) * 2016-12-28 2017-05-10 吴晓军 Industrial character identification method based on convolution neural network
CN106875352A (en) * 2017-01-17 2017-06-20 北京大学深圳研究生院 A kind of enhancement method of low-illumination image
CN108734667A (en) * 2017-04-14 2018-11-02 Tcl集团股份有限公司 A kind of image processing method and system
CN107169450A (en) * 2017-05-15 2017-09-15 中国科学院遥感与数字地球研究所 The scene classification method and system of a kind of high-resolution remote sensing image
CN109522945A (en) * 2018-10-31 2019-03-26 中国科学院深圳先进技术研究院 One kind of groups emotion identification method, device, smart machine and storage medium
CN109801224A (en) * 2018-12-04 2019-05-24 北京奇艺世纪科技有限公司 A kind of image processing method, device, server and storage medium
CN109934293A (en) * 2019-03-15 2019-06-25 苏州大学 Image-recognizing method, device, medium and obscure perception convolutional neural networks
CN110264424A (en) * 2019-06-20 2019-09-20 北京理工大学 A kind of fuzzy retinal fundus images Enhancement Method based on generation confrontation network
CN110428011A (en) * 2019-08-06 2019-11-08 华南理工大学 A kind of deep learning image fault classification method towards video transmission quality
CN110956201A (en) * 2019-11-07 2020-04-03 江南大学 Image distortion type classification method based on convolutional neural network
CN111127435A (en) * 2019-12-25 2020-05-08 福州大学 No-reference image quality evaluation method based on double-current convolutional neural network

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112541877A (en) * 2020-12-24 2021-03-23 广东宜教通教育有限公司 Condition-based generation of deblurring method, system, device and medium for countermeasure network
CN112541877B (en) * 2020-12-24 2024-03-19 广东宜教通教育有限公司 Defuzzification method, system, equipment and medium for generating countermeasure network based on condition
CN113469083A (en) * 2021-07-08 2021-10-01 西安电子科技大学 SAR image target classification method and system based on anti-sawtooth convolution neural network
CN113469083B (en) * 2021-07-08 2024-05-31 西安电子科技大学 SAR image target classification method and system based on antialiasing convolutional neural network

Also Published As

Publication number Publication date
CN111815529B (en) 2023-02-07

Similar Documents

Publication Publication Date Title
CN112233038B (en) True image denoising method based on multi-scale fusion and edge enhancement
CN109584248B (en) Infrared target instance segmentation method based on feature fusion and dense connection network
CN108229468B (en) Vehicle appearance feature recognition and vehicle retrieval method and device, storage medium and electronic equipment
EP3937481A1 (en) Image display method and device
CN111402146B (en) Image processing method and image processing apparatus
CN107274445B (en) Image depth estimation method and system
CN111915525B (en) Low-illumination image enhancement method capable of generating countermeasure network based on improved depth separation
CN111754446A (en) Image fusion method, system and storage medium based on generation countermeasure network
WO2022021999A1 (en) Image processing method and image processing apparatus
US10614736B2 (en) Foreground and background detection method
US10706558B2 (en) Foreground and background detection method
CN111260738A (en) Multi-scale target tracking method based on relevant filtering and self-adaptive feature fusion
EP4226322A1 (en) Segmentation for image effects
CN109903315B (en) Method, apparatus, device and readable storage medium for optical flow prediction
CN110532959B (en) Real-time violent behavior detection system based on two-channel three-dimensional convolutional neural network
CN110866879A (en) Image rain removing method based on multi-density rain print perception
CN111582074A (en) Monitoring video leaf occlusion detection method based on scene depth information perception
CN112115979A (en) Fusion method and device of infrared image and visible image
CN113034417A (en) Image enhancement system and image enhancement method based on generation countermeasure network
CN111274988B (en) Multispectral-based vehicle weight identification method and device
CN112949453A (en) Training method of smoke and fire detection model, smoke and fire detection method and smoke and fire detection equipment
CN116977674A (en) Image matching method, related device, storage medium and program product
CN111914938A (en) Image attribute classification and identification method based on full convolution two-branch network
Chen et al. Visual depth guided image rain streaks removal via sparse coding
CN115272437A (en) Image depth estimation method and device based on global and local features

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant