CN111507910B - Single image antireflection method, device and storage medium - Google Patents
Single image antireflection method, device and storage medium Download PDFInfo
- Publication number
- CN111507910B CN111507910B CN202010193974.1A CN202010193974A CN111507910B CN 111507910 B CN111507910 B CN 111507910B CN 202010193974 A CN202010193974 A CN 202010193974A CN 111507910 B CN111507910 B CN 111507910B
- Authority
- CN
- China
- Prior art keywords
- image
- loss function
- network
- reflection
- background image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 230000000694 effects Effects 0.000 claims abstract description 26
- 238000012549 training Methods 0.000 claims abstract description 24
- 238000000605 extraction Methods 0.000 claims abstract description 14
- 230000006870 function Effects 0.000 claims description 152
- 238000000926 separation method Methods 0.000 claims description 31
- 238000004590 computer program Methods 0.000 claims description 23
- 230000008485 antagonism Effects 0.000 claims description 15
- 238000012545 processing Methods 0.000 claims description 9
- 238000011156 evaluation Methods 0.000 claims description 8
- 241000006302 Usnea Species 0.000 claims description 4
- 238000013528 artificial neural network Methods 0.000 claims description 4
- 230000009466 transformation Effects 0.000 claims description 4
- 230000001953 sensory effect Effects 0.000 abstract description 7
- 238000010586 diagram Methods 0.000 description 10
- 230000006872 improvement Effects 0.000 description 6
- 230000004913 activation Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000003667 anti-reflective effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 239000011521 glass Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 238000011158 quantitative evaluation Methods 0.000 description 2
- 238000001028 reflection method Methods 0.000 description 2
- 230000016776 visual perception Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000000087 stabilizing effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G06T5/90—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The invention discloses a method, a device and a storage medium for single image antireflection, wherein the method comprises the following steps: obtaining a background image and a corresponding reflection image through manual shooting, and obtaining a reflection image according to superposition of the background image and the reflection image; inputting the reflective image into a pretrained VGG-19 network for super-column feature extraction to obtain a feature set; inputting the feature set into a preset generation network to obtain a prediction background image and a prediction reflection image; inputting the predicted background image and the background image into a preset authentication network to calculate an authentication loss function of the authentication network; performing iterative computation for multiple times until the joint loss function and the authentication loss function of the generated network are converged, and finishing the training of the generated network and the authentication network; and selecting a plurality of reflection images to perform anti-reflection treatment so as to quantitatively evaluate the anti-reflection effect. The invention can extract the high-level sensory information of the image and add the high-level sensory information into the training of generating an countermeasure network, thereby effectively solving the anti-reflection problem of the single image.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a method, an apparatus, and a storage medium for single image light reflection.
Background
Reflection removal of a single image typically uses a priori information that is set in advance. First, the most common approach is to separate the layers of the image by finding minimized edges and corner points using the sparse nature of the natural image gradients, for example, using the constraint of gradient sparsity in combination with the data fidelity term of the laplace domain to suppress image reflection. However, this approach relies on low-level heuristics, and is limited in cases where high-level analysis of the results of the image is required, and so on. Another a priori knowledge is that the image of the reflective layer is typically unfocused, smooth. Algorithms based on this assumption cannot be applied to cases where the reflected image also has a very strong contrast. None of these methods effectively utilize the high-level sensory information of the image and fail to solve the anti-reflection problem of the high-contrast retroreflective image.
Disclosure of Invention
The embodiment of the invention aims to provide a method, a device and a storage medium for single image antireflection, which can effectively extract high-level sensory information of images, add the high-level sensory information into network training, combine the advantages of generating an countermeasure network, effectively solve the problem of single image antireflection, and still have satisfactory antireflection effect on high-contrast reflection images.
To achieve the above object, an embodiment of the present invention provides a method for single image light-removing, comprising the steps of:
obtaining a background image and a corresponding reflection image through manual shooting, and obtaining a reflection image according to superposition of the background image and the reflection image;
inputting the reflective image into a pretrained VGG-19 network for super-column feature extraction to obtain a feature set;
inputting the feature set into a preset generation network to obtain a prediction background image and a prediction reflection image; wherein the joint loss function of the generation network comprises a reconstruction loss function, an antagonism loss function and a separation loss function of the supercolumn feature space;
inputting the predicted background image and the background image into a preset authentication network to calculate an authentication loss function of the authentication network;
performing repeated iterative computation until the joint loss function and the authentication loss function are converged, and completing the training of the generation network and the authentication network;
and selecting a plurality of the reflection images to perform anti-reflection treatment so as to quantitatively evaluate the anti-reflection effect.
Preferably, the obtaining a reflective image according to the superposition of the background image and the reflective image specifically includes:
acquiring a first gray value of the background image;
acquiring a second gray value of the reflected image;
and carrying out weighted calculation on the first gray level value and the second gray level value to obtain the reflective image.
Preferably, the convolutional layers of the VGG-19 network include conv1_2, conv2_2, conv3_2, conv4_2, and conv5_2.
Preferably, the generating network comprises an input layer with a convolution kernel of 1×1 and 8 hole convolution layers with a convolution kernel of 3×3; the final layer of cavity convolution layer generates two three-channel RGB images by linear transformation.
Preferably, the joint loss function of the generating network includes a reconstruction loss function, an antagonism loss function and a separation loss function of the supercolumn feature space, and specifically includes:
the expression of the reconstruction loss function of the supercolumn feature space is that
Wherein L is feat (θ) is the reconstruction loss function of the supercolumn feature space, I, T and f T (I; θ) the retroreflective image, the background image and the predictive background image, λ, respectively l For the impact weight of the layer i convolution layer, omega is the set of image data for the training, I.I 1 Representing the taking of a 1-norm, i.e. vector, of a vector of the convolutional result of the neural networkSum of absolute values of elements, Φ l (x) A convolution operation of a first layer convolution layer of the VGG-19 network is represented, and θ represents a generated network parameter;
the expression of the counterdamage function is
Wherein L is adv (θ) is the counterdamage function, D (I, x) represents the probability that x is the background image corresponding to the retroreflective image I, derived from the output of the authentication network;
the expression of the separation loss function is
Wherein L is excl (θ) is the separation loss function,
λ T and lambda is R First and second normalized parameters, respectively, |·|| F Is Luo Beini Usnea norm, by which is expressed the multiplication of elements, N is the image downsampling parameter, N is greater than or equal to 1 and less than or equal to N, and N is the maximum value of the image downsampling parameter; f (f) R (I; θ) are the predicted reflectances, respectively,>
for predicting the modulus of the gradient of the background image +.>
A module for predicting a gradient of the reflected image;
the joint loss function of the generating network is L (theta) =w 1 L feat (θ)+w 2 L adv (θ)+w 3 L excl (θ); wherein L (θ) is the joint loss function, w 1 、w 2 And w 3 And the reconstructed loss function, the antagonism loss function and the coefficient corresponding to the separation loss function of the supercolumn feature space are respectively.
Preferably, the authentication loss function of the authentication network is L disc (θ)＝log D(I；f T (I; θ)) -log D (I, T); wherein L is disc And (θ) is the discrimination loss function.
Preferably, the selecting a plurality of light-reflecting images for light-reflecting treatment to quantitatively evaluate the light-reflecting effect specifically includes:
selecting a plurality of reflection images to perform anti-reflection processing, and calculating peak signal-to-noise ratio and structural similarity between the predicted background image and the background image generated by the generating network to quantitatively evaluate the anti-reflection effect
Another embodiment of the present invention provides a single image retroreflective apparatus, comprising:
the image set acquisition module is used for acquiring a background image and a corresponding reflection image through manual shooting, and obtaining a reflection image according to superposition of the background image and the reflection image;
the feature extraction module is used for inputting the reflective image into a pretrained VGG-19 network to perform super-column feature extraction to obtain a feature set;
the prediction generation module is used for inputting the feature set into a preset generation network to obtain a prediction background image and a prediction reflection image; wherein the joint loss function of the generation network comprises a reconstruction loss function, an antagonism loss function and a separation loss function of the supercolumn feature space;
the identification module is used for inputting the prediction background image and the background image into a preset identification network so as to calculate an identification loss function of the identification network;
the training module is used for completing the training of the generating network and the identifying network through repeated iterative computation until the joint loss function and the identifying loss function are converged;
and the evaluation module is used for selecting a plurality of the reflection images to perform the anti-reflection treatment so as to quantitatively evaluate the anti-reflection effect.
A further embodiment of the invention correspondingly provides a device for using the method for single image antireflection, comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the method for single image antireflection according to any one of the above.
Another embodiment of the present invention provides a computer readable storage medium, where the computer readable storage medium includes a stored computer program, where the computer program when executed controls a device in which the computer readable storage medium is located to perform a method for single image light-removal according to any one of the above.
Compared with the prior art, the method, the device and the storage medium for removing the reflection of the single image provided by the embodiment of the invention can obtain the predictive background image which is close to the real background image by effectively extracting the high-grade sensory information of the image by means of the deep convolutional neural network and combining with the optimization characteristic of the generated countermeasure network, effectively solve the problem of the reflection of the image in the image acquisition and still have a satisfactory reflection removing effect on the reflection image with high contrast.
Drawings
FIG. 1 is a flow chart of a method for single image retroreflective according to an embodiment of the present invention;
FIG. 2 is a simplified flow chart of a method for single image retroreflective according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a generating network according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of an authentication network according to an embodiment of the present invention;
FIG. 5 is a graph of the retroreflective contrast effect of 4 sets of retroreflective images, background images, predicted background images, and retroreflective images provided in accordance with one embodiment of the present invention;
FIG. 6 is a schematic structural diagram of a single image retroreflective apparatus according to an embodiment of the present invention;
fig. 7 is a schematic diagram of an apparatus for a method of using single image anti-reflection according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, a flowchart of a method for single image light removal according to an embodiment of the invention is shown, where the method includes steps S1 to S6:
s1, acquiring a background image and a corresponding reflection image through manual shooting, and obtaining a reflection image according to superposition of the background image and the reflection image;
s2, inputting the reflective image into a pretrained VGG-19 network for super-column feature extraction to obtain a feature set;
s3, inputting the feature set into a preset generation network to obtain a prediction background image and a prediction reflection image; wherein the joint loss function of the generation network comprises a reconstruction loss function, an antagonism loss function and a separation loss function of the supercolumn feature space;
s4, inputting the prediction background image and the background image into a preset authentication network to calculate an authentication loss function of the authentication network;
s5, performing repeated iterative computation until the joint loss function and the authentication loss function are converged, and completing training of the generation network and the authentication network;
s6, selecting a plurality of the reflection images to perform anti-reflection treatment so as to quantitatively evaluate the anti-reflection effect.
Specifically, a background image and a corresponding reflection image are obtained through manual shooting, and a reflection image is obtained according to superposition of the background image and the reflection image. Because the original image without reflection is difficult to obtain in real life, the background image is manufactured by adopting a manual mode in the invention, and the method comprises the following steps: background image an indoor image is selected, the object is placed on one side of a transparent glass (preferably the dark side) and the photographing lens is on the other side of the glass. Then fixing the object and lens positions, and shooting the image to obtain a background image without reflection. The reflected image may be selected as an outdoor image. After obtaining the background image and the reflection image, setting the two images to the same size H W3, and then carrying out image superposition to obtain the reflection image. The final dataset contained 2000 reflectance images and their corresponding background and reflectance images.
And inputting the reflective image into a pretrained VGG-19 network to perform super-column feature extraction to obtain a feature set. The supercolumn features have a total of 1472 dimensions, and then three channels of the reflectorized image are connected with the supercolumn features to form a 1475-dimensional feature set, denoted as Φ (x), where x is the input reflectorized image.
Inputting the feature set into a preset generation network to obtain a prediction background image and a prediction reflection image; wherein, the joint loss function of the generating network comprises a reconstruction loss function, an antagonism loss function and a separation loss function of the supercolumn characteristic space;
the predicted background image and the background image are input into a preset authentication network to calculate an authentication loss function of the authentication network. The authentication network is introduced to judge two images inputted, and to output probabilities that the two images originate from the dataset.
Performing repeated iterative computation until the joint loss function and the authentication loss function are converged, and finishing the training of generating a network and the authentication network; during the training process, the output probability of the discrimination network may affect the joint loss function of the generation network to optimize the generation network. Typically, when the output probability of the authentication network is 0.5, it means that both functions converge. Preferably, the training parameters are: max_epoch=250, batch_size=1; the optimization mode is Adam optimization algorithm, and the learning rate is 10 -4 。
In order to evaluate the merits of the method of the present invention, a plurality of retroreflective images are selected for retroreflective treatment to quantitatively evaluate the retroreflective effect.
For a clearer understanding of the implementation process of the method of the present invention, refer to fig. 2, which is a simple flow chart of a single image antireflection method provided in this embodiment of the present invention.
According to the method for removing the reflection of the single image, provided by the embodiment 1, the high-level sensory information of the image is effectively extracted by means of the deep convolutional neural network, and the optimized characteristic of the countermeasure network is combined and generated, so that the predicted background image which is close to the real background image can be obtained, the problem of image reflection in image acquisition is effectively solved, and the satisfactory reflection removing effect is still achieved for the reflection image with high contrast.
As an improvement of the above solution, the obtaining a reflective image according to the superposition of the background image and the reflective image specifically includes:
acquiring a first gray value of the background image;
acquiring a second gray value of the reflected image;
and carrying out weighted calculation on the first gray level value and the second gray level value to obtain the reflective image.
Specifically, the retroreflective image I can be regarded as a linear superposition of the background image T and the retroreflective image R. Therefore, the reflective image can be obtained by acquiring the first gray value of the background image and the second gray value of the reflective image, and performing weighted calculation on the first gray value and the second gray value, and expressed as a mathematical expression: i= (1- α) ×t+α×r; wherein I is a reflective image, T is a background image, R is a reflective image, α is a weighting parameter corresponding to the reflective image, and α e [0,1], preferably α=0.5.
As an improvement to the above, the convolutional layers of the VGG-19 network include conv1_2, conv2_2, conv3_2, conv4_2, and conv5_2.
Specifically, the convolutional layers of the VGG-19 network include conv1_2, conv2_2, conv3_2, conv4_2, and conv5_2.VGG-19 networks are pre-trained on ImageNet to extract supercolumn features of an input image, which is beneficial in that the input adds useful features that abstract the visual perception of large datasets (e.g., imageNet). The supercolumn feature for a given pixel location is a stack of active cells on a network selected convolutional layer for that location.
As an improvement of the above scheme, the generating network includes an input layer with a convolution kernel of 1×1 and 8 hole convolution layers with a convolution kernel of 3×3; the final layer of cavity convolution layer generates two three-channel RGB images by linear transformation.
Specifically, referring to fig. 3, a schematic structural diagram of a generating network according to this embodiment of the present invention is provided. As can be seen from fig. 3, the generating network includes an input layer with a convolution kernel of 1×1 and 8 hole convolution layers with a convolution kernel of 3×3; the final layer of cavity convolution layer generates two three-channel RGB images by linear transformation. The input layer can reduce 1475-dimensional characteristics output by the VGG-19 network to 64-dimensional characteristics, and the expansion rate of the 8 cavity convolution layers is in the range of 1 to 128. The number of feature layers that generate all intermediate layers of the network is 64.
As an improvement of the above solution, the joint loss function of the generating network includes a reconstruction loss function, an antagonism loss function and a separation loss function of the supercolumn feature space, and specifically includes:
the expression of the reconstruction loss function of the supercolumn feature space is that
Wherein L is feat (θ) is a reconstruction loss function of the supercolumn feature space, Φ l For the first layer convolution layers, I, T and f of the VGG-19 network T (I; θ) the retroreflective image, the background image and the predictive background image, λ, respectively l For the impact weight of the layer i convolution layer, omega is the set of image data for the training, I.I 1 Representing the vector taking a 1-norm, i.e. the sum of the absolute values of the vector elements, of the convolutional result of the neural network Φ l (x) A convolution operation of a first layer convolution layer of the VGG-19 network is represented, and θ represents a generated network parameter;
the expression of the counterdamage function is
Wherein L is adv (θ) is the counterdamage function, D (I, x) represents the probability that x is the background image corresponding to the reflective image I, and is derived from the authentication networkObtaining the product;
the expression of the separation loss function is
Wherein L is excl (θ) is the separation loss function,
λ T and lambda is R First and second normalized parameters, respectively, |·|| F Is Luo Beini Usnea norm, by which is expressed the multiplication of elements, N is the image downsampling parameter, N is greater than or equal to 1 and less than or equal to N, and N is the maximum value of the image downsampling parameter; f (f) R (I; θ) are the predicted reflectances, respectively,>
for predicting the modulus of the gradient of the background image +.>
A module for predicting a gradient of the reflected image;
the joint loss function of the generating network is L (theta) =w 1 L feat (θ)+w 2 L adv (θ)+w 3 L excl (θ); wherein L (θ) is the joint loss function, w 1 、w 2 And w 3 And the reconstructed loss function, the antagonism loss function and the coefficient corresponding to the separation loss function of the supercolumn feature space are respectively.
Specifically, the reconstruction loss function of the supercolumn feature space, also called Feature reconstruction loss, is used to measure the distance between the predictive background image generated by the generating network and the background image T in the supercolumn space. Typically, the distance of the predicted image from the target image is calculated at the selected VGG-19 network layer. The expression of the reconstruction loss function of the supercolumn feature space is
Wherein L is feat (θ) is the reconstruction loss function of the supercolumn feature space, I, T and f T (I; θ) are respectively a reflection image, a background image and a predicted background image, lambda l For the impact weight of the layer i convolution layer, omega is the set of image data for the training, I.I 1 Representing the vector taking a 1-norm, i.e. the sum of the absolute values of the vector elements, of the convolutional result of the neural network Φ l (x) Representing the convolution operation of the first layer convolution layer of the VGG-19 network, θ represents the generation network parameters.
The function of countering losses, also called the universal loss, is to make the generated predictive background map f T (I; θ) is more different from the reflected image I. The expression of the counterloss function is
Wherein L is adv And (theta) is an anti-loss function, D (I, x) represents the probability that x is a background image corresponding to the reflective image I, and the probability is obtained by the output of the discrimination network.
The separation loss function, also called the emission loss, is designed according to the rule that the reflection image is found at the image edge by observation. The two layers of the retroreflective image are viewed to find that the edges of the background layer and the reflective layer generally do not coincide. The edges in the retroreflective image I can only be created by the background image or the reflective image, and not by the superposition of the two. Therefore, the invention proposes to minimize the gradient spatial correlation of the reflection layer and the background layer predicted by the generation network, and consider calculating normalized gradient information on a plurality of resolutions of the two layers, so as to calculate the image edge correlation as a separation loss function.
The expression of the separation loss function is
Wherein L is excl (θ) is the separation loss function, +.>
λ T And lambda is R First and second normalized parameters, respectively, |·|| F Is Luo Beini Usneius (Frobenius) norm, by which we mean element multiplication, n is image downsamplingParameters, N is more than or equal to 1 and less than or equal to N, wherein N is the maximum value of the downsampling parameters of the image; f (f) R (I; θ) are respectively predicted reflectograms, < >>
For predicting the modulus of the gradient of the background image +.>
A module for predicting a gradient of the reflected image; f (f) T And f R All pass through 2 n-1 Is used for bilinear interpolation downsampling. Preferably, n=3, < >>
Generating a joint loss function of the network as L (θ) =w 1 L feat (θ)+w 2 L adv (θ)+w 3 L excl (θ); wherein L (θ) is a joint loss function, w 1 、w 2 And w 3 The coefficients corresponding to the reconstruction loss function, the antagonism loss function and the separation loss function of the supercolumn feature space are used for balancing the influence capacity of each loss function on the generation network. Preferably, w 1 ＝20，w 2 ＝100，w 3 ＝1。
As an improvement of the above scheme, the authentication loss function of the authentication network is L disc (θ)＝log D(I；f T (I; θ)) -log D (I, T); wherein L is disc And (θ) is the discrimination loss function.
The authentication network is constructed by first merging the predicted background image and the background image input to the authentication network to obtain a layered input image by merging channels. If the predicted background image and the background image are both c×w×h in size, where C is the number of channels of the image, and W and H are the width and height of the image, respectively, then the dimensions of the resulting layered image after channel merging will be 2c×w×h. The resulting stacked input image is then passed through a plurality of cascaded downsampling units. The processing of these downsampling units causes the input stacked image to be a progressively smaller feature map. These downsampling units consist of a convolution layer with a convolution step of 2, a batch normalization layer and a nonlinear activation layer in series. The convolution layer with the step length of 2 reduces the size of the input image to one half of the original size, which plays a role in downsampling; the batch normalization layer performs the functions of stabilizing training and accelerating model convergence by normalizing the input data of one batch to normalized data with the mean value of 0 and the variance of 1; while the addition of the nonlinear activation layer prevents the model from being degenerated into a simple linear model, improves the descriptive capacity of the model, preferably a leak Re LU unit with a slope of 0.02 is used as the nonlinear activation unit. After passing through a plurality of downsampling units, a characteristic diagram with dimensions of C multiplied by W multiplied by H is obtained; wherein C is the number of channels of the feature map, W is less than or equal to 4, the width of the feature map, and H is less than or equal to 4, the height of the feature map. After the feature map is obtained, it is mapped to a scalar ranging from 0 to 1 to indicate the probability that the source of the input image is the dataset, a convolution layer is used that contains a convolution kernel of dimension C x W x H. After processing through this one convolution layer, a scalar value will be obtained, which will be input into the Sigmoid function, resulting in a probability value in the range 0 to 1. Referring to fig. 4, a schematic diagram of an authentication network according to this embodiment of the present invention is shown.
Specifically, the authentication loss function of the authentication network is L disc (θ)＝log D(I；f T (I; θ)) -log D (I, T); wherein L is disc (θ) is the discrimination loss function, D (I, x) represents the probability that x is the background image corresponding to the reflective image I, i.e., D (I; f) T (I; θ)) represents a predictive background map f T (I; θ) probability of being derived from the background image in the dataset, D (I, T) represents probability of the background image T being derived from the background image in the dataset.
As an improvement of the above solution, the selecting a plurality of reflective images for the anti-reflective treatment to quantitatively evaluate the anti-reflective effect specifically includes:
selecting a plurality of reflection images to perform anti-reflection processing, and calculating peak signal-to-noise ratio and structural similarity between the predicted background image and the background image generated by the generating network to quantitatively evaluate the anti-reflection effect
Specifically, a plurality of reflection images are selected for reflection removal processing, and peak signal-to-noise ratio and structural similarity between a prediction background image and a background image generated by a generating network are calculated so as to quantitatively evaluate the reflection removal effect. Wherein, peak signal-to-noise ratio, also called PSNR, is abbreviated. Structural similarity, also known as Structural Similarity Index, is abbreviated as SSIM. Referring to fig. 5, there are 4 sets of retroreflective images, background images, predicted background images, and retroreflective contrast effect maps of the retroreflective images provided in this embodiment of the invention. Referring to table 1, there is a quantitative evaluation table corresponding to the evaluation index of the retroreflective effect of fig. 5. As can be seen from fig. 5 and table 1, the method of the present invention has a good effect on single image retroreflective.
Table 1 quantitative evaluation table for image retroreflective effect evaluation index
| Retroreflective/background image | PSNR/SSIM | Background image/predictive background image | PSNR/SSIM |
| First retroreflective image/first background image | 15.93/0.54 | First background image/first predictive background image | 23.85/0.82 |
| Second retroreflective image/second background image | 14.70/0.53 | Second background image/second predictive background image | 25.40/0.87 |
| Third reflective image/third background image | 14.45/0.54 | Third background image/third predictive background image | 23.86/0.83 |
| Fourth retroreflective image/fourth background image | 15.52/0.58 | Fourth background image/fourth predictive background image | 22.74/0.79 |
| Average value of | 15.15/0.55 | Average value of | 23.96/0.83 |
Referring to fig. 6, a schematic structural diagram of a single image light-removing device according to an embodiment of the present invention includes:
the image set acquisition module 11 is used for acquiring a background image and a corresponding reflection image through manual shooting, and obtaining a reflection image according to superposition of the background image and the reflection image;
the feature extraction module 12 is used for inputting the reflective image into a pretrained VGG-19 network to perform super-column feature extraction to obtain a feature set;
the prediction generation module 13 is configured to input the feature set into a preset generation network to obtain a prediction background map and a prediction reflection map; wherein the joint loss function of the generation network comprises a reconstruction loss function, an antagonism loss function and a separation loss function of the supercolumn feature space;
an authentication module 14, configured to input the predicted background image and the background image to a preset authentication network, so as to calculate an authentication loss function of the authentication network;
a training module 15, configured to complete training of the generating network and the identifying network by performing iterative computation for multiple times until the joint loss function and the identifying loss function both converge;
the evaluation module 16 is configured to select a plurality of the reflection images for performing the anti-reflection treatment, so as to quantitatively evaluate the anti-reflection effect.
The single image light-removing device provided by the embodiment of the invention can realize all the processes of the single image light-removing method described in any one of the embodiments, and the functions and the realized technical effects of each module and unit in the device are respectively the same as those of the single image light-removing method described in the embodiment, and are not repeated here.
Referring to fig. 7, a schematic diagram of an apparatus for using a single image light-reflection method according to an embodiment of the present invention includes a processor 10, a memory 20, and a computer program stored in the memory 20 and configured to be executed by the processor 10, where the processor 10 implements the single image light-reflection method according to any one of the embodiments.
By way of example, a computer program may be partitioned into one or more modules/units that are stored in the memory 20 and executed by the processor 10 to perform the present invention. One or more of the modules/units may be a series of computer program instruction segments capable of performing a specified function, the instruction segments being for describing the execution of a computer program in a single image retroreflective method. For example, the computer program may be partitioned into an image set acquisition module, a feature extraction module, a prediction generation module, an authentication module, a training module, and an evaluation module, each module functioning specifically as follows:
the image set acquisition module 11 is used for acquiring a background image and a corresponding reflection image through manual shooting, and obtaining a reflection image according to superposition of the background image and the reflection image;
the feature extraction module 12 is used for inputting the reflective image into a pretrained VGG-19 network to perform super-column feature extraction to obtain a feature set;
the prediction generation module 13 is configured to input the feature set into a preset generation network to obtain a prediction background map and a prediction reflection map; wherein the joint loss function of the generation network comprises a reconstruction loss function, an antagonism loss function and a separation loss function of the supercolumn feature space;
an authentication module 14, configured to input the predicted background image and the background image to a preset authentication network, so as to calculate an authentication loss function of the authentication network;
a training module 15, configured to complete training of the generating network and the identifying network by performing iterative computation for multiple times until the joint loss function and the identifying loss function both converge;
the evaluation module 16 is configured to select a plurality of the reflection images for performing the anti-reflection treatment, so as to quantitatively evaluate the anti-reflection effect.
The device using the method for removing the reflection of the single image can be a computing device such as a desktop computer, a notebook computer, a palm computer, a cloud server and the like. The means for using the single image retroreflective method may include, but is not limited to, a processor, a memory. It will be appreciated by those skilled in the art that the schematic diagram 7 is merely an example of an apparatus using a single image retroreflective method, and is not limited to the apparatus using a single image retroreflective method, and may include more or less components than illustrated, or may combine some components, or different components, e.g., the apparatus using a single image retroreflective method may further include an input-output device, a network access device, a bus, etc.
The processor 10 may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. The general purpose processor may be a microprocessor or the processor 10 may be any conventional processor or the like, and the processor 10 is a control center of the apparatus using the single image retroreflective method, and connects the respective parts of the entire apparatus using the single image retroreflective method using various interfaces and lines.
The memory 20 may be used to store the computer program and/or module, and the processor 10 implements various functions of the apparatus using the single image retro-reflective method by running or executing the computer program and/or module stored in the memory 20 and invoking data stored in the memory 20. The memory 20 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, application programs required for at least one function, and the like; the storage data area may store data created according to program use, or the like. In addition, the memory 20 may include high-speed random access memory, and may also include nonvolatile memory, such as a hard disk, memory, plug-in hard disk, smart Media Card (SMC), secure Digital (SD) Card, flash Card (Flash Card), at least one disk storage device, flash memory device, or other volatile solid state storage device.
Wherein the means integrated module of the method for using single image anti-reflection may be stored in a computer readable storage medium if implemented in the form of a software functional unit and sold or used as a separate product. Based on such understanding, the present invention may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the steps of each method embodiment may be implemented. The computer program comprises computer program code, and the computer program code can be in a source code form, an object code form, an executable file or some intermediate form and the like. The computer readable medium may include: any entity or device capable of carrying computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the content of the computer readable medium can be appropriately increased or decreased according to the requirements of the jurisdiction's jurisdiction and the patent practice, for example, in some jurisdictions, the computer readable medium does not include electrical carrier signals and telecommunication signals according to the jurisdiction and the patent practice.
The embodiment of the invention also provides a computer readable storage medium, which comprises a stored computer program, wherein the computer program is used for controlling equipment where the computer readable storage medium is located to execute the method for single image antireflection according to any embodiment.
In summary, the method, the device and the storage medium for removing reflection of a single image provided by the embodiment of the invention consider a reflection separation task as a separation and evaluation task of a layer, a convolution layer of a generating network uses cavity convolution to increase a visual field without losing detail characteristics, and meanwhile, a loss function of the convolution layer fully considers high-level characteristics, image gradient characteristics and the difference between a predicted background image and a reflection image of the image; the high-level features are obtained through VGG-19 network, and can abstract visual perception of the data set; the identification network designs a loss function based on the difference between the predicted background image and the input reflective image, so that the predicted background image and the background image are more similar; finally, through verification, the invention has good anti-reflection effect on the image, and particularly has satisfactory anti-reflection effect on the high-contrast reflection image.
While the foregoing is directed to the preferred embodiments of the present invention, it will be appreciated by those skilled in the art that changes and modifications may be made without departing from the principles of the invention, such changes and modifications are also intended to be within the scope of the invention.
Claims (7)
1. A method for single image retroreflective comprising the steps of:
obtaining a background image and a corresponding reflection image through manual shooting, and obtaining a reflection image according to superposition of the background image and the reflection image;
inputting the reflective image into a pretrained VGG-19 network for super-column feature extraction to obtain a feature set;
inputting the feature set into a preset generation network to obtain a prediction background image and a prediction reflection image; wherein the joint loss function of the generation network comprises a reconstruction loss function, an antagonism loss function and a separation loss function of the supercolumn feature space;
inputting the predicted background image and the background image into a preset authentication network to calculate an authentication loss function of the authentication network;
performing repeated iterative computation until the joint loss function and the authentication loss function are converged, and completing the training of the generation network and the authentication network;
selecting a plurality of the reflection images to perform anti-reflection treatment so as to quantitatively evaluate the anti-reflection effect;
the joint loss function of the generation network comprises a reconstruction loss function, an antagonism loss function and a separation loss function of the supercolumn feature space, and specifically comprises the following steps:
the expression of the reconstruction loss function of the supercolumn feature space is that
Wherein L is feat (θ) is the reconstruction loss function of the supercolumn feature space, I, T and f T (I; θ) the retroreflective image, the background image and the predictive background image, λ, respectively l For the impact weight of the layer i convolution layer, omega is the set of image data for the training, I.I 1 Representing the vector taking a 1-norm, i.e. the sum of the absolute values of the vector elements, of the convolutional result of the neural network Φ l (x) Watch (watch)Convolution operation of a first layer convolution layer of the VGG-19 network is shown, and θ represents generation network parameters;
the expression of the counterdamage function is
Wherein L is adv (θ) is the counterdamage function, D (I, x) represents the probability that x is the background image corresponding to the retroreflective image I, derived from the output of the authentication network;
the expression of the separation loss function is
Wherein L is excl (θ) is the separation loss function, < ->
λ T And lambda is R First and second normalized parameters, respectively, |·|| F Is Luo Beini Usnea norm, by which is expressed the multiplication of elements, N is the image downsampling parameter, N is greater than or equal to 1 and less than or equal to N, and N is the maximum value of the image downsampling parameter; f (f) R (I; θ) are the predicted reflectances, respectively,>
for predicting the modulus of the gradient of the background image +.>
A module for predicting a gradient of the reflected image;
the joint loss function of the generating network is L (theta) =w 1 L feat (θ)+w 2 L adv (θ)+w 3 L excl (θ); wherein L (θ) is the joint loss function, w 1 、w 2 And w 3 The reconstructed loss function, the counterloss function and the coefficient corresponding to the separation loss function of the supercolumn feature space are respectively obtained;
and obtaining a reflective image according to the superposition of the background image and the reflective image, wherein the method specifically comprises the following steps: acquiring a first gray value of the background image; acquiring a second gray value of the reflected image; weighting the first gray value and the second gray value to obtain the reflective image;
the authentication loss function of the authentication network is L disc (θ)＝logD(I；f T (I; θ)) -log d (I, T); wherein L is disc And (θ) is the discrimination loss function.
2. The method of single image retroreflective of claim 1, wherein the convolutional layer of the VGG-19 network comprises conv1_2, conv2_2, conv3_2, conv4_2, and conv5_2.
3. The method of single image retroreflective according to claim 1, wherein the generating network comprises an input layer having a convolution kernel of 1 x 1 and 8 hole convolution layers having a convolution kernel of 3 x 3; the final layer of cavity convolution layer generates two three-channel RGB images by linear transformation.
4. The method for single image retroreflective sheeting of claim 1 wherein the selecting a plurality of retroreflective images to be retroreflective to quantitatively evaluate the retroreflective effects comprises:
and selecting a plurality of reflection images to perform anti-reflection processing, and calculating peak signal-to-noise ratio and structural similarity between the predicted background image and the background image generated by the generating network so as to quantitatively evaluate the anti-reflection effect.
5. A single image retroreflective apparatus comprising:
the image set acquisition module is used for acquiring a background image and a corresponding reflection image through manual shooting, and obtaining a reflection image according to superposition of the background image and the reflection image;
the feature extraction module is used for inputting the reflective image into a pretrained VGG-19 network to perform super-column feature extraction to obtain a feature set;
the prediction generation module is used for inputting the feature set into a preset generation network to obtain a prediction background image and a prediction reflection image; wherein the joint loss function of the generation network comprises a reconstruction loss function, an antagonism loss function and a separation loss function of the supercolumn feature space;
the identification module is used for inputting the prediction background image and the background image into a preset identification network so as to calculate an identification loss function of the identification network;
the training module is used for completing the training of the generating network and the identifying network through repeated iterative computation until the joint loss function and the identifying loss function are converged;
the evaluation module is used for selecting a plurality of the reflection images to carry out anti-reflection treatment so as to quantitatively evaluate the anti-reflection effect;
the joint loss function of the generation network comprises a reconstruction loss function, an antagonism loss function and a separation loss function of the supercolumn feature space, and specifically comprises the following steps:
the expression of the reconstruction loss function of the supercolumn feature space is that
Wherein L is feat (θ) is the reconstruction loss function of the supercolumn feature space, I, T and f T (I; θ) the retroreflective image, the background image and the predictive background image, λ, respectively l For the impact weight of the layer i convolution layer, omega is the set of image data for the training, I.I 1 Representing the vector taking a 1-norm, i.e. the sum of the absolute values of the vector elements, of the convolutional result of the neural network Φ l (x) A convolution operation of a first layer convolution layer of the VGG-19 network is represented, and θ represents a generated network parameter;
the expression of the counterdamage function is
Wherein L is adv (θ) is the counterdamage function, D (I, x) represents x isThe probability of the background image corresponding to the reflective image I is obtained by the output of the identification network; />
The expression of the separation loss function is
Wherein L is excl (θ) is the separation loss function, < ->
λ T And lambda is R First and second normalized parameters, respectively, |·|| F Is Luo Beini Usnea norm, by which is expressed the multiplication of elements, N is the image downsampling parameter, N is greater than or equal to 1 and less than or equal to N, and N is the maximum value of the image downsampling parameter; f (f) R (I; θ) are the predicted reflectances, respectively,>
for predicting the modulus of the gradient of the background image +.>
A module for predicting a gradient of the reflected image;
the joint loss function of the generating network is L (theta) =w 1 L feat (θ)+w 2 L adv (θ)+w 3 L excl (θ); wherein L (θ) is the joint loss function, w 1 、w 2 And w 3 The reconstructed loss function, the counterloss function and the coefficient corresponding to the separation loss function of the supercolumn feature space are respectively obtained;
and obtaining a reflective image according to the superposition of the background image and the reflective image, wherein the method specifically comprises the following steps: acquiring a first gray value of the background image; acquiring a second gray value of the reflected image; weighting the first gray value and the second gray value to obtain the reflective image;
the authentication loss function of the authentication network is L disc (θ)＝logD(I；f T (I；θ)) -log (I, T); wherein L is disc And (θ) is the discrimination loss function.
6. An apparatus for using a single image retroreflective method comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, the processor implementing the single image retroreflective method of any of claims 1-4 when the computer program is executed.
7. A computer readable storage medium, characterized in that the computer readable storage medium comprises a stored computer program, wherein the computer program, when run, controls a device in which the computer readable storage medium is located to perform the method of single image retroreflective according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010193974.1A CN111507910B (en) | 2020-03-18 | 2020-03-18 | Single image antireflection method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010193974.1A CN111507910B (en) | 2020-03-18 | 2020-03-18 | Single image antireflection method, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111507910A CN111507910A (en) | 2020-08-07 |
| CN111507910B true CN111507910B (en) | 2023-06-06 |
Family
ID=71864034
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010193974.1A Active CN111507910B (en) | 2020-03-18 | 2020-03-18 | Single image antireflection method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111507910B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112085671A (en) * | 2020-08-19 | 2020-12-15 | 北京影谱科技股份有限公司 | Background reconstruction method and device, computing equipment and storage medium |
| CN112198483A (en) * | 2020-09-28 | 2021-01-08 | 上海眼控科技股份有限公司 | Data processing method, device and equipment for satellite inversion radar and storage medium |
| CN112634161B (en) * | 2020-12-25 | 2022-11-08 | 南京信息工程大学滨江学院 | Reflected light removing method based on two-stage reflected light eliminating network and pixel loss |
| CN112907466A (en) * | 2021-02-01 | 2021-06-04 | 南京航空航天大学 | Nondestructive testing reflection interference removing method and device and computer readable storage medium |
| CN112802076A (en) * | 2021-03-23 | 2021-05-14 | 苏州科达科技股份有限公司 | Reflection image generation model and training method of reflection removal model |
| WO2022222080A1 (en) * | 2021-04-21 | 2022-10-27 | 浙江大学 | Single-image reflecting layer removing method based on position perception |
| CN114926705A (en) * | 2022-05-12 | 2022-08-19 | 网易（杭州）网络有限公司 | Cover design model training method, medium, device and computing equipment |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109993124B (en) * | 2019-04-03 | 2023-07-14 | 深圳华付技术股份有限公司 | Living body detection method and device based on video reflection and computer equipment |
| CN110188776A (en) * | 2019-05-30 | 2019-08-30 | 京东方科技集团股份有限公司 | Image processing method and device, the training method of neural network, storage medium |
| CN110473154B (en) * | 2019-07-31 | 2021-11-16 | 西安理工大学 | Image denoising method based on generation countermeasure network |
| CN110675336A (en) * | 2019-08-29 | 2020-01-10 | 苏州千视通视觉科技股份有限公司 | Low-illumination image enhancement method and device |
| CN110827217B (en) * | 2019-10-30 | 2022-07-12 | 维沃移动通信有限公司 | Image processing method, electronic device, and computer-readable storage medium |
-
2020
- 2020-03-18 CN CN202010193974.1A patent/CN111507910B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN111507910A (en) | 2020-08-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111507910B (en) | Single image antireflection method, device and storage medium | |
| CN109949255B (en) | Image reconstruction method and device | |
| US11093832B2 (en) | Pruning redundant neurons and kernels of deep convolutional neural networks | |
| US10891537B2 (en) | Convolutional neural network-based image processing method and image processing apparatus | |
| US20230153615A1 (en) | Neural network distillation method and apparatus | |
| US9152888B2 (en) | System and method for automated object detection in an image | |
| Zheng | Gradient descent algorithms for quantile regression with smooth approximation | |
| US20130343619A1 (en) | Density estimation and/or manifold learning | |
| EP4006776A1 (en) | Image classification method and apparatus | |
| CN112529146B (en) | Neural network model training method and device | |
| CN111899203B (en) | Real image generation method based on label graph under unsupervised training and storage medium | |
| CN112561028A (en) | Method for training neural network model, and method and device for data processing | |
| Yu et al. | Toward faster and simpler matrix normalization via rank-1 update | |
| CN115131218A (en) | Image processing method, image processing device, computer readable medium and electronic equipment | |
| Pichel et al. | A new approach for sparse matrix classification based on deep learning techniques | |
| EP4033446A1 (en) | Method and apparatus for image restoration | |
| CN114648787A (en) | Face image processing method and related equipment | |
| CN111667495A (en) | Image scene analysis method and device | |
| Mao et al. | Convolutional feature frequency adaptive fusion object detection network | |
| US20230073175A1 (en) | Method and system for processing image based on weighted multiple kernels | |
| CN112070853A (en) | Image generation method and device | |
| CN116152542A (en) | Training method, device, equipment and storage medium for image classification model | |
| CN111914996A (en) | Method for extracting data features and related device | |
| US20240161245A1 (en) | Image optimization | |
| CN116403064B (en) | Picture processing method, system, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |