CN112396674A - Rapid event image filling method and system based on lightweight generation countermeasure network - Google Patents

Rapid event image filling method and system based on lightweight generation countermeasure network Download PDF

Info

Publication number
CN112396674A
CN112396674A CN202011133015.7A CN202011133015A CN112396674A CN 112396674 A CN112396674 A CN 112396674A CN 202011133015 A CN202011133015 A CN 202011133015A CN 112396674 A CN112396674 A CN 112396674A
Authority
CN
China
Prior art keywords
event
sequence
loss
discriminator
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011133015.7A
Other languages
Chinese (zh)
Other versions
CN112396674B (en
Inventor
刘盛
程豪豪
黄圣跃
金坤
叶焕然
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang University of Technology ZJUT
Original Assignee
Zhejiang University of Technology ZJUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang University of Technology ZJUT filed Critical Zhejiang University of Technology ZJUT
Priority to CN202011133015.7A priority Critical patent/CN112396674B/en
Publication of CN112396674A publication Critical patent/CN112396674A/en
Application granted granted Critical
Publication of CN112396674B publication Critical patent/CN112396674B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/40—Filling a planar surface by adding surface attributes, e.g. colour or texture
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a rapid event image filling method and a rapid event image filling system based on a lightweight generation countermeasure network, wherein the rapid event image filling method based on the lightweight generation countermeasure network comprises the following steps: constructing a lightweight generation countermeasure network; acquiring training data, wherein the training data comprises a plurality of pairs of matched loss event images and non-loss event images; optimizing the lightweight generation countermeasure network by using the training data to obtain optimal network parameters; and acquiring a loss event image to be filled, inputting the loss event image to the light weight generation countermeasure network based on the optimal network parameters, and acquiring a filling event image output by the light weight generation countermeasure network. The rapid event image filling method and system based on the lightweight generation countermeasure network fully utilize the sparse characteristic of the event image and improve the authenticity of an image filling structure and the fineness of the structure.

Description

Rapid event image filling method and system based on lightweight generation countermeasure network
Technical Field
The application belongs to the technical field of image processing, and particularly relates to a rapid event image filling method and system based on a lightweight generation countermeasure network.
Background
Event-based cameras (Event-based cameras, or simply Event cameras, abbreviated EB., sometimes also referred to as DVS (Dynamic Vision Sensor)) are a new class of sensors. Unlike a traditional camera which takes a complete image, an event camera takes an "event" which can be simply understood as "change of pixel brightness", that is, the event camera outputs the change of pixel brightness.
Currently, event cameras are capable of generating sparse event streams and capturing high-speed motion information, however, as temporal resolution increases, spatial resolution decreases dramatically. Although generating an antagonistic network has a significant effect on conventional image inpainting, the direct use of it for event padding can overwhelm the fast response characteristics of event cameras, and the sparsity of event streams is not fully exploited.
Disclosure of Invention
The application aims to provide a rapid event image filling method and system based on a lightweight generation countermeasure network, which make full use of the sparse characteristic of an event image and improve the authenticity of an image filling structure and the fineness of the structure.
In order to achieve the purpose, the technical scheme adopted by the application is as follows:
a rapid event image filling method based on a lightweight-generated countermeasure network comprises the following steps:
constructing a lightweight generation countermeasure network;
acquiring training data, wherein the training data comprises a plurality of pairs of matched loss event images and non-loss event images;
optimizing the lightweight generation countermeasure network by using the training data to obtain optimal network parameters;
obtaining a loss event image to be filled, inputting the loss event image to a light weight generation countermeasure network based on optimal network parameters, and obtaining a filling event image output by the light weight generation countermeasure network;
wherein the lightweight generation countermeasure network includes a generator and a discriminator, the generator including an encoder, a decoder, and two residual blocks connected between the encoder and the decoder, the encoder including three 3D convolutions, the encoder downsampling an image twice, the decoder including three 3D transposed convolutions, the decoder upsampling an image twice; the event frame discriminator is of a PatchGAN structure, the convolution in the event frame discriminator is a 2D convolution, the event sequence discriminator is of a PatchGAN structure, and the convolution in the event sequence discriminator is a 3D convolution.
Several alternatives are provided below, but not as an additional limitation to the above general solution, but merely as a further addition or preference, each alternative being combinable individually for the above general solution or among several alternatives without technical or logical contradictions.
Preferably, the convolution in the residual block uses a dilation convolution with a dilation factor of 2.
Preferably, the optimizing the lightweight generative countermeasure network by using the training data to obtain optimal network parameters includes:
taking P pairs of matched loss event images and non-loss event images based on training data;
inputting P loss event images into the generator as a loss event image sequence to obtain a filling event image sequence output by the generator, wherein each filling event image in the filling event image sequence corresponds to each loss event image in the loss event image sequence as input;
taking P pieces of non-loss event images as a non-loss event image sequence, according to the non-loss event image sequence and the filling event image sequence, firstly performing back propagation of a discriminator based on a total loss function of the discriminator, and then performing back propagation of a generator based on the total loss function of the generator;
and repeating the training until the network parameters optimal for the lightweight generation countermeasure network are obtained.
Preferably, the total loss function of the discriminator includes:
Figure BDA0002735767210000021
wherein L isDAs a function of the total loss of the arbiter,
Figure BDA0002735767210000022
as a loss function of the event sequence discriminator,
Figure BDA0002735767210000023
as a loss function of the event frame discriminator,
Figure BDA0002735767210000024
is a weight parameter of the event sequence discriminator,
Figure BDA0002735767210000025
is the weight parameter of the event frame discriminator;
loss function of the event sequence discriminator
Figure BDA0002735767210000026
Loss function of sum event frame discriminator
Figure BDA0002735767210000027
The following were used:
Figure BDA0002735767210000028
Figure BDA0002735767210000029
wherein, IgtRepresenting a sequence of lossless event images, Pdata(Igt) Indicates no lossDistribution of the sequence of event images, E [. sup. ]]Indicating the expected value, logD, of the distribution functions(Igt) Representing the probability, logD, that the event sequence discriminator discriminated as a non-lost event imagef(Igt) Representing the probability of the event frame discriminator discriminating as an unrepaired event image, IinRepresenting a sequence of loss event images, Pdata(Iin) Represents the distribution of the loss event image sequence, log (1-D)s(G(Iin) Log (1-D)) represents the probability that the event sequence discriminator discriminated as a padded event image output by the generatorf(G(Iin) ) represents the probability that the event frame discriminator discriminated the padded event image output by the generator.
Preferably, the total loss function of the generator comprises:
LG＝λ1L1+λpLperc+λsLstyle+λgLg
wherein L isGTo the total loss function of the generator, L1Is L1Loss function, λ1Is L1Weight parameter of the loss function, LpercAs a function of perceptual loss, λpAs weight parameter of the perceptual loss function, LstyleAs a function of the loss of style, λsAs a weight parameter of the style loss function, LgTo generate a generator opposition loss function, λgA weight parameter for the generator counter loss function;
the generator fighting loss function LgThe following were used:
Figure BDA0002735767210000031
wherein G denotes a generator, D denotes a discriminator, IinRepresenting a sequence of loss event images, Pdata(Iin) Representing the distribution of the loss event image sequence, E [. + ]]Expected value, G (I), representing distribution functionin) Sequence of padded event images, logD, representing the output of the generators(G(Iin) Means that the event sequence discriminator willFilling up the probability, logD, of an event image being discriminated as an unreleased event imagef(G(Iin) Represents the probability that the event frame discriminator discriminated the shim event image as an unreduced event image;
said L1Loss function L1The following were used:
Figure BDA0002735767210000032
wherein, IgtRepresenting a sequence of lossless event images, IpredA sequence of shim event images representing the generator output;
the perceptual loss function LpercThe following were used:
Figure BDA0002735767210000033
wherein phi isjIs the activation map of the jth layer of the pre-trained VGG-19 network, phij(Igt) Representing the corresponding activation graph sequence obtained after the non-loss event image sequence is input into the j layer of the VGG-19 networkj(Ipred) Representing a corresponding activation graph sequence obtained after the filling event image sequence is input into a j layer of a VGG-19 network; n is a radical ofjRepresenting the number of characteristic channels of a j-th network in the VGG-19 network;
the style loss function LstyleThe following were used:
Figure BDA0002735767210000041
wherein,
Figure BDA0002735767210000042
is based on an activation map phijC of constructionj×CjThe matrix of the Gram is a matrix of,
Figure BDA0002735767210000043
representing activation corresponding to a sequence of lossless event imagesA plurality of Gram matrices constructed by the graph sequence,
Figure BDA0002735767210000044
a plurality of Gram matrices constructed from the sequence of activation maps corresponding to the sequence of shim event images is represented.
The application also provides a rapid event image filling system based on a lightweight generation countermeasure network, which comprises:
a first module for constructing a lightweight generative confrontation network;
a second module for obtaining training data, the training data comprising a plurality of pairs of matched loss event images and non-loss event images;
a third module, configured to optimize the lightweight generation countermeasure network using the training data to obtain an optimal network parameter;
the fourth module is used for acquiring a loss event image to be filled, inputting the loss event image to the light weight generation countermeasure network based on the optimal network parameters, and obtaining a filling event image output by the light weight generation countermeasure network;
wherein the lightweight generation countermeasure network includes a generator and a discriminator, the generator including an encoder, a decoder, and two residual blocks connected between the encoder and the decoder, the encoder including three 3D convolutions, the encoder downsampling an image twice, the decoder including three 3D transposed convolutions, the decoder upsampling an image twice; the event frame discriminator is of a PatchGAN structure, the convolution in the event frame discriminator is a 2D convolution, the event sequence discriminator is of a PatchGAN structure, and the convolution in the event sequence discriminator is a 3D convolution.
Preferably, the convolution in the residual block uses a dilation convolution with a dilation factor of 2.
Preferably, the third module optimizes the lightweight generation countermeasure network by using the training data to obtain optimal network parameters, and performs the following operations:
taking P pairs of matched loss event images and non-loss event images based on training data;
inputting P loss event images into the generator as a loss event image sequence to obtain a filling event image sequence output by the generator, wherein each filling event image in the filling event image sequence corresponds to each loss event image in the loss event image sequence as input;
taking P pieces of non-loss event images as a non-loss event image sequence, according to the non-loss event image sequence and the filling event image sequence, firstly performing back propagation of a discriminator based on a total loss function of the discriminator, and then performing back propagation of a generator based on the total loss function of the generator;
and repeating the training until the network parameters optimal for the lightweight generation countermeasure network are obtained.
Preferably, the total loss function of the discriminator includes:
Figure BDA0002735767210000051
wherein L isDAs a function of the total loss of the arbiter,
Figure BDA0002735767210000052
as a loss function of the event sequence discriminator,
Figure BDA0002735767210000053
as a loss function of the event frame discriminator,
Figure BDA0002735767210000054
is a weight parameter of the event sequence discriminator,
Figure BDA0002735767210000055
is the weight parameter of the event frame discriminator;
loss function of the event sequence discriminator
Figure BDA0002735767210000056
Loss function of sum event frame discriminator
Figure BDA0002735767210000057
The following were used:
Figure BDA0002735767210000058
Figure BDA0002735767210000059
wherein, IgtRepresenting a sequence of lossless event images, Pdata(Igt) Representing the distribution of the sequence of unreleased event images, E [. sup. ]]Indicating the expected value, logD, of the distribution functions(Igt) Representing the probability, logD, that the event sequence discriminator discriminated as a non-lost event imagef(Igt) Representing the probability of the event frame discriminator discriminating as an unrepaired event image, IinRepresenting a sequence of loss event images, Pdata(Iin) Represents the distribution of the loss event image sequence, log (1-D)s(G(Iin) Log (1-D)) represents the probability that the event sequence discriminator discriminated as a padded event image output by the generatorf(G(Iin) ) represents the probability that the event frame discriminator discriminated the padded event image output by the generator.
Preferably, the total loss function of the generator comprises:
LG＝λ1L1+λpLperc+λsLstyle+λgLg
wherein L isGTo the total loss function of the generator, L1Is L1Loss function, λ1Is L1Weight parameter of the loss function, LpercAs a function of perceptual loss, λpAs weight parameter of the perceptual loss function, LstyleAs a function of the loss of style, λsWeight parameter as a function of style loss，LgTo generate a generator opposition loss function, λgA weight parameter for the generator counter loss function;
the generator fighting loss function LgThe following were used:
Figure BDA00027357672100000510
wherein G denotes a generator, D denotes a discriminator, IinRepresenting a sequence of loss event images, Pdata(Iin) Representing the distribution of the loss event image sequence, E [. + ]]Expected value, G (I), representing distribution functionin) Sequence of padded event images, logD, representing the output of the generators(G(Iin) Representing the probability, logD, that the event sequence discriminator discriminated the padded event image as an unreduced event imagef(G(Iin) Represents the probability that the event frame discriminator discriminated the shim event image as an unreduced event image;
said L1Loss function L1The following were used:
Figure BDA0002735767210000061
wherein, IgtRepresenting a sequence of lossless event images, IpredA sequence of shim event images representing the generator output;
the perceptual loss function LpercThe following were used:
Figure BDA0002735767210000062
wherein phi isjIs the activation map of the jth layer of the pre-trained VGG-19 network, phij(Igt) Representing the corresponding activation graph sequence obtained after the non-loss event image sequence is input into the j layer of the VGG-19 networkj(Ipred) Representing a corresponding activation graph sequence obtained after the filling event image sequence is input into a j layer of a VGG-19 network; n is a radical ofjRepresenting the number of characteristic channels of a j-th network in the VGG-19 network;
the style loss function LstyleThe following were used:
Figure BDA0002735767210000063
wherein,
Figure BDA0002735767210000064
is based on an activation map phijC of constructionj×CjThe matrix of the Gram is a matrix of,
Figure BDA0002735767210000065
representing a plurality of Gram matrices constructed from a sequence of activation maps corresponding to a sequence of non-lost event images,
Figure BDA0002735767210000066
a plurality of Gram matrices constructed from the sequence of activation maps corresponding to the sequence of shim event images is represented.
In order to overcome the defects of large size, parameter redundancy and low inference speed of a traditional image restoration model and the problem of time consistency reduction of results caused by 2D convolution, the rapid event image filling method and system based on the lightweight generation countermeasure network, which are provided by the application, construct a shallow 3D generator to fully utilize the sparse characteristic of an event image, and simultaneously, in order to ensure the authenticity of event filling results and the fineness of the structure, add L into original countermeasure loss1Loss, perception loss, and grid loss. And finally, an event sequence discriminator is provided, and the time consistency of the result is improved.
Drawings
FIG. 1 is a flow chart of a method for rapid event image fill based on lightweight generative confrontation networks according to the present application;
FIG. 2 is a schematic diagram of a lightweight generative countermeasure network as constructed herein;
FIG. 3 is a diagram illustrating an example of a loss event image according to the present application;
fig. 4 is a padded event image output by the countermeasure network for fig. 3, lightweight according to the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the present application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.
In one embodiment, a rapid event image filling method based on a lightweight generation countermeasure network is provided, and is used in the field of image processing, in particular to filling of event camera images with impaired spatial resolution.
As shown in fig. 1, a method for filling up a rapid event image of a countermeasure network based on lightweight generation comprises the following steps:
and step S1, constructing a lightweight generation countermeasure network.
Since the direct application of the generation of the countermeasure network to fill up the event image will overwhelm the characteristic of fast response of the event camera, and the sparsity of the event stream cannot be fully utilized, the embodiment constructs a lightweight generation countermeasure network to facilitate the application of a high dynamic response scene.
As shown in fig. 2, the lightweight generative countermeasure network constructed by the present embodiment includes a generator and an arbiter. Wherein the generator comprises an encoder and a decoder, and two residual blocks connected between the encoder and the decoder.
Conventional image inpainting networks are too deep for the event image, which results in too slow an inference speed. Therefore, the encoder of the invention comprises three 3D convolutions, only two downsampling are carried out on the image, the characteristic channel is expanded to be 2 times of the front layer after each downsampling, the corresponding decoder comprises three 3D transposition convolutions, only 2 upsampling are carried out on the image, and the characteristic channel is reduced to be 2 times of the front layer after each upsampling.
Meanwhile, due to the sparsity of the event image, the shallow network does not cause the generation quality of the event image to be too low, so that only two residual blocks are used between the encoder and the decoder. In order to increase the receptive field, the regular convolution in the residual layer is replaced by the expansion convolution with the expansion factor of 2, and meanwhile, in order to improve the generalization capability, more space-time information is reserved, and the 2D expansion convolution is replaced by the 3D expansion convolution. The present embodiment uses instance normalization at all layers of the network.
In order to improve the time consistency and quality of the filling event images, the classifiers constructed in the embodiment include an event frame classifier and an event sequence classifier. The event frame arbiter is of a PatchGAN structure, and the convolution in the event frame arbiter is a 2D convolution, the event sequence arbiter is of a PatchGAN structure, and the convolution in the event sequence arbiter is a 3D convolution.
The discriminator uses a 70 × 70PatchGAN structure for discriminating whether or not an overlapping image block of size 70 × 70 is authentic. The event frame discriminator uses a 2D convolution, the purpose of which is to focus on the spatial feature consistency of the event frames. Although the use of 3D convolution in the generator can retain more spatio-temporal information, the blurring of the image edges can also occur, and an event sequence discriminator is introduced for this purpose, namely, the 3D convolution is used for improving the quality of the generated image, and the event sequence discriminator focuses on the time dependence and the correlation of pixel change. Finally, to enhance training stability, spectral normalization is applied to the discriminators.
The present embodiment uses conventional convolution, dilation convolution and transpose convolution. Taking 5 × 5 input feature map as an example, a convolution kernel with the size of 3 × 3 is adopted, the step size is 1, a conventional convolution outputs a feature map with the size of 3 × 3, and then the feature map is taken as an input of an expansion convolution, when the number of intervals of convolution kernel points is 1 (0 is filled between the convolution kernel points, the kernel size is 5 × 5, the parameter quantity is unchanged, the receptive field is large), meanwhile, the feature map edge filling number (0 is filled) is set to be 2, the step size is 1, and the size of the feature map output by the expansion convolution is 3 × 3 at this moment; for the transposed convolution, 3 × 3 feature maps can be restored to a size of 5 × 5 by only setting the feature map edge padding number to 2 and performing conventional convolution on other parameters.
And step S2, acquiring training data, wherein the training data comprises a plurality of pairs of matched loss event images and non-loss event images.
The present embodiment mainly fills up images of the event camera, so the present embodiment takes the event camera as an example for explanation, and for the event camera, the output thereof can be regarded as an event { e }iA continuous stream of ∈ N. Each event eiCan be represented using the following form:
ei＝(xi,yi,ti,pi) (1)
wherein (x)i,yi) Representing the spatial position, t, of the pixel generating the eventiTime coordinate, p, representing a change in brightnessiE { -1, 1} represents the positive or negative change in intensity at the pixel that caused the event, i is the event index number.
An event frame F is obtained by adding all events between times t and t + τ at the pixel level during an exposure time interval Δ t-t + ττ(t), therefore the event frame can be represented as:
Figure BDA0002735767210000081
wherein Et,τ＝{ei|ti∈[t,t+τ]}. In this way, an event frame can be represented as a grayscale image of size 1 x w x h that integrates all events occurring within a particular time interval into a single channel. Based on the way of accumulating the event frames, the embodiment generates M when the matched loss event image and the non-loss event image are generated1Accumulating the events into an event frame as a loss event image, and converting M2Accumulating the events into an event frame as an unreleased event image, wherein M2At least M180 times higher to ensure that the validity of the event image is not lost.
For example, in the present embodiment, the loss event image is accumulated by taking 100 events as 1 frame, and the non-loss event image is accumulated by taking 7500 events as 1 frame. Fig. 3 is an accumulated loss event image of 1 frame with 100 events, whereby a low resolution event image with a frame rate of about 2000FPS can be obtained.
And step S3, optimizing the lightweight generation countermeasure network by using training data to obtain the optimal network parameters.
In the training of the lightweight generation countermeasure network, in order to avoid insufficient video memory or low video memory utilization rate, the present embodiment uses a plurality of event frames as a sequence segment input for training.
Specifically, in the present embodiment, P (P >1, for example, 8) pairs of matched loss event images and non-loss event images are taken based on the training data.
And inputting the P loss event images into the generator as a loss event image sequence to obtain a filling event image sequence output by the generator, wherein each filling event image in the filling event image sequence corresponds to each loss event image in the loss event image sequence as input.
Taking P pieces of non-loss event images as a non-loss event image sequence, according to the non-loss event image sequence and the filling event image sequence, firstly performing back propagation of a discriminator based on a total loss function of the discriminator, and then performing back propagation of a generator based on the total loss function of the generator.
And repeating the training until the network parameters optimal for the lightweight generation countermeasure network are obtained.
The discriminators of the embodiment have two types, and in order to strengthen the correlation between the two discriminators, the embodiment combines the training of the two discriminators, and when the discriminators are updated and lost, the losses of the two discriminators are summed and then propagated reversely instead of updating the losses respectively. Thus constructing the overall loss function of the arbiter comprises:
Figure BDA0002735767210000091
wherein L isDAs a function of the total loss of the arbiter,
Figure BDA0002735767210000092
as a loss function of the event sequence discriminator,
Figure BDA0002735767210000093
as a loss function of the event frame discriminator,
Figure BDA0002735767210000094
is a weight parameter of the event sequence discriminator,
Figure BDA0002735767210000095
is the weight parameter of the event frame discriminator. This embodiment is preferred
Figure BDA0002735767210000096
Loss function of the event sequence discriminator
Figure BDA0002735767210000097
Loss function of sum event frame discriminator
Figure BDA0002735767210000098
The following were used:
Figure BDA0002735767210000099
Figure BDA00027357672100000910
wherein, IgtRepresenting a sequence of lossless event images, Pdata(Igt) Representing the distribution of the sequence of unreleased event images, E [. sup. ]]Indicating the expected value, logD, of the distribution functions(Igt) Representing sequence of eventsProbability of discriminator judging as no loss event image, logDf(Igt) Representing the probability of the event frame discriminator discriminating as an unrepaired event image, IinRepresenting a sequence of loss event images, Pdata(Iin) Represents the distribution of the loss event image sequence, log (1-D)s(G(Iin) Log (1-D)) represents the probability that the event sequence discriminator discriminated as a padded event image output by the generatorf(G(Iin) ) represents the probability that the event frame discriminator discriminated the padded event image output by the generator.
In order to ensure the authenticity and quality of the event image sequence to be padded, the embodiment comprehensively considers various losses of the generator, and the total loss function of the generator comprises the following steps:
LG＝λ1L1+λpLperc+λsLstyle+λgLg (6)
wherein L isDAs a function of the total loss of the discriminator, L1Is L1Loss function, λ1Is L1Weight parameter of the loss function, LpercAs a function of perceptual loss, λpAs weight parameter of the perceptual loss function, LstyleAs a function of the loss of style, λsAs a weight parameter of the style loss function, LgTo generate a generator opposition loss function, λgTo generate weight parameters for the counter-loss function of the generator. Preferred λ for this embodiment1＝1，λg＝λp＝0.1，λs＝250。
The lightweight generation countermeasure network fills each image in the input image sequence and outputs the image, and BCELoss (binary cross entropy loss) is used to make the distribution of the filling event image sequence close to that of the real label, so the generator countermeasure loss function L is adoptedgThe following were used:
Figure BDA0002735767210000101
wherein G represents a generator and D represents an arbiterOther device, IinRepresenting a sequence of loss event images, Pdata(Iin) Representing the distribution of the loss event image sequence, E [. + ]]Expected value, G (I), representing distribution functionin) Sequence of padded event images, logD, representing the output of the generators(G(Iin) Representing the probability, logD, that the event sequence discriminator discriminated the padded event image as an unreduced event imagef(G(Iin) Represents the probability that the event frame discriminator discriminated the shim event image as an unreduced event image;
in order to fully utilize the sparsity characteristic of an event image, L is added into an original generator loss function1Loss, L1Loss focusing on pixel level features, L used in this embodiment1Loss function L1The following were used:
Figure BDA0002735767210000102
wherein, IgtRepresenting a sequence of lossless event images, IpredRepresenting the sequence of shim event images output by the generator.
L1The loss function can cause blurring of the result while ensuring the pixel-level characteristics of the generator, so this embodiment introduces perceptual and lattice losses to preserve the image content. Perception loss target image I to be generatedpredNormalized to be closer to the real tag I in VGG subspacegtSaid perceptual loss function LpercThe following were used:
Figure BDA0002735767210000103
wherein phi isjIs the activation map of the jth layer of the pre-trained VGG-19 network, phij(Igt) Representing the corresponding activation graph sequence obtained after the non-loss event image sequence is input into the j layer of the VGG-19 networkj(Ipred) Representing a corresponding activation graph sequence obtained after the filling event image sequence is input into a j layer of a VGG-19 network; n is a radical ofjIndicating the number of characteristic channels of the layer j network in the VGG-19 network.
Unlike perceptual loss, to better recover detailed texture, style loss first applies autocorrelation (Gram matrix) to the features. The style loss may measure the difference between activation map covariances, also calculated using VGG. Given size Cj×Hj×WjThe style loss can be calculated by:
Figure BDA0002735767210000111
wherein,
Figure BDA0002735767210000112
is based on an activation map phijC of constructionj×CjThe matrix of the Gram is a matrix of,
Figure BDA0002735767210000113
representing a plurality of Gram matrices constructed from a sequence of activation maps corresponding to a sequence of non-lost event images,
Figure BDA0002735767210000114
a plurality of Gram matrices constructed from the sequence of activation maps corresponding to the sequence of shim event images is represented.
The embodiment mainly fills up the event image sequence with high time resolution and low spatial resolution or the event image sequence with normal time resolution and damaged spatial resolution. The method comprises the steps of firstly accumulating events under a high time resolution condition to obtain event frames, sending event sequence images into a generator, and outputting filled event image sequences after the event sequence images pass through the generator. And then, the filled event image sequence and the real label are jointly sent into two discriminators, the discriminators discriminate the authenticity and feed back the result to the generator, so that the time consistency and the image quality of the filled event image sequence are ensured.
And step S4, obtaining the loss event image to be filled, inputting the loss event image to the light weight generation countermeasure network based on the optimal network parameters, and obtaining the filling event image output by the light weight generation countermeasure network.
In order to use the lightweight generated countermeasure network obtained by training to the maximum extent, in the present embodiment, when performing loss event image filling, P pieces of loss event images are similarly input into the lightweight generated countermeasure network as one loss event image sequence to be filled, and the corresponding output is the filled event image sequence. As shown in fig. 4, in the event image which is output after the countermeasure network is generated in a lightweight manner for the loss event image shown in fig. 3 in this embodiment, it can be seen from the figure that the reality and fineness of the structure of the countermeasure network output image filling structure in the lightweight generation of the present invention are high, and the image can be restored to a large extent.
The method adopts the light-weight generation countermeasure network to fill the event images, and is smaller than the countermeasure network model used in the traditional image restoration and has higher deduction speed; the 3D convolution and event sequence discriminator is used, so that the time consistency and the quality of the filling result can be effectively improved. The method is suitable for capturing objects with rapid motion, fully retains the characteristic of high dynamic response of an event camera, and can be applied to ultra-high-speed human motion capture and high-frame-rate scenes.
The embodiment uses a shallow 3D generator to fully utilize the sparse characteristic of the event image, and meanwhile, in order to ensure the authenticity of the event filling result and the fineness of the structure, L is added into the original damage tolerance1Loss, perception loss, and grid loss. And finally, an event frame discriminator and an event sequence discriminator are used, so that the time consistency of the result is improved. The method has a small model, the inferred speed can reach 500FPS, the requirement of an event camera for capturing a high-speed moving object can be basically met, and the method can also be used for a high-dynamic response scene.
In another embodiment, there is also provided a lightweight-generated countermeasure network-based fast event image fill-in system, including:
a first module for constructing a lightweight generative confrontation network;
a second module for obtaining training data, the training data comprising a plurality of pairs of matched loss event images and non-loss event images;
a third module, configured to optimize the lightweight generation countermeasure network using the training data to obtain an optimal network parameter;
the fourth module is used for acquiring a loss event image to be filled, inputting the loss event image to the light weight generation countermeasure network based on the optimal network parameters, and obtaining a filling event image output by the light weight generation countermeasure network;
wherein the lightweight generation countermeasure network includes a generator and a discriminator, the generator including an encoder, a decoder, and two residual blocks connected between the encoder and the decoder, the encoder including three 3D convolutions, the encoder downsampling an image twice, the decoder including three 3D transposed convolutions, the decoder upsampling an image twice; the event frame discriminator is of a PatchGAN structure, the convolution in the event frame discriminator is a 2D convolution, the event sequence discriminator is of a PatchGAN structure, and the convolution in the event sequence discriminator is a 3D convolution.
For specific limitations of the rapid event image filling system based on the lightweight-generated countermeasure network, reference may be made to the above limitations of the rapid event image filling method based on the lightweight-generated countermeasure network, and details thereof are not repeated here. The various modules described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In another embodiment, the convolution in the residual block uses a dilation convolution with a dilation factor of 2.
In another embodiment, the third module optimizes the network parameters optimized by the lightweight generative countermeasure network using the training data, and performs the following operations:
taking P pairs of matched loss event images and non-loss event images based on training data;
inputting P loss event images into the generator as a loss event image sequence to obtain a filling event image sequence output by the generator, wherein each filling event image in the filling event image sequence corresponds to each loss event image in the loss event image sequence as input;
taking P pieces of non-loss event images as a non-loss event image sequence, according to the non-loss event image sequence and the filling event image sequence, firstly performing back propagation of a discriminator based on a total loss function of the discriminator, and then performing back propagation of a generator based on the total loss function of the generator;
and repeating the training until the network parameters optimal for the lightweight generation countermeasure network are obtained.
In another embodiment, the total loss function of the discriminator comprises:
Figure BDA0002735767210000131
wherein L isDAs a function of the total loss of the arbiter,
Figure BDA0002735767210000132
as a loss function of the event sequence discriminator,
Figure BDA0002735767210000133
as a loss function of the event frame discriminator,
Figure BDA0002735767210000134
is a weight parameter of the event sequence discriminator,
Figure BDA0002735767210000135
is the weight parameter of the event frame discriminator;
loss function of the event sequence discriminator
Figure BDA0002735767210000136
Things of harmonyLoss function of frame discriminator
Figure BDA0002735767210000137
The following were used:
Figure BDA0002735767210000138
Figure BDA0002735767210000139
wherein, IgtRepresenting a sequence of lossless event images, Pdata(Igt) Representing the distribution of the sequence of unreleased event images, E [. sup. ]]Indicating the expected value, logD, of the distribution functions(Igt) Representing the probability, logD, that the event sequence discriminator discriminated as a non-lost event imagef(Igt) Representing the probability of the event frame discriminator discriminating as an unrepaired event image, IinRepresenting a sequence of loss event images, Pdata(Iin) Represents the distribution of the loss event image sequence, log (1-D)s(G(Iin) Log (1-D)) represents the probability that the event sequence discriminator discriminated as a padded event image output by the generatorf(G(Iin) ) represents the probability that the event frame discriminator discriminated the padded event image output by the generator.
In another embodiment, the total loss function of the generator comprises:
LG＝λ1L1+λpLperc+λsLstyle+λgLg
wherein L isGTo the total loss function of the generator, L1Is L1Loss function, λ1Is L1Weight parameter of the loss function, LpercAs a function of perceptual loss, λpAs weight parameter of the perceptual loss function, LstyleAs a function of the loss of style, λsAs a weight parameter of the style loss function, LgTo generate a generator opposition loss function, λgFor the life of a living beingA weight parameter of the resultant opposition loss function;
the generator fighting loss function LgThe following were used:
Figure BDA00027357672100001310
wherein G denotes a generator, D denotes a discriminator, IinRepresenting a sequence of loss event images, Pdata(Iin) Representing the distribution of the loss event image sequence, E [. + ]]Expected value, G (I), representing distribution functionin) Sequence of padded event images, logD, representing the output of the generators(G(Iin) Representing the probability, logD, that the event sequence discriminator discriminated the padded event image as an unreduced event imagef(G(Iin) Represents the probability that the event frame discriminator discriminated the shim event image as an unreduced event image;
said L1Loss function L1The following were used:
Figure BDA0002735767210000141
wherein, IgtRepresenting a sequence of lossless event images, IpredA sequence of shim event images representing the generator output;
the perceptual loss function LpercThe following were used:
Figure BDA0002735767210000142
wherein phi isjIs the activation map of the jth layer of the pre-trained VGG-19 network, phij(Igt) Representing the corresponding activation graph sequence obtained after the non-loss event image sequence is input into the j layer of the VGG-19 networkj(Ipred) Representing a corresponding activation graph sequence obtained after the filling event image sequence is input into a j layer of a VGG-19 network; n is a radical ofjRepresenting the number of characteristic channels of a j-th network in the VGG-19 network;
the style loss function LstyleThe following were used:
Figure BDA0002735767210000143
wherein,
Figure BDA0002735767210000144
is based on an activation map phijC of constructionj×CjThe matrix of the Gram is a matrix of,
Figure BDA0002735767210000145
representing a plurality of Gram matrices constructed from a sequence of activation maps corresponding to a sequence of non-lost event images,
Figure BDA0002735767210000146
a plurality of Gram matrices constructed from the sequence of activation maps corresponding to the sequence of shim event images is represented.
It should be understood that, although the steps in the flowchart of fig. 1 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in fig. 1 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims (10)

1. A rapid event image filling method based on a lightweight generation countermeasure network is characterized by comprising the following steps:
constructing a lightweight generation countermeasure network;
acquiring training data, wherein the training data comprises a plurality of pairs of matched loss event images and non-loss event images;
optimizing the lightweight generation countermeasure network by using the training data to obtain optimal network parameters;
obtaining a loss event image to be filled, inputting the loss event image to a light weight generation countermeasure network based on optimal network parameters, and obtaining a filling event image output by the light weight generation countermeasure network;
wherein the lightweight generation countermeasure network includes a generator and a discriminator, the generator including an encoder, a decoder, and two residual blocks connected between the encoder and the decoder, the encoder including three 3D convolutions, the encoder downsampling an image twice, the decoder including three 3D transposed convolutions, the decoder upsampling an image twice; the event frame discriminator is of a PatchGAN structure, the convolution in the event frame discriminator is a 2D convolution, the event sequence discriminator is of a PatchGAN structure, and the convolution in the event sequence discriminator is a 3D convolution.
2. The method of claim 1, wherein the convolution in the residual block employs an extended convolution with an extension factor of 2.
3. The method of claim 1, wherein optimizing the lightweight generative confrontation network with the training data to obtain optimal network parameters comprises:
taking P pairs of matched loss event images and non-loss event images based on training data;
inputting P loss event images into the generator as a loss event image sequence to obtain a filling event image sequence output by the generator, wherein each filling event image in the filling event image sequence corresponds to each loss event image in the loss event image sequence as input;
taking P pieces of non-loss event images as a non-loss event image sequence, according to the non-loss event image sequence and the filling event image sequence, firstly performing back propagation of a discriminator based on a total loss function of the discriminator, and then performing back propagation of a generator based on the total loss function of the generator;
and repeating the training until the network parameters optimal for the lightweight generation countermeasure network are obtained.
4. The method of claim 3, wherein the overall loss function of the discriminator comprises:
Figure FDA0002735767200000021
wherein L isDAs a function of the total loss of the arbiter,
Figure FDA0002735767200000022
as a loss function of the event sequence discriminator,
Figure FDA0002735767200000023
as a loss function of the event frame discriminator,
Figure FDA0002735767200000024
is a weight parameter of the event sequence discriminator,
Figure FDA0002735767200000025
is the weight parameter of the event frame discriminator;
loss function of the event sequence discriminator
Figure FDA0002735767200000026
Loss function of sum event frame discriminator
Figure FDA0002735767200000027
The following were used:
Figure FDA0002735767200000028
Figure FDA0002735767200000029
wherein, IgtRepresenting a sequence of lossless event images, Pdata(Igt) Representing the distribution of the sequence of unreleased event images, E [. sup. ]]Expressing the expected value, log D, of the distribution functions(Igt) Representing the probability of the event sequence discriminator discriminating as an unrepaired event image, log Df(Igt) Representing the probability of the event frame discriminator discriminating as an unrepaired event image, IinRepresenting a sequence of loss event images, Pdata(Iin) Represents the distribution of the loss event image sequence, log (1-D)s(G(Iin) Log (1-D)) represents the probability that the event sequence discriminator discriminated as a padded event image output by the generatorf(G(Iin) ) represents the probability that the event frame discriminator discriminated as the fill-in event image output by the generatorAnd (4) rate.
5. The lightweight-based generation fast event image fill-in method against networks of claim 3, wherein the total loss function of the generator comprises:
LG＝λ1L1+λpLperc+λsLstyle+λgLg
wherein L isGTo the total loss function of the generator, L1Is L1Loss function, λ1Is L1Weight parameter of the loss function, LpercAs a function of perceptual loss, λpAs weight parameter of the perceptual loss function, LstyleAs a function of the loss of style, λsAs a weight parameter of the style loss function, LgTo generate a generator opposition loss function, λgA weight parameter for the generator counter loss function;
the generator fighting loss function LgThe following were used:
Figure FDA00027357672000000210
wherein G denotes a generator, D denotes a discriminator, IinRepresenting a sequence of loss event images, Pdata(Iin) Representing the distribution of the loss event image sequence, E [. + ]]Expected value, G (I), representing distribution functionin) Sequence of padded event images, log D, representing the generator outputs(G(Iin) Log D) represents the probability that the event sequence discriminator discriminated the padded event image as an unreduced event imagef(G(Iin) Represents the probability that the event frame discriminator discriminated the shim event image as an unreduced event image;
said L1Loss function L1The following were used:
Figure FDA00027357672000000211
wherein, IgtRepresenting a sequence of lossless event images, IpredA sequence of shim event images representing the generator output;
the perceptual loss function LpercThe following were used:
Figure FDA0002735767200000031
wherein phi isjIs the activation map of the jth layer of the pre-trained VGG-19 network, phij(Igt) Representing the corresponding activation graph sequence obtained after the non-loss event image sequence is input into the j layer of the VGG-19 networkj(Ipred) Representing a corresponding activation graph sequence obtained after the filling event image sequence is input into a j layer of a VGG-19 network; n is a radical ofjRepresenting the number of characteristic channels of a j-th network in the VGG-19 network;
the style loss function LstyleThe following were used:
Figure FDA0002735767200000032
wherein,
Figure FDA0002735767200000033
is based on an activation map phijC of constructionj×CjThe matrix of the Gram is a matrix of,
Figure FDA0002735767200000034
representing a plurality of Gram matrices constructed from a sequence of activation maps corresponding to a sequence of non-lost event images,
Figure FDA0002735767200000035
a plurality of Gram matrices constructed from the sequence of activation maps corresponding to the sequence of shim event images is represented.
6. A lightweight-generated confrontation network-based fast event image fill-in system, the lightweight-generated confrontation network-based fast event image fill-in system comprising:
a first module for constructing a lightweight generative confrontation network;
a second module for obtaining training data, the training data comprising a plurality of pairs of matched loss event images and non-loss event images;
a third module, configured to optimize the lightweight generation countermeasure network using the training data to obtain an optimal network parameter;
the fourth module is used for acquiring a loss event image to be filled, inputting the loss event image to the light weight generation countermeasure network based on the optimal network parameters, and obtaining a filling event image output by the light weight generation countermeasure network;
wherein the lightweight generation countermeasure network includes a generator and a discriminator, the generator including an encoder, a decoder, and two residual blocks connected between the encoder and the decoder, the encoder including three 3D convolutions, the encoder downsampling an image twice, the decoder including three 3D transposed convolutions, the decoder upsampling an image twice; the event frame discriminator is of a PatchGAN structure, the convolution in the event frame discriminator is a 2D convolution, the event sequence discriminator is of a PatchGAN structure, and the convolution in the event sequence discriminator is a 3D convolution.
7. The lightweight-based generation confrontation network fast event image fill-in system of claim 6, wherein the convolution in the residual block employs a dilation convolution with a dilation factor of 2.
8. The light-weight generation countermeasure network-based fast event image fill-in system of claim 6, wherein the third module, utilizing the training data to optimize the light-weight generation countermeasure network for optimal network parameters, performs the following operations:
taking P pairs of matched loss event images and non-loss event images based on training data;
inputting P loss event images into the generator as a loss event image sequence to obtain a filling event image sequence output by the generator, wherein each filling event image in the filling event image sequence corresponds to each loss event image in the loss event image sequence as input;
taking P pieces of non-loss event images as a non-loss event image sequence, according to the non-loss event image sequence and the filling event image sequence, firstly performing back propagation of a discriminator based on a total loss function of the discriminator, and then performing back propagation of a generator based on the total loss function of the generator;
and repeating the training until the network parameters optimal for the lightweight generation countermeasure network are obtained.
9. The lightweight-based generation confrontation network fast event image fill-in system of claim 8 wherein the overall loss function of the discriminator comprises:
Figure FDA0002735767200000041
wherein L isDAs a function of the total loss of the arbiter,
Figure FDA0002735767200000042
as a loss function of the event sequence discriminator,
Figure FDA0002735767200000043
as a loss function of the event frame discriminator,
Figure FDA0002735767200000044
is a weight parameter of the event sequence discriminator,
Figure FDA0002735767200000045
weight parameter for event frame discriminatorCounting;
loss function of the event sequence discriminator
Figure FDA0002735767200000046
Loss function of sum event frame discriminator
Figure FDA0002735767200000047
The following were used:
Figure FDA0002735767200000048
Figure FDA0002735767200000049
wherein, IgtRepresenting a sequence of lossless event images, Pdata(Igt) Representing the distribution of the sequence of unreleased event images, E [. sup. ]]Expressing the expected value, log D, of the distribution functions(Igt) Representing the probability of the event sequence discriminator discriminating as an unrepaired event image, log Df(Igt) Representing the probability of the event frame discriminator discriminating as an unrepaired event image, IinRepresenting a sequence of loss event images, Pdata(Iin) Represents the distribution of the loss event image sequence, log (1-D)s(G(Iin) Log (1-D)) represents the probability that the event sequence discriminator discriminated as a padded event image output by the generatorf(G(Iin) ) represents the probability that the event frame discriminator discriminated the padded event image output by the generator.
10. The lightweight-based generation confrontation network fast event image fill-in system of claim 8 wherein the generator's total loss function comprises:
LG＝λ1L1+λpLperc+λsLstyle+λgLg
wherein L isGTo the total loss function of the generator, L1Is L1Loss function, λ1Is L1Weight parameter of the loss function, LpercAs a function of perceptual loss, λpAs weight parameter of the perceptual loss function, LstyleAs a function of the loss of style, λsAs a weight parameter of the style loss function, LgTo generate a generator opposition loss function, λgA weight parameter for the generator counter loss function;
the generator fighting loss function LgThe following were used:
Figure FDA0002735767200000051
wherein G denotes a generator, D denotes a discriminator, IinRepresenting a sequence of loss event images, Pdata(Iin) Representing the distribution of the loss event image sequence, E [. + ]]Expected value, G (I), representing distribution functionin) Sequence of padded event images, log D, representing the generator outputs(G(Iin) Log D) represents the probability that the event sequence discriminator discriminated the padded event image as an unreduced event imagef(G(Iin) Represents the probability that the event frame discriminator discriminated the shim event image as an unreduced event image;
said L1Loss function L1The following were used:
Figure FDA0002735767200000052
wherein, IgtRepresenting a sequence of lossless event images, IpredA sequence of shim event images representing the generator output;
the perceptual loss function LpercThe following were used:
Figure FDA0002735767200000053
wherein phi isjIs the activation map of the jth layer of the pre-trained VGG-19 network, phij(Igt) Representing the corresponding activation graph sequence obtained after the non-loss event image sequence is input into the j layer of the VGG-19 networkj(Ipred) Representing a corresponding activation graph sequence obtained after the filling event image sequence is input into a j layer of a VGG-19 network; n is a radical ofjRepresenting the number of characteristic channels of a j-th network in the VGG-19 network;
the style loss function LstyleThe following were used:
Figure FDA0002735767200000054
wherein,
Figure FDA0002735767200000055
is based on an activation map phijC of constructionj×CjThe matrix of the Gram is a matrix of,
Figure FDA0002735767200000056
representing a plurality of Gram matrices constructed from a sequence of activation maps corresponding to a sequence of non-lost event images,
Figure FDA0002735767200000057
a plurality of Gram matrices constructed from the sequence of activation maps corresponding to the sequence of shim event images is represented.
CN202011133015.7A 2020-10-21 2020-10-21 Rapid event image filling method and system based on lightweight generation countermeasure network Active CN112396674B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011133015.7A CN112396674B (en) 2020-10-21 2020-10-21 Rapid event image filling method and system based on lightweight generation countermeasure network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011133015.7A CN112396674B (en) 2020-10-21 2020-10-21 Rapid event image filling method and system based on lightweight generation countermeasure network

Publications (2)

Publication Number Publication Date
CN112396674A true CN112396674A (en) 2021-02-23
CN112396674B CN112396674B (en) 2024-10-18

Family

ID=74596029

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011133015.7A Active CN112396674B (en) 2020-10-21 2020-10-21 Rapid event image filling method and system based on lightweight generation countermeasure network

Country Status (1)

Country Link
CN (1) CN112396674B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114266786A (en) * 2021-12-21 2022-04-01 北京工业大学 Gastric lesion segmentation method and system based on generation countermeasure network
CN115860054A (en) * 2022-07-21 2023-03-28 广州工商学院 Sparse codebook multiple access coding and decoding system based on generation countermeasure network

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109559287A (en) * 2018-11-20 2019-04-02 北京工业大学 A kind of semantic image restorative procedure generating confrontation network based on DenseNet
CN110930418A (en) * 2019-11-27 2020-03-27 江西理工大学 Retina blood vessel segmentation method fusing W-net and conditional generation confrontation network
CN111402179A (en) * 2020-03-12 2020-07-10 南昌航空大学 Image synthesis method and system combining countermeasure autoencoder and generation countermeasure network
CN111695435A (en) * 2020-05-19 2020-09-22 东南大学 Driver behavior identification method based on deep hybrid coding and decoding neural network

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109559287A (en) * 2018-11-20 2019-04-02 北京工业大学 A kind of semantic image restorative procedure generating confrontation network based on DenseNet
CN110930418A (en) * 2019-11-27 2020-03-27 江西理工大学 Retina blood vessel segmentation method fusing W-net and conditional generation confrontation network
CN111402179A (en) * 2020-03-12 2020-07-10 南昌航空大学 Image synthesis method and system combining countermeasure autoencoder and generation countermeasure network
CN111695435A (en) * 2020-05-19 2020-09-22 东南大学 Driver behavior identification method based on deep hybrid coding and decoding neural network

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
ALEX ZIHAO ZHU等: ""EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras"", ARVIV PREPRINT ARXIV:1802.06898, 31 August 2018 (2018-08-31), pages 3 *
CHE SUN等: "’Adversarial 3D Convolutional Auto-Encoder for Abnormal Event Detection in Videos"", 《IEEE TRANSACTIONS ON MULTIMEDIA》, vol. 23, 10 September 2020 (2020-09-10), pages 3292 - 3305 *
DMITRY ULYANOV等: "\'Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesis\'", 《2017 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION》, 31 December 2017 (2017-12-31), pages 4106 - 4113 *
杨东升等: ""基于双生成器生成对抗网络的电力系统暂态稳定评估方法"", 《电网技术》, vol. 45, no. 8, 14 October 2020 (2020-10-14), pages 2394 - 2945 *
王万良等: ""生成式对抗网络研究进展"", 《通信学报》, vol. 39, no. 2, 28 February 2018 (2018-02-28), pages 2018032 - 1 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114266786A (en) * 2021-12-21 2022-04-01 北京工业大学 Gastric lesion segmentation method and system based on generation countermeasure network
CN114266786B (en) * 2021-12-21 2024-09-13 北京工业大学 Stomach illness change segmentation method and system based on generation countermeasure network
CN115860054A (en) * 2022-07-21 2023-03-28 广州工商学院 Sparse codebook multiple access coding and decoding system based on generation countermeasure network
CN115860054B (en) * 2022-07-21 2023-09-26 广州工商学院 Sparse codebook multiple access coding and decoding system based on generation countermeasure network

Also Published As

Publication number Publication date
CN112396674B (en) 2024-10-18

Similar Documents

Publication Publication Date Title
CN111028150B (en) Rapid space-time residual attention video super-resolution reconstruction method
Khan et al. Fhdr: Hdr image reconstruction from a single ldr image using feedback network
CN111445418B (en) Image defogging processing method and device and computer equipment
CN111986084B (en) Multi-camera low-illumination image quality enhancement method based on multi-task fusion
CN112541877B (en) Defuzzification method, system, equipment and medium for generating countermeasure network based on condition
CN112750201B (en) Three-dimensional reconstruction method, related device and equipment
CN112801027B (en) Vehicle target detection method based on event camera
CN112653899A (en) Network live broadcast video feature extraction method based on joint attention ResNeSt under complex scene
CN110225260B (en) Three-dimensional high dynamic range imaging method based on generation countermeasure network
CN112288632B (en) Single image super-resolution method and system based on simplified ESRGAN
CN110555527A (en) Method and equipment for generating delayed shooting video
CN113344773B (en) Single picture reconstruction HDR method based on multi-level dual feedback
CN112396674A (en) Rapid event image filling method and system based on lightweight generation countermeasure network
CN114463176B (en) Image super-resolution reconstruction method based on improved ESRGAN
CN117576402B (en) Deep learning-based multi-scale aggregation transducer remote sensing image semantic segmentation method
CN116469007A (en) Forest fire identification method
Ren et al. A lightweight object detection network in low-light conditions based on depthwise separable pyramid network and attention mechanism on embedded platforms
CN115439849B (en) Instrument digital identification method and system based on dynamic multi-strategy GAN network
CN114119428B (en) Image deblurring method and device
CN114882405B (en) Video saliency detection method based on space-time double-flow pyramid network architecture
CN116523743A (en) Game super-resolution method based on cyclic neural network
CN115330633A (en) Image tone mapping method and device, electronic equipment and storage medium
CN115187488A (en) Image processing method and device, electronic device and storage medium
CN114677282A (en) Image super-resolution reconstruction method and system
Lu et al. EV-LFV: Synthesizing Light Field Event Streams from an Event Camera and Multiple RGB Cameras

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant