CN112396674A - Rapid event image filling method and system based on lightweight generation countermeasure network - Google Patents

Abstract

The invention discloses a method and system for fast event image filling based on a lightweight generation confrontation network, wherein the fast event image filling method based on a lightweight generation confrontation network comprises: constructing a lightweight generation confrontation network; acquiring training data, the training The data includes multiple pairs of matching loss event images and non-loss event images; use the training data to optimize the lightweight generative adversarial network to obtain optimal network parameters; obtain the loss event images to be filled, and input them to the network based on the optimal In the light-weight generative adversarial network of parameters, the padded event images output by the light-weight generative adversarial network are obtained. The method and system for fast event image filling based on the light-weight generative confrontation network of the present invention make full use of the sparse characteristics of the event image to improve the authenticity of the image filling structure and the fineness of the structure.

Description

A method and system for fast event image filling based on lightweight generative adversarial network

technical field

The present application belongs to the technical field of image processing, and in particular relates to a method and system for fast event image filling based on a lightweight generative confrontation network.

Background technique

Event-based Camera (Event-based Camera, or simply Event Camera, abbreviated as EB. Sometimes also called DVS (Dynamic Vision Sensor, dynamic vision sensor)) is a new type of sensor. Different from the traditional camera that shoots a complete image, the event camera shoots an "event", which can be simply understood as "change in pixel brightness", that is, what the event camera outputs is the change in pixel brightness.

Currently, event cameras are able to generate sparse event streams and capture high-speed motion information, however, with increasing temporal resolution, spatial resolution decreases dramatically. Although generative adversarial networks have achieved remarkable results on traditional image inpainting, direct use of them for event padding burys the fast response properties of event cameras, and the sparsity of event streams is also underutilized.

SUMMARY OF THE INVENTION

The purpose of this application is to provide a fast event image filling method and system based on a lightweight generative adversarial network, making full use of the sparse characteristics of event images, and improving the authenticity of the image filling structure and the fineness of the structure.

In order to achieve the above-mentioned purpose, the technical scheme adopted in this application is:

A fast event image filling method based on lightweight generative adversarial network, the fast event image filling method based on lightweight generative adversarial network, comprising:

Build a lightweight generative adversarial network;

acquiring training data, the training data including multiple pairs of matching loss event images and non-loss event images;

Using the training data to optimize the lightweight generative adversarial network to obtain optimal network parameters;

Obtain the loss event image to be filled, input it into the light-weight generative adversarial network based on the optimal network parameters, and obtain the filled event image output by the light-weight generative adversarial network;

Wherein, the lightweight generative adversarial network includes a generator and a discriminator, the generator includes an encoder, a decoder, and two residual blocks connected between the encoder and the decoder, the encoder includes three 3D convolutions, the encoder downsamples the image twice, the decoder includes three 3D transposed convolutions, the decoder upsamples the image twice; the discriminator includes event frame discrimination device and event sequence discriminator, the event frame discriminator is a PatchGAN structure, and the convolution in the event frame discriminator is a 2D convolution, the event sequence discriminator is a PatchGAN structure, and the convolution in the event sequence discriminator is a 3D convolution.

Several optional methods are also provided below, which are not intended to be additional limitations on the above-mentioned overall solution, but are merely further additions or optimizations. On the premise of no technical or logical contradiction, each optional method can be independently implemented for the above-mentioned overall solution. The combination can also be a combination between multiple optional ways.

Preferably, the convolution in the residual block adopts a dilated convolution with a dilation factor of 2.

Preferably, using the training data to optimize the lightweight generative adversarial network to obtain optimal network parameters, including:

Take P pairs of matching loss event images and non-loss event images based on the training data;

Input the P loss event images as a loss event image sequence into the generator, and obtain the padded event image sequence output by the generator, each padded event image in the padded event image sequence and the input loss event image Each loss event image in the image sequence corresponds to;

Taking P unlost event images as a sequence of unlossed event images, according to the unlossed event image sequence and the filled event image sequence, the back-propagation of the discriminator is first performed based on the total loss function of the discriminator, and then based on the generator's total loss function. The total loss function performs back-propagation of the generator;

The training is repeated until the optimal network parameters of the light-weight generative adversarial network are obtained.

Preferably, the total loss function of the discriminator includes:

为事件序列判别器的损失函数，

为事件帧判别器的损失函数，

为事件序列判别器的权重参数，

为事件帧判别器的权重参数；where _LD is the total loss function of the discriminator,

is the loss function of the event sequence discriminator,

is the loss function of the event frame discriminator,

is the weight parameter of the event sequence discriminator,

is the weight parameter of the event frame discriminator;

和事件帧判别器的损失函数

如下：The loss function of the event sequence discriminator

and the loss function of the event frame discriminator

as follows:

Among them, I _gt represents the sequence of non-lost event images, P _data (I _gt ) represents the distribution of the sequence of non-lost event images, E[*] represents the expected value of the distribution function, and logD _s (I _gt ) represents the event sequence discriminator discriminated as not The probability of loss event image, logD _f (I _gt ) represents the probability that the event frame discriminator discriminates as a non-lost event image, I _in represents the loss event image sequence, P _data (I _in ) represents the distribution of the loss event image sequence, log( 1-D _s (G(I _in ))) represents the probability that the event sequence discriminator discriminates as the padded event image output by the generator, log(1-D _f (G(I _in ))) represents the event frame discriminator discriminant is the probability of filling the event image output by the generator.

Preferably, the total loss function of the generator includes:

L _G =λ ₁ L ₁ +λ _p L _perc +λ _s L _style +λ _g L _g

where L _G is the total loss function of the generator, L ₁ is the L ₁ loss function, λ ₁ is the weight parameter of the L ₁ loss function, L _perc is the perceptual loss function, λ _p is the weight parameter of the perceptual loss function, L _style is the style loss function, λ _s is the weight parameter of the style loss function, L _g is the generator confrontation loss function, and λ _g is the weight parameter of the generator confrontation loss function;

The generator adversarial loss function L _g is as follows:

Among them, G represents the generator, D represents the discriminator, I _in represents the loss event image sequence, P _data (I _in ) represents the distribution of the loss event image sequence, E[*] represents the expected value of the distribution function, G(I _in ) represents the sequence of padded event images output by the generator, logD _s (G(I _in )) represents the probability that the event sequence discriminator will discriminate the padded event images as unloss event images, and logD _f (G(I _in )) represents the event frame discrimination probability that the padded event image will be discriminated as a non-lost event image;

_The L1 _loss function L1 is as follows:

where I _gt represents the sequence of unlossed event images, and I _pred represents the sequence of padded event images output by the generator;

The perceptual loss function L _perc is as follows:

where φ _j is the activation map of the jth layer of the pre-trained VGG-19 network, φ _j (I _gt ) represents the corresponding activation map sequence obtained after inputting the unloss event image sequence into the jth layer of the VGG-19 network, φ j _j (I _pred ) represents the corresponding activation map sequence obtained after the padded event image sequence is input to the jth layer of the VGG-19 network; N _j represents the number of feature channels of the jth layer network in the VGG-19 network;

The style loss function L _style is as follows:

是根据激活图φ_j构造的C_j×C_jGram矩阵，

表示根据未损失事件图像序列对应的激活图序列构造的多个Gram矩阵，

表示根据填补事件图像序列对应的激活图序列构造的多个Gram矩阵。in,

is the C _j ×C _j Gram matrix constructed from the activation map φ _j ,

represents multiple Gram matrices constructed from the activation map sequence corresponding to the unloss event image sequence,

Represents multiple Gram matrices constructed from activation map sequences corresponding to padded event image sequences.

The present application also provides a light-weight generative adversarial network-based fast event image filling system, and the light-weight generative adversarial network-based fast event image filling system includes:

The first module is used to build a lightweight generative adversarial network;

a second module, configured to acquire training data, the training data including multiple pairs of matching loss event images and non-loss event images;

The third module is used to optimize the lightweight generative adversarial network by using the training data to obtain optimal network parameters;

The fourth module is used to obtain the loss event image to be filled, and input it into the light-weight generative adversarial network based on the optimal network parameters to obtain the filled-in event image output by the light-weight generative adversarial network;

Wherein, the lightweight generative adversarial network includes a generator and a discriminator, the generator includes an encoder, a decoder, and two residual blocks connected between the encoder and the decoder, the encoder includes three 3D convolutions, the encoder downsamples the image twice, the decoder includes three 3D transposed convolutions, the decoder upsamples the image twice; the discriminator includes event frame discrimination device and event sequence discriminator, the event frame discriminator is a PatchGAN structure, and the convolution in the event frame discriminator is a 2D convolution, the event sequence discriminator is a PatchGAN structure, and the convolution in the event sequence discriminator is a 3D convolution.

Preferably, the convolution in the residual block adopts a dilated convolution with a dilation factor of 2.

Preferably, the third module uses the training data to optimize the lightweight generative adversarial network to obtain optimal network parameters, and performs the following operations:

Take P pairs of matching loss event images and non-loss event images based on the training data;

Input the P loss event images as a loss event image sequence into the generator, and obtain the padded event image sequence output by the generator, each padded event image in the padded event image sequence and the input loss event image Each loss event image in the image sequence corresponds to;

Taking P unlost event images as a sequence of unlossed event images, according to the unlossed event image sequence and the filled event image sequence, the back-propagation of the discriminator is first performed based on the total loss function of the discriminator, and then based on the generator's total loss function. The total loss function performs back-propagation of the generator;

The training is repeated until the optimal network parameters of the light-weight generative adversarial network are obtained.

Preferably, the total loss function of the discriminator includes:

为事件序列判别器的损失函数，

为事件帧判别器的损失函数，

为事件序列判别器的权重参数，

为事件帧判别器的权重参数；where _LD is the total loss function of the discriminator,

is the loss function of the event sequence discriminator,

is the loss function of the event frame discriminator,

is the weight parameter of the event sequence discriminator,

is the weight parameter of the event frame discriminator;

和事件帧判别器的损失函数

如下：The loss function of the event sequence discriminator

and the loss function of the event frame discriminator

as follows:

Among them, I _gt represents the sequence of non-lost event images, P _data (I _gt ) represents the distribution of the sequence of non-lost event images, E[*] represents the expected value of the distribution function, and logD _s (I _gt ) represents the event sequence discriminator discriminated as not The probability of loss event image, logD _f (I _gt ) represents the probability that the event frame discriminator discriminates as a non-lost event image, I _in represents the loss event image sequence, P _data (I _in ) represents the distribution of the loss event image sequence, log( 1-D _s (G(I _in ))) represents the probability that the event sequence discriminator discriminates as the padded event image output by the generator, log(1-D _f (G(I _in ))) represents the event frame discriminator discriminant is the probability of filling the event image output by the generator.

Preferably, the total loss function of the generator includes:

L _G =λ ₁ L ₁ +λ _p L _perc +λ _s L _style +λ _g L _g

where L _G is the total loss function of the generator, L ₁ is the L ₁ loss function, λ ₁ is the weight parameter of the L ₁ loss function, L _perc is the perceptual loss function, λ _p is the weight parameter of the perceptual loss function, L _style is the style loss function, λ _s is the weight parameter of the style loss function, L _g is the generator confrontation loss function, and λ _g is the weight parameter of the generator confrontation loss function;

The generator adversarial loss function L _g is as follows:

Among them, G represents the generator, D represents the discriminator, I _in represents the loss event image sequence, P _data (I _in ) represents the distribution of the loss event image sequence, E[*] represents the expected value of the distribution function, G(I _in ) represents the sequence of padded event images output by the generator, logD _s (G(I _in )) represents the probability that the event sequence discriminator will discriminate the padded event images as unloss event images, and logD _f (G(I _in )) represents the event frame discrimination probability that the padded event image will be discriminated as a non-lost event image;

_The L1 _loss function L1 is as follows:

where I _gt represents the sequence of unlossed event images, and I _pred represents the sequence of padded event images output by the generator;

The perceptual loss function L _perc is as follows:

where φ _j is the activation map of the jth layer of the pre-trained VGG-19 network, φ _j (I _gt ) represents the corresponding activation map sequence obtained after inputting the unloss event image sequence into the jth layer of the VGG-19 network, φ j _j (I _pred ) represents the corresponding activation map sequence obtained after the padded event image sequence is input to the jth layer of the VGG-19 network; N _j represents the number of feature channels of the jth layer network in the VGG-19 network;

The style loss function L _style is as follows:

是根据激活图φ_j构造的C_j×C_jGram矩阵，

表示根据未损失事件图像序列对应的激活图序列构造的多个Gram矩阵，

表示根据填补事件图像序列对应的激活图序列构造的多个Gram矩阵。in,

is the C _j ×C _j Gram matrix constructed from the activation map φ _j ,

represents multiple Gram matrices constructed from the activation map sequence corresponding to the unloss event image sequence,

Represents multiple Gram matrices constructed from activation map sequences corresponding to padded event image sequences.

In order to overcome the shortcomings of traditional image inpainting models such as large size, redundant parameters, and slow inference speed, as well as the problem of 2D convolution resulting in decreased time consistency of results, the present application provides a method and system for fast event image filling based on lightweight generative adversarial networks. , a shallow 3D generator is constructed to take full advantage of the sparse nature of event images, and at the same time, in order to ensure the authenticity of the filled event results and the fineness of the structure, L1 _loss , perceptual loss and style loss are added to the original adversarial loss . Finally, an event sequence discriminator is proposed to improve the temporal consistency of the results.

Description of drawings

FIG. 1 is a flowchart of the fast event image filling method based on a lightweight generative adversarial network of the present application;

2 is a schematic structural diagram of a lightweight generative adversarial network constructed by the application;

FIG. 3 is an example diagram of an embodiment of a loss event image of the present application;

FIG. 4 is the padded event image output by the lightweight generative adversarial network of the application for FIG. 3 .

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein in the specification of the present application are for the purpose of describing specific embodiments only, and are not intended to limit the present application.

In one of the embodiments, a fast event image filling method based on a lightweight generative adversarial network is provided, which is used in the field of image processing, especially the filling of event camera images with damaged spatial resolution.

As shown in Figure 1, a fast event image filling method based on lightweight generative adversarial network includes the following steps:

Step S1, constructing a lightweight generative adversarial network.

Since the direct application of the generative adversarial network to fill in the event image will bury the fast response characteristics of the event camera, and the sparsity of the event stream cannot be fully utilized, this embodiment builds a lightweight generative adversarial network to facilitate high dynamic response scenarios. Applications.

As shown in Figure 2, the lightweight generative adversarial network constructed in this embodiment includes a generator and a discriminator. The generator includes an encoder and a decoder, and two residual blocks connected between the encoder and the decoder.

The network of traditional image inpainting is too deep for event images, which leads to slow inference speed. Therefore, the encoder of the present invention includes three 3D convolutions, and only performs two downsampling on the image. After each downsampling, the feature channel is expanded to be twice that of the previous layer, and the corresponding decoder includes three 3D transposed volumes. product, and only upsampling the image twice, after each upsampling, the feature channel is reduced to 2 times that of the previous layer.

At the same time, due to the sparseness of the event image, the shallow network will not cause the generation quality of the event image to be too low, so this embodiment only uses two residual blocks between the encoder and the decoder. In order to increase the receptive field, a dilated convolution with a dilation factor of 2 is used to replace the regular convolution in the residual layer. At the same time, in order to improve the generalization ability and retain more spatiotemporal information, a 3D dilated convolution is used to replace the 2D dilated convolution. This embodiment uses instance normalization across all layers of the network.

In order to improve the temporal consistency and quality of the filled event images, the discriminator constructed in this embodiment includes an event frame discriminator and an event sequence discriminator. The event frame discriminator is a PatchGAN structure, and the convolution in the event frame discriminator is a 2D convolution, the event sequence discriminator is a PatchGAN structure, and the convolution in the event sequence discriminator is a 3D convolution.

The discriminator uses a 70×70 PatchGAN structure, which is used to discriminate whether overlapping image patches of size 70×70 are real. The event frame discriminator uses 2D convolutions, which aim to focus on the spatial feature consistency of event frames. Although the use of 3D convolution in the generator can retain more spatiotemporal information, it also makes the edge of the image appear blurry. For this reason, an event sequence discriminator is introduced, that is, using 3D convolution to improve the quality of the generated image, the event sequence discriminator focuses on Focus on the temporal dependencies and correlations of pixel changes. Finally, to enhance training stability, spectral normalization is applied to the discriminator.

This embodiment uses regular convolution, dilated convolution and transposed convolution. Taking the 5×5 input feature map as an example, a 3×3 convolution kernel is used, and the stride is 1. The conventional convolution will output a 3×3 feature map, which is then used as the input of the dilated convolution. When the interval of the kernel points is 1 (0 is filled between the convolution kernel points, the kernel size becomes 5 × 5, the parameter amount remains unchanged, and the receptive field becomes larger), and the number of feature map edge fills (fill 0) is set to is 2, the stride is 1, and the size of the feature map output by the dilated convolution is 3 × 3; for the transposed convolution, only the number of edge padding of the feature map is set to 2, and other parameters are the same as those of the conventional convolution. The 3×3 feature map can be restored to 5×5 size.

Step S2: Acquire training data, where the training data includes multiple pairs of matching loss event images and non-loss event images.

This embodiment mainly fills in the image of the event camera, so this embodiment takes the event camera as an example for description. For the event camera, the output can be regarded as a continuous stream of events {e _i }∈N. Each event e _i can be represented using the following form:

e _i =(x _i ,y _i ,t _i ,p _i ) (1)

where (x _i , y _i ) denotes the spatial position of the pixel that generated the event, t _i denotes the temporal coordinate of the luminance change, p _i ∈ {-1, 1} denotes the positive or negative change in intensity at the pixel that caused the event, and i is Event index sequence number.

In the exposure time interval Δt = t + τ, the event frame F _τ (t) is obtained by adding all events between times t and t + τ at the pixel level, so the event frame can be expressed as:

where E _t,τ ={e _i |t _i ∈[t,t+τ]}. In this way, an event frame can be represented as a grayscale image of size 1 × w × h that integrates all events that occurred within a specific time interval into a single channel. Based on the method of event frame accumulation, when generating matching loss event images and non-loss event images, M ₁ events are accumulated into one event frame as the loss event image, and M ₂ events are accumulated into one event frame as the loss event image. Unlost event images, where M ₂ is at least 80 times larger than M ₁ , to guarantee the validity of un-loss event images.

For example, in this embodiment, the loss event image is accumulated with 100 events as one frame, while the non-loss event image is accumulated with 7500 events as one frame. Figure 3 is an accumulated loss event image with 100 events as one frame, so a low-resolution event image with a frame rate of about 2000 FPS can be obtained.

Step S3, using the training data to optimize the lightweight generative adversarial network to obtain optimal network parameters.

In the training of the light-weight generative adversarial network, in order to avoid insufficient video memory or low utilization of video memory, in this embodiment, multiple event frames are used as a sequence segment input for training.

Specifically, in this embodiment, based on the training data, P (P>1, for example, a value of 8) is used to pair matching loss event images and non-loss event images.

Input the P loss event images as a loss event image sequence into the generator, and obtain the padded event image sequence output by the generator, each padded event image in the padded event image sequence and the input loss event image Each loss event image in the image sequence corresponds to.

Taking P unlost event images as a sequence of unlossed event images, according to the unlossed event image sequence and the filled event image sequence, the back-propagation of the discriminator is first performed based on the total loss function of the discriminator, and then based on the generator's total loss function. The total loss function does the back-propagation of the generator.

The training is repeated until the optimal network parameters of the light-weight generative adversarial network are obtained.

There are two types of discriminators in this embodiment. In order to strengthen the correlation between the two discriminators, this embodiment simultaneously trains the two discriminators. When updating the discriminator loss, the losses of the two discriminators are calculated first. and then backpropagate, instead of updating the losses individually. So the total loss function for building the discriminator consists of:

为事件序列判别器的损失函数，

为事件帧判别器的损失函数，

为事件序列判别器的权重参数，

为事件帧判别器的权重参数。本实施例优选

where _LD is the total loss function of the discriminator,

is the loss function of the event sequence discriminator,

is the loss function of the event frame discriminator,

is the weight parameter of the event sequence discriminator,

is the weight parameter of the event frame discriminator. This embodiment is preferred

和事件帧判别器的损失函数

如下：The loss function of the event sequence discriminator

and the loss function of the event frame discriminator

as follows:

Among them, I _gt represents the sequence of non-lost event images, P _data (I _gt ) represents the distribution of the sequence of non-lost event images, E[*] represents the expected value of the distribution function, and logD _s (I _gt ) represents the event sequence discriminator discriminated as not The probability of loss event image, logD _f (I _gt ) represents the probability that the event frame discriminator discriminates as a non-lost event image, I _in represents the loss event image sequence, P _data (I _in ) represents the distribution of the loss event image sequence, log( 1-D _s (G(I _in ))) represents the probability that the event sequence discriminator discriminates as the padded event image output by the generator, log(1-D _f (G(I _in ))) represents the event frame discriminator discriminant is the probability of filling the event image output by the generator.

In order to ensure the authenticity and quality of the filled event image sequence, this embodiment comprehensively considers multiple losses of the generator, and the total loss function of the generator used includes:

L _G =λ ₁ L ₁ +λ _p L _perc +λ _s L _style +λ _g L _g (6)

where L _D is the total loss function of the discriminator, L ₁ is the L ₁ loss function, λ ₁ is the weight parameter of the L ₁ loss function, L _perc is the perceptual loss function, λ _p is the weight parameter of the perceptual loss function, L _style is the style loss function, λ _s is the weight parameter of the style loss function, L _g is the generator confrontation loss function, and λ _g is the weight parameter of the generator confrontation loss function. In this embodiment, it is preferable that λ ₁ =1, λ _g =λ _p =0.1, and λ _s =250.

The light-weight generative adversarial network fills each image in the input image sequence and outputs it, and uses BCELoss (binary cross entropy loss) to make the distribution of the filled event image sequence close to the distribution of the real label, so the generator adversarial loss is used. The function L _g is as follows:

Among them, G represents the generator, D represents the discriminator, I _in represents the loss event image sequence, P _data (I _in ) represents the distribution of the loss event image sequence, E[*] represents the expected value of the distribution function, G(I _in ) represents the sequence of padded event images output by the generator, logD _s (G(I _in )) represents the probability that the event sequence discriminator will discriminate the padded event images as unloss event images, and logD _f (G(I _in )) represents the event frame discrimination probability that the padded event image will be discriminated as a non-lost event image;

In order to make full use of the sparsity feature of the event image, L ₁ loss is added to the original generator loss function. The L ₁ loss focuses on pixel-level features. The L ₁ loss function L ₁ used in this embodiment is as follows:

where I _gt represents the sequence of unloss event images and I _pred represents the sequence of padded event images output by the generator.

The L1 _loss function will cause blurring of the result while ensuring the pixel-level features of the generator, so this embodiment introduces perceptual loss and style loss to preserve the image content. The perceptual loss normalizes the generated target image I _pred to be closer to the true label I _gt in the VGG subspace, and the perceptual loss function L _perc is as follows:

where φ _j is the activation map of the jth layer of the pre-trained VGG-19 network, φ _j (I _gt ) represents the corresponding activation map sequence obtained after inputting the unloss event image sequence into the jth layer of the VGG-19 network, φ j _j (I _pred ) represents the corresponding activation map sequence obtained after the padded event image sequence is input to the jth layer of the VGG-19 network; N _j represents the number of feature channels of the jth layer network in the VGG-19 network.

Unlike the perceptual loss, the style loss first applies the autocorrelation (Gram matrix) to the features in order to recover the detailed texture better. Style loss, which measures the difference between activation map covariances, is also computed using VGG. Given an activation map of size C _j × H _j × W _j , the style loss can be computed in the following way:

是根据激活图φ_j构造的C_j×C_jGram矩阵，

表示根据未损失事件图像序列对应的激活图序列构造的多个Gram矩阵，

表示根据填补事件图像序列对应的激活图序列构造的多个Gram矩阵。in,

is the C _j ×C _j Gram matrix constructed from the activation map φ _j ,

represents multiple Gram matrices constructed from the activation map sequence corresponding to the unloss event image sequence,

Represents multiple Gram matrices constructed from activation map sequences corresponding to padded event image sequences.

This embodiment mainly implements filling in event image sequences with high temporal resolution and low spatial resolution, or event image sequences with normal temporal resolution but impaired spatial resolution. First, the events are accumulated under the condition of high time resolution to obtain the event frame, and the event sequence image is sent to the generator. After the generator, the filled event image sequence is output. Afterwards, the padded event image sequence and the ground truth are sent to two discriminators together, and the discriminator discriminates the authenticity and feeds the result back to the generator, thus ensuring the temporal consistency and image quality of the padded event image sequence.

Step S4: Obtain the loss event image to be filled, and input it into the light-weight generative adversarial network based on the optimal network parameters to obtain the filled-in event image output by the light-weight generative adversarial network.

In order to use the light-weight generative adversarial network obtained by training to the greatest extent, when the loss event image filling is performed in this embodiment, P loss event images are also input into the light-weight generative adversarial network as a loss event image sequence for filling, corresponding to The output is a sequence of padded event images. As shown in FIG. 4 , the light-weight generative adversarial network of the present embodiment outputs the event image after filling for the loss event image shown in FIG. 3 . From the figure, it can be seen that the light-weight generative adversarial network output image of the present application is actually filled in the structure. The fineness of nature and structure is high, and the image can be restored to a greater extent.

This application uses a lightweight generative adversarial network to fill in event images, which is smaller than the adversarial network model used in traditional image inpainting, and has faster inference speed; using 3D convolution and event sequence discriminator can effectively improve the time to fill in results Consistency and quality. It is suitable for capturing fast-moving things, and fully retains the high dynamic response characteristics of event cameras, such as ultra-high-speed human motion capture and high frame rate scenes.

This embodiment uses a shallow 3D generator to take full advantage of the sparseness of event images. At the same time, in order to ensure the authenticity of the filled event results and the fineness of the structure, L1 _loss , perceptual loss and style are added to the original adversarial loss. loss. Finally, an event frame discriminator and an event sequence discriminator are used to improve the temporal consistency of the results. The model of the invention is small, and the inference speed can reach 500FPS, which can basically meet the requirements of the event camera for capturing high-speed moving objects, and can also be used in high dynamic response scenarios.

In another embodiment, a light-weight generative adversarial network-based fast event image filling system is also provided, and the light-weight generative adversarial network-based fast event image filling system includes:

The first module is used to build a lightweight generative adversarial network;

a second module, configured to acquire training data, the training data including multiple pairs of matching loss event images and non-loss event images;

The third module is used to optimize the lightweight generative adversarial network by using the training data to obtain optimal network parameters;

The fourth module is used to obtain the loss event image to be filled, and input it into the light-weight generative adversarial network based on the optimal network parameters to obtain the filled-in event image output by the light-weight generative adversarial network;

Wherein, the lightweight generative adversarial network includes a generator and a discriminator, the generator includes an encoder, a decoder, and two residual blocks connected between the encoder and the decoder, the encoder includes three 3D convolutions, the encoder downsamples the image twice, the decoder includes three 3D transposed convolutions, the decoder upsamples the image twice; the discriminator includes event frame discrimination device and event sequence discriminator, the event frame discriminator is a PatchGAN structure, and the convolution in the event frame discriminator is a 2D convolution, the event sequence discriminator is a PatchGAN structure, and the convolution in the event sequence discriminator is a 3D convolution.

For the specific definition of the light-weight generative adversarial network-based fast event image filling system, please refer to the definition of the light-weight generative adversarial network-based fast event image filling method above, which will not be repeated here. The above-mentioned modules can be implemented in whole or in part by software, hardware and combinations thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

In another embodiment, the convolution in the residual block adopts a dilated convolution with a dilation factor of 2.

In another embodiment, the third module, using the training data to optimize the lightweight generative adversarial network to obtain optimal network parameters, performs the following operations:

Take P pairs of matching loss event images and non-loss event images based on the training data;

Input the P loss event images as a loss event image sequence into the generator, and obtain the padded event image sequence output by the generator, each padded event image in the padded event image sequence and the input loss event image Each loss event image in the image sequence corresponds to;

Taking P unlost event images as a sequence of unlossed event images, according to the unlossed event image sequence and the filled event image sequence, the back-propagation of the discriminator is first performed based on the total loss function of the discriminator, and then based on the generator's total loss function. The total loss function performs back-propagation of the generator;

The training is repeated until the optimal network parameters of the light-weight generative adversarial network are obtained.

In another embodiment, the total loss function of the discriminator includes:

为事件序列判别器的损失函数，

为事件帧判别器的损失函数，

为事件序列判别器的权重参数，

为事件帧判别器的权重参数；where _LD is the total loss function of the discriminator,

is the loss function of the event sequence discriminator,

is the loss function of the event frame discriminator,

is the weight parameter of the event sequence discriminator,

is the weight parameter of the event frame discriminator;

和事件帧判别器的损失函数

如下：The loss function of the event sequence discriminator

and the loss function of the event frame discriminator

as follows:

Among them, I _gt represents the sequence of non-lost event images, P _data (I _gt ) represents the distribution of the sequence of non-lost event images, E[*] represents the expected value of the distribution function, and logD _s (I _gt ) represents the event sequence discriminator discriminated as not The probability of loss event image, logD _f (I _gt ) represents the probability that the event frame discriminator discriminates as a non-lost event image, I _in represents the loss event image sequence, P _data (I _in ) represents the distribution of the loss event image sequence, log( 1-D _s (G(I _in ))) represents the probability that the event sequence discriminator discriminates as the padded event image output by the generator, log(1-D _f (G(I _in ))) represents the event frame discriminator discriminant is the probability of filling the event image output by the generator.

In another embodiment, the overall loss function of the generator includes:

L _G =λ ₁ L ₁ +λ _p L _perc +λ _s L _style +λ _g L _g

where L _G is the total loss function of the generator, L ₁ is the L ₁ loss function, λ ₁ is the weight parameter of the L ₁ loss function, L _perc is the perceptual loss function, λ _p is the weight parameter of the perceptual loss function, L _style is the style loss function, λ _s is the weight parameter of the style loss function, L _g is the generator confrontation loss function, and λ _g is the weight parameter of the generator confrontation loss function;

The generator adversarial loss function L _g is as follows:

Among them, G represents the generator, D represents the discriminator, I _in represents the loss event image sequence, P _data (I _in ) represents the distribution of the loss event image sequence, E[*] represents the expected value of the distribution function, G(I _in ) represents the sequence of padded event images output by the generator, logD _s (G(I _in )) represents the probability that the event sequence discriminator will discriminate the padded event images as unloss event images, and logD _f (G(I _in )) represents the event frame discrimination probability that the padded event image will be discriminated as a non-lost event image;

_The L1 _loss function L1 is as follows:

where I _gt represents the sequence of unlossed event images, and I _pred represents the sequence of padded event images output by the generator;

The perceptual loss function L _perc is as follows:

where φ _j is the activation map of the jth layer of the pre-trained VGG-19 network, φ _j (I _gt ) represents the corresponding activation map sequence obtained after inputting the unloss event image sequence into the jth layer of the VGG-19 network, φ j _j (I _pred ) represents the corresponding activation map sequence obtained after the padded event image sequence is input to the jth layer of the VGG-19 network; N _j represents the number of feature channels of the jth layer network in the VGG-19 network;

The style loss function L _style is as follows:

是根据激活图φ_j构造的C_j×C_jGram矩阵，

表示根据未损失事件图像序列对应的激活图序列构造的多个Gram矩阵，

表示根据填补事件图像序列对应的激活图序列构造的多个Gram矩阵。in,

is the C _j ×C _j Gram matrix constructed from the activation map φ _j ,

represents multiple Gram matrices constructed from the activation map sequence corresponding to the unloss event image sequence,

Represents multiple Gram matrices constructed from activation map sequences corresponding to padded event image sequences.

It should be understood that although the various steps in the flowchart of FIG. 1 are shown in sequence according to the arrows, these steps are not necessarily executed in the sequence shown by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order, and these steps may be performed in other orders. Moreover, at least a part of the steps in FIG. 1 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed and completed at the same time, but may be executed at different times. The execution of these sub-steps or stages The sequence is also not necessarily sequential, but may be performed alternately or alternately with other steps or sub-steps of other steps or at least a portion of a phase.

The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity, all possible combinations of the technical features in the above-described embodiments are not described. However, as long as there is no contradiction between the combinations of these technical features, All should be regarded as the scope described in this specification.

The above-mentioned embodiments only represent several embodiments of the present application, and the descriptions thereof are specific and detailed, but should not be construed as a limitation on the scope of the invention patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of the patent of the present application shall be subject to the appended claims.

Claims (10)

1. A rapid event image filling method based on a lightweight generation countermeasure network is characterized by comprising the following steps:

constructing a lightweight generation countermeasure network;

acquiring training data, wherein the training data comprises a plurality of pairs of matched loss event images and non-loss event images;

optimizing the lightweight generation countermeasure network by using the training data to obtain optimal network parameters;

obtaining a loss event image to be filled, inputting the loss event image to a light weight generation countermeasure network based on optimal network parameters, and obtaining a filling event image output by the light weight generation countermeasure network;

wherein the lightweight generation countermeasure network includes a generator and a discriminator, the generator including an encoder, a decoder, and two residual blocks connected between the encoder and the decoder, the encoder including three 3D convolutions, the encoder downsampling an image twice, the decoder including three 3D transposed convolutions, the decoder upsampling an image twice; the event frame discriminator is of a PatchGAN structure, the convolution in the event frame discriminator is a 2D convolution, the event sequence discriminator is of a PatchGAN structure, and the convolution in the event sequence discriminator is a 3D convolution.

2. The method of claim 1, wherein the convolution in the residual block employs an extended convolution with an extension factor of 2.

3. The method of claim 1, wherein optimizing the lightweight generative confrontation network with the training data to obtain optimal network parameters comprises:

taking P pairs of matched loss event images and non-loss event images based on training data;

inputting P loss event images into the generator as a loss event image sequence to obtain a filling event image sequence output by the generator, wherein each filling event image in the filling event image sequence corresponds to each loss event image in the loss event image sequence as input;

taking P pieces of non-loss event images as a non-loss event image sequence, according to the non-loss event image sequence and the filling event image sequence, firstly performing back propagation of a discriminator based on a total loss function of the discriminator, and then performing back propagation of a generator based on the total loss function of the generator;

and repeating the training until the network parameters optimal for the lightweight generation countermeasure network are obtained.

4. The method of claim 3, wherein the overall loss function of the discriminator comprises:

wherein L is_DAs a function of the total loss of the arbiter,

as a loss function of the event sequence discriminator,

as a loss function of the event frame discriminator,

is a weight parameter of the event sequence discriminator,

is the weight parameter of the event frame discriminator;

loss function of the event sequence discriminator

Loss function of sum event frame discriminator

The following were used:

wherein, I_gtRepresenting a sequence of lossless event images, P_data(I_gt) Representing the distribution of the sequence of unreleased event images, E [. sup. ]]Expressing the expected value, log D, of the distribution function_s(I_gt) Representing the probability of the event sequence discriminator discriminating as an unrepaired event image, log D_f(I_gt) Representing the probability of the event frame discriminator discriminating as an unrepaired event image, I_inRepresenting a sequence of loss event images, P_data(I_in) Represents the distribution of the loss event image sequence, log (1-D)_s(G(I_in) Log (1-D)) represents the probability that the event sequence discriminator discriminated as a padded event image output by the generator_f(G(I_in) ) represents the probability that the event frame discriminator discriminated as the fill-in event image output by the generatorAnd (4) rate.

5. The lightweight-based generation fast event image fill-in method against networks of claim 3, wherein the total loss function of the generator comprises:

L_G＝λ₁L₁+λ_pL_perc+λ_sL_style+λ_gL_g

wherein L is_GTo the total loss function of the generator, L₁Is L₁Loss function, λ₁Is L₁Weight parameter of the loss function, L_percAs a function of perceptual loss, λ_pAs weight parameter of the perceptual loss function, L_styleAs a function of the loss of style, λ_sAs a weight parameter of the style loss function, L_gTo generate a generator opposition loss function, λ_gA weight parameter for the generator counter loss function;

the generator fighting loss function L_gThe following were used:

wherein G denotes a generator, D denotes a discriminator, I_inRepresenting a sequence of loss event images, P_data(I_in) Representing the distribution of the loss event image sequence, E [. + ]]Expected value, G (I), representing distribution function_in) Sequence of padded event images, log D, representing the generator output_s(G(I_in) Log D) represents the probability that the event sequence discriminator discriminated the padded event image as an unreduced event image_f(G(I_in) Represents the probability that the event frame discriminator discriminated the shim event image as an unreduced event image;

said L₁Loss function L₁The following were used:

wherein, I_gtRepresenting a sequence of lossless event images, I_predA sequence of shim event images representing the generator output;

the perceptual loss function L_percThe following were used:

wherein phi is_jIs the activation map of the jth layer of the pre-trained VGG-19 network, phi_j(I_gt) Representing the corresponding activation graph sequence obtained after the non-loss event image sequence is input into the j layer of the VGG-19 network_j(I_pred) Representing a corresponding activation graph sequence obtained after the filling event image sequence is input into a j layer of a VGG-19 network; n is a radical of_jRepresenting the number of characteristic channels of a j-th network in the VGG-19 network;

the style loss function L_styleThe following were used:

wherein,

is based on an activation map phi_jC of construction_j×C_jThe matrix of the Gram is a matrix of,

representing a plurality of Gram matrices constructed from a sequence of activation maps corresponding to a sequence of non-lost event images,

a plurality of Gram matrices constructed from the sequence of activation maps corresponding to the sequence of shim event images is represented.

6. A lightweight-generated confrontation network-based fast event image fill-in system, the lightweight-generated confrontation network-based fast event image fill-in system comprising:

a first module for constructing a lightweight generative confrontation network;

a second module for obtaining training data, the training data comprising a plurality of pairs of matched loss event images and non-loss event images;

a third module, configured to optimize the lightweight generation countermeasure network using the training data to obtain an optimal network parameter;

the fourth module is used for acquiring a loss event image to be filled, inputting the loss event image to the light weight generation countermeasure network based on the optimal network parameters, and obtaining a filling event image output by the light weight generation countermeasure network;

wherein the lightweight generation countermeasure network includes a generator and a discriminator, the generator including an encoder, a decoder, and two residual blocks connected between the encoder and the decoder, the encoder including three 3D convolutions, the encoder downsampling an image twice, the decoder including three 3D transposed convolutions, the decoder upsampling an image twice; the event frame discriminator is of a PatchGAN structure, the convolution in the event frame discriminator is a 2D convolution, the event sequence discriminator is of a PatchGAN structure, and the convolution in the event sequence discriminator is a 3D convolution.

7. The lightweight-based generation confrontation network fast event image fill-in system of claim 6, wherein the convolution in the residual block employs a dilation convolution with a dilation factor of 2.

8. The light-weight generation countermeasure network-based fast event image fill-in system of claim 6, wherein the third module, utilizing the training data to optimize the light-weight generation countermeasure network for optimal network parameters, performs the following operations:

taking P pairs of matched loss event images and non-loss event images based on training data;

inputting P loss event images into the generator as a loss event image sequence to obtain a filling event image sequence output by the generator, wherein each filling event image in the filling event image sequence corresponds to each loss event image in the loss event image sequence as input;

taking P pieces of non-loss event images as a non-loss event image sequence, according to the non-loss event image sequence and the filling event image sequence, firstly performing back propagation of a discriminator based on a total loss function of the discriminator, and then performing back propagation of a generator based on the total loss function of the generator;

and repeating the training until the network parameters optimal for the lightweight generation countermeasure network are obtained.

9. The lightweight-based generation confrontation network fast event image fill-in system of claim 8 wherein the overall loss function of the discriminator comprises:

wherein L is_DAs a function of the total loss of the arbiter,

as a loss function of the event sequence discriminator,

as a loss function of the event frame discriminator,

is a weight parameter of the event sequence discriminator,

weight parameter for event frame discriminatorCounting;

loss function of the event sequence discriminator

Loss function of sum event frame discriminator

The following were used:

wherein, I_gtRepresenting a sequence of lossless event images, P_data(I_gt) Representing the distribution of the sequence of unreleased event images, E [. sup. ]]Expressing the expected value, log D, of the distribution function_s(I_gt) Representing the probability of the event sequence discriminator discriminating as an unrepaired event image, log D_f(I_gt) Representing the probability of the event frame discriminator discriminating as an unrepaired event image, I_inRepresenting a sequence of loss event images, P_data(I_in) Represents the distribution of the loss event image sequence, log (1-D)_s(G(I_in) Log (1-D)) represents the probability that the event sequence discriminator discriminated as a padded event image output by the generator_f(G(I_in) ) represents the probability that the event frame discriminator discriminated the padded event image output by the generator.

10. The lightweight-based generation confrontation network fast event image fill-in system of claim 8 wherein the generator's total loss function comprises:

L_G＝λ₁L₁+λ_pL_perc+λ_sL_style+λ_gL_g

wherein L is_GTo the total loss function of the generator, L₁Is L₁Loss function, λ₁Is L₁Weight parameter of the loss function, L_percAs a function of perceptual loss, λ_pAs weight parameter of the perceptual loss function, L_styleAs a function of the loss of style, λ_sAs a weight parameter of the style loss function, L_gTo generate a generator opposition loss function, λ_gA weight parameter for the generator counter loss function;

the generator fighting loss function L_gThe following were used:

wherein G denotes a generator, D denotes a discriminator, I_inRepresenting a sequence of loss event images, P_data(I_in) Representing the distribution of the loss event image sequence, E [. + ]]Expected value, G (I), representing distribution function_in) Sequence of padded event images, log D, representing the generator output_s(G(I_in) Log D) represents the probability that the event sequence discriminator discriminated the padded event image as an unreduced event image_f(G(I_in) Represents the probability that the event frame discriminator discriminated the shim event image as an unreduced event image;

said L₁Loss function L₁The following were used:

wherein, I_gtRepresenting a sequence of lossless event images, I_predA sequence of shim event images representing the generator output;

the perceptual loss function L_percThe following were used:

wherein phi is_jIs the activation map of the jth layer of the pre-trained VGG-19 network, phi_j(I_gt) Representing the corresponding activation graph sequence obtained after the non-loss event image sequence is input into the j layer of the VGG-19 network_j(I_pred) Representing a corresponding activation graph sequence obtained after the filling event image sequence is input into a j layer of a VGG-19 network; n is a radical of_jRepresenting the number of characteristic channels of a j-th network in the VGG-19 network;

the style loss function L_styleThe following were used:

wherein,

is based on an activation map phi_jC of construction_j×C_jThe matrix of the Gram is a matrix of,

representing a plurality of Gram matrices constructed from a sequence of activation maps corresponding to a sequence of non-lost event images,

a plurality of Gram matrices constructed from the sequence of activation maps corresponding to the sequence of shim event images is represented.

Images