CN114885144A - High frame rate 3D video generation method and device based on data fusion - Google Patents

Abstract

The application discloses a high frame rate 3D video generation method and device based on data fusion, wherein the method comprises the following steps: acquiring video and event data lower than a preset frame rate from an event camera, combining the video and the event data in pairs to generate a plurality of groups of adjacent image frames, calculating to obtain a timestamp set of all intermediate frames, intercepting event streams from two boundary frames to expected intermediate frames, inputting the event streams into a preset impulse neural network for forward propagation to obtain an event stream data feature vector, splicing the event stream data feature vector with the adjacent image frames, inputting the event stream data feature vector into a preset multi-mode fusion network for forward propagation to obtain all intermediate frames, generating high frame rate video higher than a second preset frame rate, and performing forward propagation by using a preset 3D depth estimation network to obtain all high frame rate depth maps, thereby forming the high frame rate 3D video. Therefore, the technical problem that the generated image quality is low due to the fact that the initial brightness value of each pixel point is lacked because only the event stream is used as input in the related technology is solved.

Description

Method and device for generating high frame rate 3D video based on data fusion

technical field

The present application relates to the technical fields of computer vision and neuromorphic computing, and in particular, to a method and device for generating high frame rate 3D video based on data fusion.

Background technique

On the one hand, traditional cameras are limited by the frame rate, and the professional high-speed cameras required to shoot high-frame-rate videos are extremely expensive; There are certain defects in realizing high-speed 3D observation.

Related technologies use pure event streams to generate videos, convert event streams into grid-like tensor representations in a stacking manner, and use deep learning methods to generate images to achieve high-speed 3D observation.

However, the related art only uses the event stream as input, lacks the initial brightness value of each pixel, and only relies on the brightness change record to estimate the brightness is an underdetermined problem, which in turn leads to low quality of the generated image, which needs to be improved.

SUMMARY OF THE INVENTION

The present application provides a method and device for generating high frame rate 3D video based on data fusion, so as to solve the problem that the related art only uses event stream as input, lacks the initial brightness value of each pixel point, thus resulting in low quality of the generated image. technical problem.

The embodiment of the first aspect of the present application provides a high frame rate 3D video generation method based on data fusion, including the following steps: acquiring video and event data with a frame rate lower than a preset frame rate from an event camera; The frames are combined in pairs to generate multiple groups of adjacent image frames, and calculate the timestamp set of all intermediate frames expected to be obtained; intercept the first event stream and the second event stream from the two boundary frames to the expected intermediate frame according to the timestamp set. event stream, and input the first event stream and the second event stream into a preset spiking neural network for forward propagation, and obtain the first event stream data feature vector and the second event stream data feature vector; splicing the phase The adjacent image frames, the first event stream data feature vector and the second event stream data feature vector are input to the preset multi-modal fusion network for forward propagation, and all intermediate frames are obtained. A high frame rate video with a preset frame rate; based on the high frame rate video, forward propagation is performed using a preset 3D depth estimation network to obtain all high frame rate depth maps, and combine all the high frame rate depth maps, Compose high frame rate 3D video.

Optionally, in an embodiment of the present application, before inputting the first event stream and the second event stream into the preset spiking neural network for forward propagation, the method further includes: based on the Spike Response model as a Neuron dynamics model to construct the spiking neural network.

Optionally, in an embodiment of the present application, the multi-modal fusion network includes a coarse synthesis sub-network and a fine-tuning sub-network, wherein the coarse synthesis sub-network uses the first U-Net structure, and the input of the input layer The number of channels is 64+2×k, the number of output channels of the output layer is k, and the fine-tuning sub-network uses the second U-Net structure, the number of input channels of the input layer is 3×k, and the number of output channels of the output layer is k, k is the channel number of the image frame of the video whose frame rate is lower than the preset frame rate.

Optionally, in an embodiment of the present application, the 3D depth estimation network uses a third U-Net structure, and the number of input channels of the input layer is 3×k, and the number of output channels of the output layer is 1.

Optionally, in an embodiment of the present application, the calculation formula of the timestamp set of all intermediate frames is:

Among them, N is the total number of frames of the input low frame rate video, n is the multiple of the expected frame rate increase, and t _j is the timestamp of the jth frame of the input low frame rate video.

An embodiment of the second aspect of the present application provides a high frame rate 3D video generation device based on data fusion, including: a first acquisition module, configured to acquire video and event data with a frame rate lower than a preset frame rate from an event camera; a calculation module, For combining adjacent image frames in the video in pairs, generating multiple groups of adjacent image frames, and calculating a set of timestamps expected to obtain all intermediate frames; a second acquisition module, used for intercepting according to the set of timestamps From the two boundary frames to the first event stream and the second event stream of the expected intermediate frame, and input the first event stream and the second event stream to the preset spiking neural network for forward propagation to obtain the first event The stream data feature vector and the second event stream data feature vector; the fusion module is used for splicing the adjacent image frames, the first event stream data feature vector and the second event stream data feature vector, and input to the pre- The set multi-modal fusion network performs forward propagation, obtains all intermediate frames, and generates a high frame rate video higher than the second preset frame rate; the generation module is used for using the preset 3D video based on the high frame rate video. The depth estimation network performs forward propagation to obtain all high frame rate depth maps, and combines all the high frame rate depth maps to form a high frame rate 3D video.

Optionally, in an embodiment of the present application, it further includes: a building module for building the spiking neural network based on the Spike Response model as a neuron dynamics model.

Optionally, in an embodiment of the present application, the multi-modal fusion network includes a coarse synthesis sub-network and a fine-tuning sub-network, wherein the coarse synthesis sub-network uses the first U-Net structure, and the input of the input layer The number of channels is 64+2×k, the number of output channels of the output layer is k, and the fine-tuning sub-network uses the second U-Net structure, the number of input channels of the input layer is 3×k, and the number of output channels of the output layer is k, k is the channel number of the image frame of the video whose frame rate is lower than the preset frame rate.

Optionally, in an embodiment of the present application, the 3D depth estimation network uses a third U-Net structure, and the number of input channels of the input layer is 3×k, and the number of output channels of the output layer is 1.

Optionally, in an embodiment of the present application, the calculation formulas of the first event flow and the second event flow are:

Among them, N is the total number of frames of the input low frame rate video, n is the multiple of the expected frame rate increase, and t _j is the timestamp of the jth frame of the input low frame rate video.

An embodiment of a third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the program to achieve The method for generating high frame rate 3D video based on data fusion as described in the above embodiments.

Embodiments of the fourth aspect of the present application provide a computer-readable storage medium, where the computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute the high-speed data fusion-based high-level data fusion described in the foregoing embodiments. Frame rate 3D video generation method.

In this embodiment of the present application, event data can be used to provide inter-frame motion information, a spiking neural network can be used to encode the event stream, all intermediate frames can be obtained through a multi-modal fusion network, and a high-frame-rate video can be generated, and then a 3D depth estimation network can be used to form a high-speed video. Frame rate 3D video enables effective stereoscopic observation of high-speed scenes. By using event streams and low frame rate video image frames as input, multi-modal data information can be better used, thereby improving the quality of high frame rate 3D video. As a result, the technical problem in the related art that only the event stream is used as an input and the initial brightness value of each pixel is lacking, thus resulting in a lower quality of the generated image, is solved.

Additional aspects and advantages of the present application will be set forth, in part, in the following description, and in part will be apparent from the following description, or learned by practice of the present application.

Description of drawings

The above and/or additional aspects and advantages of the present application will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

1 is a flowchart of a method for generating a high frame rate 3D video based on data fusion provided according to an embodiment of the present application;

2 is a flowchart of a method for generating a high frame rate 3D video based on data fusion according to an embodiment of the present application;

3 is a schematic diagram of low frame rate video data and event stream data of a high frame rate 3D video generation method based on data fusion according to an embodiment of the present application;

4 is a schematic diagram of intermediate frame video data of a method for generating high frame rate 3D video based on data fusion according to an embodiment of the present application;

5 is a schematic diagram of an input event stream, a low frame rate video, and the generated high frame rate video data of a method for generating a high frame rate 3D video based on data fusion according to an embodiment of the present application;

6 is a high frame rate depth map under a 10-fold frame rate improvement of a method for generating a high frame rate 3D video based on data fusion according to an embodiment of the present application;

7 is a schematic structural diagram of a device for generating high frame rate 3D video based on data fusion provided according to an embodiment of the present application;

FIG. 8 is a schematic structural diagram of an electronic device provided according to an embodiment of the present application

Detailed ways

The following describes in detail the embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to be used to explain the present application, but should not be construed as a limitation to the present application.

The following describes the method and apparatus for generating a high frame rate 3D video based on data fusion according to the embodiments of the present application with reference to the accompanying drawings. Aiming at the technical problem that the related art mentioned in the above-mentioned Background Art Center only uses the event stream as input and lacks the initial brightness value of each pixel, thus resulting in low quality of the generated image, the present application provides a data fusion-based A high frame rate 3D video generation method, in which event data can be used to provide inter-frame motion information, a spiking neural network is used to encode the event stream, and all intermediate frames are obtained through a multi-modal fusion network to generate a high frame rate video , and then use the 3D depth estimation network to form high frame rate 3D video to achieve effective stereoscopic observation for high-speed scenes. By using event streams and low frame rate video image frames as input, multi-modal data information can be better used, and then Improve the quality of high frame rate 3D videos. As a result, the technical problem in the related art that only the event stream is used as an input and the initial brightness value of each pixel is lacking, thus resulting in a lower quality of the generated image, is solved.

Specifically, FIG. 1 is a schematic flowchart of a method for generating a high frame rate 3D video based on data fusion provided by an embodiment of the present application.

As shown in Figure 1, the high frame rate 3D video generation method based on data fusion includes the following steps:

In step S101, video and event data below a preset frame rate are acquired from the event camera.

In the actual execution process, the embodiment of the present application can acquire video and event data with a lower frame rate than a preset frame rate from the event camera, realize the acquisition of raw data, and lay a data foundation for the subsequent generation of high frame rate video.

Understandably, an event camera is a bio-inspired sensor that works very differently from a traditional camera. Unlike a traditional camera that captures the absolute light intensity of a scene at a fixed frame rate, an event camera only changes when the scene light intensity changes. Compared with traditional cameras, event cameras have the advantages of high dynamic range, high temporal resolution, and no motion blur, which is conducive to ensuring the generation of high frame rate video.

As a new type of visual sensor, the event camera cannot directly apply various algorithms of traditional cameras and images. The event camera has no concept of frame rate. Each pixel of the event camera works asynchronously. When a change in light intensity is detected, an event is output. The event is a quadruple (x, y, t, p), including the horizontal and vertical coordinates of the pixel (x, y), the timestamp t and the event polarity p (where, p=-1 indicates that the light intensity of the pixel decreases , p=1 indicates that the light intensity of the pixel increases), and the event data output by all the pixels is aggregated to form an event list composed of events as the event stream data output by the camera. The event camera has no concept of frame rate, and each pixel works asynchronously, and outputs an event when a change in light intensity is detected. The event data output by all pixel points is aggregated to form an event list composed of several events, which is used as the event stream data output by the camera.

The preset frame rate can be set correspondingly by those skilled in the art, which is not specifically limited here.

In step S102, adjacent image frames in the video are combined in pairs to generate multiple groups of adjacent image frames, and a set of time stamps of all intermediate frames expected to be obtained is calculated.

As a possible implementation method, in this embodiment of the present application, adjacent image frames in a low frame rate video can be combined in pairs to generate multiple groups of adjacent image frames, and for each group of adjacent image frames, the calculation is expected to obtain The timestamp set T of all intermediate frames, denoted as:

T={τ ¹ _1,2 ,τ ² _1,2 ,...,τ ⁿ _1,2 ,τ ¹ _2,3 ,τ ² _2,3 ,...,τ ⁿ _2,3 ,... ,τ ¹ _N-1,N ,τ ² _N-1,N ,...,τ ⁿ _N-1,N }.

Optionally, in an embodiment of the present application, the calculation formula of the timestamp set of all intermediate frames is:

Among them, N is the total number of frames of the input low frame rate video, n is the multiple of the expected frame rate increase, and t _j is the timestamp of the jth frame of the input low frame rate video.

Specifically, the calculation formula for obtaining the timestamps of all intermediate frames may be as follows:

Among them, N is the total number of frames of the input low frame rate video, n is the multiple of the expected frame rate improvement, and t _j is the timestamp of the jth frame of the input low frame rate video.

In the embodiment of the present application, the set of timestamps of all intermediate frames can be expected to be obtained by calculation, so as to realize the preprocessing of the data, and provide a basis for the subsequent data fusion.

In step S103, intercept the first event stream and the second event stream from the two boundary frames to the desired intermediate frame according to the timestamp set, and input the first event stream and the second event stream into a preset spiking neural network for Forward propagation to obtain the first event stream data feature vector and the second event stream data feature vector.

Further, this embodiment of the present application may intercept the first event stream ε ₁ and the second event stream ε ₂ from the two boundary frames to the expected intermediate frame according to the intermediate frame timestamp set calculated in step S102 , and convert the first event stream ε 1 to the desired intermediate frame. The event stream and the second event stream are input to the preset spiking neural network for forward propagation to obtain the first event stream data feature vector F ₁ and the second event stream data feature vector F ₂ . By using the spiking neural network to encode the event stream in the embodiments of the present application, the effect of denoising the event stream data can be better achieved, thereby improving the quality of the generated video.

Wherein, the calculation formulas of the first event flow ε ₁ and the second event flow ε ₂ can be respectively as follows:

Among them, τ ⁱ _j,j+1 is the time stamp of the desired intermediate frame, and t _j and t _j+1 are the time stamps of the adjacent input low frame rate video frames of the desired intermediate frame.

It should be noted that the preset spiking neural network will be described in detail below.

Optionally, in an embodiment of the present application, before inputting the first event stream and the second event stream into a preset spiking neural network for forward propagation, the method further includes: using a Spike Response model as a neuron dynamics model to build a spiking neural network.

The spiking neural network is described in detail here.

It is understandable that the spiking neural network is the third generation of artificial neural network. The neurons in the spiking neural network are not activated in each iterative propagation, but are activated when their membrane potential reaches a certain value. When a neuron is activated, the spiking neural network will generate a signal that is transmitted to other neurons, increasing or decreasing its membrane potential, so the simulated neurons of the spiking neural network are closer to reality and are more suitable for processing timing pulse signals.

In the actual execution process, the embodiment of the present application may use the Spike Response model as a neuron dynamics model to construct a spiking convolutional neural network.

Specifically, a spiking neural network can include an input convolutional layer, a hidden convolutional layer, and an output convolutional layer. Among them, the number of input channels of the input convolution layer is 2, corresponding to the positive events and negative events of the event stream, the size of the convolution kernel is 3 × 3, the stride is 1, and the number of output channels is 16; the hidden convolution layer The number of input channels is 16, the size of the convolution kernel is 3 × 3, the stride is 1, and the number of output channels is 16; the number of input channels of the output convolution layer is 16, the size of the convolution kernel is 3 × 3, the step The length is 1, and the number of output channels is 32.

In step S104, the adjacent image frames, the first event stream data feature vector and the second event stream data feature vector are spliced, and input to a preset multi-modal fusion network for forward propagation to obtain all intermediate frames, and generate higher than High frame rate video of the second preset frame rate.

As a possible implementation manner, in this embodiment of the present application, the adjacent image frames of the low frame rate video obtained from step S102 and the first event stream data feature vector F ₁ and the second event stream data feature obtained from step S103 can be _The vector F2 is spliced and input to the preset multi-modal fusion network for forward propagation, and an intermediate frame is generated to complete the calculation of a single high frame rate image frame.

Specifically, in the embodiment of the present application, the adjacent image frames of the low frame rate video and the event stream data feature vectors F ₁ and F ₂ can be spliced together, and input into the coarse synthesis sub-network to obtain a coarse output result; then the coarse output result is combined with The input adjacent image frames are stitched together and input to the fine-tuning sub-network to get the final output.

Further, in this embodiment of the present application, the above steps may be repeated for the timestamp of each expected intermediate frame calculated in step S102 to complete the calculation of all intermediate frames, thereby generating a high frame rate video higher than the second preset frame rate.

It should be noted that the preset multimodal fusion network will be described in detail below.

Optionally, in an embodiment of the present application, the multimodal fusion network includes a coarse synthesis sub-network and a fine-tuning sub-network, wherein the coarse synthesis sub-network uses the first U-Net structure, and the number of input channels of the input layer is 64. +2×k, the number of output channels of the output layer is k, and the fine-tuning sub-network uses the second U-Net structure, the number of input channels of the input layer is 3×k, the number of output channels of the output layer is k, and k is lower than The number of channels of the image frame of the video with the preset frame rate.

The multimodal fusion network is elaborated here.

Understandably, the data fusion network consists of a coarse synthesis sub-network and a fine-tuned sub-network. Among them, the coarse synthesis sub-network uses the first U-Net structure, the number of input channels of the input layer is 64+2×k, and the number of output channels of the output layer is k; the fine-tuning sub-network uses the second U-Net structure, and the number of input channels of the input layer is k The number of input channels is 3×k, and the number of output channels of the output layer is k.

Among them, k is the channel number of the image frame of the low frame rate video input in step S101, that is, when the image frame of the low frame rate video input in step S101 is a grayscale image, k=1, when the input in step S101 When the image frame of the low frame rate video is an RGB image, k=3.

In step S105, based on the high frame rate video, forward propagation is performed using a preset 3D depth estimation network to obtain all high frame rate depth maps, and all high frame rate depth maps are combined to form a high frame rate 3D video.

In the actual execution process, the embodiment of the present application can splicing the high frame rate image frame obtained in the above steps with the adjacent high frame rate image frames before and after, and use the preset 3D depth estimation network for forward propagation to generate a A series of high frame rate depth maps, and the generated series of high frame rate depth maps are combined to form a high frame rate 3D video to achieve high frame rate 3D video generation. In this embodiment of the present application, event data can be used to provide inter-frame motion information, a spiking neural network can be used to encode the event stream, all intermediate frames can be obtained through a multi-modal fusion network, and a high-frame-rate video can be generated, and then a 3D depth estimation network can be used to form a high-speed video. Frame rate 3D video enables effective stereoscopic observation of high-speed scenes. By using event streams and low frame rate video image frames as input, multi-modal data information can be better used, thereby improving the quality of high frame rate 3D video.

Optionally, in an embodiment of the present application, the 3D depth estimation network uses a third U-Net structure, and the number of input channels of the input layer is 3×k, and the number of output channels of the output layer is 1.

The construction of the 3D depth estimation network is elaborated here.

Specifically, the 3D depth estimation network constructed in the embodiment of the present application can use the third U-Net structure, the number of input channels of the input layer is 3×k, and the number of output channels of the output layer is 1, where k is the input in step S101 The number of channels of the image frame of the low frame rate video, that is, when the image frame of the low frame rate video input in step S101 is a grayscale image, k=1, and when the image frame of the low frame rate video input in step S101 is For RGB images, k=3.

Hereinafter, with reference to FIGS. 2 to 7 , the embodiments of the present application will be described in detail with an embodiment. As shown in Figure 2, the embodiment of the present application includes the following steps:

Step S201 : acquiring low frame rate video data and event stream data. In the actual execution process, the embodiments of the present application can acquire video and event data at a frame rate from an event camera, realize the acquisition of original data, and lay a data foundation for subsequent generation of high-frame rate videos.

It can be understood that the event camera does not have the concept of frame rate, and each pixel of it works asynchronously. When a change in light intensity is detected, an event is output, and each event is a quadruple (x, y, t, p), Including the horizontal and vertical coordinates of the pixel (x, y), the time stamp t and the event polarity p (where, p=-1 indicates that the light intensity of the pixel decreases, and p=1 indicates that the light intensity of the pixel increases). The event data output by the pixel points can be aggregated to form an event list composed of events, which can be used as the event stream data output by the camera.

For example, as shown in FIG. 3 , the frame rate of the low frame rate video obtained from the event camera in this embodiment of the present application may be 20 FPS (Frames Per Second, the number of frames transmitted per second), a total of 31 frames, and the corresponding event stream continues The time is 1500ms.

Step S202: data preprocessing. In this embodiment of the present application, adjacent image frames in the low frame rate video may be combined in pairs, and for each group of adjacent image frames, the time stamp set T of all intermediate frames expected to be obtained is calculated as:

T={τ ¹ _1,2 ,τ ² _1,2 ,...,τ ⁿ _1,2 ,τ ¹ _2,3 ,τ ² _2,3 ,...,τ ⁿ _2,3 ,... ,τ ¹ _N-1,N ,τ ² _N-1,N ,...,τ ⁿ _N-1,N },

Among them, the calculation formula of each expected intermediate frame timestamp is as follows:

Among them, N is the total number of frames of the input low frame rate video, n is the multiple of the expected frame rate improvement, and t _j is the timestamp of the jth frame of the input low frame rate video.

For example, the input low frame rate video in this embodiment of the present application may include N=31 frames of images and the frame rate is 20FPS, and the timestamp of the jth frame of the input low frame rate video is t _j =(j-1)×50ms. If a high frame rate video with a frame rate increase of n=10 times is obtained, the calculated timestamp set of all intermediate frames can be T={0,5,10,15,20,...,1495}, including 300 elements.

Step S203: constructing a spiking neural network. In the actual execution process, the embodiment of the present application may use the SpikeResponse model as a neuron dynamics model to construct a spiking convolutional neural network.

Specifically, a spiking neural network can include an input convolutional layer, a hidden convolutional layer, and an output convolutional layer. Among them, the number of input channels of the input convolution layer is 2, corresponding to the positive events and negative events of the event stream, the size of the convolution kernel is 3 × 3, the stride is 1, and the number of output channels is 16; the hidden convolution layer The number of input channels is 16, the size of the convolution kernel is 3 × 3, the stride is 1, and the number of output channels is 16; the number of input channels of the output convolution layer is 16, the size of the convolution kernel is 3 × 3, the step The length is 1, and the number of output channels is 32.

Step S204: Event stream encoding calculation. In this embodiment of the present application, according to the timestamp τ ⁱ _j,j+1 of the intermediate frame calculated in step S202 , intercept the event streams ε ₁ , ε ₂ from the two boundary frames to the expected intermediate frame, and combine ε ₁ , ε ₂ The spiking neural network obtained in step S203 is respectively input to carry out forward propagation to obtain event stream data feature vectors F ₁ and F ₂ .

Among them, the calculation formulas of event streams ε ₁ and ε ₂ from two boundary frames to expected intermediate frames are as follows:

where τ ⁱ _j,j+1 is the timestamp of the expected intermediate frame, and t _j and t _j+1 are the timestamps of the input low frame rate video frames adjacent to the expected intermediate frame.

For example, taking the timestamp of the 15th expected intermediate frame, that is, the 5th frame inserted in the 2nd and 3rd frames of the input low frame rate video in the embodiment of the present application, τ ⁵ _2,3 =75ms is For example, event streams ε ₁ and ε ₂ from two boundary frames to desired intermediate frames can be as shown in Table 1. Among them, Table 1 and Table 2 are the data tables of event streams ε ₁ and ε ₂ , respectively.

Table I

Step S205: building a multimodal fusion network. Understandably, the data fusion network consists of a coarse synthesis sub-network and a fine-tuned sub-network. Among them, the coarse synthesis sub-network uses the U-Net structure, the number of input channels of the input layer is 64+2×k, and the number of output channels of the output layer is k; the fine-tuning sub-network uses the U-Net structure, and the number of input channels of the input layer is 3×k, the number of output channels of the output layer is k.

Wherein, k is the channel number of the image frame of the low frame rate video input in step S201, that is, when the image frame of the low frame rate video input in step S201 is a grayscale image, k=1, when the input in step S201 When the image frame of the low frame rate video is an RGB image, k=3.

For example, in this embodiment of the present application, the image frame of the low frame rate video input in step S201 can be input as a grayscale image, that is, k=1. The number of output channels is 1; the number of input channels of the fine-tuning sub-network input layer is 3; the number of output channels of the output layer is 1.

Step S206: Single high frame rate image frame calculation. As a possible implementation manner, in this embodiment of the present application, the adjacent image frames of the low frame rate video obtained from step S202 and the first event stream data feature vector F ₁ and the second event stream data feature obtained from step S203 can be _The vector F2 is spliced and input to the preset multi-modal fusion network for forward propagation, and an intermediate frame is generated to complete the calculation of a single high frame rate image frame.

Specifically, in the embodiment of the present application, the adjacent image frames of the low frame rate video and the event stream data feature vectors F ₁ and F ₂ can be spliced together, and input into the coarse synthesis sub-network to obtain a coarse output result; then the coarse output result is combined with The input adjacent image frames are stitched together and input to the fine-tuning sub-network to get the final output.

For example, taking the 15th expected intermediate frame as an example, the generated intermediate frame is shown in FIG. 4 .

Step S207: Calculate all high frame rate image frames. Further, in this embodiment of the present application, the above steps S302 to S306 may be repeated for the timestamp of each expected intermediate frame calculated in step S302 to complete the calculation of all intermediate frames.

For example, in this embodiment of the present application, a low frame rate video can be input including N=31 frames of images. If a high frame rate video with a frame rate increased by n=10 times is obtained, steps S202 to S206 need to be repeated a total of 300 times.

In this embodiment of the present application, all intermediate frames obtained in step S207 are combined to form a high frame rate video, so as to realize the generation of a high frame rate video.

Wherein, taking obtaining a high frame rate video with a frame rate increase of n=10 times as an example, the input event stream, the low frame rate video and the generated high frame rate video may be as shown in FIG. 5 .

Step S208: 3D depth estimation network construction. Specifically, the 3D depth estimation network constructed in the embodiment of the present application can use the third U-Net structure, the number of input channels of the input layer is 3×k, and the number of output channels of the output layer is 1, where k is the input in step S201 The number of channels of the image frame of the low frame rate video, that is, when the image frame of the low frame rate video input in step S201 is a grayscale image, k=1, and when the image frame of the low frame rate video input in step S201 is For RGB images, k=3.

Step S209: high frame rate 3D depth estimation calculation.

Step S210: data post-processing. In the actual execution process, the embodiment of the present application can splicing the high frame rate image frame obtained in the above steps with the adjacent high frame rate image frames before and after, and use the preset 3D depth estimation network for forward propagation to generate a A series of high frame rate depth maps, and the generated series of high frame rate depth maps are combined to form a high frame rate 3D video to achieve high frame rate 3D video generation.

For example, as shown in FIG. 6 , the embodiment of the present application can realize the generation of a high frame rate depth map video with a frame rate increase of 10 times, and realize effective stereoscopic scene observation in a high-speed environment.

According to the method for generating high frame rate 3D video based on data fusion proposed in the embodiment of the present application, event data can be used to provide inter-frame motion information, a spiking neural network can be used to encode the event stream, and all intermediate frames can be obtained through a multi-modal fusion network. , generate a high frame rate video, and then use the 3D depth estimation network to form a high frame rate 3D video to achieve effective stereo observation for high-speed scenes. By using event streams and low frame rate video image frames as input, you can better use Modal data information, thereby improving the quality of high frame rate 3D video. As a result, the technical problem in the related art that only the event stream is used as an input and the initial brightness value of each pixel is lacking, thus resulting in a lower quality of the generated image, is solved.

Next, the device for generating high frame rate 3D video based on data fusion proposed according to the embodiments of the present application will be described with reference to the accompanying drawings.

FIG. 7 is a schematic block diagram of an apparatus for generating a high frame rate 3D video based on data fusion according to an embodiment of the present application.

As shown in FIG. 7 , the high frame rate 3D video generation device 10 based on data fusion includes: a first acquisition module 100 , a calculation module 200 , a second acquisition module 300 , a fusion module 400 and a generation module 500 .

Specifically, the first obtaining module 100 is configured to obtain video and event data with a frame rate lower than a preset frame rate from the event camera.

The calculation module 200 is configured to combine adjacent image frames in the video in pairs to generate multiple groups of adjacent image frames, and calculate a set of timestamps expected to obtain all intermediate frames.

The second acquiring module 300 is configured to intercept the first event stream and the second event stream from the two boundary frames to the desired intermediate frame according to the timestamp set, and input the first event stream and the second event stream to the preset pulse The neural network performs forward propagation to obtain the first event stream data feature vector and the second event stream data feature vector.

The fusion module 400 is used for splicing adjacent image frames, the first event stream data feature vector and the second event stream data feature vector, and inputting them to a preset multi-modal fusion network for forward propagation, obtaining all intermediate frames, and generating High frame rate video above the second preset frame rate.

The generating module 500 is used for forward propagation based on the high frame rate video using a preset 3D depth estimation network to obtain all high frame rate depth maps, and combining all the high frame rate depth maps to form a high frame rate 3D video.

Optionally, in an embodiment of the present application, the apparatus 10 for generating a high frame rate 3D video based on data fusion further includes: a building block.

Among them, the building block is used to build a spiking neural network based on the Spike Response model as a neuron dynamics model.

Optionally, in an embodiment of the present application, the multimodal fusion network includes a coarse synthesis sub-network and a fine-tuning sub-network, wherein the coarse synthesis sub-network uses the first U-Net structure, and the number of input channels of the input layer is 64. +2×k, the number of output channels of the output layer is k, and the fine-tuning sub-network uses the second U-Net structure, the number of input channels of the input layer is 3×k, the number of output channels of the output layer is k, and k is lower than The number of channels of the image frame of the video with the preset frame rate.

Optionally, in an embodiment of the present application, the 3D depth estimation network uses a third U-Net structure, and the number of input channels of the input layer is 3×k, and the number of output channels of the output layer is 1.

Optionally, in an embodiment of the present application, the calculation formula of the timestamp set of all intermediate frames is:

Among them, N is the total number of frames of the input low frame rate video, n is the multiple of the expected frame rate increase, and t _j is the timestamp of the jth frame of the input low frame rate video.

It should be noted that, the foregoing explanations of the embodiment of the method for generating high frame rate 3D video based on data fusion are also applicable to the apparatus for generating high frame rate 3D video based on data fusion in this embodiment, which will not be repeated here.

According to the high frame rate 3D video generation device based on data fusion proposed in the embodiment of the present application, the event data can be used to provide inter-frame motion information, the spiking neural network can be used to encode the event stream, and all intermediate frames can be obtained through the multi-modal fusion network. , generate a high frame rate video, and then use the 3D depth estimation network to form a high frame rate 3D video to achieve effective stereo observation for high-speed scenes. By using event streams and low frame rate video image frames as input, you can better use Modal data information, thereby improving the quality of high frame rate 3D video. As a result, the technical problem in the related art that only the event stream is used as an input and the initial brightness value of each pixel is lacking, thus resulting in a lower quality of the generated image, is solved.

FIG. 8 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device may include:

Memory 801 , processor 802 , and computer programs stored on memory 801 and executable on processor 802 .

When the processor 802 executes the program, the method for generating a high frame rate 3D video based on data fusion provided in the above embodiments is implemented.

Further, the electronic device also includes:

The communication interface 803 is used for communication between the memory 801 and the processor 802 .

The memory 801 is used to store computer programs that can be executed on the processor 802 .

The memory 801 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

If the memory 801, the processor 802 and the communication interface 803 are independently implemented, the communication interface 803, the memory 801 and the processor 802 can be connected to each other through a bus and complete communication with each other. The bus may be an Industry Standard Architecture (referred to as ISA) bus, a Peripheral Component (referred to as PCI) bus, or an Extended Industry Standard Architecture (referred to as EISA) bus or the like. The bus can be divided into address bus, data bus, control bus and so on. For convenience of representation, only one thick line is used in FIG. 8, but it does not mean that there is only one bus or one type of bus.

Optionally, in terms of specific implementation, if the memory 801, the processor 802 and the communication interface 803 are integrated on one chip, the memory 801, the processor 802 and the communication interface 803 can communicate with each other through the internal interface.

The processor 802 may be a central processing unit (Central Processing Unit, CPU for short), or a specific integrated circuit (Application Specific Integrated Circuit, ASIC for short), or is configured to implement one or more of the embodiments of the present application integrated circuit.

This embodiment also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the above method for generating a high frame rate 3D video based on data fusion.

In the description of this specification, description with reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc., mean specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or N of the embodiments or examples. Furthermore, those skilled in the art may combine and combine the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples, without conflicting each other.

In addition, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In the description of the present application, "N" means at least two, such as two, three, etc., unless otherwise expressly and specifically defined.

Any process or method description in the flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or N more executable instructions for implementing custom logical functions or steps of the process , and the scope of the preferred embodiments of the present application includes alternative implementations in which the functions may be performed out of the order shown or discussed, including performing the functions substantially concurrently or in the reverse order depending upon the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present application belong.

The logic and/or steps represented in flowcharts or otherwise described herein, for example, may be considered an ordered listing of executable instructions for implementing the logical functions, may be embodied in any computer-readable medium, For use with, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions from an instruction execution system, apparatus, or apparatus) or equipment. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or apparatus. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections (electronic devices) with one or N wires, portable computer disk cartridges (magnetic devices), random access memory (RAM), Read Only Memory (ROM), Erasable Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program may be printed, as the paper or other medium may be optically scanned, for example, followed by editing, interpretation, or other suitable medium as necessary process to obtain the program electronically and then store it in computer memory.

It should be understood that various parts of this application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the N steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented by any one of the following techniques known in the art, or a combination thereof: discrete with logic gates for implementing logic functions on data signals Logic circuits, application specific integrated circuits with suitable combinational logic gates, Programmable Gate Arrays (PGA), Field Programmable Gate Arrays (FPGA), etc.

Those skilled in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the program can be stored in a computer-readable storage medium. When executed, one or a combination of the steps of the method embodiment is included.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, and the like. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limitations to the present application. Embodiments are subject to variations, modifications, substitutions and variations.

Claims (10)

1. a high frame rate 3D video generation method based on data fusion, is characterized in that, comprises the following steps: Get video and event data below the preset frame rate from the event camera; Combining adjacent image frames in the video in pairs, generating multiple groups of adjacent image frames, and calculating the time stamp sets expected to obtain all intermediate frames; Intercept the first event stream and the second event stream from the two boundary frames to the desired intermediate frame according to the timestamp set, and input the first event stream and the second event stream into the preset spiking neural network for preprocessing Propagating in the direction to obtain the first event stream data feature vector and the second event stream data feature vector; Splicing the adjacent image frames, the first event stream data feature vector and the second event stream data feature vector, and inputting them to a preset multi-modal fusion network for forward propagation, obtaining all intermediate frames, and generating High frame rate video higher than the second preset frame rate; Based on the high frame rate video, forward propagation is performed using a preset 3D depth estimation network to obtain all high frame rate depth maps, and all the high frame rate depth maps are combined to form a high frame rate 3D video. 2. The method according to claim 1, wherein before inputting the first event stream and the second event stream to the preset spiking neural network for forward propagation, the method further comprises: The spiking neural network is constructed based on the Spike Response model as the neuron dynamics model. 3. The method according to claim 1, wherein the multimodal fusion network comprises a coarse synthesis sub-network and a fine-tuning sub-network, wherein the coarse synthesis sub-network uses a first U-Net structure, and the input layer The number of input channels is 64+2×k, the number of output channels of the output layer is k, and the fine-tuning sub-network uses the second U-Net structure, the number of input channels of the input layer is 3×k, and the number of output channels of the output layer is k. The number is k, where k is the channel number of the image frame of the video whose frame rate is lower than the preset frame rate. 4 . The method according to claim 1 , wherein the 3D depth estimation network uses a third U-Net structure, and the number of input channels of the input layer is 3×k, and the number of output channels of the output layer is 1. 5 . 5. The method according to any one of claims 1-4, wherein the calculation formula of the time stamp sets of all intermediate frames is:

Among them, N is the total number of frames of the input low frame rate video, n is the multiple of the expected frame rate increase, and t _j is the timestamp of the jth frame of the input low frame rate video. 6. A high frame rate 3D video generation device based on data fusion, is characterized in that, comprising: The first acquisition module is used to acquire video and event data below the preset frame rate from the event camera; A calculation module, for combining adjacent image frames in the video in pairs, generating multiple groups of adjacent image frames, and calculating the time stamp sets expected to obtain all intermediate frames; The second acquiring module is configured to intercept the first event stream and the second event stream from the two boundary frames to the expected intermediate frame according to the timestamp set, and input the first event stream and the second event stream to the pre- The set spiking neural network performs forward propagation to obtain the first event stream data feature vector and the second event stream data feature vector; The fusion module is used for splicing the adjacent image frames, the first event stream data feature vector and the second event stream data feature vector, and inputting them to a preset multi-modal fusion network for forward propagation to obtain All intermediate frames, generate a high frame rate video higher than the second preset frame rate; The generation module is used for forward propagation based on the high frame rate video using a preset 3D depth estimation network to obtain all high frame rate depth maps, and combining all the high frame rate depth maps to form a high frame rate 3D video. 7 . The apparatus according to claim 6 , further comprising: a building module for building the spiking neural network based on the SpikeResponse model as a neuron dynamics model. 8 . 8. The device according to any one of claims 6-7, wherein the calculation formula of the time stamp sets of all intermediate frames is:

Among them, N is the total number of frames of the input low frame rate video, n is the multiple of the expected frame rate increase, and t _j is the timestamp of the jth frame of the input low frame rate video. 9. An electronic device, characterized in that it comprises: a memory, a processor and a computer program stored on the memory and running on the processor, the processor executing the program to realize the method as claimed in the claim The method for generating high frame rate 3D video based on data fusion according to any one of requirements 1-5. 10. A computer-readable storage medium on which a computer program is stored, characterized in that the program is executed by a processor to implement the data fusion-based high frame as claimed in any one of claims 1-5 Rate 3D video generation method.

Images