CN114584713B - Pulse camera simulation method and device, control equipment and readable storage medium - Google Patents

Abstract

The application provides a pulse camera simulation method, a pulse camera simulation device, control equipment and a readable storage medium, wherein the method comprises the following steps: extracting a key frame sequence of a video to be simulated, and converting the key frame sequence into an intensity map sequence; increasing the frame rate of the intensity map sequence according to the working clock frequency of the pulse camera; acquiring an integral distribution model, and simulating the intensity map sequence according to the integral distribution model; and outputting the simulated pulse data. Therefore, the imaging process of the pulse camera is simulated, the existing image and video data are converted into pulse data, and the problem that the real pulse data are seriously deficient is solved.

Description

A pulse camera simulation method, device, control device and readable storage medium

technical field

The present application relates to the field of computer technology, and in particular, to a pulse camera simulation method, device, control device, and readable storage medium.

Background technique

The pulse camera adopts the sensing principle inspired by biological vision, has ultra-high temporal resolution and biologically inspired perception mechanism, breaks through the limitation of fixed exposure time of traditional cameras, and has stronger detection ability for high-speed moving targets. This feature makes the pulse camera a huge advantage in high-speed motion scenarios such as autonomous driving.

However, when the pulse camera is used in specific applications, it is not as a separate device to directly output the final result, but as a provider of high-speed images, providing pulse data for other modules to call. Therefore, in order to put the pulse camera into practical application, it is necessary to provide a large amount of real pulse data for simulation. At present, the real pulse data is seriously lacking. Therefore, how to develop a pulse camera simulation algorithm to convert the existing image and video data into pulse data is an urgent problem to be solved.

SUMMARY OF THE INVENTION

The problem solved by the present application is to convert the existing image and video data into pulse data, so as to solve the problem of lack of pulse data.

In order to solve the above problems, a first aspect of the present application provides a pulse camera simulation method, including:

Extract the key frame sequence of the video to be simulated, and convert the key frame sequence into an intensity map sequence;

Increase the frame rate of the intensity map sequence according to the working clock frequency of the pulse camera;

obtaining a point distribution model, and simulating the intensity map sequence according to the point distribution model;

The simulated pulse data is output.

Preferably, after the frame rate of the intensity map sequence is increased according to the working clock frequency of the pulse camera, the method further includes:

Brightness adjustment is performed by multiplying the intensity map sequence after the frame rate has been increased by a transform coefficient.

Preferably, the value range of the transform coefficient is (1-3)×10 ^-4 .

Preferably, in the simulation of the intensity map sequence according to the integral release model, a simulation time step and an excitation threshold are determined, each simulation time step is integrated by the integral release model, and when the integration result exceeds the excitation threshold A pulse is fired after the threshold, and the simulation is completed based on the fired pulse.

Preferably, in the conversion of the key frame sequence into an intensity map sequence, when the key frame sequence is a color image sequence, the key frame sequence is converted into intensity maps of R channel, G channel and B channel sequence; if the key frame sequence is a grayscale image sequence, convert the key frame sequence into a single-channel intensity map sequence.

Preferably, the point distribution model is:

In the model, C _pd is the conversion capacitance, λ is the conversion coefficient, N ₁ ( t ) is the Poisson model, G ( t ) is the luminance intensity of the intensity map sequence, t is the time step, and η is the linear coefficient.

Preferably, in the output of the simulated pulse data, the pulse data is encoded into binary pulse data and then output.

A second aspect of the present application provides a pulse camera simulation device, which includes:

a frame extraction module for extracting a key frame sequence of the video to be simulated, and converting the key frame sequence into an intensity map sequence;

a frame rate boosting module, configured to boost the frame rate of the intensity map sequence according to the working clock frequency of the pulse camera;

a simulation module, configured to obtain an integral distribution model, and simulate the intensity map sequence according to the integral distribution model;

The output module is used to output the simulated pulse data.

A third aspect of the present application provides an electronic device, characterized by comprising a computer-readable storage medium storing a computer program and a processor, and when the computer program is read and executed by the processor, the above-mentioned The pulse camera simulation method described above.

A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is read and executed by a processor, the above-mentioned Pulse camera simulation method.

A fifth aspect of the present application provides a computer program product, including a computer program, characterized in that, the computer program is executed by a processor to implement the above-mentioned pulse camera simulation method.

In this application, by simulating the imaging process of the pulse camera, the existing image and video data are converted into pulse data, which solves the problem of serious lack of real pulse data.

In the present application, by introducing transform coefficients, the luminance value of the intensity map sequence can be amplified, thereby making up for the problem of insufficient luminance of the video to be simulated.

In this application, a Poisson model is introduced into the integral release model to perform a noise addition operation, so as to load the noise caused by scattered light and increase the accuracy of the simulation.

Description of drawings

1 is an example diagram of an image of a control group and an image of an experimental group according to an embodiment of the present application;

2 is an example diagram of an image of a control group and an image of an experimental group according to another embodiment of the present application;

3 is a structural block diagram of a pulse camera simulation device according to an embodiment of the present application;

4 is a structural block diagram of a pulse camera simulation device according to another embodiment of the present application;

FIG. 5 is a structural block diagram of a control device according to an embodiment of the present application.

Detailed ways

In order to make the above objects, features and advantages of the present application more obvious and easy to understand, specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the application will be more thoroughly understood, and will fully convey the scope of the application to those skilled in the art.

It should be noted that, unless otherwise specified, the technical or scientific terms used in this application should have the usual meanings understood by those skilled in the art to which this application belongs.

Although the current field of machine vision has achieved great success, it still faces many challenges in high-speed scenarios. For example, in the process of autonomous driving, for a moving target that appears suddenly, the imaging mode of a traditional camera with a fixed exposure time interval causes imaging blur and a data delay of tens of milliseconds. For cars traveling at high speeds, this problem can have very dangerous consequences.

Pulse cameras use a sensing principle inspired by biological vision, where each pixel independently senses the intensity of light and converts it into a charge accumulation. Immediately after the accumulated charge exceeds the trigger threshold, the pixel fires a pulse and resets the voltage. The excited pulses are read out with a high-speed scan clock. With its ultra-high temporal resolution and bio-inspired perception mechanism, the pulse camera breaks through the limitation of fixed exposure time of traditional cameras, so it has a stronger detection capability for high-speed moving targets. This imaging principle has brought about a significant improvement in temporal resolution and dynamic range, making it easy to solve many problems faced by traditional cameras in high-speed motion scenes.

However, when the pulse camera is used in specific applications, it is not as a separate device to directly output the final result, but as a provider of high-speed images, providing pulse data for other modules to call. For example, after outputting the pulse data, the control module or the computing module analyzes it through the trained deep neural network model, and outputs the corresponding features, so as to make automatic driving decisions.

This way of use means that if you want to put the pulse camera into practical application, you need to provide a large amount of real pulse data for training the deep neural network model, so that it can be used in practice. However, due to the novelty and high manufacturing cost of pulse cameras, such cameras are currently difficult to popularize on a large scale. This leads to a serious lack of real pulse data. Existing large-scale image and video data are completely different in format from pulse data, thus greatly limiting the research and applications based on pulse cameras.

Under this circumstance, developing a pulse camera simulation algorithm to convert existing image and video data into pulse data is an urgent problem to be solved.

In view of the above problems, the present application provides a new pulse camera simulation solution, which can convert existing image and video data into pulse data, and solve the problem of serious shortage of real pulse data.

The embodiment of the present application provides a pulse camera simulation method, which can be executed by a pulse camera simulation device, and the pulse camera simulation device can be integrated in electronic equipment such as pads, computers, servers, computers, server clusters, and data centers. As shown in FIG. 1, it is a flowchart of a pulse camera simulation method according to an embodiment of the present application; wherein, the pulse camera simulation method includes:

S100, extracting a key frame sequence of the video to be simulated, and converting the key frame sequence into an intensity map sequence;

The video to be simulated may be a given video or a given image sequence.

In one embodiment, the video to be simulated is an image captured by an imaging device; or a key frame image extracted from a video; or an image generated from a synthesis software.

In an embodiment, when the video to be simulated is in a video format, an image sequence is obtained by extracting key frames of the video to be simulated, and the image sequence is a sequence of key frames.

In one embodiment, the sequence of key frames is a sequence of images at 60 frames per second.

Wherein, when the video to be simulated is in a video format, the video is generally a 60FPS video. In this way, all frames of the video are directly extracted as a key frame sequence.

In one embodiment, when the video to be simulated is a given image sequence, the image sequence is directly used as the key frame sequence; if the image sequence fails to reach 60 frames per second, the key frame sequence is obtained by duplicating the image sequence. frame sequence.

For example, if the image sequence is a single image, a key frame sequence of 60 frames per second is generated by duplication.

Wherein, when the video to be simulated is in a video format, if the video is a 120FPS video, key frames can be extracted by means of interval extraction. In this application, the setting or extraction method of key frames is not limited.

In an embodiment, in the conversion of the key frame sequence into an intensity map sequence, when the key frame sequence is a color image sequence, the key frame sequence is converted into R channel, G channel and B channel Channel intensity map sequence; if the key frame sequence is a grayscale image sequence, convert the key frame sequence into a single channel intensity map sequence.

Wherein, when the key frame sequence is a color image sequence, the key frame sequence is converted into three intensity map sequences, which are the intensity map sequences of the R channel, the G channel and the B channel respectively. When the key frame sequence is a grayscale image sequence, the key frame sequence is converted into a single-channel intensity map sequence.

Among them, for color images, each pixel value has RGB values of three color channels, and the values of R channel, G channel and B channel are respectively reserved, and the intensity map sequence of R channel, G channel and B channel is obtained.

S200, increasing the frame rate of the intensity map sequence according to the working clock frequency of the pulse camera;

Among them, the transformed intensity map sequence needs to increase the frame rate according to the working clock frequency of the pulse camera. If the working clock of the pulse camera is 1MHz, the intensity map sequence needs to be interpolated to 1000000FPS. The gray value of the intensity map of each frame is used as the equivalent brightness of the pulse simulation.

For example, for an intensity map sequence of 60 frames per second, the working clock frequency of Vidar-I is 10MHz, so 166666 (10000000/60-1) frames of images need to be added between adjacent key frames.

Among them, by inserting frames, the frame rate is increased to the working clock frequency of the pulse camera.

In one embodiment, the frame interpolation method adopts bilinear interpolation.

In one embodiment, the frame interpolation method is a linear interpolation algorithm; or an algorithm based on a deep neural network; or an algorithm based on optical flow.

S400, obtaining an integral distribution model, and simulating the intensity map sequence according to the integral distribution model;

S500, output the pulse data after simulation.

In this application, by simulating the imaging process of the pulse camera, the existing image and video data are converted into pulse data, which solves the problem of serious lack of real pulse data.

In an embodiment, as shown in FIG. 2, in S200, after increasing the frame rate of the intensity map sequence according to the working clock frequency of the pulse camera, the method further includes:

S300: Multiply the intensity map sequence after the frame rate has been increased by a transformation coefficient to adjust the brightness.

Based on the principle of the pulse camera, it can be known that for a pixel, a corresponding pulse will be output only after the accumulated threshold of the pixel value. Therefore, if the pixel value in the intensity map sequence is too low, the corresponding pulse will be generated sparsely. Images in nature have strong brightness, but after they are taken as photos or videos, the corresponding brightness will be greatly reduced due to the imaging principle, and direct simulation will make the simulation pulses sparse and distorted.

In order to solve this problem, it is necessary to adjust the brightness of the intensity map sequence, that is, to increase the corresponding brightness. Each pixel value on the intensity map represents the brightness intensity of the light in the equivalent field of view of the pulse camera; by introducing a transformation coefficient, the brightness value of the intensity map sequence can be enlarged, thereby making up for the lack of brightness of the video to be simulated.

The specific value of the transform coefficient may be obtained according to the actual situation or by means of multiple measurements. For example, an image of the same scene is acquired by a pulse camera and a normal camera, and then the image of the normal camera is used as the video to be simulated based on the preset transformation coefficients for simulation, and the transformation coefficients are adjusted based on the simulation results until the simulation results meet the requirements. The transform coefficients may also be determined by other means, such as historical data or the like.

。In one embodiment, the value range of the transform coefficient is

.

In one embodiment, the point distribution model is:

In the model, C _pd is the conversion capacitance, λ is the conversion coefficient, N ₁ ( t ) is the Poisson model, G ( t ) is the luminance intensity of the intensity map sequence, t is the time step, and η is the linear coefficient.

The integral release model introduces the Poisson model to add noise, so as to load the noise caused by scattered light and increase the accuracy of the simulation.

In an embodiment, in the S400, a point distribution model is obtained, and the intensity map sequence is simulated according to the point distribution model,

In one embodiment, in the S400, a point distribution model is obtained, and in simulating the intensity map sequence according to the point distribution model, a simulation time step and an excitation threshold are determined, and each simulation is performed by the point distribution model. The time step is integrated and a pulse is fired after the result of the integration exceeds the excitation threshold, and the simulation is completed based on the fired pulse.

Among them, the total simulation time T and the simulation time step are determined, and the simulation time step is integrated through the integral distribution model; after each time step, the integral result is judged. If the integral result exceeds the threshold, a pulse is issued and the integral result is cleared. If the integration result does not exceed the threshold, the integration result is retained; proceed to the next time step until the total simulation time is reached.

Specifically: determine the total simulation time T, and use the luminance intensity map sequence to perform pulse simulation. Each pixel value on the brightness intensity map represents the brightness intensity of the light in the equivalent field of view of the pulsed camera. Substitute the brightness intensity of the light into the integral excitation model to perform integral and noise addition operations.

Taking a pixel p as an example to illustrate the generation process of the pulse signal. The luminance intensity of pixel p is Gp. In a simulation time step δt , the actual luminance intensity received on pixel p of the pulse camera photosensitive chip is Gp+N ₁ ( _t ) due to the influence of scattering noise.

Each camera of the pulse camera can be regarded as an integral firing neuron, and the amount of change in the neuron's membrane potential in each time step is:

Among them, η is the linear coefficient of pulse excitation frequency and light intensity of each pixel of the pulse camera, λ is the linear coefficient of pixel gray value and brightness intensity, and C _pd is the capacitance parameter of each pixel photosensitive circuit.

After the neuron membrane potential is updated, it will be judged whether the current membrane potential exceeds the excitation threshold θ. If it exceeds, the membrane potential will be reset and a pulse will be issued; if not, the above accumulation process will continue.

Among them, the simulation time length T determines how long the pulse sequence needs to be generated. The simulation time T can be further divided into N _Tr , where N is the number of simulation time units, and _Tr is the length of the simulation time unit. The simulation time unit can be further divided into K simulation time steps, each simulation time step is the smallest time scale in the simulation process, and the length of the time step is twice the reciprocal of the working clock frequency of the pulse camera, denoted as δ _t .

In order to improve the authenticity of the simulated pulse data, noise is added in the simulation process. According to the different sources of noise, the noise is divided into scattered noise (denoted as N ₁ ) and inherent noise (denoted as N ₂ ). Scattering noise is caused by diffuse light in natural scenes, and is generally modeled by the Poisson model. The inherent noise is caused by the manufacturing process of the pulse camera itself. When there is no optical signal input, there will still be a pulse signal generated. Therefore, this type of noise is independent of the optical signal. In the simulation process, statistical methods are used to obtain the distribution law of this kind of noise, and then modeling is carried out.

In one embodiment, in the case that the pulse readout frequency of the pulse camera is lower than the operating frequency, if there is one, multiple pulses are excited in the simulation time unit, and only one is read out.

For example, the pulse readout frequency of a pulse camera is 40,000 Hz, so the excited pulse cannot be read out in real time. This in turn results in that, even if multiple pulses are fired within one simulation time unit, only one can be read out.

In one embodiment, inherent noise is introduced, and the inherent noise is initialized as a noise matrix in advance, and after the integration process of each time unit is completed, the excited pulse is determined according to the noise matrix.

For example, within the simulation time T, the inherent noise is initialized in advance to the noise matrix N2. In the matrix, 1 represents the presence of noise, and 0 represents the absence of noise. After the integration process of each time unit is completed, the value of the corresponding position in the noise matrix N2 needs to be read. If the value of the position at this moment is 1, no matter whether there is an excited pulse at the current moment, one pulse is read. If the value of this position of the noise matrix N2 at this moment is 0, it will not affect the simulation result at the current moment.

In an implementation manner, in the output simulated pulse data, the pulse data is encoded into binary pulse data and then output.

In one embodiment, the output of the simulated pulse data includes direct readout of the pulse signal.

In an embodiment, in the output emulated pulse data, the pulse signal is encoded and then read out, and the encoding is quaternary encoding, octal encoding or hexadecimal encoding.

For ease of understanding, in this application, the implementation principle of the simulation method is explained: it is assumed that a visual scene is recorded simultaneously with a pulse camera and a traditional camera, and S and G are used to represent the pulse sequence and image respectively; the goal of this application is to The simulated pulse sequence Ŝ is generated from the video sequence G and the difference between the simulated pulse sequence Ŝ and the real pulse sequence S is minimized. Among them, the difference between the simulated pulse sequence Ŝ and the real pulse sequence S needs to be measured from the pulse excitation frequency and the distribution of the pulse sequence, respectively.

In order to generate the simulated pulse sequence Ŝ from the video sequence G , the relationship between the image gray level and the pulse frequency must first be established, specifically:

Firstly, the relationship between the image gray level G and the brightness intensity I is established, and then the relationship between the brightness intensity I and the pulse firing frequency f _s is deduced. In conventional CCD cameras, the luminance intensity of the scene usually has a positive linear relationship with the gray level of the image. Therefore, the relationship between the conventional camera imaging result G and the brightness intensity I of the scene can be recorded as: G = λI , where λ is a constant.

Among them, the pulse release principle of the pulse camera is as follows: each pixel of the sensor chip of the pulse camera is composed of three circuit modules, namely an integrating circuit, a reset circuit and a readout circuit. Correspondingly, the generation of the pulse also includes three states, namely integration, reset and readout. In the integrating state, the photodiode converts the optical signal into a photocurrent I _ph . With the progress of the photoelectric conversion process, the voltage on the capacitor C _pd will continue to drop, while the transistor node voltage V _tr will gradually increase. When the node voltage exceeds the trigger threshold θ, the comparator will toggle. After the reset circuit detects the inversion signal, the integrating circuit enters the reset state immediately. The reset circuit will generate a reset signal and input it to the triode, which resets the photodiode and re-enters a new integral state. At the same time, the inverted signal stored in the reset circuit will be read out by the readout circuit, and the information in the reset circuit will be cleared. The clock frequency of the readout circuit is 40KHz. Therefore, the clock frequency of the readout circuit limits the maximum pulse delivery frequency of the pulse chip, which is 40KHz. This in turn determines the temporal resolution of the pulse camera to be 25 us .

Based on the above analysis, the conditions under which a pixel can trigger a pulse are:

Among them, Δt is the integration time, the trigger threshold is θ= V _dd - V _ref , V _dd is the supply voltage, and V _ref is the reference voltage. From the excitation pulse data, it can be seen that higher light intensity leads to higher pulse firing frequency. Assuming that the light intensity felt by the pixel is stable, the pulse frequency of the pixel is:

The pulse excitation conditions are:

According to the sensing principle of the pulse camera, the integration time required for a pixel to excite a pulse is:

Based on the pulse firing principle of the pulse camera, it can be known that there is a linear relationship between the pulse firing frequency f _s and the photocurrent intensity I _ph . Furthermore, assuming that the relationship between the photocurrent intensity I _ph and the light intensity is I _ph =R( I ), it can be deduced that the relationship between f _s and I is:

Since it is very difficult to directly measure the relationship between the photocurrent intensity I _ph and the light intensity I , this paper adopts the relationship between the light intensity I and the pulse firing frequency f _s , and then estimates the relationship between the photocurrent intensity I _ph and the light intensity through the experimental results. Intensity I relationship. The specific experimental process is not repeated here.

The experimental results show that when the light intensity I is less than the turning point _IS (ie 11000Lux ), the pulse firing frequency is approximately linearly positively correlated with the light intensity; when the light intensity is greater than 11000Lux, the pulse firing frequency reaches the maximum excitation frequency f _max =1/ T _r = 40KHz, and no longer varies with increasing light intensity. From the experimental results, it can be inferred that when the light intensity is less than 11000Lux, there is also a linear positive correlation between the photocurrent intensity I _ph and the light intensity I. And the value of the linear coefficient η between I and f _s was determined to be 1.09e-13. Combining the relationship between image grayscale and light intensity above, it can be deduced that the image grayscale G and

The relationship between the pulse firing frequency f _s is:

Therefore, given the gray value G of a pixel, the pulse excitation frequency f _s of each pixel can be simulated based on this formula. However, the pulses generated by the simulation of this model are ideal data, and the influence of noise is not considered.

To this end, noise analysis and modeling are also required:

In a real scene, the photosensitive process is affected by diffuse reflection, which in turn affects the pulse interval generated by the simulation under the ideal model. Over a period of time, the number of scattered photons reaching the photosensitive chip is random. The Poisson distribution is often used to model the distribution of random events and is used to model image noise. Therefore, this paper adopts the Poisson model to simulate the scattering noise in the simulation process. Assuming that the probability of the number of scattered photons arriving at the pixel is k within a unit time δ _t (δ _t = 2/ CLK ) is _γ , then within a minimum pulse interval Tr , the probability distribution function of the Poisson noise model can be expressed as :

In addition, the inherent noise causes pulsed cameras to excite even when no light is incident.

Sending pulses, it is determined by experiments (the specific experimental method will not be repeated here) that the pulse firing interval approximately obeys the Gaussian distribution. Therefore, based on the experimental results, this paper proposes a Gaussian model to simulate the inherent noise in the simulation process, as shown in the following formula:

In this application, in order to verify the simulation effect of the simulation method and obtain the pulse sequence of the real scene recorded by the pulse camera, firstly, the pulse data is restored to a grayscale image by a reconstruction algorithm (Texture-From-Window, TFW). Through the simulation method, the grayscale image is simulated as the video to be simulated, and the simulated pulse sequence is obtained. Calculate the similarity of the real pulse sequence to the simulated pulse sequence.

In this application, two metric rules are used to calculate the similarity between the pulse sequence and the simulated pulse sequence.

In the first measure, the pulse sequence is regarded as a matrix of dimension N×M (M=W×H, where W and H are the spatial resolution of the pulse camera), and the similarity of the pulse excitation frequency for each pixel is calculated.

The second measurement method is KL divergence, which evaluates the similarity of the distribution between the pulse sequence and the simulated pulse sequence.

In the calculation results, the similarity of scene 1, scene 2, scene 3 and scene 4 in the first measurement method are 0.991, 0.993, 0.994 and 0.993 respectively; in the second measurement method, the similarity of scene 1, scene 2, scene 3 and scene The results for 4 were 0.081, 0.079, 0.057, 0.044, respectively. It can be seen from the calculation results that the pulse data obtained by the simulation method of the present application is very close to the real pulse data, which greatly improves the accuracy of the simulation.

An embodiment of the present application provides a pulse camera simulation apparatus, which is used for executing the pulse camera simulation method described in the above content of the present application, and the pulse camera simulation apparatus is described in detail below.

As shown in Figure 3, the pulse camera simulation device includes:

The frame extraction module 101 is used to extract the key frame sequence of the video to be simulated, and convert the key frame sequence into an intensity map sequence;

a frame rate boosting module 102, configured to boost the frame rate of the intensity map sequence according to the working clock frequency of the pulse camera;

A simulation module 104, configured to obtain a point distribution model, and simulate the intensity map sequence according to the point distribution model;

The output module 105 is used for outputting the simulated pulse data.

In one embodiment, as shown in Figure 4, the device further includes:

The brightness adjustment module 103 is configured to multiply the intensity map sequence after the frame rate has been increased by a transform coefficient to perform brightness adjustment.

In one embodiment, the simulation module 104 is further used for:

A simulation time step and excitation threshold are determined, each simulation time step is integrated by the integral firing model, and a pulse is issued after the integration result exceeds the excitation threshold, and the simulation is completed based on the issued pulse.

In one embodiment, the frame extraction module 101 is further configured to:

When the key frame sequence is a color image sequence, the key frame sequence is converted into an intensity map sequence of R channel, G channel and B channel; when the key frame sequence is a grayscale image sequence, all The keyframe sequence is converted into a single-channel intensity map sequence.

In one embodiment, the point distribution model is:

In the model, C _pd is the conversion capacitance, λ is the conversion coefficient, N ₁ ( t ) is the Poisson model, G ( t ) is the luminance intensity of the intensity map sequence, t is the time step, and η is the linear coefficient.

In one embodiment, the output module 105 is further configured to:

In the output of the simulated pulse data, the pulse data is encoded into binary pulse data and then output.

The pulse camera simulation device provided by the above embodiments of the present application and the pulse camera simulation method provided by the embodiments of the present application are based on the same inventive concept, and have the same beneficial effects as the methods adopted, run or implemented by the stored applications.

The internal function and structure of the pulse camera simulation device are described above. As shown in FIG. 5 , in practice, the pulse camera simulation device can be implemented as a control device, including: a memory 301 and a processor 303 .

The memory 301 may be configured to store programs.

Additionally, the memory 301 may also be configured to store various other data to support operations on the control device. Examples of such data include instructions for any application or method operating on the control device, contact data, phonebook data, messages, pictures, videos, etc.

Memory 301 may be implemented by any type of volatile or non-volatile storage device or combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EEPROM), Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.

The processor 303, coupled to the memory 301, is used for executing the program in the memory 301 for:

Extract the key frame sequence of the video to be simulated, and convert the key frame sequence into an intensity map sequence;

Increase the frame rate of the intensity map sequence according to the working clock frequency of the pulse camera;

obtaining a point distribution model, and simulating the intensity map sequence according to the point distribution model;

The simulated pulse data is output.

In one embodiment, the processor 303 is specifically configured to:

Brightness adjustment is performed by multiplying the intensity map sequence after the frame rate has been increased by a transform coefficient.

In one embodiment, the processor 303 is specifically configured to:

A simulation time step and excitation threshold are determined, each simulation time step is integrated by the integral firing model, and a pulse is issued after the integration result exceeds the excitation threshold, and the simulation is completed based on the issued pulse.

In one embodiment, the processor 303 is specifically configured to:

When the key frame sequence is a color image sequence, the key frame sequence is converted into an intensity map sequence of R channel, G channel and B channel; when the key frame sequence is a grayscale image sequence, all The keyframe sequence is converted into a single-channel intensity map sequence.

In one embodiment, the point distribution model is:

In the model, C _pd is the conversion capacitance, λ is the conversion coefficient, N ₁ ( t ) is the Poisson model, G ( t ) is the luminance intensity of the intensity map sequence, t is the time step, and η is the linear coefficient.

In one embodiment, the processor 303 is specifically configured to:

The pulse data is encoded into binary pulse data and then output.

In this application, only some components are schematically shown in FIG. 5 , which does not mean that the control device only includes the components shown in FIG. 5 .

The control device provided by this embodiment is based on the same inventive concept as the pulse camera simulation method provided by the embodiment of the present application, and has the same beneficial effect as the method adopted, executed or implemented by the stored application program.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include non-persistent memory in computer readable media, random access memory (RAM) and/or non-volatile memory in the form of, for example, read only memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

The present application also provides a computer-readable storage medium corresponding to the pulse camera simulation method provided by the foregoing embodiments, and a computer program (ie, a program product) is stored thereon, and when the computer program is run by a processor, it will be executed The pulse camera simulation method provided by any of the foregoing embodiments.

Computer-readable media includes both persistent and non-permanent, removable and non-removable media, and storage of information may be implemented by any method or technology. Information may be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash Memory or other memory technology, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridges, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

The computer-readable storage medium provided by the above-mentioned embodiments of the present application and the pulse camera simulation method provided by the embodiments of the present application are based on the same inventive concept, and have the same beneficial effects as the methods adopted, run or implemented by the stored application programs.

It should be noted that, in the description provided herein, numerous specific details are set forth. It will be understood, however, that the embodiments of the present application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

It should also be noted that the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device comprising a series of elements includes not only those elements, but also Other elements not expressly listed, or which are inherent to such a process, method, article of manufacture, or apparatus are also included. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article of manufacture, or device that includes the element.

It will be appreciated by those skilled in the art that the embodiments of the present application may be provided as a method, a system or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The above descriptions are merely examples of the present application, and are not intended to limit the present application. Various modifications and variations of this application are possible for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the scope of the claims of this application.

Claims (7)

1. A pulse camera simulation method is characterized by comprising the following steps:

extracting a key frame sequence of a video to be simulated, and converting the key frame sequence into an intensity map sequence;

increasing the frame rate of the intensity map sequence according to the working clock frequency of the pulse camera;

acquiring an integral distribution model, and simulating the intensity map sequence according to the integral distribution model;

outputting the simulated pulse data;

wherein the point issuing model is as follows:

in the model, the model is divided into a plurality of models,C _pd in order to switch the capacitance of the capacitor,λin order to convert the coefficients of the image,N ₁ (t) In order to be a poisson model,G(t) Is the intensity of the luminance of the sequence of intensity maps,tin order to be the time step,ηis a linear coefficient;

and in the simulation of the intensity map sequence according to the integral issuing model, determining simulation time steps and an excitation threshold value, integrating each simulation time step through the integral issuing model, issuing a pulse after an integration result exceeds the excitation threshold value, and finishing the simulation based on the issued pulse.

2. The method of claim 1, wherein after increasing the frame rate of the sequence of intensity maps according to the operating clock frequency of the pulse camera, the method further comprises:

and multiplying the intensity map sequence after the frame rate is improved by a transformation coefficient to perform brightness adjustment.

3. The method according to any of claims 1-2, wherein the converting the sequence of key frames into a sequence of intensity maps is performed by converting the sequence of key frames into a sequence of intensity maps for an R-channel, a G-channel and a B-channel in case the sequence of key frames is a sequence of color images; and converting the key frame sequence into a single-channel intensity map sequence under the condition that the key frame sequence is a gray image sequence.

4. The method according to any one of claims 1-2, wherein the output of the simulated pulse data is performed after the pulse data is encoded into binary pulse data and output.

5. A pulse camera simulation apparatus, comprising:

the frame extraction module is used for extracting a key frame sequence of a video to be simulated and converting the key frame sequence into an intensity map sequence;

the frame rate increasing module is used for increasing the frame rate of the intensity map sequence according to the working clock frequency of the pulse camera;

the simulation module is used for acquiring an integral distribution model and simulating the intensity map sequence according to the integral distribution model;

wherein the point issuing model is as follows:

in the model, the model is divided into a plurality of models,C _pd to convert electricityThe volume of the liquid to be treated is,λin order to convert the coefficients of the image,N ₁ (t) In order to be a poisson model,G(t) Is the intensity of the luminance of the sequence of intensity maps,tin order to be the time step,ηis a linear coefficient;

in the simulation of the intensity map sequence according to the integral issuing model, a simulation time step and an excitation threshold value are determined, integration is performed on each simulation time step through the integral issuing model, a pulse is issued after an integration result exceeds the excitation threshold value, and the simulation is completed based on the issued pulse;

and the output module is used for outputting the simulated pulse data.

6. An electronic device, comprising a computer-readable storage medium storing a computer program and a processor, the computer program, when read and executed by the processor, implementing the method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that it stores a computer program which, when read and executed by a processor, implements the method according to any one of claims 1-4.

Images