CN114584703B - Imaging method, device, equipment and storage medium of bionic pulse camera - Google Patents
Imaging method, device, equipment and storage medium of bionic pulse camera Download PDFInfo
- Publication number
- CN114584703B CN114584703B CN202210096416.2A CN202210096416A CN114584703B CN 114584703 B CN114584703 B CN 114584703B CN 202210096416 A CN202210096416 A CN 202210096416A CN 114584703 B CN114584703 B CN 114584703B
- Authority
- CN
- China
- Prior art keywords
- pulse
- imaging
- point
- image sequence
- calculating
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/207—Analysis of motion for motion estimation over a hierarchy of resolutions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Studio Devices (AREA)
- Image Analysis (AREA)
Abstract
The application discloses an imaging method, an imaging device, imaging equipment and a storage medium of a bionic pulse camera, wherein the method comprises the following steps: reconstructing a basic image sequence according to a preset pulse camera imaging algorithm and a continuous pulse array in a preset time period; calculating motion vectors at any pixel position at any time from the basic image sequences to form motion fields between the basic image sequences; calculating the contribution weight of each pulse of an imaging grid target imaging point according to the motion field and the pulse array; and reconstructing a high-resolution image sequence according to the contribution weight of the imaging point of the imaging grid target to each pulse. According to the method provided by the embodiment of the application, the method can be applied to a scene moving at a high speed, the movement details of objects in the scene are clearly reconstructed, and the obtained image not only avoids movement blurring, but also has a high signal-to-noise ratio.
Description
Technical Field
The invention relates to the technical field of computational imaging, in particular to an imaging method, an imaging device, imaging equipment and a storage medium of a bionic pulse camera.
Background
Conventional digital cameras typically perform photographic imaging at a fixed frame rate, with each frame image being generated as follows: in a certain exposure time window, each pixel of the image sensor performs photoelectric conversion and accumulation on incident light, and the total illumination amount of the pixel is obtained through analog-to-digital conversion after exposure is finished. The length of the exposure time is usually determined by the illumination intensity, and when the light is weak, the exposure time is usually increased to suppress the influence of dark current noise. This method cannot effectively image a high-speed object, and often results in imaging blur of a high-speed moving object. In recent years, the unique neuron connection structure of biological fovea and the integral distribution model of ganglion cells provide a new idea for visual sampling. Through simulation and abstraction of the fovea of the retina, a pulsed camera comprising photoreceptors, integrators and threshold comparators is presented. The pulse camera represents visual information in a pulse array form, can continuously record light intensity change, has no concepts of frame rate and exposure time, breaks through the limitations of the traditional camera, can capture and record high-speed motion, and can reconstruct texture details in a scene.
In the prior art, there are two reconstruction algorithms for pulse cameras, a reconstruction algorithm based on pulse interval and a reconstruction algorithm based on time window averaging. The reconstruction algorithm based on the pulse interval utilizes the characteristic that the pulse interval is reduced along with the increase of the light intensity, and reconstructs the light intensity in a short time by using the front pulse and the rear pulse. Although the algorithm can draw the contour of high-speed motion, the reconstructed signal is usually not stable enough, and the pixel value has obvious fluctuation in the time direction; the time window averaging based reconstruction algorithm uses the property that the pulse delivery frequency increases with increasing light intensity to estimate the average light intensity within the time window. Although the signal-to-noise ratio of the reconstructed image is improved to a certain extent, when the object has motion, the average in the time window can cause the motion blur of the reconstructed image.
Therefore, in the prior art, a method for reconstructing an image by using a pulse camera still cannot obtain a clear image in a high-speed motion scene.
Disclosure of Invention
The embodiment of the application provides an imaging method, an imaging device, imaging equipment and a storage medium of a bionic pulse camera. The following presents a simplified summary in order to provide a basic understanding of some aspects of the disclosed embodiments. This summary is not an extensive overview and is intended to neither identify key/critical elements nor delineate the scope of such embodiments. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.
In a first aspect, an embodiment of the present application provides an imaging method for a biomimetic pulse camera, including:
reconstructing a basic image sequence according to a preset pulse camera imaging algorithm and a continuous pulse array in a preset time period;
calculating motion vectors at any pixel position at any time from the basic image sequence to form motion fields between the basic image sequence;
calculating the contribution weight of each pulse of an imaging grid target imaging point according to the motion field and the pulse array;
and reconstructing a high-resolution image sequence according to the contribution weight of the imaging point of the imaging grid target to each pulse.
In an optional embodiment, reconstructing a basic image sequence according to a preset pulse camera imaging algorithm and a continuous pulse array in a preset time period includes:
reconstructing a basic image sequence according to a pulse interval algorithm and a continuous pulse array in a preset time period; or the like, or, alternatively,
and reconstructing a basic image sequence according to a reconstruction algorithm based on time window average and a continuous pulse array in a preset time period.
In an optional embodiment, calculating the contribution weight of the imaging grid target imaging point to each pulse according to the motion field and the pulse array comprises:
calculating the motion trail of a target imaging point according to the motion field between the basic image sequences;
calculating the contribution duration of each pulse of the target imaging point according to the motion track of the target imaging point and the position relation of the pixel to which each pulse belongs;
and calculating the contribution weight of the imaging grid target imaging point to each pulse according to the contribution duration of the target imaging point to each pulse.
In an optional embodiment, reconstructing a high resolution image sequence according to the contribution weight of the imaging point of the imaging grid target to each pulse comprises:
constructing a prior constraint model based on local smoothness, and constructing a data fidelity constraint model according to contribution weight of imaging grid target imaging points to each pulse;
and calculating the brightness value of each pixel point in the image to be reconstructed according to the prior constraint model and the data fidelity constraint model to obtain a reconstructed high-resolution image sequence.
In an optional embodiment, calculating, according to the prior constraint model and the data fidelity constraint model, a brightness value of each pixel point in the image to be reconstructed to obtain a reconstructed high-resolution image sequence includes:
obtaining a first objective function according to the combination of the prior constraint model and the data fidelity constraint model;
and solving the first objective function in an iterative optimization solving mode, and calculating the brightness value of each pixel point in the image to be reconstructed to obtain a reconstructed high-resolution image sequence.
In an optional embodiment, reconstructing a high resolution image sequence according to the contribution weight of the imaging point of the imaging grid target to each pulse comprises:
establishing a relation model between grid pixel values and pulses according to the contribution weight of the imaging grid target imaging point to each pulse;
and solving a second objective function corresponding to the relation model in an iterative optimization solving mode, and calculating the brightness value of each pixel point in the image to be reconstructed to obtain a reconstructed high-resolution image sequence.
In an optional embodiment, establishing a relation model between the grid pixel values and the pulses according to the contribution weight of the imaging grid target imaging point to each pulse comprises establishing the relation model according to the following formula:
θ＝α·W s I
wherein, W s Representing the contribution weight of each target imaging point in the imaging grid to each pulse, I representing the image brightness matrix to be solved, α representing the photoelectric conversion rate of the pulse camera, and θ representing the pulse emission threshold.
In a second aspect, an embodiment of the present application provides an imaging apparatus of a biomimetic pulse camera, including:
the first reconstruction module is used for reconstructing a basic image sequence according to a preset pulse camera imaging algorithm and a continuous pulse array in a preset time period;
the motion field determining module is used for calculating motion vectors at any pixel position at any time from the basic image sequences to form motion fields between the basic image sequences;
the weight determining module is used for calculating the contribution weight of each pulse of the imaging point of the imaging grid target according to the motion field and the pulse array;
and the second reconstruction module is used for reconstructing a high-resolution image sequence according to the contribution weight of the imaging point of the imaging grid target to each pulse.
In a third aspect, an embodiment of the present application provides an imaging device of a bionic pulse camera, which includes a processor and a memory storing program instructions, where the processor is configured to execute the imaging method of the bionic pulse camera provided in the foregoing embodiment when executing the program instructions.
In a fourth aspect, the present application provides a computer-readable medium, on which computer-readable instructions are stored, where the computer-readable instructions are executed by a processor to implement the imaging method of the biomimetic pulse camera provided in the foregoing embodiment.
The technical scheme provided by the embodiment of the application can have the following beneficial effects:
according to the imaging method of the bionic pulse camera, provided by the embodiment of the application, a high-resolution image sequence can be reconstructed from a low-resolution binary pulse data array by utilizing the relative motion between a pulse camera sensor and an external scene. The method can be applied to scenes moving at high speed, generates images with high signal-to-noise ratio and no blurring, realizes better visual effect, and solves the problem that objects moving at high speed cannot be clearly imaged in the prior art.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
FIG. 1 is a flow diagram illustrating an imaging method of a biomimetic pulse camera in accordance with an exemplary embodiment;
FIG. 2 is a schematic diagram illustrating a method of imaging a biomimetic pulse camera in accordance with an exemplary embodiment;
FIG. 3 is a diagram illustrating a mapping of object point intensity to pulse according to an exemplary embodiment;
FIG. 4 is a schematic diagram illustrating reconstructed images from a different reconstruction algorithm according to an exemplary embodiment;
FIG. 5 is a schematic diagram of an imaging apparatus of a biomimetic pulse camera according to an exemplary embodiment;
FIG. 6 is a schematic diagram of a configuration of an imaging device of a biomimetic pulse camera, according to an exemplary embodiment;
FIG. 7 is a schematic diagram illustrating a computer storage medium in accordance with an exemplary embodiment.
Detailed Description
The following description and the drawings sufficiently illustrate specific embodiments of the invention to enable those skilled in the art to practice them.
It should be understood that the described embodiments are only some embodiments of the invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present invention. Rather, they are merely examples of systems and methods consistent with certain aspects of the invention, as detailed in the appended claims.
In the prior art, a traditional digital camera is used for shooting and imaging, and the method cannot effectively image the object moving at high speed, so that the imaging of the object moving at high speed is blurred. In recent years, the unique neuron connection structure of biological fovea and the integral distribution model of ganglion cells provide a new idea for visual sampling. Through simulation and abstraction of the fovea of the retina, a novel bionic pulse camera is provided. The bionic pulse camera can continuously record the change of light intensity, has no concepts of frame rate and exposure time, breaks through the limitation of the traditional camera, can capture and record high-speed motion, and can reconstruct texture details in a scene.
Therefore, the embodiment of the application provides an imaging method of a bionic pulse camera, which comprises the steps of firstly estimating a basic reconstructed image sequence from a pulse array, and then determining the motion trail of each pixel point in an image by using an optical flow estimation method. Next, a mapping relationship between the scene light intensity and the pulse is determined based on the motion information. Then, establishing a constraint model; and finally, deducing the brightness value of each pixel point in the image to be reconstructed according to the mapping relation between the light intensity and the pulse and the constraint model so as to reconstruct an image sequence with high resolution. The method solves the problem that the object moving at high speed cannot be clearly imaged in the prior art.
The imaging method of the bionic pulse camera provided by the embodiment of the present application will be described in detail below with reference to fig. 1. Referring to fig. 1, the method specifically includes the following steps.
S101, reconstructing a basic image sequence according to a preset pulse camera imaging algorithm and a continuous pulse array in a preset time period.
Specifically, in order to break the limitation of the conventional camera, the embodiment of the present disclosure performs shooting imaging by using a bionic pulse camera, in which each pixel includes a photoreceptor, an integrator, and a threshold comparator. The photoreceptor is responsible for converting an optical signal into an electric signal, the integrator accumulates the converted photo-generated charges, the threshold comparator repeatedly compares the accumulated charges at an ultrahigh frequency, and when the accumulated charges reach a preset threshold, the pixel performs pulse distribution and the integrator is emptied. The signal sequence we call the output of a single pixel is a "pulse sequence", and the cross section of the pulse array at a certain moment is a "pulse matrix". The pulse array is composed of two symbols "0" and "1", where "1" indicates that there is a pulse delivered at the spatio-temporal location; a "0" indicates that the spatiotemporal location is unpulsed. Through this bionical formula pulse camera, can last the change of recording light intensity, do not have the notion of frame rate and exposure time, broken through the limitation of traditional camera.
Further, after a pixel-point-based continuous pulse array of the object to be imaged is acquired, a basic image sequence is reconstructed according to the acquired continuous pulse array in a preset time period. In a possible implementation, the basic image sequence may be reconstructed by using a pulse interval algorithm, or the basic reconstructed image sequence may be estimated by using a reconstruction algorithm based on time window averaging.
In one embodiment, the basic image sequence is reconstructed by using a pulse spacing algorithm, wherein the pulse spacing is the time between two adjacent pulses at the same pixel position, and is equivalent to the integration time for accumulating the next pulse by an integrator. Generally, an integrator of a pixel in a high-light area can accumulate photo-generated charges to a pulse release threshold value in a short time, and the pulse interval is small; the integrator for the pixel with the weak illumination area takes longer to accumulate the charge to the pulse delivery threshold, with a larger pulse interval. According to the characteristic that the average light intensity between pulses is inversely proportional to the size of pulse interval, reconstructing a basic image sequence
S102 calculates a motion vector at an arbitrary pixel position at an arbitrary time from the base image sequence to form a motion field between the base image sequences.
In one possible implementation, the current frame is calculated by using an optical flow estimation algorithm such as Horn-Schunck and the like
And a motion vector at an arbitrary pixel position at an arbitrary time between the preceding and succeeding r frames of images, each imageMotion vectors in voxel positions form a motion field, also called optical flow field>
Thereby determining the relative motion relationship between the images. Wherein it is present>
Indicating light intensity information in the horizontal and vertical directions, respectively.
The optical flow field is calculated in order to find the correspondence between the image sequences. The optical flow field is understood to be the motion field and the apparent motion of the image intensity pattern is the optical flow.
In an exemplary scene, if a triangle in a captured scene is falling freely, and the camera is kept still, the positions of the triangle in the captured images at different times are different, and the corresponding relationship of the same object on different images is found by using an optical flow method, which is mostly based on the principle of light intensity consistency, that is, assuming that the light intensity values of the same object are the same. Based on the assumption of consistency of light intensity, solving the light stream (relative motion) between two frames by using a light stream method, further determining the corresponding relation of images between different frames based on the light stream (relative motion), and obtaining the corresponding position of each pixel point (x, y) in the current frame on other frames
And finding out the positions of the same object point in the scene on the images at different time points, and determining the motion track of the object.
S103, according to the motion field and the pulse array, calculating the contribution weight of the imaging grid target imaging point to each pulse.
In a possible implementation manner, firstly, a motion trajectory of a target imaging point is obtained according to a motion field between basic image sequences, and a contribution duration of the target imaging point to each pulse is calculated according to the motion trajectory of the target imaging point and a position relation of a pixel to which each pulse belongs.
FIG. 3 is a graph illustrating object point intensity versus pulse in accordance with an exemplary embodimentThe mapping relationship is schematically shown in fig. 3, and according to the motion trail information, any point I in the current frame can be determined k (x, y) the duration of action on any one pulse. I.e. how long the light intensity at this spatio-temporal position can contribute to each pulse.
Assuming that the sensor coordinate to which the pulse s belongs is (m, n), the integration time of the pulse starts at t s And ends at t e . If I k (x, y) at t k Acting on pulse s at time + Δ t, the following constraint should be satisfied:
the target imaging point I can be solved from the relationship k The duration of the contribution of (x, y) to the pulse s, denoted c s (x, y). Wherein, t k + Δ t is a time point, and all time points satisfying the set of inequalities constitute a contribution period.
And further, calculating the contribution weight of the imaging grid target imaging point to each pulse according to the contribution duration of the target imaging point to each pulse.
For any imaging grid (r, c), it corresponds to a rectangular region Ω of the imaging grid r，c . Calculating the contribution weight of the imaging grid (r, c) to the pulse s based on the contribution duration of the target imaging point light intensity of the region to be imaged of the imaging grid to each pulse, wherein the specific calculation form is that the region omega is positioned r，c Inner pair c s (x, y) is integrated as shown in the following equation:
according to the illumination consistency, the light intensity signals on the motion tracks are assumed to be equal. Light intensity I k (x, y) acting on the pulse means I k (x, y) and their relative positions along the motion trajectory fall within the light intensity accumulation region of the pulse, i.e. at the pulse accumulation time t s To t e Within, on the imaging grid (r, c)And obtaining the contribution weight of the target imaging point in the imaging grid to the pulse s in the covered area.
And S104, reconstructing a high-resolution image sequence according to the contribution weight of the imaging point of the imaging grid target to each pulse.
In an optional embodiment, reconstructing a high resolution image sequence according to the contribution weight of the imaging point of the imaging grid target to each pulse comprises:
constructing a prior constraint model based on local smoothness, and constructing a data fidelity constraint model according to contribution weight of imaging grid target imaging points to each pulse; and calculating the brightness value of each pixel point in the image to be reconstructed according to the prior constraint model and the data fidelity constraint model to obtain a reconstructed high-resolution image sequence.
Specifically, a priori constraint model based on local smoothness is constructed, due to the influence of factors such as dark current, certain noise may exist in recorded pulse signals, in order to improve the signal-to-noise ratio of a reconstructed image, a priori constraint is established based on the assumption of local smoothness, and the priori constraint model is constructed according to the following formula:
wherein E is s A term of a-priori constraints is represented,
representing partial derivation of the horizontal direction of the image>
And I represents an image brightness matrix to be solved.
According to the contribution weight of each imaging grid point (r, c) to each pulse s, a relation model between the imaging grid image pixel value and any pulse s can be established, namely the weighted average value of a group of specific pixels in the imaging grid is equal to the pulse sending threshold value of the pulse sensor pixel, so that a data fidelity constraint model is formed, wherein the specific pixels refer to a group of pixels which act on the pulse sensor pixel in the image. A data fidelity constraint model is constructed according to the following formula:
wherein E is d Representing data fidelity constraints, W s Representing the contribution weight of a target imaging point in an imaging grid to each pulse, I representing an image brightness matrix to be solved, α representing the photoelectric conversion rate of the pulse camera, and θ representing the pulse emission threshold.
Further, a first objective function is obtained according to the combination of the prior constraint model and the data fidelity constraint model.
Specifically, the first objective function is obtained by combining the following formulas:
wherein, lambda represents a weight coefficient for adjusting the strength of the two constraint terms,
representing data fidelity constraint term>
Representing a priori constraint terms.
Furthermore, the first objective function is solved through an iterative optimization solving method, the brightness value of each pixel point in the image to be reconstructed is calculated, and the reconstructed high-resolution image sequence is obtained. The initial state of the image solution can be obtained by the basic reconstruction image through algorithms such as bilinear interpolation and the like.
Optionally, establishing a relationship model between the grid pixel values and the pulses according to the contribution weight of the imaging grid target imaging point to each pulse; and solving a second objective function corresponding to the relation model in an iterative optimization solving mode, and calculating the brightness value of each pixel point in the image to be reconstructed to obtain a reconstructed high-resolution image sequence.
Specifically, based on the weight of each pulse s contributed by each imaging grid point (r, c), a model of the relationship between the imaging grid image pixel values and any pulse s can be built, i.e., the weighted average of a particular set of pixels in the imaging grid is equal to the pulse delivery threshold of the pulse sensor pixel.
The relational model can be expressed generally as a system of linear equations:
θ＝α·W s I
wherein, W s Representing the contribution weight of each target imaging point in the imaging grid to each pulse, I representing the image brightness matrix to be solved, α representing the photoelectric conversion rate of the pulse camera, and θ representing the pulse emission threshold.
Further, a second objective function is constructed according to the relationship model:
the second objective function can be solved through iterative optimization solving methods such as a linear least square method or a gradient descent method, the brightness value of each pixel point in the image to be reconstructed is calculated, and the reconstructed high-resolution image sequence is obtained. The initial state of the image solution can be obtained by the basic reconstruction image through algorithms such as bilinear interpolation and the like.
According to the imaging method of the bionic pulse camera, provided by the embodiment of the application, the motion details in a scene can be reconstructed at high definition, and the reconstructed image not only avoids motion blur but also has a higher signal-to-noise ratio. Fig. 4 is a schematic diagram illustrating a reconstructed image obtained by a different reconstruction algorithm according to an exemplary embodiment, and as shown in fig. 4, the reconstructed image obtained based on a pulse interval algorithm, the reconstructed image obtained based on a time window averaging algorithm, and the reconstructed image obtained based on the method of the embodiment of the present application are respectively shown from left to right. It can be seen that the method of the embodiment of the application can reconstruct an image sequence with higher resolution than that of a pulse sensor.
In order to facilitate understanding of the imaging method of the bionic pulse camera provided in the embodiment of the present application, the following description is made with reference to fig. 2. As shown in fig. 2, the method includes the following steps.
Firstly, basic light intensity inference is carried out on different time points in a scene to obtain a basic reconstruction image sequence, and then the motion trail of an object in the scene is determined based on the light intensity consistency assumption to obtain an optical flow field between images. And then, analyzing the mapping relation between the light intensity of the object point and the pulse according to the motion trail of the object to obtain the contribution weight of the imaging grid target imaging point to each pulse. Further, a data fidelity constraint model is built according to the contribution weight of the imaging point of the imaging grid target to each pulse, a priori constraint model based on local smoothness is built, and a constraint equation set is obtained. And solving a constraint equation set by an iterative optimization solution method, and calculating the brightness value of each pixel point in the image to be reconstructed to obtain a reconstructed high-resolution image sequence.
According to the imaging method of the bionic pulse camera, provided by the embodiment of the application, a high-resolution image sequence can be reconstructed from a low-resolution binary pulse data array by utilizing the relative motion between a pulse camera sensor and an external scene. The method can be applied to scenes moving at high speed, generates images with high signal-to-noise ratio and no blurring, realizes better visual effect, and solves the problem that objects moving at high speed cannot be clearly imaged in the prior art.
An embodiment of the present application further provides an imaging device of a biomimetic pulse camera, where the imaging device is configured to perform the imaging method of the biomimetic pulse camera according to the foregoing embodiment, and as shown in fig. 5, the imaging device includes:
a first reconstructing module 501, configured to reconstruct a basic image sequence according to a preset pulse camera imaging algorithm and a continuous pulse array in a preset time period;
a motion field determining module 502, configured to calculate a motion vector at any pixel position at any time from the basic image sequences to form a motion field between the basic image sequences;
a weight determining module 503, configured to calculate a contribution weight of each pulse to the imaging point of the imaging grid target according to the motion field and the pulse array;
and a second reconstructing module 504, configured to reconstruct a high-resolution image sequence according to the contribution weight of the imaging point of the imaging grid target to each pulse.
It should be noted that, when the imaging apparatus of the biomimetic pulse camera provided in the above embodiment executes the imaging method of the biomimetic pulse camera, only the division of the above functional modules is taken as an example, in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the above described functions. In addition, the imaging device of the bionic pulse camera provided by the above embodiment and the imaging method embodiment of the bionic pulse camera belong to the same concept, and the implementation process is detailed in the method embodiment, which is not described herein again.
The embodiment of the present application further provides an electronic device corresponding to the imaging method of the biomimetic pulse camera provided in the foregoing embodiment, so as to execute the imaging method of the biomimetic pulse camera.
Please refer to fig. 6, which illustrates a schematic diagram of an electronic device according to some embodiments of the present application. As shown in fig. 6, the electronic apparatus includes: a processor 600, a memory 601, a bus 602 and a communication interface 603, wherein the processor 600, the communication interface 603 and the memory 601 are connected through the bus 602; the memory 601 stores a computer program that can be executed on the processor 600, and the processor 600 executes the computer program to perform the imaging method of the bionic pulse camera provided in any of the foregoing embodiments of the present application.
The Memory 601 may include a high-speed Random Access Memory (RAM) and may further include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is implemented through at least one communication interface 603 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like may be used.
The electronic device provided by the embodiment of the application and the imaging method of the bionic pulse camera provided by the embodiment of the application have the same inventive concept and have the same beneficial effects as the method adopted, operated or realized by the electronic device.
Referring to fig. 7, the computer readable storage medium is an optical disc 700, on which a computer program (i.e., a program product) is stored, and when the computer program is executed by a processor, the computer program performs the imaging method of the biomimetic pulse camera according to any of the embodiments.
It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory, or other optical and magnetic storage media, which are not described in detail herein.
The computer-readable storage medium provided by the above embodiment of the present application and the imaging method of the biomimetic pulse camera provided by the embodiment of the present application are based on the same inventive concept, and have the same beneficial effects as the method adopted, operated or implemented by the application program stored in the computer-readable storage medium.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above examples only show several embodiments of the present invention, and the description thereof is specific and detailed, but not to be construed as limiting the scope of the present invention. It should be noted that various changes and modifications can be made by those skilled in the art without departing from the spirit of the invention, and these changes and modifications are all within the scope of the invention. Therefore, the protection scope of the present patent should be subject to the appended claims.
Claims (10)
1. An imaging method of a bionic pulse camera is characterized by comprising the following steps:
reconstructing a basic image sequence according to a preset pulse camera imaging algorithm and a continuous pulse array in a preset time period;
calculating motion vectors at any pixel position at any time from the basic image sequences to form motion fields between the basic image sequences;
calculating the contribution weight of the imaging grid target imaging point to each pulse according to the motion field and the pulse array, wherein the contribution weight comprises the following steps: calculating the motion trail of a target imaging point according to the motion field between the basic image sequences; calculating the contribution duration of each pulse of the target imaging point according to the motion track of the target imaging point and the position relation of the pixel to which each pulse belongs; calculating the contribution weight of the imaging grid target imaging point to each pulse according to the contribution duration of the target imaging point to each pulse;
and reconstructing a high-resolution image sequence according to the contribution weight of the imaging point of the imaging grid target to each pulse.
2. The method of claim 1, wherein reconstructing the basic image sequence according to a preset pulse camera imaging algorithm and a continuous pulse array within a preset time period comprises:
reconstructing a basic image sequence according to a pulse interval algorithm and a continuous pulse array in a preset time period; or the like, or, alternatively,
and reconstructing a basic image sequence according to a reconstruction algorithm based on time window average and a continuous pulse array in a preset time period.
3. The method of claim 1, wherein calculating the duration of contribution of the target imaging point to each pulse according to the motion trajectory of the target imaging point and the position relationship of the pixel to which each pulse belongs comprises:
solving a target imaging point I according to the following constraint relation k Duration of contribution of (x, y) to pulse s:
wherein the sensor coordinate to which the pulse s belongs is (m, n), and the accumulated time of the pulse is separatedStarting from t s And ends at t e If I k (x, y) at t k The + delta t moment acts on the pulse s, and the target imaging point I can be solved from the relation k The duration of the contribution of (x, y) to the pulse s, denoted c s (x, y) wherein t k + Δ t is a time point, and the time points satisfying the set of inequalities constitute a contribution duration, an optical flow field
4. The method of claim 1, wherein reconstructing the high resolution image sequence based on the weights of the contributions of the imaging points of the imaging grid to the respective pulses comprises:
constructing a prior constraint model based on local smoothness, and constructing a data fidelity constraint model according to contribution weight of imaging grid target imaging points to each pulse;
and calculating the brightness value of each pixel point in the image to be reconstructed according to the prior constraint model and the data fidelity constraint model to obtain a reconstructed high-resolution image sequence.
5. The method according to claim 4, wherein calculating the brightness value of each pixel point in the image to be reconstructed according to the prior constraint model and the data fidelity constraint model to obtain the reconstructed high-resolution image sequence comprises:
obtaining a first objective function according to the combination of the prior constraint model and the data fidelity constraint model;
and solving the first objective function in an iterative optimization solving mode, and calculating the brightness value of each pixel point in the image to be reconstructed to obtain a reconstructed high-resolution image sequence.
6. The method of claim 1, wherein reconstructing the high resolution image sequence based on the weights of the contributions of the imaging points of the imaging grid to the respective pulses comprises:
establishing a relation model between grid pixel values and pulses according to the contribution weight of the imaging grid target imaging point to each pulse;
and solving a second objective function corresponding to the relation model in an iterative optimization solving mode, and calculating the brightness value of each pixel point in the image to be reconstructed to obtain a reconstructed high-resolution image sequence.
7. The method of claim 6, wherein modeling a relationship between grid pixel values and pulses based on the weights of contributions of imaging grid target imaging points to respective pulses comprises modeling the relationship according to the following equation:
θ＝α·W s I
wherein, W s Representing the contribution weight of each target imaging point in the imaging grid to each pulse, I representing the image brightness matrix to be solved, α representing the photoelectric conversion rate of the pulse camera, and θ representing the pulse emission threshold.
8. An imaging device of a bionic pulse camera, comprising:
the first reconstruction module is used for reconstructing a basic image sequence according to a preset pulse camera imaging algorithm and a continuous pulse array in a preset time period;
a motion field determining module, configured to calculate a motion vector at any pixel position at any time from the basic image sequence, and form a motion field between the basic image sequences;
a weight determining module, configured to calculate a contribution weight of an imaging point of the imaging grid target to each pulse according to the motion field and the pulse array, including: calculating the motion trail of a target imaging point according to the motion field between the basic image sequences; calculating the contribution duration of each pulse of the target imaging point according to the motion track of the target imaging point and the position relation of the pixel to which each pulse belongs; calculating the contribution weight of the imaging grid target imaging point to each pulse according to the contribution duration of the target imaging point to each pulse;
and the second reconstruction module is used for reconstructing a high-resolution image sequence according to the contribution weight of the imaging point of the imaging grid target to each pulse.
9. An imaging device of a biomimetic pulse camera, comprising a processor and a memory storing program instructions, the processor being configured to perform the imaging method of the biomimetic pulse camera according to any of claims 1 to 7 when executing the program instructions.
10. A computer readable medium having computer readable instructions stored thereon which are executed by a processor to implement the imaging method of a biomimetic pulse camera as claimed in any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210096416.2A CN114584703B (en) | 2022-01-26 | 2022-01-26 | Imaging method, device, equipment and storage medium of bionic pulse camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210096416.2A CN114584703B (en) | 2022-01-26 | 2022-01-26 | Imaging method, device, equipment and storage medium of bionic pulse camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114584703A CN114584703A (en) | 2022-06-03 |
| CN114584703B true CN114584703B (en) | 2023-04-07 |
Family
ID=81772046
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210096416.2A Active CN114584703B (en) | 2022-01-26 | 2022-01-26 | Imaging method, device, equipment and storage medium of bionic pulse camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114584703B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115687911B (en) * | 2022-06-13 | 2023-06-02 | 北京融合未来技术有限公司 | Signal lamp detection method, device and system based on pulse signals |
| CN115246559A (en) * | 2022-07-22 | 2022-10-28 | 鄂尔多斯市国源矿业开发有限责任公司 | Industrial belt longitudinal tearing identification method |
| CN116456202B (en) * | 2023-04-25 | 2023-12-15 | 北京大学 | Pulse camera and color imaging method and device thereof |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113034634A (en) * | 2021-03-04 | 2021-06-25 | 北京大学 | Adaptive imaging method, system and computer medium based on pulse signal |
| CN113067979A (en) * | 2021-03-04 | 2021-07-02 | 北京大学 | Imaging method, device, equipment and storage medium based on bionic pulse camera |
| CN113327266A (en) * | 2021-04-26 | 2021-08-31 | 北京大学 | Optical flow estimation method and device based on pulse camera, computer equipment and medium |
-
2022
- 2022-01-26 CN CN202210096416.2A patent/CN114584703B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113034634A (en) * | 2021-03-04 | 2021-06-25 | 北京大学 | Adaptive imaging method, system and computer medium based on pulse signal |
| CN113067979A (en) * | 2021-03-04 | 2021-07-02 | 北京大学 | Imaging method, device, equipment and storage medium based on bionic pulse camera |
| CN113327266A (en) * | 2021-04-26 | 2021-08-31 | 北京大学 | Optical flow estimation method and device based on pulse camera, computer equipment and medium |
Non-Patent Citations (3)
| Title |
|---|
| 《High-Speed Motion Scene Reconstruction for Spike Camera via Motion Aligned Filtering》;Jing Zhao;《2020 IEEE International Symposium on Circuits and Systems 》;20200928;全文 * |
| 《Learning to Reconstruct Dynamic Scene from Continuous Spike Stream》;Jing Zhao;《2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition》;20211102;全文 * |
| 《Motion Estimation for Spike Camera Data Sequence via Spike Interval Analysis》;Jing Zhao;《2020 IEEE International Conference on Visual Communications and Image Processing》;20201229;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114584703A (en) | 2022-06-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114584703B (en) | Imaging method, device, equipment and storage medium of bionic pulse camera | |
| CN113067979A (en) | Imaging method, device, equipment and storage medium based on bionic pulse camera | |
| Xu et al. | Motion deblurring with real events | |
| WO2021208122A1 (en) | Blind video denoising method and device based on deep learning | |
| EP3046320B1 (en) | Method for generating an hdr image of a scene based on a tradeoff between brightness distribution and motion | |
| CN102970464B (en) | Image processing apparatus and image processing method | |
| CN110599400B (en) | EPI-based light field image super-resolution method | |
| CN110097509B (en) | Restoration method of local motion blurred image | |
| EP2608529B1 (en) | Camera and method for optimizing the exposure of an image frame in a sequence of image frames capturing a scene based on level of motion in the scene | |
| CN113034634B (en) | Adaptive imaging method, system and computer medium based on pulse signal | |
| KR101664123B1 (en) | Apparatus and method of creating high dynamic range image empty ghost image by using filtering | |
| Zhao et al. | High-speed motion scene reconstruction for spike camera via motion aligned filtering | |
| WO2022184167A1 (en) | Imaging method and apparatus, device, and storage medium | |
| Zhao et al. | Super resolve dynamic scene from continuous spike streams | |
| CN113329146B (en) | Pulse camera simulation method and device | |
| US8213496B2 (en) | Image processing device, image processing method, and image processing program | |
| Zhang et al. | Generalizing Event-Based Motion Deblurring in Real-World Scenarios | |
| CN111798484B (en) | Continuous dense optical flow estimation method and system based on event camera | |
| Zhao et al. | Learning to super-resolve dynamic scenes for neuromorphic spike camera | |
| Chang et al. | 1000 FPS HDR Video with a Spike-RGB Hybrid Camera | |
| Akhil et al. | Single Image HDR Synthesis Using a Densely Connected Dilated ConvNet. | |
| CN112487874A (en) | Method and system for eliminating background noise based on event camera | |
| CN111210390A (en) | Motion blur restoration method based on Golay sequence complementary code word set | |
| Dong et al. | High-speed scene reconstruction from low-light spike streams | |
| CN116091337A (en) | Image enhancement method and device based on event signal nerve coding mode |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |