WO2022184167A1

WO2022184167A1 - Imaging method and apparatus, device, and storage medium

Info

Publication number: WO2022184167A1
Application number: PCT/CN2022/079318
Authority: WO
Inventors: 熊瑞勤; 赵菁; 黄铁军
Original assignee: 脉冲视觉(北京)科技有限公司
Priority date: 2021-03-04
Filing date: 2022-03-04
Publication date: 2022-09-09

Abstract

The present application discloses an imaging method and apparatus, device, and storage medium. The imaging method comprises: obtaining reconstructed image sequences according to a pulse array within a preset time period; according to relative motion relationships between the image sequences, performing filtering in the time dimension on each pixel point of image frames in the image sequences that meets a specified condition to update a light intensity value of each pixel point to obtain updated light intensity values; and generating updated images according to the updated light intensity value of each pixel point. The imaging method provided by the embodiments of the present application solves the problem of difficulty of traditional cameras in imaging a high-speed moving object clearly, and enables the generation of a clear image having a high signal-to-noise ratio.

Description

Imaging method, apparatus, device and storage medium

technical field

The present application relates to the technical field of image processing, and in particular, to an imaging method, apparatus, device, and storage medium.

Background of the Invention

Traditional digital cameras usually shoot and image at a fixed frame rate, and each frame of image is generated in the following way: within a certain exposure time window, each pixel of the image sensor performs photoelectric conversion and accumulation of incident light, and after exposure, the analog-to-digital Convert to get the total amount of light for this pixel. Usually the length of the exposure time is determined by the light intensity. When the light is weak, it is often necessary to increase the exposure time to suppress the influence of dark current noise. This method cannot effectively image high-speed moving objects, which often leads to blurred imaging of high-speed moving objects. In recent years, the unique neuronal connection structure of the biological fovea and the integral firing model of ganglion cells have provided new ideas for visual sampling. Through the simulation and abstraction of the fovea, a pulsed camera including a photoreceptor, an integrator, and a threshold comparator is proposed. Pulse cameras represent visual information in the form of pulse arrays and can continuously record changes in light intensity without the concept of frame rate and exposure time, breaking through the limitations of traditional cameras. The texture details in the scene can be reconstructed.

In the prior art, the image reconstruction algorithm of the pulse camera includes two types: the image reconstruction algorithm based on the pulse interval and the image reconstruction algorithm based on the time window averaging. Among them, the image reconstruction algorithm based on the pulse interval utilizes the characteristic that the pulse interval decreases with the increase of the light intensity, and reconstructs the light intensity in a short period of time with the help of the two pulses before and after. Although this algorithm can describe the outline of high-speed moving objects, the reconstructed signal is usually not stable enough, and the pixel value has obvious fluctuations in the time direction; the image reconstruction algorithm based on the time window average uses the pulse firing frequency to increase with the increase of light intensity. This feature is used to estimate the average light intensity within a time window. Although the algorithm improves the signal-to-noise ratio of the reconstructed image to a certain extent, when there is motion of the object, the average in the time window will cause motion blur in the reconstructed image.

Therefore, in the prior art method for image reconstruction using a pulse camera, a clear image cannot be obtained in a high-speed motion scene.

SUMMARY OF THE INVENTION

Embodiments of the present application provide an imaging method, apparatus, device, and storage medium, which can improve the quality of reconstructed images and obtain clear images in high-speed motion scenes. In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended to be an extensive review, nor is it intended to identify key/critical elements or delineate the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simplified form as a prelude to the detailed description that follows.

In the first aspect, the embodiments of the present application provide an imaging method, comprising:

Obtain a reconstructed image sequence according to the pulse array within a preset time period;

According to the relative motion relationship between the image sequences, filtering in the time dimension is performed on each pixel point of the image frame in the image sequence that meets the specified condition to update the light intensity value of each pixel point, and obtain the updated light intensity value;

An updated image is generated according to the updated light intensity value of each pixel.

In one embodiment, the specified condition includes time k, which is within a preset time period.

In one embodiment, according to the relative motion relationship between the image sequences, filtering in the time dimension is performed on each pixel of the image frame in the image sequence that satisfies the specified condition to update the light intensity value of each pixel to obtain the updated light intensity values, including:

According to the relative motion relationship, determine the corresponding pixel points of each pixel point in the image frame at time t of the image sequence;

According to the light intensity value of the corresponding pixel point, each pixel point is filtered in the time dimension to obtain the updated light intensity value.

In one embodiment, according to the light intensity value of the corresponding pixel point, filtering in the time dimension is performed on each pixel point to obtain the updated light intensity value, including:

According to the light intensity value of the corresponding pixel point, the following formula is used to filter each pixel point in the time dimension to obtain the updated light intensity value,

Among them, I _k (x, y) represents the updated light intensity value at the (x, y) pixel point in the image frame at time k, C represents the regularization parameter,

Represents the filter weight of the image frame at time t when reconstructing the updated image,

represents the light intensity value of the pixel in the image frame at time k at the corresponding pixel point in the image frame at time t, and r represents the radius of the filtering time window.

In one embodiment, the regularization parameter C is determined by the following formula:

In one embodiment,

Determined by the following formula:

Among them, σ is the setting parameter.

In one embodiment, the setting parameters are determined by the following formula:

σ=(2*r+1)/3.

In one embodiment, the imaging method further includes:

If the corresponding pixel is an integer pixel, directly take the light intensity value at the corresponding pixel;

If the corresponding pixel is a sub-pixel, the light intensity value of the corresponding pixel is obtained by interpolation.

According to the relative motion relationship, the autoregressive model of the pixels in the image sequence in the time direction is adaptively learned to determine the model parameters of the autoregressive model;

Determine the corresponding pixel points of each pixel point in the image frame at time t of the image sequence;

According to the model parameters and the light intensity value of the corresponding pixel point, each pixel point is filtered in the time dimension to obtain the updated light intensity value.

In one embodiment, the formula of the autoregressive model is:

Among them, I _k represents the theoretical light intensity value at time k; I _t represents the theoretical light intensity value at time t; T _k represents the set of adjacent moments at time k; P _k→t (x, y) is the image frame at time k The corresponding pixel position of the pixel point (x, y) in the image frame at time t; α _t is the model parameter; Δ _k (x, y) is the model error on the (x, y) pixel point.

In one embodiment, according to the relative motion relationship, adaptive learning is performed on the autoregressive model of the pixel points in the image sequence in the time direction to determine the model parameters of the autoregressive model, including:

According to the relative motion relationship, the autoregressive model is adaptively learned by the least square method to solve the objective function to determine the model parameters.

In one embodiment, the objective function is:

Among them, W represents the set of pixels in the spatial range centered on (x, y);

Represents the estimated light intensity value at time k;

represents the estimated light intensity value at time t.

In one embodiment, according to the model parameters and the light intensity value of the corresponding pixel point, filtering in the time dimension is performed on each pixel point to obtain the updated light intensity value, including:

According to the model parameters and the light intensity value of the corresponding pixel point, the following formula is used to filter each pixel point in the time dimension to obtain the updated light intensity value,

Among them, I′ _k represents the updated light intensity value at time k.

In one embodiment, the imaging method further includes: obtaining the relative motion between the image sequences according to any one or more of optical flow method, pixel point matching, pixel point motion alignment, and pixel point relative position offset estimation relation.

In one embodiment, before obtaining the reconstructed image sequence according to the pulse array within a preset time period, the imaging method further includes:

According to the pulse signal within a preset time period, the pixel-based pulse array of the object to be imaged is obtained.

In one embodiment, a reconstructed image sequence is obtained according to the pulse array within a preset time period, including:

According to the pulse array, the image sequence is reconstructed using the pulse interval algorithm.

In a second aspect, an embodiment of the present application provides an imaging device, including:

a reconstruction module, configured to obtain a reconstructed image sequence according to the pulse array within a preset time period;

The update module is used to filter each pixel point of the image frame that meets the specified condition in the image sequence in the time dimension according to the relative motion relationship between the image sequences to update the light intensity value of each pixel point, and obtain the updated light intensity value ;

The imaging module is configured to generate an updated image according to the updated light intensity value of each pixel point.

In a third aspect, embodiments of the present application provide an electronic device, including a processor and a memory storing program instructions, where the processor is configured to execute the imaging method provided by the foregoing embodiments when executing the program instructions.

In a fourth aspect, the embodiments of the present application provide an imaging device, including the electronic device provided by the above embodiments.

In a fifth aspect, the embodiments of the present application provide a computer-readable medium on which computer-readable instructions are stored, and the computer-readable instructions can be executed by a processor to implement the imaging method provided by the foregoing embodiments.

In the imaging method provided by the embodiment of the present application, a reconstructed image sequence is obtained according to a pulse array within a preset time period; according to the relative motion relationship between the image sequences, each pixel of an image frame that meets a specified condition in the image sequence is analyzed. Filtering in the time dimension is performed to update the light intensity value of each pixel point to obtain the updated light intensity value; an updated image is generated according to the updated light intensity value of each pixel point. The method proposes to filter the image pixels along the moving trajectory of the object in the temporal dimension, which can be applied to high-speed moving scenes to generate images with high signal-to-noise ratio and no blur, and achieve better visual effects. The problem of clear imaging of high-speed moving objects.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of the present application.

Brief Description of Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description serve to explain the principles of the application.

FIG. 1 is a schematic flowchart of an imaging method according to an exemplary embodiment of the present application.

FIG. 2 is a schematic diagram of a pulse array according to an exemplary embodiment of the present application.

FIG. 3 is a schematic diagram of motion alignment according to an exemplary embodiment of the present application.

FIG. 4 is a schematic diagram of calculating a light intensity value according to an interpolation method according to an exemplary embodiment of the present application.

FIG. 5 is a schematic flowchart of an imaging method according to another exemplary embodiment of the present application.

FIG. 6 is a schematic flowchart of an imaging method according to another exemplary embodiment of the present application.

FIG. 7 is a schematic flowchart of an imaging method according to another exemplary embodiment of the present application.

FIG. 8 is a schematic flowchart of an imaging method according to another exemplary embodiment of the present application.

FIG. 9 is a schematic diagram of an imaging picture taken by different imaging methods according to an exemplary embodiment of the present application.

FIG. 10 is a schematic diagram of an imaging picture taken by different imaging methods according to another exemplary embodiment of the present application.

FIG. 11 is a schematic structural diagram of an imaging device according to an exemplary embodiment of the present application.

FIG. 12 is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present application.

FIG. 13 is a schematic diagram of a computer storage medium according to an exemplary embodiment of the present application.

MODES OF IMPLEMENTING THE INVENTION

In order to have a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings. In the following technical description, for the convenience of explanation, numerous details are provided to provide a thorough understanding of the disclosed embodiments. However, one or more embodiments may be practiced without these details. In other instances, well-known structures and devices may be shown simplified in order to simplify the drawings. Obviously, the embodiments described in the present application are only a part of the embodiments of the present application, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict.

In the process of realizing the present application, the applicant found that traditional cameras cannot effectively image high-speed moving objects, which often leads to blurred images of high-speed moving objects. Therefore, inspired by the visual sampling mechanism of the biological retina, new cameras that collect pulse array signals have entered people's field of vision, including sensors that emit pulse signals based on changes in light intensity, such as Dynamic Vision Sensor (DVS), asynchronous time-based sensors. Image sensor (Asynchronous Time-based image Sensor, ATIS), dynamic active pixel vision sensor (Dynamic and Active Pixel Vision Sensor, DAVIS), etc., sensors that emit signals based on the cumulative intensity of light intensity, such as light intensity cumulative sensors, etc. The sensor of this camera collects the information of the light signal in a certain area for a certain time, and has the advantages of high dynamic range and high time resolution.

While the imaging algorithm of the pulse camera based on the pulse signal can describe the contour of the high-speed moving object, the reconstructed signal is usually not stable and perfect, and the pixel value in the time direction has obvious fluctuations, or the reconstructed image has motion blur. question. Therefore, a new imaging method is urgently needed to solve the above problem of unclear imaging of pulse cameras in high-speed motion scenes.

Applicants have also found that pulse arrays have correlations in the temporal direction, and that correlations behave differently in different spatial locations of the image. For example, in a smooth motion area with relatively continuous motion, the temporal correlation of the image content along the motion trajectory is relatively strong; while for an area with relatively complex local motion, the image content at a certain moment may be different from the image content at the previous and subsequent times. There is a corresponding relationship, and even occlusion or disappearance may occur. At this time, the correlation of the time direction of the image content along the motion trajectory is relatively weak.

In order to reasonably utilize the correlation in the time direction of the pulse array to improve the quality of the reconstructed image, based on the above, an embodiment of the present application proposes an imaging method.

As shown in FIG. 1, the imaging method includes the following contents.

S110: Obtain a reconstructed image sequence according to the pulse array within a preset time period.

In a possible implementation manner, before step S110 is performed, the imaging method further includes: acquiring a pixel-based pulse array of the object to be imaged according to the pulse signal within a preset time period.

Specifically, in order to break the limitations of traditional cameras, the embodiments of the present application may use a bionic pulse camera for imaging, and each pixel in the bionic pulse camera includes a photoreceptor, an integrator, and a threshold comparator. Among them, the photoreceptor is responsible for converting the optical signal into an electrical signal, the integrator accumulates the converted photo-generated charges, and the threshold comparator repeatedly compares the accumulated charges at an ultra-high frequency. When the accumulated charges reach a preset threshold , the pixel is pulsed and the integrator is cleared. Figure 2 is a schematic diagram of a pulse array. As shown in Figure 2, the pulse camera represents visual information in the form of "H*W*T" pulse array, "H*W" is the spatial resolution of the pulse camera, and "T" is the sampling times. We call the signal sequence output by a single pixel as a "pulse sequence", and the cross-section of the pulse array at a certain moment is a "pulse matrix". The pulse array consists of two symbols, "0" and "1", where "1" (solid dot) indicates that there is a pulse firing at this spatiotemporal position; "0" (open dot) means that there is no pulse firing at this spatiotemporal position. The bionic pulse camera can continuously record changes in light intensity without the concept of frame rate and exposure time, breaking through the limitations of traditional cameras.

Further, after acquiring the pixel-based pulse array of the object to be imaged, a basic image sequence is reconstructed according to the acquired pulse array within a preset time period. In a possible implementation manner, according to the pulse array, a pulse interval algorithm can be used to reconstruct the image sequence, or other algorithms can be used to estimate the basic reconstructed image sequence according to the pulse array.

In one embodiment, the image sequence is reconstructed using a pulse interval algorithm, and the pulse interval is the time interval between two adjacent pulses at the same pixel position, which is equivalent to the integration time used by the integrator to accumulate the next pulse. Generally speaking, the integrator of the pixel in the strong illumination area can accumulate the photogenerated charge to the pulse emission threshold in a shorter time, and the pulse interval is small; the integrator of the pixel in the weak illumination area takes a longer time to accumulate the charge to Threshold of pulse release, pulse interval is larger. According to the characteristic that the average light intensity between pulses is inversely proportional to the size of the pulse interval, the basic image sequence is reconstructed

According to this step, the basic image sequence can be reconstructed according to the pulse array using the pulse interval algorithm.

S120: According to the relative motion relationship between the image sequences, perform filtering in the time dimension on each pixel of the image frame in the image sequence that meets the specified condition to update the light intensity value of each pixel to obtain an updated light intensity value.

Specifically, the specified condition may be a certain moment, or an image frame with a certain feature. For example, the specified condition includes time k, which is within a preset time period.

According to the relative motion relationship between the image sequences, the motion alignment of the image sequences can be performed to obtain the corresponding relationship between pixels in different image frames in the image sequence. When the camera position is fixed, with the movement of the object, a certain point in the object has different pixel positions in the image frames at different times. Therefore, for a certain pixel in the image frame at time k, corresponding pixels in the image frame at other times can be obtained. According to the correspondence between each pixel in different image frames in the image sequence, the motion trajectory of each pixel in different image frames can be obtained.

Based on the motion trajectory of the pixel point, the light intensity value of the pixel point at different times can be determined. For example, for each pixel in the image frame at time k, filtering in the time dimension can be performed on the pixel based on the corresponding light intensity values of the pixel in the image frames at different times to update the light intensity of the pixel. Intensity value, get updated light intensity value. An updated image can be generated according to the updated light intensity value of each pixel in the image frame at time k.

S130: Generate an updated image according to the updated light intensity value of each pixel.

The imaging method provided by the embodiment of the present application utilizes the temporal direction correlation of the pulse array to improve the quality of the reconstructed image.

In the embodiment of the present application, according to the relative motion relationship between the image sequences, filtering in the time dimension is performed on each pixel point of the image frame in the image sequence that meets the specified condition to update the light intensity value of each pixel point, so as to obtain the updated light Intensity value, and then generate an updated image according to the updated light intensity value of each pixel, so that the quality of the reconstructed image can be improved, a clear image in a high-speed motion scene can be obtained, and a better visual effect can be achieved. The problem of clear imaging of moving objects.

According to an embodiment of the present application, step S120 includes: determining the corresponding pixel points of each pixel point in the image frame at time t of the image sequence according to the relative motion relationship; Dimensional filtering to get updated light intensity values.

Specifically, for each pixel in the image frame at time k, the corresponding pixel of the pixel in the image frame at other times may be determined based on the relative motion relationship. The light intensity value of the corresponding pixel point can be considered as the light intensity value corresponding to the pixel point in the image frame at time k at other time points in the image frame.

According to an embodiment of the present application, the imaging method further includes: obtaining the relative relationship between the image sequences according to any one or more of optical flow method, pixel point matching, pixel point motion alignment, and pixel point relative position offset estimation. sports relationship.

For example, the relative motion relationship between image sequences is obtained according to the optical flow method.

In a possible implementation, the current frame is calculated using an optical flow algorithm such as Horn–Schunck

The optical flow field between the r frame images before and after it is used to determine the relative motion relationship between the images.

Among them, the calculation of the optical flow field is to find the correspondence between the image sequences. The optical flow field can be understood as a motion field, and the apparent motion of the image brightness pattern is the optical flow.

In an exemplary scene, assuming that a triangle is in free fall in the captured scene, then if the camera is kept still, its position in the images captured at different times is different, and the optical flow method is used to find the same object in different images. The optical flow method is mostly based on the principle of light intensity consistency, that is, it is assumed that the same object has the same light intensity value. For example, the (x, y) point of the first image corresponds to the (x', y') point of the second image, and UV_t^k(x, y) is the optical flow at the (x, y) point (describes the two relative motion at that point between frames).

According to the above steps, the optical flow method can be used to solve the optical flow (relative motion) between two frames based on the assumption of light intensity consistency, and further, based on the optical flow (relative motion), the correspondence between images between different frames can be determined, That is, motion alignment, get each pixel (x, y) in the current frame in other

the corresponding position on the frame

Figure 3 is a schematic diagram of motion alignment. As shown in Figure 3, the positions of free-falling triangles on different images are different. Based on the optical flow information, we can determine the correspondence between the images and find the same object point in the scene at different time points. The position on the object determines the trajectory of the object.

In one embodiment, after obtaining the corresponding relationship between the pixels in different frames (according to the relative motion relationship, after determining the corresponding pixels of each pixel in the image frame at time t of the image sequence), the imaging method further includes: Obtain the light intensity value of the pixel point of the current frame at the corresponding pixel point on other frame images (obtain the light intensity value of the corresponding pixel point).

In one embodiment, acquiring the light intensity value of the corresponding pixel point includes: if the corresponding pixel point is an integer pixel point, directly obtaining the light intensity value at the corresponding pixel point; if the corresponding pixel point is a sub-pixel point, using an interpolation method Get the light intensity value of the corresponding pixel point.

Specifically, if the pixel of the current frame corresponds to the pixel in other frame images, the light intensity value at the corresponding pixel is directly taken; if the pixel of the current frame corresponds to the pixel in other frame images is a sub-pixel point, then the light intensity value at the corresponding pixel point is obtained by interpolation.

Figure 4 is a schematic diagram of calculating the light intensity value using the interpolation method. As shown in Figure 4, assuming that after the motion is aligned, the corresponding point of the current point on other frame images is not at an integer position, then it is necessary to use the surrounding whole point. light intensity value to estimate the light intensity value at that point. Taking bilinear interpolation as an example, P is a non-integer point, and the light intensity value of this point is unknown, then the brightness information of the surrounding integer points Q ₁₁ , Q ₁₂ , Q ₂₁ , and Q ₂₂ can be used to interpolate the light intensity of point P. light intensity value. The specific methods are as follows:

where R ₁ =(x,y ₁ )

where R ₂ =(x,y ₂ )

Among them, f(.) represents the light intensity value of a certain point.

According to an embodiment of the present application, according to the light intensity value of the corresponding pixel point, filtering each pixel point in the time dimension to obtain the updated light intensity value includes: according to the light intensity value of the corresponding pixel point, through the following formula, for each pixel point The pixel points are filtered in the time dimension to obtain the updated light intensity value,

Represents the filter weight of the image frame at time t when reconstructing the updated image (reconstructing the image frame at time k),

Weights

The selection of is gradually reduced with the increase of the time interval between two frames, which can be calculated by the following expression or other similar expressions.

Wherein, σ can be a set parameter, and its value is determined by the size of the time window, for example, σ=(2*r+1)/3, and r is the radius of the filter time window.

According to an embodiment of the present application,

Optionally, according to the light intensity value of the corresponding pixel point, the following formula can be used to filter each pixel point in the time dimension to obtain the updated light intensity value,

where T represents the total number of frames in the image sequence,

1≤k≤T.

It can be determined according to the method provided in the foregoing embodiment, or determined by other methods according to the actual situation.

In the embodiment of the present application, according to the corresponding relationship between the pixels in different frames, the motion trajectory of the object can be obtained, and the signal is filtered along the time dimension along the motion trajectory of the object, and the image frame at time k can be updated (x, y) The light intensity value on the pixel point.

In this embodiment, motion trajectory filtering in the time dimension can be performed on the basic reconstructed image sequence to improve the stability of the signal, thereby improving the quality of the reconstructed image. Among them, the motion trajectory filtering along the time dimension realizes the utilization of the time correlation of the pulse array, and the motion alignment ensures the accuracy of the time direction correlation used for the moving object.

FIG. 5 is an imaging method provided by another embodiment of the present application, specifically an imaging method based on a bionic pulse camera. FIG. 5 is an example of the embodiment of FIG. 1 . As shown in FIG. 5 , the method specifically includes the following contents.

S510: Obtain a reconstructed image sequence according to the pulse array within a preset time period.

S520: Obtain the relative motion relationship between the image sequences according to the optical flow method.

S530: Perform motion alignment on the image sequence according to the relative motion relationship to obtain the corresponding relationship between pixels in different image frames.

S540: According to the correspondence between the pixels of different image frames, perform filtering on the pixels of the image sequence in the time dimension, and update the light intensity values of the pixels of the image to be reconstructed to generate an updated image.

Specifically, according to the imaging method provided by the embodiments of the present application, one frame of images or multiple frames of images in the image sequence can be reconstructed.

In order to facilitate the understanding of the imaging method based on the bionic pulse camera provided by the embodiment of the present application, the following description is made with reference to FIG. 6 . As shown in Figure 6, the method includes the following contents.

First, a bionic pulse camera is used for imaging, and a pixel-based pulse array of the object to be imaged is obtained according to the pulse signal within a preset time period. Then, based on the pulse array, a pulse interval algorithm is used to obtain a reconstructed image sequence (basic reconstruction).

Further, the optical flow estimation algorithm such as Horn-Schunck is used to calculate the optical flow field between the current frame and its previous and previous frame images, so as to determine the relative motion relationship between the images.

According to the above steps, the optical flow method can be used to solve the optical flow (relative motion) between two frames based on the assumption of light intensity consistency, and further, based on the optical flow (relative motion), the correspondence between images between different frames can be determined, i.e. motion alignment. Get each pixel (x, y) in the current frame in other

the corresponding position on the frame

After obtaining the correspondence between the pixels of different frames, the method further includes: acquiring the light intensity values of the pixels of the current frame at the corresponding pixels of other frame images.

Finally, according to the correspondence between the pixels in different frames, the motion trajectory of the object can be obtained, and the signal is filtered in the time dimension along the motion trajectory of the object, and the light intensity value of the pixel is updated to generate an updated image.

In the embodiment of the present application, firstly, the image sequence reconstruction is performed on the scene at different times according to the characteristics of the pulse array; then, on the basis of the reconstructed image sequence, the optical flow estimation algorithm is used to determine the motion trajectory of the object along the time direction, along the respective The motion track performs motion alignment on the image sequence; finally, the reconstructed image sequence is subjected to time dimension motion track filtering to improve the stability of the signal, thereby improving the quality of the reconstructed image, obtaining clear images in high-speed motion scenes, and achieving better visual effects. , which solves the problem that high-speed moving objects cannot be clearly imaged in the prior art. The imaging method based on the bionic pulse camera provided in the embodiment of the present application can be applied to a high-speed motion scene, and can generate an image with a high signal-to-noise ratio and no blur.

According to an embodiment of the present application, step S120 includes: performing adaptive learning on the autoregressive model of the pixels in the image sequence in the time direction according to the relative motion relationship to determine the model parameters of the autoregressive model; The corresponding pixel points in the image frame at time t of the sequence; according to the model parameters and the light intensity value of the corresponding pixel point, each pixel point is filtered in the time dimension to obtain the updated light intensity value.

Specifically, the autoregressive model can be set in advance as required.

In one embodiment, the formula of the autoregressive model is:

Among them, I _k represents the theoretical light intensity value (image theoretical value) at time k; I _t represents the theoretical light intensity value (image theoretical value) at time t; T _k represents the set of adjacent moments at time k; P _k→t ( x, y) is the corresponding pixel position of the pixel (x, y) in the image frame at time k in the image frame at time t; α _t is the model parameter; Δ _k (x, y) is the pixel at (x, y) ) model error on pixels, which may correspond to some image details or noise.

According to an embodiment of the present application, according to the relative motion relationship, adaptive learning is performed on the autoregressive model of the pixel points in the image sequence in the time direction, so as to determine the model parameters of the autoregressive model, including: according to the relative motion relationship, through the least two The method of multiplying to solve the objective function performs adaptive learning on the autoregressive model to determine the model parameters.

Specifically, the pixels on the image sequence can be adaptively selected, the autoregressive model can be adaptively learned, and the model parameters of the autoregressive model can be determined. For example, the objective function is solved by the least squares method to obtain the model parameters of the autoregressive model. The objective function can be:

Represents the estimated light intensity value at time k (image estimated value);

Indicates the estimated light intensity value (image estimated value) at time t.

In this step, in different image regions, the temporal correlation of the image content along the motion trajectory is different. Therefore, it is necessary to adjust the parameters of the autoregressive model adaptively according to the content of the local region of the image sequence, so as to determine the suitable local Filter weights for regions. When filtering the pixels in the pulse array in the temporal direction, the filter weight is related to the content of the pulse array, and the filter weight at different spatial positions of an image may be different.

Therefore, the present application assumes that pixels with similar spatial positions have relatively consistent temporal correlations, so the model parameters can be adaptively selected according to the data at the local spatial positions, and can be obtained by optimizing the above objective function.

According to an embodiment of the present application, according to the model parameters and the light intensity value of the corresponding pixel point, filtering each pixel point in the time dimension to obtain the updated light intensity value, including: according to the model parameter and the light intensity value of the corresponding pixel point, Through the following formula, each pixel is filtered in the time dimension to obtain the updated light intensity value,

Among them, I′ _k represents the updated light intensity value at time k, or is called image adaptive imaging value. Specifically, on the basis of obtaining the model parameters, this formula is used to filter the pixel points in the time dimension, and the obtained result is used as the updated light intensity value on the (x, y) pixel point in the image frame at time k, which can be finally obtained. Adaptive imaging of the object is obtained.

In other embodiments, an autoregressive model of each pixel in the time direction may be established according to the motion trajectory of each pixel of the image sequence. For example, the motion trajectory of each pixel of the image sequence may be determined according to the relative motion relationship between the image sequences, and reference may be made to the description in the foregoing embodiments for the determination of the motion trajectory. Then the pixels on the image sequence are adaptively selected, and the autoregressive model is adaptively learned to determine the model parameters.

FIG. 7 is an imaging method provided by another embodiment of the present application, specifically an adaptive imaging method based on a pulse signal. FIG. 7 is an example of the embodiment of FIG. 1 . As shown in FIG. 7 , the method specifically includes the following contents.

S710: Obtain a reconstructed image sequence according to the pulse array within a preset time period.

S720: Determine the motion trajectory of each pixel of the image sequence according to the relative motion relationship between the image sequences.

S730: Establish an autoregressive model of each pixel in the time direction according to the motion trajectory of each pixel in the image sequence.

Specifically, an appropriate autoregressive model can be established according to the motion trajectory of each pixel point.

S740: Adaptively select pixels on the image sequence, perform adaptive learning on the autoregressive model, and determine model parameters of the autoregressive model.

S750: Perform filtering in the time dimension on the pixels of the image sequence according to the model parameters, and update the light intensity values of the pixels of the image to be reconstructed to generate an updated image.

In order to facilitate the understanding of the adaptive imaging method based on the pulse signal provided by the embodiment of the present application, the following description is made with reference to FIG. 8 . As shown in Figure 8, the method includes the following contents.

First, a basic reconstructed image sequence (reconstructed basic image) is estimated from the pulse array based on the pulse interval information. The basic reconstructed image is generated by the pulse array, and a pulse spacing algorithm or other reconstruction method can be used. According to the characteristic that the average optical flow between pulses is inversely proportional to the size of the pulse interval, the basic image sequence is reconstructed:

in,

represents the reconstructed image sequence,

represents each reconstructed image. Then, the motion trajectory of each pixel point of the image is determined (establishing the motion trajectory) according to the basic reconstructed image sequence by means of methods such as optical flow.

Next, an autoregressive model in the time dimension is established along the direction of the motion trajectory (establishing an autoregressive model), and the model parameters are adaptively adjusted according to the temporal correlation of neighboring pixels (adaptive learning model parameters).

Finally, according to the obtained model, the motion trajectory filtering of the time dimension (time dimension filtering) is performed on the image pixels to reconstruct a higher quality image.

The present application establishes an autoregressive model between image pixels at different times along the motion trajectory of the object, and adaptively adjusts the model parameters according to the content of the image sequence, so as to accurately utilize the temporal direction correlation of the image signal, thereby improving the reconstructed image. the quality of.

The adaptive imaging method based on pulse signals of the present application obtains a pixel-based pulse array of an object to be imaged according to the pulse signals within a preset time period; obtains a reconstructed image sequence within a preset time period according to the pulse array; The relative motion relationship between the image sequences determines the motion trajectory of each pixel in the reconstructed image sequence; according to the motion trajectory of each pixel, the autoregressive model of each pixel in the time direction is established; the pixels on the reconstructed image sequence are selected , the autoregressive model is adaptively learned, and the model parameters of the autoregressive model are determined; according to the model parameters, the pixel points of the reconstructed image sequence are filtered in the time dimension, and the light intensity value of the pixel points is updated to generate an updated image.

The present application adaptively utilizes the temporal direction correlation of the pulse array to improve the quality of the reconstructed image. In order to reasonably utilize the temporal correlation of the pulse array, firstly, an autoregressive model along the time direction of the motion trajectory is established; then, the model parameters are adaptively adjusted according to the correlation structure of the local spatial position of the pulse array , in order to improve the accuracy of the autoregressive model; finally, the basic reconstructed image is filtered in the time dimension by the established local adaptive autoregressive model to improve the stability of the signal, thereby improving the quality of the reconstructed image.

Among them, the locally adaptive autoregressive model ensures the accuracy of the used time-direction correlation, effectively reduces the influence of outliers, and ensures the effectiveness and robustness of the filtering algorithm.

Hereinafter, the imaging method provided by the present application, such as an imaging method based on a biomimetic pulse camera, or an adaptive imaging method based on a pulse signal, will be further described through specific application scenarios.

In an exemplary scene, an experiment is performed on real pulse array data captured by a pulse camera to test the reconstruction performance of the imaging method provided in this embodiment in a high-speed motion scene. FIG. 9 is a schematic diagram of a car traveling at a speed of 100 km/h shot by different imaging methods according to an exemplary embodiment. As shown in FIG. 9 , (a) is an initial pulse matrix image, (b) is an image reconstructed using the pulse interval method, and (c) is an image reconstructed using the imaging method provided by the embodiment of the present application. It can be seen from the pictures that, according to the image reconstruction algorithm of the embodiment of the present application, a clear image of a vehicle traveling at a high speed can be obtained, and a better visual effect can be achieved.

FIG. 10 is a schematic diagram of a free-falling ragdoll photographed by different imaging methods according to an exemplary embodiment. (a) is an initial pulse matrix image, (b) is an image reconstructed using the pulse interval method, and (c) is an image reconstructed using the imaging method provided by the embodiments of the present application. It can be seen from the images that the embodiments of the present application can clearly reconstruct the motion details in the scene, and the reconstructed images not only avoid motion blur but also have a high signal-to-noise ratio.

An embodiment of the present application further provides an imaging apparatus, which is used for executing the imaging method of the foregoing embodiment. For details not disclosed in the imaging device of this embodiment, please refer to the description of the imaging method in the above-mentioned embodiment. As shown in FIG. 11 , the apparatus includes: a reconstruction module 1110 , an update module 1120 and an imaging module 1130 .

The reconstruction module 1110 is used for obtaining the reconstructed image sequence according to the pulse array in the preset time period; the updating module 1120 is used for, according to the relative motion relationship between the image sequences, for the image frames that meet the specified conditions in the image sequence Each pixel of the pixel is filtered in the time dimension to update the light intensity value of each pixel to obtain an updated light intensity value; the imaging module 1130 is configured to generate an updated image according to the updated light intensity value of each pixel.

In one embodiment, the updating module 1120 is configured to: determine the corresponding pixel points of each pixel point in the image frame at time t of the image sequence according to the relative motion relationship; Filtering in the time dimension to get updated light intensity values.

In one embodiment, the update module 1120 is configured to: perform filtering on each pixel in the time dimension according to the light intensity value of the corresponding pixel point by the following formula to obtain the updated light intensity value,

In one embodiment,

Determined by the following formula:

Among them, σ is the setting parameter.

σ=(2*r+1)/3.

In one embodiment, the imaging device further includes: a first acquiring module 1140, configured to acquire the light intensity value of the corresponding pixel point.

In one embodiment, the first obtaining module 1140 is configured to: if the corresponding pixel is an integer pixel, directly obtain the light intensity value at the corresponding pixel; if the corresponding pixel is a sub-pixel, obtain the corresponding pixel by interpolation The light intensity value of the pixel.

In one embodiment, the updating module 1120 is configured to: perform adaptive learning on the autoregressive model of the pixel points in the image sequence in the time direction according to the relative motion relationship, so as to determine the model parameters of the autoregressive model; The corresponding pixel points in the image frame at time t of the image sequence; according to the model parameters and the light intensity value of the corresponding pixel point, each pixel point is filtered in the time dimension to obtain the updated light intensity value.

In one embodiment, the formula of the autoregressive model is:

In one embodiment, the update module 1120 is configured to: perform adaptive learning on the autoregressive model by solving the objective function by the least squares method according to the relative motion relationship, so as to determine the model parameters.

In one embodiment, the objective function is:

Represents the estimated light intensity value at time k;

represents the estimated light intensity value at time t.

In one embodiment, the update module 1120 is configured to: filter each pixel in the time dimension to obtain the updated light intensity value by using the following formula according to the model parameters and the light intensity value of the corresponding pixel point,

Among them, I′ _k represents the updated light intensity value at time k.

In one embodiment, the imaging device further includes: a second acquisition module 1150, configured to: according to any one or more of optical flow method, pixel point matching, pixel point motion alignment and pixel point relative position offset estimation, Obtain the relative motion relationship between image sequences.

In one embodiment, the imaging device further includes: a third acquisition module 1160, configured to: acquire a pixel-based pulse array of the object to be imaged according to the pulse signal within a preset time period.

In one embodiment, the reconstruction module 1110 is configured to: reconstruct an image sequence by using a pulse interval algorithm according to the pulse array.

It should be noted that, when the imaging apparatus provided in the above-mentioned embodiments executes the imaging method, only the division of the above-mentioned functional modules is used as an example. The internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the imaging apparatus and the imaging method embodiments provided by the above embodiments belong to the same concept, and the specific implementation process and effects thereof are described in the description of the method embodiment section, which will not be repeated here.

The embodiment of the present application further provides an electronic device corresponding to the imaging method provided by the foregoing embodiments, so as to execute the foregoing imaging method.

Please refer to FIG. 12 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. As shown in FIG. 12 , the electronic device includes: a processor 1210 and a memory 1220 . The memory 1220 stores a computer program that can be executed on the processor 1210. When the processor 1210 runs the computer program, the imaging method provided by any of the foregoing embodiments of the present application is executed. Further, the electronic device also includes a bus 1230 and a communication interface 1240 . The processor 1210 , the communication interface 1240 and the memory 1220 are connected through the bus 1230 .

The memory 1220 may include a high-speed random access memory (RAM: Random Access Memory), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The communication connection between the network element of the system and at least one other network element is implemented through at least one communication interface 1240 (which may be wired or wireless), which may use the Internet, a wide area network, a local network, a metropolitan area network, and the like.

The bus 1230 may be an ISA bus, a PCI bus, an EISA bus, or the like. The bus can be divided into address bus, data bus, control bus and so on. The memory 1220 is used to store a program, and the processor 1210 executes the program after receiving the execution instruction. The imaging method based on the bionic pulse camera disclosed in any of the foregoing embodiments of the present application may be applied to the processor 1210, or Implemented by processor 1210 .

The processor 1210 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above-mentioned method can be completed by an integrated logic circuit of hardware in the processor 1210 or an instruction in the form of software. The above-mentioned processor 900 can be a general-purpose processor, including a central processing unit (Central Processing Unit, referred to as CPU), a network processor (Network Processor, referred to as NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component. The processor 1210 may implement or execute the methods, steps, and logical block diagrams disclosed in the embodiments of this application. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the methods disclosed in the embodiments of the present application may be directly executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory 1220, and the processor 1210 reads the information in the memory 1220, and completes the steps of the above method in combination with its hardware.

The electronic device provided by the embodiment of the present application and the imaging method provided by the embodiment of the present application are based on the same inventive concept, and have the same beneficial effects as the method adopted, operated or realized.

Embodiments of the present application further provide an imaging device, including the electronic device provided by the foregoing embodiments. For example, the imaging device may be a camera, a monitor, or the like.

Embodiments of the present application also provide a computer-readable storage medium corresponding to the imaging method provided by the foregoing embodiments. Please refer to FIG. 13 , the computer-readable storage medium shown is an optical disc 1300, on which a computer program ( That is, a program product), when the computer program is executed by the processor, the imaging method provided by any of the foregoing embodiments will be executed.

It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory Memory (RAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash Memory or other optical and magnetic storage media will not be repeated here.

The computer-readable storage medium provided by the above embodiments of the present application and the imaging methods provided by the embodiments of the present application are based on the same inventive concept, and have the same beneficial effects as the methods used, run or implemented by the application programs stored thereon.

The above are only the preferred specific embodiments of the present application, but the protection scope of the present application is not limited to this. Substitutions should be covered within the protection scope of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

An imaging method, characterized in that, comprising:

Obtain a reconstructed image sequence according to the pulse array within a preset time period;

According to the relative motion relationship between the image sequences, filtering in the time dimension is performed on each pixel of the image frame in the image sequence that satisfies the specified condition to update the light intensity value of each pixel to obtain the updated light intensity value;

An updated image is generated according to the updated light intensity value of each pixel point.
The method according to claim 1, wherein the specified condition includes time k, and the time k is within the preset time period.
The method according to claim 2, wherein, according to the relative motion relationship between the image sequences, filtering in the time dimension is performed on each pixel of the image frame in the image sequence that meets the specified condition to Update the light intensity value of each pixel to obtain the updated light intensity value, including:

According to the relative motion relationship, determine the corresponding pixel points of each pixel point in the image frame at time t of the image sequence;

According to the light intensity value of the corresponding pixel point, filtering in the time dimension is performed on each pixel point to obtain the updated light intensity value.
The method according to claim 3, wherein, according to the light intensity value of the corresponding pixel point, filtering the pixel points in the time dimension to obtain the updated light intensity value, comprising:

According to the light intensity value of the corresponding pixel point, the following formula is used to filter each pixel point in the time dimension to obtain the updated light intensity value,

Among them, I k (x, y) represents the updated light intensity value at the (x, y) pixel point in the image frame at the k moment, C represents the regularization parameter,
represents the filter weight of the image frame at time t when reconstructing the updated image,
represents the light intensity value of the pixel point in the image frame at time k at the corresponding pixel point in the image frame at time t, and r represents the radius of the filtering time window.
The method according to claim 4, wherein the regularization parameter C is determined by the following formula:
The method according to claim 4 or 5, wherein,
Determined by the following formula:

Among them, σ is the setting parameter.
The method according to claim 6, wherein the set parameter is determined by the following formula:

σ=(2*r+1)/3.
The method according to any one of claims 3 to 7, further comprising:

If the corresponding pixel is an integer pixel, directly take the light intensity value at the corresponding pixel;

If the corresponding pixel point is a sub-pixel point, the light intensity value of the corresponding pixel point is obtained by an interpolation method.
The method according to claim 2, wherein, according to the relative motion relationship between the image sequences, filtering in the time dimension is performed on each pixel of the image frame in the image sequence that meets the specified condition to Update the light intensity value of each pixel to obtain the updated light intensity value, including:

According to the relative motion relationship, adaptive learning is performed on the autoregressive model of the pixel points in the image sequence in the time direction, so as to determine the model parameters of the autoregressive model;

Determine the corresponding pixel points of each pixel point in the image frame at time t of the image sequence;

According to the model parameter and the light intensity value of the corresponding pixel point, filtering in the time dimension is performed on each pixel point to obtain the updated light intensity value.
The method according to claim 9, wherein the formula of the autoregressive model is:

Among them, I k represents the theoretical light intensity value at time k; I t represents the theoretical light intensity value at time t; T k represents the set of adjacent moments at time k; P k→t (x, y) is the corresponding pixel position of the pixel point (x, y) in the image frame at the time k at the time t; α t is the model parameter; Δ k (x, y) is the pixel at (x, y) , y) the model error at the pixel point.
The method according to claim 10, wherein the autoregressive model of the pixels in the image sequence in the time direction is adaptively learned according to the relative motion relationship to determine the autoregressive model model parameters, including:

According to the relative motion relationship, the autoregressive model is adaptively learned by means of least squares method to solve the objective function, so as to determine the model parameters.
The method according to claim 11, wherein the objective function is:

Among them, W represents the set of pixels in the spatial range centered on (x, y);
represents the estimated light intensity value at time k;
represents the estimated light intensity value at the time t.
The method according to any one of claims 10 to 12, wherein, according to the model parameter and the light intensity value of the corresponding pixel point, filtering the pixel points in the time dimension to Get the updated light intensity value, including:

According to the model parameters and the light intensity value of the corresponding pixel point, the following formula is used to filter each pixel point in the time dimension to obtain the updated light intensity value,

Wherein, I′ k represents the updated light intensity value at time k.
The method according to any one of claims 1 to 13, further comprising:

The relative motion relationship between the image sequences is obtained according to any one or more of the optical flow method, pixel point matching, pixel point motion alignment, and pixel point relative position offset estimation.
The method according to any one of claims 1 to 14, characterized in that before obtaining the reconstructed image sequence according to the pulse array within a preset time period, the method further comprises:

According to the pulse signal within the preset time period, the pixel-based pulse array of the object to be imaged is acquired.
The method according to any one of claims 1 to 15, wherein the obtaining a reconstructed image sequence according to the pulse array within a preset time period comprises:

From the pulse array, the image sequence is reconstructed using a pulse spacing algorithm.
An imaging device, characterized in that, comprising:

a reconstruction module, configured to obtain a reconstructed image sequence according to the pulse array within a preset time period;

an update module, configured to perform filtering in the time dimension on each pixel point of the image frame in the image sequence that meets the specified condition according to the relative motion relationship between the image sequences to update the light intensity value of each pixel point , get the updated light intensity value;

An imaging module, configured to generate an updated image according to the updated light intensity value of each pixel point.
An electronic device, characterized by comprising a processor and a memory storing program instructions, the processor is configured to perform the imaging according to any one of claims 1 to 16 when executing the program instructions method.
An imaging device, characterized by comprising the electronic device as claimed in claim 18 .
A computer-readable medium having computer-readable instructions stored thereon, the computer-readable instructions being executable by a processor to implement the imaging method according to any one of claims 1 to 16.