CN117294963A - Image stabilizing method based on fusion of dynamic vision sensor and image sensor - Google Patents
Image stabilizing method based on fusion of dynamic vision sensor and image sensor Download PDFInfo
- Publication number
- CN117294963A CN117294963A CN202311291832.9A CN202311291832A CN117294963A CN 117294963 A CN117294963 A CN 117294963A CN 202311291832 A CN202311291832 A CN 202311291832A CN 117294963 A CN117294963 A CN 117294963A
- Authority
- CN
- China
- Prior art keywords
- image
- motion
- gradient
- motion compensation
- dynamic vision
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 230000000087 stabilizing effect Effects 0.000 title claims abstract description 10
- 230000004927 fusion Effects 0.000 title description 5
- 230000033001 locomotion Effects 0.000 claims abstract description 222
- 239000013598 vector Substances 0.000 claims abstract description 110
- 230000003287 optical effect Effects 0.000 claims abstract description 30
- 230000008859 change Effects 0.000 claims abstract description 28
- 230000000694 effects Effects 0.000 claims abstract description 14
- 238000013507 mapping Methods 0.000 claims abstract description 12
- 238000003384 imaging method Methods 0.000 claims description 20
- 230000007246 mechanism Effects 0.000 claims description 16
- 239000011159 matrix material Substances 0.000 claims description 14
- 230000008569 process Effects 0.000 claims description 11
- 238000009825 accumulation Methods 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 claims description 5
- 238000010276 construction Methods 0.000 claims description 4
- 238000005282 brightening Methods 0.000 claims description 2
- 238000006243 chemical reaction Methods 0.000 claims description 2
- 238000011156 evaluation Methods 0.000 claims description 2
- 230000004044 response Effects 0.000 claims description 2
- 238000012634 optical imaging Methods 0.000 claims 1
- 238000012544 monitoring process Methods 0.000 abstract description 2
- 230000003068 static effect Effects 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 238000011897 real-time detection Methods 0.000 description 3
- 230000007423 decrease Effects 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000006641 stabilisation Effects 0.000 description 2
- 238000011105 stabilization Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000004397 blinking Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000001747 exhibiting effect Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 230000009012 visual motion Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/47—Image sensors with pixel address output; Event-driven image sensors; Selection of pixels to be read out based on image data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/681—Motion detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/682—Vibration or motion blur correction
- H04N23/685—Vibration or motion blur correction performed by mechanical compensation
- H04N23/687—Vibration or motion blur correction performed by mechanical compensation by shifting the lens or sensor position
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/71—Circuitry for evaluating the brightness variation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/95—Computational photography systems, e.g. light-field imaging systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/50—Control of the SSIS exposure
- H04N25/57—Control of the dynamic range
- H04N25/58—Control of the dynamic range involving two or more exposures
- H04N25/587—Control of the dynamic range involving two or more exposures acquired sequentially, e.g. using the combination of odd and even image fields
- H04N25/589—Control of the dynamic range involving two or more exposures acquired sequentially, e.g. using the combination of odd and even image fields with different integration times, e.g. short and long exposures
Abstract
The invention provides an image stabilizing method for fusing a dynamic vision sensor and an image sensor. Firstly, a motion vector estimation and motion compensation system is built, then the mapping relation of sensor pixels is calibrated, then an image gradient field is pre-built, motion vector estimation and motion compensation are carried out, after the motion compensation, the dynamic vision sensor still keeps running, and the motion compensation quantity is continuously corrected through brightness change information generated subsequently, so that stable shooting in a longer exposure time is realized. Compared with the optical anti-shake system based on the angular gyroscope commonly used in the existing photographing anti-shake field, the invention can realize real-time monitoring of the anti-shake compensation effect, thereby realizing closed-loop control of motion compensation and realizing more stable photographing anti-shake effect.
Description
Technical Field
The invention relates to the technical field of computer vision motion vector estimation, in particular to an image stabilizing method based on fusion of a dynamic vision sensor and an image sensor.
Background
In recent years, dynamic vision sensors (or event cameras) have shown great application potential in the field of high-speed moving object recognition and tracking by virtue of their high time resolution capability. The dynamic vision sensor is different from the traditional image sensor in that the dynamic vision sensor only responds to the position where brightness change occurs in an image field, so that a large amount of static redundant information is eliminated in a dynamic object tracking scene, and the possibility is provided for a higher-speed motion vector estimation algorithm.
Currently, there are many methods for estimating an image feature motion vector or an optical flow based on a dynamic vision sensor. Often, they send the recorded event stream data into a neural network, or through iterative optimization, calculate the motion vector of the whole object or the optical flow of the local feature, so as to realize the non-real-time estimation of the motion vector of the object.
In the field of photography, it is critical to maintain camera stability during exposure, otherwise significant motion smear blur can occur. Also, when a high-speed moving object is photographed, even if the camera remains stationary, motion blur is generated in the photographed photograph due to the movement of the object.
In order to solve the problem of motion blur caused by camera motion, a current common method is to use an angular gyroscope to monitor rotation of a camera rigid body in real time and use a motor to adjust the inclination angle of a lens so that an optical axis always aims at a target. However, this method is limited by the accuracy and detection frequency of the angle gyroscope, and the angle gyroscope mounted on the mobile phone platform tends to have large zero bias and floating bias in view of cost, so that the stabilizing effect is poor.
In order to solve the problem of motion blur caused by object motion, the camera can only be rotated at the same speed as the object motion by depending on experience of photographers, so as to ensure that the target object is always at the image fixed position, and then a 'track-and-shoot' effect with clear target and motion blur background can be generated.
The existing image motion vector estimation method (or optical flow estimation method) based on the dynamic vision sensor cannot realize real-time estimation and compensation of the image feature motion vector whether the method is based on a neural network or an iterative optimization method. The reason for this is two.
1. And (5) a data acquisition link. Since the dynamic vision sensor can only respond to changes in brightness, a negative event is generated at a darkened location and a positive event is generated at a brightened location. Edges where there is a change in brightness will trigger events successively in adjacent pixels during motion. The current motion vector estimation algorithm based on the dynamic vision sensor uses the events to reversely infer the image motion. Therefore, these methods using only dynamic vision sensor data, in principle, require that the amount of motion of an object exceeds at least two pixels, and it is possible to estimate the magnitude and direction of the motion vector.
2. And (5) an algorithm operation link. Most of the current motion vector estimation methods based on dynamic vision sensors adopt a neural network method or an iterative optimization algorithm to achieve the minimum motion vector estimation error, so that the demand on calculation force is large, and the real-time demand on the sub-millisecond level cannot be achieved in time consumption.
In summary, based on the existing motion vector estimation algorithm of the dynamic sensor, the real-time detection and compensation of the shake and the object motion vector in the exposure process of the camera cannot be realized, so that the effective closed-loop image stabilizing function cannot be realized.
Disclosure of Invention
In order to solve the problems in the prior art, the invention provides an image stabilizing method based on fusion of a dynamic vision sensor and an image sensor, which is used for realizing image characteristic motion vector estimation and motion compensation. The method is used for overcoming the defect of low speed of the traditional motion vector estimation method based on the dynamic vision sensor, realizing the real-time detection of the image characteristic motion vector, simultaneously overcoming the problem of motion compensation accumulated drift caused by the error of the angle gyroscope in the current optical anti-shake technology based on the angle gyroscope, and realizing the reliable closed-loop image stabilization in the long-time exposure process.
The technical scheme of the invention is as follows:
the invention firstly provides an image stabilizing method for fusing a dynamic vision sensor and an image sensor, which comprises the following steps:
s1, building a motion vector estimation and motion compensation system; the motion vector estimation and motion compensation system includes: the system comprises an image sensor for final imaging and initial gradient field image acquisition, a dynamic vision sensor for recording brightness change information generated by motion in the exposure process, a light splitting device, an optical lens, a motion compensation mechanical mechanism for adjusting the dip angle of the imaging optical system and a motion control system for calculating motion compensation quantity, wherein the control system collects the brightness change information obtained by the image sensor and the dynamic vision sensor, calculates the motion compensation quantity and transmits the motion compensation quantity to the motion compensation mechanical mechanism, and the light splitting device divides an imaging light beam into two parts, one part transmits and images the imaging light beam to an image sensor plane, and the other part reflects and images the imaging light beam to the dynamic vision sensor plane;
s2, calibrating the pixel mapping relation of the dynamic vision sensor and the image sensor;
s3, pre-constructing a scene gradient field;
s4, the image sensor immediately starts exposure, and meanwhile, the dynamic vision sensor is started to acquire brightness change information in exposure time; obtaining pre-constructed scene gradient field information according to the S3, and calculating to obtain a motion vector by combining brightness change information obtained in real time by a dynamic vision sensor;
s5, according to the motion vector obtained in the S4, giving an adjustment instruction to a motion compensation mechanical mechanism through a conversion relation between the image motion vector and the inclination angle of the optical axis, and performing motion compensation;
s6, after the motion compensation is performed in S5, the dynamic vision sensor still keeps running, and the motion compensation amount is continuously corrected through brightness change information generated subsequently, so that stable shooting in a longer exposure time is realized.
According to a preferred embodiment of the present invention, the S3 is specifically:
3.1 Before the formal exposure, the image sensor acquires a clear scene image without motion blur with extremely short exposure time;
3.2 A) converting the clear scene image into a gray scale image, the value of each pixel in the gray scale image being proportional to the absolute intensity I of light impinging on the pixel;
3.3 Calculating a gradient field of the scene image by using a sobel operator; the specific calculation method of the gradient field is as follows: the sobel operator consists of two sets of 3×3 matrices, which areAnd->The two matrixes are respectively convolved with the gray level diagram I to obtain gradient +.>And a gradient in the y directionIs a convolution symbol;
3.4 Gradient amplitude field is defined asThe gradient direction field is defined asThereby obtaining a gradient amplitude field image and a gradient direction field image; the gradient amplitude field image matrix and the gradient direction field image matrix jointly form a scene gradient pre-construction field +.>
According to a preferred embodiment of the present invention, the S4 is specifically:
the first time derivative of the luminance field has the following relation with the scene gradient field and the motion vector:
wherein the brightness gradient fieldPre-constructing field for the scene gradient acquired in S3>Taking a matrix after logarithm, wherein v is a motion vector at the moment; the luminance change information acquired by the dynamic vision sensor is a discretized sample of the first time derivative of the luminance field.
The step S4 specifically comprises the following steps:
4.1 Examining N events generated by the dynamic vision sensor within an extremely short time interval delta t; gradient field of brightnessVector addition of gradient vectors at the N event generating locations is performed according to event polarity; the meaning of the event polarity is that the brightness collected by the dynamic vision sensor changes, the brightening event polarity is positive, and the darkening event polarity is negative; vector addition is carried out according to the event polarity, and the steps are as follows: adding the brightness gradient vector at the position if the positive polarity event is detected, and subtracting if the negative polarity event is detected;
after finishing the vector accumulation, obtaining a vector sum v * The direction is the direction of the motion vector v, the vector sum v * Dividing by the number N of events generated in the Δt time interval to obtain an average motion vector
4.2 Average vectorSize and dimensions of (2)The scale factor C related to the device exists between the magnitudes of the motion vectors v, and the calibration is carried out when leaving a factory;
the calibration steps are as follows: in the selected interval [ C ] of a given scale factor C 1 ,c 2 ]In, traversing all C values in the interval in delta C steps and correlating with the average motion vectorMultiplying; using the estimated motion vector +.>Performing motion compensation every time deltat passes; after motion compensation is carried out on M deltat time intervals according to a time sequence, an average gradient evaluation method is utilized to evaluate the motion compensation effect, and M average gradient values are obtained; wherein the definition of the average gradient is: scalar accumulation is carried out on the image gradient amplitude values after motion compensation of all the positions of the events generated in the delta t time interval, and the image gradient amplitude values are divided by the total number N of the events generated in the delta t; the closer the average gradient is to the initial average gradient, the better the motion compensation effect is represented; calculating the mean square error of M average gradient values and the initial average gradient value, which is called average gradient mean square error; the selected interval C for a given scale factor C 1 ,c 2 ]Finding the value with the minimum mean square error of the average gradient by using all C values traversed by the step length of delta C, namely calibrating to obtain a scale factor C value;
4.3 Calibrating the obtained C and the average motion vector in the step 4.1)Product of>I.e. the calculated motion vector.
According to a preferred embodiment of the present invention, the S5 is specifically:
5.1 Record center O of optical lens 1 To the centre O of the spectroscopic device 2 Distance d of (2) 1 Center O of spectroscopic device 2 Distance to dynamic vision sensord 2 ；
The movement of light on the image plane can be decomposed into two orthogonal and independent xy directions; s4, calculating to obtain a motion vectorThe component sizes in the orthogonal direction xy axis are +.>And->In pixels, the compensation quantity theta along the two orthogonal xy motion directions needs to be fed back to the motion compensation mechanism x And theta y In units of angle, and +.>Andthe following relationship is satisfied:
wherein d 1 And d 2 The unit is mu m, delta is the pixel spacing of the image sensor, and the unit is mu m/pixel;
5.2 Motion vector calculated by S4)Calculating the quantity to be compensated theta of the motion compensation mechanical mechanism along the two orthogonal xy motion directions on the image plane by combining the formula in the step 5.1) x And theta y After the quick compensation of the motion compensation mechanical mechanism, the image caused by the motion is formedThe shift of the light on the surface is compensated.
The invention can be used for preventing the shake of the motion in the single long-time exposure scene in the field of photography, and is also suitable for the motion follow-up shooting when the high-speed moving object is subjected to long-time exposure. Compared with the optical anti-shake system based on the angular gyroscope commonly used in the existing photographing anti-shake field, the invention can realize real-time monitoring of the anti-shake compensation effect, thereby realizing closed-loop control of motion compensation and realizing more stable photographing anti-shake effect. Compared with the existing motion vector estimation method based on the dynamic vision sensor, the motion vector estimation method provided by the invention has the advantage of more real-time performance.
Drawings
FIG. 1 is a flow chart of motion vector estimation and motion compensation according to the present invention;
FIG. 2 is a schematic diagram of a motion vector information acquisition and motion compensation apparatus according to the present invention;
FIG. 3 is a picture for calibrating the pixel mapping relationship between an image sensor and a dynamic vision sensor acquired in step 2 of the present embodiment;
FIG. 4 is a sharp scene image and scene image gradient pre-constructed field image referred to in step 3 of an embodiment of the invention;
FIG. 5 is a superimposed graph of events generated within 0.1ms after initiation of a formal exposure referred to in step 4 of an embodiment of the invention with a clear scene image;
FIG. 6 is a schematic diagram of the motion compensation effect at different scale factors C involved in step 4 of the present invention;
FIG. 7 is a schematic diagram of the geometrical relationships and system parameters involved in step 5 of the present invention;
fig. 8 is a flow chart of the motion compensated closed loop adjustment involved in step 6 of the embodiment of the present invention.
Detailed Description
The invention is further illustrated and described below in connection with specific embodiments. The described embodiments are merely exemplary of the present disclosure and do not limit the scope. The technical features of the embodiments of the invention can be combined correspondingly on the premise of no mutual conflict.
As shown in FIG. 1, an example of motion vector estimation and compensation for an object moving at high speed during a 22ms exposure time is shown in an embodiment of the present invention.
The embodiment of the invention provides an image stabilizing method based on fusion of a dynamic visual sensor and an image sensor, which is used for realizing real-time detection and motion compensation of image characteristic motion vectors. The method mainly comprises the following six steps:
1) And (5) constructing a motion vector estimation and motion compensation system.
2) And calibrating the pixel mapping relation of the dynamic vision sensor and the image sensor.
3) Scene gradient field pre-construction.
4) Motion vector estimation.
5) And calculating a motion compensation amount.
6) Motion compensated closed loop adjustment.
Referring to fig. 1, after the formal exposure is started, the parallel system consisting of the image acquisition system and the motion compensation closed-loop control system performs image stabilization operation. The image acquisition system is responsible for an image sensor, exposure is continuously carried out until the set exposure time is over, light intensity information is acquired, and finally a clear traditional digital image is obtained. The motion compensation closed-loop control system consists of a dynamic vision sensor, a motion control system and a motion compensation mechanical mechanism, wherein the motion is detected by the dynamic vision sensor, the compensation quantity is obtained by analysis and calculation of the motion control system, and the motion compensation mechanical mechanism is used for executing motion compensation.
In this embodiment, the specific content of the step 1) is:
referring to fig. 2, a thick dotted arrow in the drawing represents a ray incidence trajectory. The motion vector estimation and motion compensation system comprises:
an image sensor 1 for final imaging and initial gradient field image acquisition;
a dynamic vision sensor 2 for recording brightness change information generated by movement during exposure;
a spectroscopic device 3 for separating a part of the imaging light to the dynamic vision sensor; the optical paths of the image sensor 1 and the dynamic vision sensor 2 are ensured to be the same during assembly, and the image sensor 1 and the dynamic vision sensor 2 are ensured to be perpendicular to incident light rays;
an optical lens 4;
a spring 5 for adjusting the inclination angle of the optical lens;
a motor 6 for adjusting the tightness of the two springs; wherein, the two sides of the optical lens 4, which are not provided with springs, are fixed, and the tightness of the two springs 5 is controlled by the motor 6, so that the inclination of the optical lens 4 in two degrees of freedom of meridian and sagittal can be adjusted, thereby controlling the direction of the incident track of light.
The motion control system 7 collects the image obtained by the image sensor 1 and the brightness change information obtained by the dynamic vision sensor 2, calculates the motion compensation amount by the method of the invention, and transmits the motion compensation amount to the motor 6.
1.1 The external environment is imaged through the optical lens 4, the imaging light beam is divided into two parts through the light splitting device 3, one part of the imaging light beam is transmitted and imaged to the plane of the image sensor 1, and the other part of the imaging light beam is reflected through the light splitting device and imaged to the plane of the dynamic vision sensor 2.
1.2 In the link of building a basic imaging system, the optical paths of the image sensor 1 and the dynamic vision sensor 2 are controlled to be the same, otherwise, the image sensor 1 and the dynamic vision sensor 2 cannot focus on objects with the same object distance accurately at the same time. And the plane of the image sensor 1 is vertical to the optical axis of the light transmitted by the light splitting device 3, and the plane of the dynamic vision sensor 2 is vertical to the optical axis of the light reflected by the light splitting device 3.
1.3 According to the working principle of the dynamic vision sensor, when the brightness change of any pixel exceeds a set threshold value, the sensor can record the change with microsecond precision, and the change information is called an event. It is noted that the dynamic vision sensor has a logarithmic mapping characteristic, and the brightness L in brightness variation refers to a value obtained by logarithmic mapping of the absolute light intensity I. A dimming is defined as positive polarity event and a dimming is defined as negative polarity event:
L(x,y,t)＝log(I(x,y,t))
I(x,y,t i ) Represented by t i Light intensity values, L (x, y, t), at time instants at dynamic vision sensor pixel coordinates (x, y) i ) Represented by t i Luminance values at time instant on dynamic vision sensor pixel coordinates (x, y), thr represents the set threshold.
The data obtained by the dynamic vision sensor is a section of event stream, and the data format of each event in the event stream is (x, y, p, t). Wherein x, y represents the position coordinates of the brightness change event occurring at the sensor plane; p represents the polarity of the event, as described above; t represents a timestamp generated by a brightness change event, and the precision can reach microsecond magnitude. The event information is sequentially injected into the event stream according to the time sequence and is sent to a subsequent processing system.
However, dynamic vision sensors cannot respond to static scenes.
1.4 The image sensor can be used for acquiring static texture information of the whole scene, and the digital expression mode of the static texture information is an M multiplied by N matrix, wherein M is the number of pixels in the length direction of the image, and N is the number of pixels in the width direction of the image.
1.5 Referring to fig. 2, the spring 5 and the motor 6 together constitute a motion compensation mechanism. The optical lens 4 is fixed on two sides without installing springs, the tightness of the two springs 5 is controlled by a motor 6, and the inclination of the optical lens 4 in two degrees of freedom of meridian and sagittal can be adjusted, so that the direction of the light incident track is controlled. When an object moves or a camera moves, the optical anti-shake method can fix the image characteristics at a certain position all the time, so that the final image obtained by exposure has no motion smear.
In this embodiment, the specific content of the step 2) is:
2.1 The main purpose of step 2) is to correspond the pixel positions of the image sensor to the pixel positions of the dynamic vision sensor, which is not a one-to-one correspondence, since the pixel sizes of the image sensor and the dynamic vision sensor tend not to be exactly identical. The mapping relation between the pixels of the image sensor and the pixels of the dynamic vision sensor can be constructed by using an image calibration method, and the following steps are specific.
2.2 A display screen is placed in the in-focus position of the optical lens 4 of fig. 2 and the relative rest of the display screen and imaging system positions is maintained. The in-focus position of the optical lens refers to a position at which imaging is made sharpest. Since the dynamic vision sensor can only respond to brightness changes, the dynamic vision sensor does not have event output when a moving scene is not played on the screen.
2.3 A blinking checkerboard image is played on the display screen and the dynamic vision sensor will produce an event response at a particular location. At the same time, the image sensor also shoots a plurality of flickering checkerboard images, and selects one piece with the clearest checkerboard outline for standby.
2.4 The event stream of the dynamic visual sensor is accumulated into an event frame for a certain time. The event frame is an image, which is an integer matrix of X Y, and the value of each position (X, Y) of the matrix is the number of events generated at that position.
2.5 Referring to fig. 3, the left image is an image with a checkerboard acquired by the image sensor, and the right image is a dynamic vision sensor checkerboard event frame obtained in step 2.4).
2.6 The two checkerboard images in fig. 3 are subjected to angular point extraction functions and homography matrix calibration functions provided by an OpenCV library, and the mapping relation from the image sensor collected image coordinates to the dynamic vision sensor coordinates is obtained. All the image sensor images described below are aligned with the dynamic vision sensor coordinates by this mapping relationship. In this embodiment, the specific content of the step 3) is:
3.1 Referring to the left hand side of fig. 4, the image sensor acquires a clear scene image without motion blur with very short exposure times prior to a formal exposure.
3.2 A clear scene image is converted to a gray scale image. Because of the linear mapping characteristic of the image sensor, the value of each pixel in the gray-scale image is proportional to the absolute light intensity I impinging on the pixel, and the gray-scale image I and the absolute light intensity field I are treated as the same by neglecting the scale factor therebetween.
3.3 A sobel operator is used to calculate the gradient field of the scene image. The specific calculation method of the gradient field is as follows: the sobel operator consists of two sets of 3×3 matrices, which areAnd->The two matrices are convolved with the gray map I, respectively, to obtain the gradient +.>And a gradient in the y directionIs a convolution symbol.
3.4 Gradient amplitude field is defined asThe gradient direction field is defined asThus, two gradient magnitude field images and gradient direction field images are obtained in fig. 4 and right. The gradient amplitude field image matrix and the gradient direction field image matrix jointly form a scene gradient pre-construction field +.>
In this embodiment, the specific content of the step 4) is:
4.1 And 3) immediately starting formal exposure by the image sensor after the scene gradient pre-constructed field is obtained after the exposure is finished, and simultaneously starting the dynamic vision sensor to acquire motion event information in the formal exposure process. In the present embodiment, the moving direction of the scene feature (human body) in the image is leftward.
4.2 Referring to fig. 5, 1194 events generated within 0.1ms after formally starting exposure are superimposed according to their coordinates (x, y) onto the clear scene image acquired in step 3.1). Comparing the middle and right panels of fig. 4, it can be observed that these events occur at positions having two characteristics: (1) The method is applied to the position with larger amplitude of the scene gradient field, namely the position with large brightness space change in the image; (2) Occurs at a location where the scene gradient field direction is nearly collinear with the direction of motion.
4.3 1194 events generated in step 4.2) can be considered as data obtained by discrete sampling of the first time derivative of the luminance field, according to the principle of operation of the dynamic vision sensor. The first time derivative of the luminance field has the following relation with the scene gradient field and the motion vector:
wherein the brightness gradient fieldPre-constructing a field for the scene gradient obtained in step 3.4)>Taking the matrix after logarithm, v is the motion vector at that moment.
4.4 Looking at events generated by the dynamic vision sensor within a very short time interval deltat, i.e. 1194 events generated within 0.1ms after formally starting exposure in this embodiment, brightness change information generated within deltat can be obtained, and a field is pre-constructed according to a known scene gradientThe direction of the motion vector v occurring within Δt can be inverted.
In the present embodiment, the luminance gradient fieldAt the 1194 event generation locationsThe value is vector added according to the event polarity, and the specific steps are as follows: the luminance gradient vector at that location is added if it is a positive event and subtracted if it is a negative event. After completion of 1194 positions of vector accumulation, vector sum v is obtained * The direction is the direction of the motion vector v.
4.5 Vector sum v) * Dividing by the total number of events 1194 to obtain an average vectorAverage vector>The magnitude of (2) is not the magnitude of the motion vector v because the dynamic vision sensor discretely samples the temporal variation of the continuous luminance field, the sampling interval being related to the dynamic vision sensor threshold. That is, the absolute amount of luminance change represented by each event is unknown and varies depending on the device characteristics and the hardware parameters involved, but is fixed for a certain device under a certain parameter setting. Therefore, the average vector +.>A scale factor C between the magnitude of the motion vector v and the magnitude of the motion vector v.
4.6 The calibration method is still described by taking the data set as an example in the embodiment. Referring to fig. 6, the estimated motion vector is utilized within 10ms after the exposure is started when setting different scale factors C are sequentially presented from left to rightThe motion compensation situation continues. According to the characteristics of the event generated by the dynamic vision sensor in the step 4.2), the event is generated at the position with larger image gradient. Thus, at any time after motion compensation, events still occur where the gradient of the image is large after motion compensation. Thus, the average gradient can quantitatively evaluate the effect of motion compensation. The definition of the average gradient is: for generation within the delta t time intervalThe motion compensated image gradient magnitude values are scalar-accumulated and divided by the total number of events generated during the Δt time interval.
The three graphs in the upper line of fig. 6 reflect the change in average gradient over time for three scale factors C. It can be seen that when c=1, the average gradient rapidly decreases after 2.5ms, indicating that motion compensation starts to gradually fail at 2.5ms after exposure starts, so that an under-compensation effect is exhibited in the event and motion compensation image overlay of 10ms after exposure starts; at c=10, the average gradient rapidly decreases after 0.1ms, exhibiting an overcompensation effect in the overlay. Whereas at c=3.2 the average gradient remains stable throughout the exposure time of 20ms, indicating that the motion compensation effect is good during this exposure time. The following description is given of how to obtain the preferred c=3.2.
4.7 In this embodiment, the scale factor C of the calibration process is selected to be 1,10]The method comprises the steps of carrying out a first treatment on the surface of the Traversing all C values within the interval in 0.1 steps and using the estimated motion vectorPerforming motion compensation once every 0.1ms, and calculating the average value of the image gradient of the position where the event is located; finally obtaining average gradient values of 200 time points in the exposure time of 20 ms; calculating the mean square error of the 200 average gradient values and the initial average gradient, which is called average gradient mean square error; for all C values, the C value at which the mean gradient mean square error is the smallest is found. In this example, the preferred C value is 3.2.
In this embodiment, the specific content of the step 5) is:
step 4 gives the size and direction of the motion vector in pixel units in the time interval of Δt=0.1 ms at each moment, and realizes the motion vector estimation at the image level. As described in step 1.5), the motion compensation is physically achieved by two orthogonal direction adjustment motors, and therefore the motion vector in pixels is associated with two orthogonal motor adjustment amounts. The method comprises the following specific steps:
5.1 Referring to FIG. 7, opticsCenter O of lens 4 1 To the centre O of the spectroscopic device 3 2 Distance d of (2) 1 Center O of spectroscopic device 3 2 Distance d to dynamic vision sensor 2 2 . The partial magnified view of the dynamic vision sensor is shown in the upper right corner of fig. 7, and the motion of light on the image plane can be broken down into two orthogonal and independent xy directions as shown. The motion vector can be calculated in the step 4The component sizes in the orthogonal direction xy axis are +.>And->(in pixels), the compensation amount θ in the two orthogonal xy movement directions needs to be fed back to the motor x And theta y (in degrees) and->And->The following relationship is satisfied:
wherein d is 1 And d 2 The delta is the pixel pitch (in μm/pixel) of the image sensor, and is given by factory measurement.
5.2 A) the motion vector calculated in step 4Calculating the compensation quantity theta of the motor along the orthogonal xy two motion directions on the image plane by combining the formula in the step 5.1) x And theta y And the control system is fed back to the motor, and after the motor is rapidly compensated, the offset of the light on the image plane caused by the movement can be compensated.
In this embodiment, the specific content of the step 6) is:
in the process of handheld shooting, the motion vector can be irregularly changed in the process of exposure, so that the motion vector initially estimated in the step 4 cannot accurately indicate the motion compensation in the process of subsequent long-time exposure, and thus the information of the dynamic vision sensor needs to be continuously acquired in the whole exposure process so as to carry out closed-loop estimation on the motion vector after the motion compensation, so that the motion compensation quantity is continuously corrected. Reference is made to fig. 8 for a motion compensated closed loop adjustment flow chart.
6.1 In this embodiment, every time the dynamic vision sensor passes 0.1ms, the motion event generated in the time interval Δθ=0.1 ms is acquired for a total of 22ms exposure time period.
6.2 Using the initial motion vectors obtained in step 4.4) and step 4.5)On the one hand, the image gradient pre-constructed field is fed back to the adjusting motor through the step 5, and on the other hand, the image gradient pre-constructed field is updated, so that the new image gradient pre-constructed field accords with the situation after motion compensation.
6.3 Acquiring a motion event generated in the next delta t=0.1 ms time interval, repeating the step 4.4) and the step 4.5) by using a new image gradient pre-constructed field, estimating a motion vector at the current moment, updating a motion compensation quantity by using the step 5, and feeding back to a motion compensation motor.
6.4 Continuously collecting motion events, calculating real-time motion vectors, and correcting motion compensation quantity until the exposure time is over.
The foregoing examples illustrate only a few embodiments of the invention and are described in detail herein without thereby limiting the scope of the invention. It will be apparent to those skilled in the art that various modifications and improvements can be made without departing from the spirit of the invention.
Claims (8)
1. The image stabilizing method for fusing the dynamic vision sensor and the image sensor is characterized by comprising the following steps of:
s1, building a motion vector estimation and motion compensation system; the motion vector estimation and motion compensation system includes: the system comprises an image sensor for final imaging and initial gradient field image acquisition, a dynamic vision sensor for recording brightness change information generated by motion in the exposure process, a light splitting device, an optical lens, a motion compensation mechanical mechanism for adjusting the dip angle of the imaging optical system and a motion control system for calculating motion compensation quantity, wherein the control system collects the brightness change information obtained by the image sensor and the dynamic vision sensor, calculates the motion compensation quantity and transmits the motion compensation quantity to the motion compensation mechanical mechanism, and the light splitting device divides an imaging light beam into two parts, one part transmits and images the imaging light beam to an image sensor plane, and the other part reflects and images the imaging light beam to the dynamic vision sensor plane;
s2, calibrating the pixel mapping relation of the dynamic vision sensor and the image sensor;
s3, pre-constructing a scene gradient field;
s4, the image sensor immediately starts exposure, and meanwhile, the dynamic vision sensor is started to acquire brightness change information in exposure time; obtaining pre-constructed scene gradient field information according to the S3, and calculating to obtain a motion vector by combining brightness change information obtained in real time by a dynamic vision sensor;
s5, according to the motion vector obtained in the S4, giving an adjustment instruction to a motion compensation mechanical mechanism through a conversion relation between the image motion vector and the inclination angle of the optical axis, and performing motion compensation;
s6, after the motion compensation is performed in S5, the dynamic vision sensor still keeps running, and the motion compensation amount is continuously corrected through brightness change information generated subsequently, so that stable shooting in a longer exposure time is realized.
2. The method for real-time estimation and motion compensation of motion vectors of image features according to claim 1, wherein in S1, the optical paths of the image sensor and the dynamic vision sensor are the same, and the imaging planes of the image sensor and the dynamic vision sensor are perpendicular to the light emitted from the spectroscopic device.
3. The method for real-time estimation and motion compensation of motion vectors according to claim 1, wherein in S1, the motion compensation mechanism adjusts the tilt angle of the compensation lens group according to the motion compensation amount given by the motion control system, so as to counteract motion blur.
4. The method for real-time estimation and motion compensation of image feature motion vectors according to claim 1, wherein S2 is specifically: placing a display screen on an object plane with clear imaging by an optical imaging system, and playing a flickering checkerboard image in the display screen; the dynamic vision sensor can sense the change of brightness, and the flickering checkerboard image can generate response on specific pixels of the dynamic vision sensor, so that the image sensor can acquire the checkerboard image in the process; after a period of time, the brightness change information perceived by the dynamic vision sensor is accumulated to form a checkerboard image; and calibrating a homography matrix by using the checkerboard images accumulated by the dynamic vision sensor and the checkerboard images acquired by the image sensor, thus obtaining the mapping relation between each pixel of the dynamic vision sensor and each pixel of the image sensor.
5. The method for real-time estimation and motion compensation of motion vectors of image features according to claim 1, wherein S3 is: before the image sensor formally starts exposure, shooting a scene to be shot by using the image sensor with a short exposure time to obtain a preview image; the image gradient size and direction are calculated for the image and stored in the memory of the motion control system.
6. The method for real-time estimation and motion compensation of image feature motion vectors according to claim 1, wherein S3 is specifically:
3.1 Before the formal exposure, the image sensor acquires a clear scene image without motion blur with extremely short exposure time;
3.2 A) converting the clear scene image into a gray scale image, the value of each pixel in the gray scale image being proportional to the absolute intensity I of light impinging on the pixel;
3.3 Calculating a gradient field of the scene image by using a sobel operator; the specific calculation method of the gradient field is as follows: the sobel operator consists of two sets of 3×3 matrices, which areAnd->The two matrixes are respectively convolved with the gray level diagram I to obtain gradient +.>And a gradient in the y direction Is a convolution symbol;
3.4 Gradient amplitude field is defined asThe gradient direction field is defined as +.>Thereby obtaining a gradient amplitude field image and a gradient direction field image; gradient amplitudeThe gradient image matrix and the gradient direction image matrix jointly form a scene gradient pre-construction field +.>
7. The method for real-time estimation and motion compensation of image feature motion vectors according to claim 6, wherein said S4 is specifically:
the first time derivative of the luminance field has the following relation with the scene gradient field and the motion vector:
wherein the brightness gradient fieldPre-constructing field for the scene gradient acquired in S3>Taking a matrix after logarithm, wherein v is a motion vector at the moment; the luminance change information acquired by the dynamic vision sensor is a discretized sample of the first time derivative of the luminance field.
The step S4 specifically comprises the following steps:
4.1 Examining N events generated by the dynamic vision sensor within an extremely short time interval delta t; gradient field of brightnessVector addition of gradient vectors at the N event generating locations is performed according to event polarity; the meaning of the event polarity is that the brightness collected by the dynamic vision sensor changes, the brightening event polarity is positive, and the darkening event polarity is negative; vector addition is carried out according to the event polarity, and the steps are as follows: adding the brightness gradient vector at the position if the positive polarity event is detected, and subtracting if the negative polarity event is detected;
after finishing the vector accumulation, obtaining a vector sum v * The direction is the direction of the motion vector v, the vector sum v * Dividing by the number N of events generated in the Δt time interval to obtain an average motion vector
4.2 Average vectorThe size of the device and the size of the motion vector v are provided with a scale factor C related to the device, and the scale factor C is calibrated when leaving a factory;
the calibration steps are as follows: in the selected interval [ C ] of a given scale factor C 1 ,c 2 ]In, traversing all C values in the interval in delta C steps and correlating with the average motion vectorMultiplying; using the estimated motion vector +.>Performing motion compensation every time deltat passes; after motion compensation is carried out on M deltat time intervals according to a time sequence, an average gradient evaluation method is utilized to evaluate the motion compensation effect, and M average gradient values are obtained; wherein the definition of the average gradient is: scalar accumulation is carried out on the image gradient amplitude values after motion compensation of all the positions of the events generated in the delta t time interval, and the image gradient amplitude values are divided by the total number N of the events generated in the delta t; the closer the average gradient is to the initial average gradient, the better the motion compensation effect is represented; calculating the mean square error of M average gradient values and the initial average gradient value, which is called average gradient mean square error; the selected interval C for a given scale factor C 1 ,c 2 ]Finding the value with the minimum mean square error of the average gradient by using all C values traversed by the step length of delta C, namely calibrating to obtain a scale factor C value;
4.3 Calibrating the obtained C and the average motion vector in the step 4.1)Product of>I.e. the calculated motion vector.
8. The method for real-time estimation and motion compensation of image feature motion vectors according to claim 1, wherein S5 is specifically:
5.1 Record center O of optical lens 1 To the centre O of the spectroscopic device 2 Distance d of (2) 1 Center O of spectroscopic device 2 Distance d to dynamic vision sensor 2 ；
The movement of light on the image plane can be decomposed into two orthogonal and independent xy directions; s4, calculating to obtain a motion vectorThe component sizes in the orthogonal direction xy axis are +.>And->In pixels, the compensation quantity theta along the two orthogonal xy motion directions needs to be fed back to the motion compensation mechanism x And theta y In units of angle, and +.>Andthe following relationship is satisfied:
wherein d 1 And d 2 The unit is mu m, delta is the pixel spacing of the image sensor, and the unit is mu m/pixel;
5.2 Motion vector calculated by S4)Calculating the quantity to be compensated theta of the motion compensation mechanical mechanism along the two orthogonal xy motion directions on the image plane by combining the formula in the step 5.1) x And theta y After the rapid compensation of the motion compensation mechanical mechanism, the offset of the light on the image plane caused by the motion is compensated.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311291832.9A CN117294963A (en) | 2023-10-08 | 2023-10-08 | Image stabilizing method based on fusion of dynamic vision sensor and image sensor |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311291832.9A CN117294963A (en) | 2023-10-08 | 2023-10-08 | Image stabilizing method based on fusion of dynamic vision sensor and image sensor |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117294963A true CN117294963A (en) | 2023-12-26 |
Family
ID=89244187
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311291832.9A Pending CN117294963A (en) | 2023-10-08 | 2023-10-08 | Image stabilizing method based on fusion of dynamic vision sensor and image sensor |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117294963A (en) |
-
2023
- 2023-10-08 CN CN202311291832.9A patent/CN117294963A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111147741B (en) | Focusing processing-based anti-shake method and device, electronic equipment and storage medium | |
| KR102143456B1 (en) | Depth information acquisition method and apparatus, and image collection device | |
| US10277809B2 (en) | Imaging device and imaging method | |
| US9531938B2 (en) | Image-capturing apparatus | |
| US20060119710A1 (en) | Systems and methods for de-blurring motion blurred images | |
| US7777781B2 (en) | Method and system for determining the motion of an imaging apparatus | |
| CN113875219B (en) | Image processing method and device, electronic equipment and computer readable storage medium | |
| JP6128109B2 (en) | Image capturing apparatus, image capturing direction control method, and program | |
| CN111246100A (en) | Anti-shake parameter calibration method and device and electronic equipment | |
| WO2018076529A1 (en) | Scene depth calculation method, device and terminal | |
| JP2941815B2 (en) | Imaging device and blur correction device | |
| JP5393877B2 (en) | Imaging device and integrated circuit | |
| CN113645397A (en) | Tracking method, device and system for moving target object | |
| CN107360360A (en) | Picture pick-up device and its control method and storage medium with image stabilizing function | |
| KR101630307B1 (en) | A digital photographing apparatus, a method for controlling the same, and a computer-readable storage medium | |
| US20200021745A1 (en) | Imaging apparatus | |
| CN117294963A (en) | Image stabilizing method based on fusion of dynamic vision sensor and image sensor | |
| US20220174217A1 (en) | Image processing method and device, electronic device, and computer-readable storage medium | |
| CN106454066B (en) | Image processing apparatus and control method thereof | |
| JP2019062340A (en) | Image shake correction apparatus and control method | |
| JP7308696B2 (en) | Image blur correction device, its control method, program, and imaging device provided with image blur correction device | |
| JP2021033015A5 (en) | ||
| JP2020036091A (en) | Imaging device and control method therefor, program, and storage medium | |
| JP2019083407A (en) | Image blur correction device and control method therefor, and imaging device | |
| JP2021118523A (en) | Image processing device and image processing method, program, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |