US20190089888A1

US20190089888A1 - Image distortion correction of a camera with a rolling shutter

Info

Publication number: US20190089888A1
Application number: US16/132,730
Authority: US
Inventors: Yael Berberian
Original assignee: Mobileye Vision Technologies Ltd
Current assignee: Mobileye Vision Technologies Ltd
Priority date: 2013-02-21
Filing date: 2018-09-17
Publication date: 2019-03-21
Also published as: US20140232887A1; EP2770728A2; US9277132B2; US20160182793A1; EP2770728A3; US10079975B2; US20190356830A1; US10834324B2

Abstract

Correcting image distortion during camera motion using a system including a processor and a camera having a rolling shutter. Multiple image frames are captured by the camera equipped with the rolling shutter. The captured image frames include a base image frame and a previous image frame. Multiple time stamps are recorded respectively for multiple corresponding image points in the previous and base image frames. For the corresponding image points, multiple ego-motions are computed responsive to the time stamps of the corresponding image points of the base image frame and the previous image frame to correct the image distortion caused by the rolling shutter.

Description

BACKGROUND

1. Technical Field

The present disclosure relates to correction for distortions arising from use of a rolling shutter, and more particularly for use in a camera based driver assistance/control system.

2. Description of Related Art

Rolling shutter is a method of image acquisition in which each frame is recorded not from a snapshot of a single point in time, but rather by scanning across the image frame, for instance row by row. With a rolling shutter, not all parts of the image are recorded at exactly the same time, even though the whole frame may be displayed at the same time during playback. The rolling shutter is in contrast with a global shutter where the entire frame is exposed for the same time window.
Ego-motion “self-motion” refers to the translation and orientation (e.g. yaw, pitch and roll) in time of moving camera. A measure of ego-motion or of the camera mounted in a vehicle is important for driver assistance and/or vehicle control systems in order to accurately detect, recognize and avoid false positive detections of: other vehicles, obstacles, lights, street signs, lane markers and/or guard rails in the road environment.
Structure-from-Motion (SfM) refers to methods for recovering three-dimensional information of a scene that has been projected onto the focal plane(s) of a moving camera or multiple cameras. The structural information derived from a SfM algorithm may take the form of a set of projection matrices, one projection matrix per image frame, representing the relationship between a specific two-dimensional point in the image plane and its corresponding three-dimensional point. SfM algorithms rely on tracking specific image features from image frame to image frame to determine structural information concerning the scene.

BRIEF SUMMARY

Various systems and methods are disclosed herein for correcting image distortion during camera motion using a system including a processor and a camera having a rolling shutter. Multiple image frames are captured by the camera equipped with the rolling shutter. The captured image frames include a base image frame and a previous image frame. Multiple time stamps are recorded respectively for multiple corresponding image points in the previous and base image frames. For the corresponding image points, multiple ego-motions are computed responsive to the time stamps or capture times of the corresponding image points of the base image frame and the previous image frame to correct the image distortion caused by the rolling shutter. The computation of the ego-motions may be performed using image data from the image frames of the camera. The corresponding image points of the base image frame and the previous image frame are image points of the same object point. The computation of the ego-motions may be responsive to the time stamp difference between the corresponding image points. Image disparities may be computed based on the ego-motions. Distance to an object being imaged may be computed based on the ego-motions. The ego-motions may be determined by using an iterative process wherein an optimization goal is to minimize the distances between the epipolar lines corresponding to the image points in the previous image and the matching image points in the base image.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is herein described, by way of example only, with reference to the accompanying drawings, wherein:

FIGS. 1 and 2 illustrate a system including a camera with a rolling shutter mountable in a vehicle, according to an aspect of the present invention.

FIG. 3 illustrates a simplified method, according to embodiments of the present invention.

The foregoing and/or other aspects will become apparent from the following detailed description when considered in conjunction with the accompanying drawing figures.

DETAILED DESCRIPTION

Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. The embodiments are described below to explain the present invention by referring to the figures.
Before explaining embodiments of the invention in detail, it is to be understood that the invention is not limited in its application to the details of design and the arrangement of the components set forth in the following description or illustrated in the drawings. The invention is capable of other embodiments or of being practiced or carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein is for the purpose of description and should not be regarded as limiting.
By way of introduction, embodiments of the present invention are based on ego-motion computations from the image data of a moving camera. When ego-motion computations are available as in driver assistance systems and/or vehicle control systems, computational resources may be saved by using the available ego-motion information. By way of example, a structure from motion (SfM) algorithm may use an ego motion matrix C which describes the ego-motion of a camera moving in real space while capturing a pair of image frames (“base” and “previous”). The SfM algorithm may use the ego motion matrix C and image point correspondences between the pair of image frames to calculate real space distances to objects being imaged. Such an ego motion matrix C describes the camera motion while capturing two image frames in which all pixels of each image frame are sampled at the same time. A rolling shutter renders the use of a single ego motion matrix C inadequate, as the motion of the camera is different for each pair of corresponding image points. Therefore, in order to fully describe the motion of the camera between two image frames, as many ego motion matrix matrices C are required as the number of corresponding pixel pairs. Such an approach would be computationally prohibitive for use in driver assistance and control systems which are required to respond in real time.
However, assuming that the ego motion matrix C between points sampled at time t=0 is known, and that the characteristics of the camera motion did not change considerably between the two image frames, the timestamps between rows may be used to compute the real ego motion between capture times of image points. In practice, because of various other distortions the captured image undergoes (e.g. distortion from optical system and/or camera calibration distortion), a different time stamp for each pixel may be obtained and used rather than a time stamp per row. In this way, the other distortions may be considered in addition to the rolling shutter distortion by adjusting the time stamp according to the known distortion per pixel.
Referring now to the drawings, reference is now made to FIGS. 1 and 2 which illustrate a system 16 including a camera or image sensor 12 with a rolling shutter mountable in a vehicle 18, according to an aspect of the present invention. Image sensor 12, imaging a field of view in the forward direction typically captures in real time a time series of image frames 15. An image 5 processor 14 may be used to process image frames 15 simultaneously and/or in parallel to serve a number of driver assistance and/or control systems. Image sensor 12 may be monochrome or black-white, i.e. without color separation. By way of example in FIG. 2, image frames 15 may be used to serve a monitor/control system 23 which may include collision warning 17, lane departure warning 19, traffic sign recognition (TSR) 21 and structure from motion (SfM) with rolling shutter distortion correction 30 according to embodiments of the present invention. Structure from motion may include for example presenting three dimensional imagery on a display to the driver, measuring distance to objects being imaged or the detection of three dimensional structures including curbs, guard rails, structural barriers, e.g. concrete lane divider. Aspects of the present invention may include exchange of information between SfM 30 and other driver assistance functions and/or systems including but not limited by FCW 17 and LDW 19. For example, a Lane Departure Warning (LDW) 19 as part of warning system 23, may respond more strongly to a lane departure towards a guard rail or a barrier rather than a lane marker or even a white line. A Forward Collision Warning (FCW) system 19 may trigger sooner if the path to either side of in-path vehicle is blocked by a guard rail or another vehicle.
The terms “camera” and “image sensor” are used herein interchangeably. The term “object” as used herein refers to an object in real space being viewed by a camera. A guard rail along the edge of a road and a lane marker in the road are examples of objects. The term “image” refers to the image of one or more objects in image space at the focal plane of camera 12. Image coordinates (x,y) in small letters refer to image space and may be in arbitrary units or numbers of picture elements in the horizontal and vertical directions with the pixel dimensions assumed.
The term “image point” as used herein refers to a point (x,y) in image space. The term “pixel” short for “picture element” and “image point” are used herein interchangeably. The term “corresponding” as used herein in the context of “ corresponding” image points refers to image points of different image frames of a time sequence which have been found to be image points of the same object point. The terms “corresponding” image points and “matching” image points are used herein interchangeably.
The term “time stamp” as used herein refers to a point in time relative to a reference time which may be selected during a time sequence of image frames. The term “time stamp” as used herein is a capture time of an image point or image row of the rolling shutter. The time stamp may be further adjusted to correct for another known distortion in the camera other than that caused by the rolling shutter in which case the time stamp used is not strictly the capture time of the image point.
The term “image motion” refers to motion of an image of an object in image space. From image frame 15 to a subsequent image frame 15 the points of the image of the object may map from one set of coordinates (x1,y1) to a different set of coordinates (x2,y2). The term “image motion” refers to the mapping of coordinates of an image from image frame to image frame or a function of the mapping. The term “projection” or “projecting” as used herein refers to camera or perspective projection unless otherwise indicated by the context.
Reference is now also made to FIG. 3 which illustrates a simplified method 30, according to embodiments of the present invention for image distortion correction using a rolling shutter. Image frames are captured (step 703) with a camera equipped with a rolling shutter. Time stamps 70 are recorded (step 707) for multiple rows in the previous and base image frames or for multiple pixels per row in both. In step 709, ego-motion is computed on a row by row or pixel by pixel basis responsive to time stamps 70, and more particularly the ego-motion may be responsive to the difference in time stamps 70 of corresponding pixels in the previous and base image frames. Ego-motion results 71 of the rows and/or of the pixels for corresponding rows and pixels. Ego-motion results may be represented by corrections to a global ego-motion matrix which would be valid for all pixels with the use of a global shutter instead of a rolling shutter. The corrected ego-motion results may be used pixel by pixel or row by row for further image processing such as computation in step 711 of image disparities for providing or displaying structure from motion (SfM). In the description that follows, method 30 is described in further detail with SfM as an example.
As previously stated, an SfM algorithm using a global ego-motion computation assumes that all the pixels in image frame 15 are sampled simultaneously. However, when camera 12 has a rolling shutter, the assumption of simultaneous sampling for all pixels in a single image frame 15 is no longer valid. The rolling shutter therefore produces an image distortion, which affects performance of structure from motion (SfM) algorithm.
A correction to the image distortion may be applied in two stages to the original structure from motion (SfM) algorithm: A. distance estimation correction and B. ego-motion correction.

A. Distance Estimation Correction

The distance correction for the rolling shutter effect of camera 12 is presented here in four main steps. The inputs include the structure from motion (SfM) distance estimation, the ego motion matrix C, time stamps approximation and time difference between the base and previous images. Each of the following steps may be performed for every pixel of the base 10 and previous images; or multiple pixels or multiple rows of the base and previous images:
1. Finding Image Coordinates in the Previous Image
The following equations relate the image coordinates of the base image to the image coordinates of the previous image when the global ego-motion of camera 12 is known.
${(\begin{matrix} x \\ y \\ f \end{matrix})}_{b} \cdot \frac{Z}{f} = {(\begin{matrix} X \\ Y \\ Z \end{matrix})}_{b}$ ${C^{- 1} (\begin{matrix} X \\ Y \\ Z \\ 1 \end{matrix})}_{b} = {(\begin{matrix} X \\ Y \\ Z \\ 1 \end{matrix})}_{p}$ ${(\begin{matrix} X \\ Y \\ Z \end{matrix})}_{p} \cdot \frac{f}{Z_{p}} = {(\begin{matrix} x \\ y \\ f \end{matrix})}_{p} = x_{p}$
where b refers to base image and p refers to the previous image, x,y are image coordinates, f is the focal length of camera 12 and X,Y,Z are real-space Cartesian coordinates of the object point being imaged relative to the camera coordinates. X is the horizontal coordinate, Y is the vertical coordinate and Z is the distance to the object in the forward direction. C is an ego-motion matrix under the assumption of a global shutter.
C⁻¹is the inverse of ego-motion matrix C.
2. Correcting the Ego Motion Matrix
The input C is the ego motion matrix that describes the real-space motion of camera 12 while capturing the previous and base image frames 15 when the rolling shutter effect is not taken into account. In other words, ego motion matrix C describes the hypothetical situation in which camera 12 does not have a rolling shutter and all the pixels are simultaneously sampled. The term “simultaneous” abbreviated as “sim” is used to describe the situation when all the pixels are sampled simultaneously. So the ego motion matrix C describes ego-motion from previous-sim image to base-sim image. The term “rolling shutter”, abbreviated “RS” is used to describe the situation when all the pixels are not sampled exactly at the same moment or not sampled simultaneously, i.e camera 12 has a rolling shutter.
In order to correct the effect of the rolling shutter, the ego motion from previous-RS image frame 15 to base-RS image frame 15 is used. Let the desired ego motion matrix be denoted C_d. Given that the time difference between the previous and base images is dt, we obtain the following expression for the desired ego-motion matrix C_d:
$C_{d} = C^{1 + ϵ} = C^{1 + \frac{t_{b} - t_{p}}{dt}}$
where t_band t_pare the base point and previous point's time stamps respectively.
To obtain a representation of the time stamp of a pixel, a polynomial approximation may be used of ego-motion variation due to the rolling shutter. The power of the ego motion matrix C is calculated approximately, using a first order Taylor expansion. The first order Taylor expansion is linear where approximation of C^ϵ for ϵ is small, yields the following result:
$C^{ϵ} \approx (\begin{matrix} I + ϵlog R & ϵ \frac{\log R}{R - I} t \\ 0 & 1 \end{matrix})$ $C = (\begin{matrix} R & t \\ 0 & 1 \end{matrix})$
where R is a rotation matrix and I is an identity matrix.
3. Calculating Distance Considering Rolling Shutter Effect
The distance is recalculated in the following way:
1. The previous coordinates x_p,y_pof the previous image frame are rotated to match the base coordinate x_b,y_bof the base image frame so that camera 12 recording the previous image and camera 12 recording the base image are parallel and have the same focus of expansion (FOE)).
2. The focus of expansion FOE is calculated by the following equation:
$FOE (x, y) = (t_{x}, t_{y}) \frac{f}{t_{z}}$
where t_x,t_y,t_zare the translation vector components.
3. The disparity, d, is calculated: d=r_b−r_p, where r_bis the distance from the pixel to the FOE in the base image and r_pis the same for the previous image.
4. Finally, the distance is calculated:
$Z_{new} = \frac{- t_{z} \cdot r_{p}}{d}$
4. Final Correction of the Distance
By now the exact distance Z_newto each object point imaged by an image point in the base image may be obtained, in real time, which is the real distance at the exact moment in which the image point was recorded. The real distance to each object point imaged by an image point in the exact moment in which the image point was recorded is not the distance sought. Instead it is desired to cancel the effect of the rolling shutting while camera 12 is moving d. What is needed is the distance obtained if camera 12 had no rolling shutter. To get the distance if camera 12 had no rolling shutter, the time stamp of the base pixel is used, to get the following ego motion matrix under assumption of the rolling shutter:
$C^{\frac{- t_{b}}{dt}}$
Multiplying the ego motion matrix in the new base world point found, gets the final world point and the desired distance from it. The corrected ego motion matrices under assumption of the rolling shutter may be used in equations of section A1 to determine the previous image coordinates. The previous image coordinates may be used to determine the distances and disparities under assumption of a rolling shutter from the equations in section A3 above.

B. Ego Motion Estimation Correction

As stated previously, the original structure from motion (SfM) algorithm includes a stage in which the ego motion matrix C is calculated. As explained in the previous section, in the presence of rolling shutter distortions, the matrix which describes the camera 12 movement from previous-RS to base-RS would be C^a, where the ego motion matrix C is the camera matrix from previous-sim image to base-sim image is:
$α = 1 + ϵ = 1 + \frac{t_{b} - t_{p}}{dt}$
and therefore ϵ is small. In essence, the rolling shutter effect correction is applied to this stage by replacing the ego motion matrix C with C^apixel by pixel or row by row and adjusting all the equations previously presented accordingly.
Ego Motion Estimation in the Structure from Motion (SfM) Algorithm
In order to find the ego motion matrix, the structure from motion (SfM) algorithm uses an iterative process where the optimization goal is to minimize the distances between the epipolar lines corresponding to the previous points and the matching base points. Each iteration includes the following steps:
1. Both cameras 12 (previous and base camera 12 orientations) are rotated so that the line connecting their centers would be perpendicular to both image planes. This rotation is done assuming that the matrix R_pand translation vector t_pobtained in the previous iteration (or given as the initial guess) describe the motion from previous to base.
2. In this position the new rotation matrix would be approximately I and the translation vector would be close to:
(O O −1)^T
Thus, after the rotation is done in the first step, the rotation matrix and translation vector between the cameras may be written as:
$R = I + Δ = (\begin{matrix} 1 & - r & y \\ r & 1 & p \\ - y & - p & 1 \end{matrix}) and t = (\begin{matrix} a \\ b \\ - 1 \end{matrix})$
where r, y, p, a, b are unknown parameters assumed to be small.
3. Let v be a point in the base image, and let u be the corresponding point in the previous image (these points are the result of an earlier point matching stage). Let v and ũ be these points in the coordinate systems of the rotated cameras (step B1). The distance between v and the epipolar line determined by ũ is:
$D = \frac{{\tilde{v}}^{T} t_{x} R \tilde{u}}{\sqrt{w_{1}^{2} + w_{2}^{2}}}$
where
w=t _x Ru=(w ₁ w ₂ w ₃)^T
D=0 is an equation with five unknowns (r, y, p, a, b); so there are as many such equations as the number of point pairs. Neglecting second-order terms, new R and t are obtained by solving a system of linear equations.
Rolling Shutter Distortion Correction
Having no rolling shutter effects, the ego motion matrix from base image to previous image is:
$C = (\begin{matrix} {\tilde{R}}_{2}^{- 1} & 0 \\ 0 & 1 \end{matrix}) P (\begin{matrix} {\tilde{R}}_{1} & 0 \\ 0 & 1 \end{matrix})$
where the first and last matrices describe the rotation of the cameras back to their original position,
$P = t_{x} R = (\begin{matrix} I + Δ & (\begin{matrix} a \\ b \\ - 1 \end{matrix}) \\ 0 & 1 \end{matrix})$
and translation vector t and rotation matrix R describe the motion between the parallel cameras-after step B1).
Considering the rolling shutter effect, the corrected ego motion matrix would be
$C^{α} = (\begin{matrix} {\tilde{R}}_{2}^{- 1} & 0 \\ 0 & 1 \end{matrix}) P_{α} (\begin{matrix} {\tilde{R}}_{1} & 0 \\ 0 & 1 \end{matrix})$
and thus:
$P_{α} = t_{α} \times R_{α} = (\begin{matrix} {\tilde{R}}_{2} & 0 \\ 0 & 1 \end{matrix}) C^{α} (\begin{matrix} {\tilde{R}}_{1}^{- 1} & 0 \\ 0 & 1 \end{matrix})$
Expanding this expression and using first order Taylor expansions gives the corrected expressions for the translation and rotation matrices, and a new equation (D=0) with the same five unknowns is obtained, only here it incorporates the time stamp difference between pixels caused by the rolling shutter.
The indefinite articles “a” and “an” is used herein, such as “a camera”, “an image frame” have the meaning of “one or more” that is “one or more cameras” or “one or more image frames”.
Although selected embodiments of the present invention have been shown and described, it is to be understood the present invention is not limited to the described embodiments. Instead, it is to be appreciated that changes may be made to these embodiments, the scope of which is defined by the claims and the equivalents thereof.

Claims

What is claimed is:

1. A system for processing images captured by a moving rolling shutter camera, the system comprising:

an interface configured to receive, from the rolling shutter camera, a plurality of images of an environment of the camera, captured while the camera is moving;

a processor configured to:

correct for a global ego-motion of the camera between a first image from the plurality of images and a second image from the plurality of images;

correct the first image for a rolling shutter distortion, to give rise to a corrected first image; and

match pixels' locations in the corrected first image with respective pixels' locations in at least the second image.

2. The system according to claim 1, wherein the global ego-motion of the camera is represented by data assuming a first capture time simultaneously for all pixels of of the first image and a second capture time simultaneously for all pixels of the second image.

3. The system according to claim 2, wherein the processor is configured to perform the correction for the global ego-motion of the camera by taking into account the data representing the egomotion of the camera between the assumed first capture time and the assumed second capture time.

4. The system according to claim 2, wherein the global ego-motion is at least one of: a translation and a rotation of the camera between the first capture time of the first image and the second capture time of the second image.

5. The system according to claim 1, wherein the correction of the first image for the rolling shutter distortion uses a plurality of different time values, for respective rows or for respective pixels in the first image.

6. The system according to claim 5, wherein the time values correspond to capture times of the respective rows or pixels during operation of the rolling shutter camera.

7. The system according to claim 5, wherein the processor is configured to perform the rolling shutter distortion correction by taking into account data representing a motion of the camera between exposure times of different rows of the rolling shutter camera.

8. The system according to claim 1, wherein the processor is further configured to:

utilize an ego-motion matrix to process the first image,

apply a global ego-motion correction to the ego-motion matrix, wherein the global ego-motion correction does not take into account different exposure times of different rows or different pixels of the rolling shutter camera with respect to the first image; and

apply a polynomial expression representing ego-motion variation between different rows or different pixels due to the rolling shutter camera.

9. The system according to claim 1, wherein the processor is further configured to adjust a time stamp associated with a picture element in the first image according to a predefined distortion per picture element model.

10. The system according to claim 1, wherein the processor is configured to:

identify a plurality of pairs of image points, wherein a first image point of a given pair of image points is an image point in the corrected first image, and a second image point of the image point pair is a point in the second image which corresponds to the first image point;

associate each of the first and the second image points in each pair of image points with respective first and second epipolar lines; and

determine a distance between the first and second epipolar lines of each pair of image points.

11. The system according to claim 1, wherein the processor is further configured to compute depth information based on locations of pixels in the corrected first image and based on locations of corresponding pixels in at least the second image.

12. The system according to claim 1, wherein the processor is further configured to determine a distance to at least a portion of an object imaged in the first image and in the second image based on a relation between corresponding pixel locations in the first and second images.

13. A method for processing images captured by a rolling shutter camera, the method comprising:

configuring an interface to receive, from the rolling shutter camera, a plurality of images, of an environment of the camera, captured while the camera is moving;

configuring a processor to:

correct for a global ego-motion between a first image from the plurality of images and a second image from the plurality of images;

correct the first image for the rolling shutter distortion, giving rise to a corrected first image; and

14. The method according to claim 13, further comprising:

representing the global ego-motion by data assuming a first capture time simultaneously for all pixels of of the first image and a second capture time simultaneously for all pixels of the second image.

15. The method according to claim 14, further comprising:

configuring the processor to perform the global ego-motion correction taking into account the data representing the global egomotion of the camera between the assumed first capture time and the assumed second capture time.

16. The method according to claim 14, wherein the global ego-motion is at least one of: a translation and a rotation of the camera between a capture time of the first image and a capture time of the second image.

17. The method according to claim 13, further comprising:

configuring the processor to use a plurality of different time values for rolling-shutter distortion correction for respective rows or for respective pixels in the first image.

18. The method according to claim 17, wherein the time values correspond to capture times of the respective rows or pixels during operation of the rolling shutter camera.

19. The method according to claim 17, further comprising:

configuring the processor to perform the rolling shutter distortion correction taking into account data representing a motion of the camera between exposure times of different rows of the rolling shutter camera.

20. The method according to claim 13, further comprising:

configuring the processor to:

utilize an ego-motion matrix to process the first image,

apply a global ego-motion correction to the ego-motion matrix, wherein the global ego-motion correction does not take into account different exposure times of different rows or different pixels of the rolling shutter camera with respect to the first image, and

21. The method according to claim 13, further comprising:

configuring the processor to adjust a time stamp associated with a pixel in the first image according to a predefined distortion per picture element model.

22. The method according to claim 13, further comprising:

configuring the processor to:

23. The method according to claim 13, further comprising:

configuring the processor to compute depth information based on locations of pixels in the corrected first image and based on locations of corresponding pixels in at least the second image.

24. The method according to claim 13, further comprising:

configuring the processor to determine a distance to at least a portion of an object imaged in the first image and in the second image based on a relation between corresponding pixel locations in the first and second images.

25. A method for processing images captured by a rolling shutter camera, the method comprising:

receiving, from the rolling shutter camera, a plurality of images, of an environment of the camera, captured while the camera is moving;

correcting for a global ego-motion between a first image from the plurality of images and a second image from the plurality of images;

correcting the first image for the rolling shutter distortion, giving rise to a corrected first image; and

matching pixels' locations in the corrected first image with respective pixels' locations in at least the second image.