TW201424366A - Method for correcting rolling shutter effect and image processing device using the same - Google Patents

Abstract

A method for correcting rolling shutter effect is provided. The method includes: obtaining multiple feature point pairs in images, wherein each of the feature point pair corresponds to a motion vector; obtaining sampling points between two consecutive images in time; setting a moving velocity and an angular velocity of an image capturing unit at every sampling point as variables; obtaining multiple estimating motion vector according to the variables, a focal length of the image capturing unit, and row locations where the feature point pairs are located; executing an optimization algorithm according to a difference between the motion vectors and the estimating motion vectors, for calculating the moving velocity and the angle velocity corresponding to the variable; changing the locations of pixels in an image according to the moving velocity and the angle velocity. Therefore, the rolling shutter effect in an image is removed.

Description

Rolling shutter correction method and image processing device

The present invention relates to a method of correcting a rolling shutter effect and an image processing apparatus using the same.

In general, the photosensitive element of a camera can be implemented by a complementary metal-oxide semiconductor (CMOS) or a charge coupled device (CCD). However, when a complementary MOS is used, only one photosensitive element on one column is exposed at each time point. Therefore, if the camera or the object being photographed is moving while shooting, a so-called rolling shutter effect is produced.

Figure 1 is a schematic diagram showing the rolling shutter effect.

As shown in FIG. 1, the camera 110 is to photograph an object 120. Ideally, in the captured image, object 120 should be upright (e.g., object 130). It is assumed here that the time taken by the camera 110 to take an image is S. In the process of photographing the object 120, the position where the camera 110 is located at the time point nS and the position at which the time point (n+1)S is located are not the same. Thus, in the image 140 actually captured, the object 150 will be tilted. This is because at the time point nS, the camera 110 captures the upper half of the object 120, and when the camera 110 is about to capture the lower half of the object 120, the position of the camera 110 has changed. In other words, if the object being photographed moves quickly, the pixel values obtained by the photosensitive elements on each column will have a horizontal displacement. Therefore, how to correct the rolling shutter effect generated during shooting is an issue of interest to those skilled in the art.

Embodiments of the present invention provide a method of correcting a rolling shutter effect and an image processing apparatus using the same, which can correct a rolling shutter effect generated at the time of shooting.

An embodiment of the present invention provides a method for correcting a rolling shutter effect, which is applicable to an image processing apparatus. The method includes: acquiring a plurality of feature point pairs of a plurality of images in a video, wherein each feature point pairing corresponds to an action vector, and the video is captured by an image capturing unit; Between adjacent images, a plurality of sampling points are obtained, wherein each sampling point corresponds to a column position; at least one moving speed and at least one angular velocity of the image capturing unit at each sampling point are set to a plurality of variables; Obtaining a plurality of estimated motion vectors of feature point pairing according to the variable, the focal length of the image capturing unit, and the column position where the feature points are paired; performing an optimization algorithm according to the difference between the motion vector and the estimated motion vector a method for calculating a moving speed corresponding to the variable and the angular velocity; and changing a position of the plurality of pixels in one image according to the moving speed and the angular velocity, thereby generating a first corrected image.

In an embodiment, the feature point pairing described above includes a second feature point pairing. The second feature point pairing includes a second feature point and a third feature point. The position of the second feature point is (x ₁ , y ₁ ), and the position of the third feature point is (x ₂ , y ₂ ). The motion vector corresponding to the second feature point pairing is (x ₂ -x ₁ , y ₂ -y ₁ ). The step of obtaining the estimated action vector of the feature point pairing according to the variable, the focal length of the image capturing unit, and the column position where the feature point is paired includes: calculating the estimated action corresponding to the second feature point pairing according to the following equation (1) The x component of the vector, and the y component corresponding to the estimated motion vector corresponding to the second feature point pairing is calculated according to the following equation (2).

Where S1 and S2 are real numbers, S1 represents a sampling point corresponding to the column position where the second feature point is located, and S2 represents a sampling point corresponding to the column position where the third feature point is located. f is the focal length of the image capturing unit. Z is the depth of field of the image. The moving speed of the image capturing unit in the x direction at the sampling point i. The moving speed of the image capturing unit in the y direction at the sampling point i. For the angular velocity of the image capture unit on the x-axis at the sampling point i, The angular velocity of the image capture unit at the sampling point i on the y-axis. The angular velocity of the image capture unit at the sampling point i on the z-axis.

In an embodiment, the step of performing the optimization algorithm according to the difference between the motion vector and the estimated motion vector comprises: generating a plurality of constraints according to the x component and the y component of the motion vector; generating the first according to the estimated motion vector a matrix, wherein the number of columns in the first matrix is greater than the number of rows in the first matrix; and subtracting the result of multiplying the first matrix by the variable according to the constraints to generate a cost function, and performing an optimization algorithm according to the cost function To obtain the moving speed and angular velocity corresponding to the variable.

In an embodiment, the cost function described above further includes a variable and a differential The result of multiplying the matrix. The value of the jth column of the jth row in the differential matrix is -1, and the value of the j+1th row of the jth column in the differential matrix is 1, and j is a positive integer.

In an embodiment, multiplying a second matrix by a third matrix is the first matrix. In the i-th column of the second matrix, only the value from the 5th (i-1)th to the 5thth row is not 0, and the ith row of the second matrix corresponds to the limit The i-th limit, where i is a positive integer. The i-th limit is corresponding to the (5(i-1)+1)th column to the 5thth column of the third matrix, and the (5(i-1)+1)th column to the 5thth column are not corresponding to the The value of the i-limited sampling interval is zero.

In an embodiment, the plurality of images includes a second image. The position of the first pixel in the second image is (x _rs , y _rs ). The step of changing the position of the pixel in an image according to the moving speed and the angular velocity corresponding to the variable, thereby generating the first corrected image comprises: calculating the displacement p _{x of the} first pixel in the x direction according to the equation (3), and according to the equation (4) Calculating the displacement p _{y of the} first pixel in the y direction, where p _x and p _y are real numbers.

Where n and S are positive integers, and the second image is exposed from time point nS to time point (n+1)S. a is a floating point number. v _x (t) is the moving speed of the image capturing unit in the x direction at time t. v _y (t) is the moving speed of the image capturing unit in the y direction at the time point t. w _x (t) is the angular velocity of the image capturing unit on the x-axis at time t. w _y (t) is the angular velocity of the image capturing unit on the y-axis at time t. w _z (t) is the angular velocity of the image capturing unit on the z-axis at time t.

In an embodiment, the above correction method further comprises: generating a trajectory according to the moving speed and the angular velocity; performing a filter operation on the captured trajectory; setting a smooth trajectory as the second variable, according to the second variable A second difference between the captured trajectories of the filter operation produces a cost function; and a second optimization algorithm is performed according to the cost function, thereby obtaining a smooth trajectory.

In an embodiment, the above correction method further comprises: changing a position of a pixel in the first corrected image according to a gap between the smooth captured track and the captured track, thereby generating a second corrected image.

In another aspect, the present invention provides an image processing apparatus including a memory and a processor. There are multiple instructions stored in the memory. The processor is coupled to the memory for executing instructions to perform a plurality of steps: acquiring a plurality of feature point pairs of the plurality of images in a video, wherein each feature point pairing corresponds to an action vector, and the video is Obtaining by an image capturing unit; obtaining a plurality of sampling points from two temporally adjacent images, wherein each sampling point corresponds to a column position; and the image capturing unit is at each sampling point The at least one moving speed and the at least one angular velocity are set to a plurality of variables; according to the variable, the focal length of the image capturing unit, and the column position where the feature points are paired, the plurality of estimated motion vectors of the feature point pairing are obtained; according to the motion vector Performing an optimization algorithm with the estimated motion vector to calculate a moving speed corresponding to the variable and the angular velocity; and changing a position of the plurality of pixels in one image according to the moving speed and the angular velocity, thereby A first corrected image is produced.

In an embodiment, the feature point pairing described above includes a second feature point pairing. The second feature point pairing includes a second feature point and a third feature point. The position of the second feature point is (x ₁ , y ₁ ), and the position of the third feature point is (x ₂ , y ₂ ). The action vector corresponding to the second feature point pairing is (x ₂ -x ₁ , y ₂ -y ₁ ). The step of obtaining the estimated action vector of the feature point pairing according to the variable, the focal length of the image capturing unit, and the column position where the feature point is paired includes: calculating the estimated action corresponding to the second feature point pairing according to the above equation (1) The x component of the vector, and the y component corresponding to the estimated motion vector corresponding to the second feature point pairing is calculated according to the above equation (2).

In an embodiment, the step of performing the optimization algorithm according to the difference between the motion vector and the estimated motion vector comprises: generating a plurality of constraints according to the x component and the y component of the motion vector; generating the first according to the estimated motion vector a matrix, wherein the number of columns in the first matrix is greater than the number of rows in the first matrix; and subtracting the result of multiplying the first matrix by the variable according to the constraints to generate a cost function, and performing an optimization algorithm according to the cost function To obtain the moving speed and angular velocity corresponding to the variable.

In an embodiment, the cost function described above further includes a result of multiplying the variable by a differential matrix. The value of the jth column of the jth row in the differential matrix is -1, and the value of the j+1th row of the jth column in the differential matrix is 1, and j is a positive integer.

In an embodiment, multiplying a second matrix by a third matrix is the first matrix. In the i-th column of the second matrix, only the value from the 5th (i-1)th to the 5thth row is not 0, and the ith row of the second matrix corresponds to the limit The i-th limit, where i is a positive integer. The i-th limit is corresponding to the (5(i-1)+1)th column to the 5thth column of the third matrix, and the (5(i-1)+1)th column to the 5thth column are not corresponding to the i limited sampling interval The value is 0.

In an embodiment, the plurality of images includes a second image. The position of the first pixel in the second image is (x _rs , y _rs ). The step of changing the position of the pixel in an image according to the moving speed and the angular velocity corresponding to the variable, thereby generating the first corrected image comprises: calculating the displacement p _{x of the} first pixel in the x direction according to the above equation (3), and according to Equation (4) above calculates the displacement p _{y of the} first pixel in the y direction, where p _x and p _y are real numbers.

In an embodiment, the multiple steps further include: generating a capture trajectory according to the moving speed and the angular velocity; performing a filter operation on the captured trajectory; setting a smooth trajectory as the second variable, according to the second variable A second difference between the extracted trajectory and the filtered trajectory produces a cost function; and a second optimization algorithm is performed based on the cost function, thereby obtaining a smooth trajectory.

In an embodiment, the multiple steps further include: changing a position of a pixel in the first corrected image according to a gap between the smooth captured track and the captured track, thereby generating a second corrected image.

Based on the above, the correction method and the image processing apparatus proposed by the embodiments of the present invention can obtain the moving speed and the angular velocity of the image capturing unit by using an optimization algorithm, thereby changing the position of the pixels in the image. In this way, the rolling shutter effect in the image can be corrected.

The above described features and advantages of the present invention will be more apparent from the following description.

2 is a schematic diagram of an image processing apparatus according to an embodiment.

Referring to FIG. 2 , the image processing apparatus 200 includes a processor 210 , a memory 220 , and an image capturing unit 230 . In the present embodiment, the image processing apparatus 200 is a camera. However, in other embodiments, the image processing device 200 can also be a personal computer, a smart phone, a tablet, a notebook, a server, or a camera.

The processor 210 is used to control the overall operation of the image processing apparatus 200. For example, the processor 210 is a central processing unit (CPU), a microprocessor (Microprocessor), a digital signal processor (DSP), a programmable controller, and a special application integrated circuit (Application Specific). Integrated Circuits (ASIC) or Programmable Logic Device (PLD).

The memory 220 can be a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, or other memory. In this embodiment, a plurality of instructions are also stored in the memory 220.

The image capturing unit 230 is configured to acquire a piece of video, and the video includes a plurality of images. For example, the image capturing unit 230 includes a complementary MOS, a shutter, and a lens. The image capture unit 230 will transmit the images to the processor 210, and the processor 210 will execute instructions in the memory 220 to remove the rolling shutter effects in the images.

FIG. 3 is a schematic diagram showing the acquisition of feature point pairing according to an embodiment.

Referring to FIG. 3, the video captured by the image capturing unit 230 includes images 310, 320, and 330. The processor 210 will acquire a plurality of feature points in the images 310, 320 and 330. For example, the processor 210 performs a Scale-Invariant Feature Transform (SIFT) to obtain a plurality of feature points and feature values corresponding to each feature point. The processor 210 compares the feature values to find matching feature points in different images. For example, image 310 includes feature points 311-313, image 320 includes feature points 321-325, and image 330 includes feature points 331, 334 and 335. The feature point 311 is matched to the feature point 321 , the feature point 312 is matched to the feature point 322 , and the feature point 313 is matched to the feature point 323 . The corresponding two feature points form a feature point pairing. For example, the feature point 311 and the feature point 321 form a feature point pair 341. Each feature point pairing corresponds to a motion vector, and two feature points corresponding to one feature point pairing represent two positions of an object. For example, an object moves from the position of the feature point 311 to the position of the feature point 321 .

In this embodiment, the processor 210 obtains 200 feature point pairs between adjacent pictures every two times. However, in other embodiments, the processor 210 may also obtain a greater or lesser number of feature point pairs, and the invention is not limited thereto. In addition, the processor 210 may also acquire feature point pairing by optical flow or motion estimation, and the present invention is not limited thereto.

4 is a schematic diagram showing the acquisition of sampling points between adjacent two pictures according to an embodiment.

In FIG. 3, the motion vector corresponding to the feature point pair 341 is between two adjacent pictures. However, when calculating an action vector caused by the rolling shutter effect, this motion vector must have a higher temporal resolution. Referring to FIG. 3 and FIG. 4, the screen 310 is captured at a time point 410, the screen 320 is captured at a time point 420, and the screen 330 is captured at a time point 430. Curve 450 represents the true displacement of an object. However, the motion vector represented by feature point pair 341 is the result of curve 450 being "quantized". In this embodiment, the processor 210 takes multiple sample points from two temporally adjacent pictures, and each sample point corresponds to a column position. For example, time point 410 and time point 420 are divided into a plurality of sampling points 441-446. Each of the sampling points 441-446 corresponds to a column position in the image 320. For example, if an image has a total of 560 column positions, the sampling point 441 corresponds to the 80th column position, that is, at the sampling point 441, the image capturing unit 230 is exposing the photosensitive position at the 80th column position. element. The processor 210 calculates the moving speed and angular velocity of the image capturing unit 230 at each of the sampling points 441 to 446.

FIG. 5 is a schematic diagram of calculating an estimated motion vector according to an embodiment.

Referring to FIG. 5, it is assumed here that the screen 310 is exposed from the time point nS to the time point (n+1)S, and the picture 320 is exposed from the time point (n+1)S to the time point (n+2)S. Where n is a positive integer and S is the time required to expose a picture. The position of the feature point 311 on the screen 310 is (x1, y1), and the position of the feature point 320 in the screen 320 is (x2, y2). Therefore, the exposure time of the feature point 311 can be expressed as (n+ay ₁ )S, and the exposure time of the feature point 321 can be expressed as (n+1+ay ₂ )S. Where a is a floating point number and its value is 1/img_rows. Img_rows is a positive integer that is represented as the number of columns in a frame. On the other hand, at the time point t, the moving speed of the image capturing unit 230 in the x direction can be expressed as v _x (t), and the moving speed in the y direction can be expressed as v _y (t). The image capturing unit 230 at the time point t, the angular velocity on the x-axis can be expressed as w _x (t); at the time point t, the angular velocity on the y-axis can be expressed as w _y (t); and at the time point t The angular velocity on the z-axis is w _z (t). And a motion vector between the feature points 311 to 321 may represent a feature point _{(x 2 -x 1, y 2} -y 1), which can be calculated through the following equation (1) and Equation (2).

Where f denotes the focal length of the image capturing unit 230. Z is the depth of field of the video in the video, and is assumed to be a constant here. The above equations (1) and (2) are calculated in the form of integrals. However, since the picture 310 and the picture 320 have been divided into a plurality of sampling points, the motion vector can be calculated in a discrete form. When calculated in discrete form, the time point (n+ay ₁ )S can be represented as a sampling point S1 which is the column position y ₁ corresponding to the feature point 311. And the time point (n+1+ay ₂ )S can be expressed as the sampling point S2 which is the column position y ₂ corresponding to the feature point 321 . Thereby, equations (1) and (2) can be rewritten as equations (3) and (4).

among them, Indicates the moving speed of the image capturing unit 230 in the x direction at the sampling point i. Indicates the moving speed of the image capturing unit 230 in the y direction at the sampling point i. Indicates the angular velocity of the image capturing unit 230 on the x-axis when sampling point i. Indicates the angular velocity of the image capturing unit 230 on the y-axis when sampling point i. Indicates the angular velocity of the image capturing unit 230 on the z-axis when sampling point i.

The processor 210 sets the moving speed and angular velocity of the image capturing unit 230 at each sampling point to a plurality of variables. Based on the variables, the focal length of the image capturing unit 230, and the column position at which the feature points are paired, the processor 210 obtains a plurality of estimated motion vectors paired by the feature points. For example, the two moving speeds and the three angular velocities at each sampling point of the image capturing unit 230 can be expressed as five variables. The left side of equations (3) and (4) is a known condition that can be obtained by the position of feature point 311 and feature point 321 . The right side of equations (3) and (4) is composed of a plurality of variables, a focal length f, and sampling points S1 and S2 corresponding to the column positions. Here, the calculated motion vector is also an estimated motion vector. In theory, the motion vector calculated on the left side of the equation should be equal to the estimated motion vector calculated on the right side of the equation. Therefore, the processor 210 performs an optimization algorithm based on the difference between the motion vectors and the estimated motion vectors to calculate the moving speed and angular velocity corresponding to the variables.

For example, 5 variables per sample point can be represented as vectors . If n sample points are taken from between two adjacent pictures, there will be 5n variables between the two pictures. In addition, if the moving speed and angular velocity between k pictures are to be calculated at a time, the number of variables will be 5kn. Here, 5kn=5N is set, where k, n, and N are positive integers. These variables can be expressed as a vector x whose dimensions are 5N-by-1 (5N-by-1).

In an embodiment, processor 210 may only use the speed of movement in the x direction and the angular velocity on one axis. The present invention does not limit which moving speeds and angular velocities to be included in the above variables.

On the other hand, the processor 210 generates a plurality of constraints based on the x and y components of the motion vector, and the number of these limits is greater than the number of variables (5N). In this embodiment, if m feature point pairs (i.e., m motion vectors) are obtained between two pictures, 2m limits (corresponding to the x component and the y component) are obtained. In addition, if k frames are taken at a time, there will be 2mk limits. Here, 2mk=M is set, and M>5N, where m and M are positive integers. These limits can be expressed as a vector b whose dimensions are M-by-1.

The processor 210 evaluates the motion vector according to the above calculation by a matrix A (also referred to as a first matrix). The number of columns in this matrix A will be greater than the number of rows in matrix A. The processor 210 generates a cost function according to the above-described limit b minus the multiplication of the matrix A and the variable b, and performs an optimization algorithm according to the cost function to obtain the moving speed and the angular velocity.

For example, the result of multiplying the matrix A by the variable x is to estimate the x and y components of the motion vector. Then the cost function in the optimization algorithm can be expressed as the following equation (5). Where the dimension of matrix A is M-multiplication -5N.

The matrix A will be divided into two matrices, denoted as A=A _D A _I . The matrix A _D (also known as the second matrix) is expressed as equation (6) and its dimension is Mx5M.

Is a vector with a dimension of 1-by-5, representing the coefficient corresponding to the ith limit in equation (3) or (4) above. Specifically, the i-th limit corresponds to two feature points, and the position of one feature point is expressed as ( , ). And for the x component limit, It is expressed as the following equation (7). And for the limitation of the y component, It is expressed as the following equation (8).

It is worth noting that the five coefficients in equation (7) are coefficients corresponding to the right side of equation (3); and the five coefficients in equation (8) are coefficients corresponding to the right side of equation (4). In other words, the ith column in the matrix A _D corresponds to the ith limit in the limit b. In the i-th column of the matrix A _D , only the value from the 5th (i-1)th to the 5thth row is not 0, and the rest is 0. The five values that are not zero are the coefficients corresponding to the ith limit (such as equation (7) or (8)).

On the other hand, the dimension of the matrix A _I is 5M-by-5N, and the i-th constraint is the (5(i-1)+1)th column to the 5thth column corresponding to the matrix A _I . The column multiplied by the variable i of the i-th limit will become a 5-by-1 vector (as shown in Figure 6). And the value of one sampling interval which is not corresponding to the i-th limit in the (5(i-1)+1)th column to the 5ith column is 0. Specifically, if the feature points of the i-th constraint corresponding to the feature points are respectively at the sampling point S1 and the sampling point S2 (as in equations (3) and (4)), the sampling point S1 and the sampling point S2 A sampling interval will be formed. In the N sampling points in FIG. 6, the coefficients of the S1th sampling point to the S1th sampling point are not 0. And w ₁ corresponds to the sampling point S1; w ₂ corresponds to the sampling point S2. Also, w ₁ and w ₂ are real numbers between 0 and 1. For example, if there are 6 sampling points between two screens and one screen has 560 column positions, the sampling point S1 corresponds to 40 column positions (between the 0th column position and the 80th column position). ), then w ₁ will be 0.5.

After the matrix A is established, the processor 210 can calculate the variable x according to equation (5). Since the calculated variable x should have a smooth change in time. Therefore, in another embodiment, the cost function shown in the above equation (5) may further include a result of multiplying the variable x by a differential matrix. For example, processor 210 may use equation (9) below as a cost function to optimize the algorithm.

G is a differential matrix in which the value of the jth column of the jth row is -1, and the value of the j+1th row of the jth column is 1 (ie, G(j,j)=-1 and G(j,j+1) ) = 1). j is a positive integer. λ is a real number and can be customized by the user.

After the variable x (i.e., the moving speed at all sampling points and the angular velocity) is calculated according to the above equation (9). The processor 210 changes the position of the plurality of pixels in one image according to the moving speed and the angular velocity corresponding to the variable x, thereby generating a first corrected image. Taking image 310 as an example, assuming that the position of a first pixel before uncorrection is (x _rs , y _rs ), the corrected position will be (x _gs , y _gs ) = (x _rs , y _rs ) + (p _x , p _y ). Where p _x is the displacement of the first pixel in the x direction; and p _y is the displacement of the first pixel in the y direction. The processor 210 calculates the displacement p _x and the displacement p _y according to the focal length, the moving speed, and the angular velocity of the image capturing device 230. For example, the processor 210 can determine the displacement p _x and the displacement p _y according to the following equations (10) and (11), where p _x and p _y are real numbers.

It is worth noting that equations (10) and (11) move all pixels to a position corresponding to the exposure time of (n+0.5)S. Thereby, all the pixels in one image are moved to the same exposure time position, so that the rolling shutter effect can be removed. Here, the image after the operations of equations (10) and (11) is also referred to as a first corrected image.

In an embodiment, the processor 210 stabilizes the first corrected image based on the calculated moving speed and angular velocity. Specifically, the processor 210 generates a capture track by the moving speed and the angular velocity, and the captured track represents a moving track caused by the image capturing unit 230 being moved by the user. For example, the processor 210 calculates the captured trajectory according to the following equation (12). , where n is a real number representing time.

Where v _u (i) represents the moving speed or angular velocity of the image capturing unit 230 at the sampling point i. u is a symbol representing an angular velocity or a moving velocity. For example, if u = x, v _x represents the moving speed of the image capturing unit 230 in the x direction. If the captured trajectory changes greatly in time, the played video will be shaken. Therefore, the processor 210 performs a filter operation on the captured track and sets a smooth captured track as the second variable. The processor 210 generates a cost function according to the difference between the second variable and the extracted trajectory calculated by the filter (also called the second gap), and a second optimization algorithm for the cost function, thereby Get the smooth capture track. For example, this cost function can be represented by the following program (13).

Where h is a filter, such as a Gaussian filter. λ ₁ and λ ₂ are real numbers, ▽ is a gradient vector, and ▽ ² is a Laplace operator. Then it is smooth to capture the track. It is worth noting that the smooth snap track is a vector representing the speed of movement or angular velocity at each sample point. The nth element in the smoothed trace can be expressed as .

Calculating a smooth capture trajectory In the future, the processor 210 will capture the trajectory according to the smoothing. And capture track The difference between the pixels in the first corrected image is changed, thereby generating a second corrected image. The first correction is assumed that image pixels may be expressed as I _gs (x, y), the pixel of the second calibration image may be expressed as _{I gss (x ', y'} ). Smooth capture track And capture track The difference between the two can be expressed as the following equation (14); the relationship between the position of the pixel in the first corrected image and the position of the pixel in the second corrected image can be expressed as the following equation (15); and the processor 210 is based on the following Equations (16) and (17) produce a second corrected image. It is worth noting that when u is expressed as the moving speed and the angular velocity, Δ t _u ( n ) can also be rewritten as Δ t _x ( n ), Δ t _y ( n ), Δ r _x ( n ), Δ r _y ( n ) and Δ r _z ( n ).

7 is a flow chart illustrating a method of correcting a rolling shutter effect, in accordance with an embodiment.

Referring to FIG. 7, in step S702, a plurality of feature point pairs of a plurality of images in a video are acquired. Each of the feature point pairs corresponds to an action vector, and the video is captured by an image capturing unit. In step S704, a plurality of sampling points are taken from two temporally adjacent images, wherein each sampling point corresponds to a column of positions. In step S706, at least one moving speed and at least one angular velocity of the image capturing unit at each sampling point are set as a plurality of variables. In step S708, according to the variables, the focal length of the image capturing unit and the column in which the feature points are paired Position, obtain multiple estimated motion vectors of feature point pairing. In step S710, an optimization algorithm is performed according to the difference between the motion vector and the estimated motion vector to calculate the moving speed and the angular velocity corresponding to the variable. In step S712, the position of the plurality of pixels in one image is changed according to the moving speed and the angular velocity corresponding to the variable, thereby generating a corrected image. In step S714, the first corrected image is made smoother based on the calculated moving speed and angular velocity, thereby generating a second corrected image. However, the steps in FIG. 7 have been described in detail above, and will not be described again here. It is to be noted that step S714 is an optional step, that is, step S714 may be omitted in an embodiment.

In an embodiment, the various steps performed by processor 210 may be implemented as one or more circuits. The invention is not limited to being implemented in the form of a soft body or a hard body.

In summary, the method for correcting rolling shutters and the image processing apparatus according to the embodiments of the present invention can obtain sampling points with high time resolution, and calculate the moving speed and angular velocity of each sampling point through an optimization algorithm. . With these moving speeds and angular velocities, the rolling shutter effect can be corrected and the video can be made smoother.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the invention, and any one of ordinary skill in the art can make some modifications and refinements without departing from the spirit and scope of the invention. The scope of the invention is defined by the scope of the appended claims.

110‧‧‧ camera

120, 130, 150 ‧ ‧ objects

200‧‧‧Image processing device

210‧‧‧ processor

220‧‧‧ memory

230‧‧‧Image capture unit

140, 310, 320, 330‧ ‧ images

311~313, 321~325, 331, 334, 335‧ ‧ feature points

341‧‧‧ Feature Point Pairing

410, 420, 430‧‧‧ time points

441~446‧‧‧ sampling points

450‧‧‧ Curve

Steps of S702, S704, S706, S708, S710, S712, S714‧‧‧ rolling shutter correction method

Figure 1 is a schematic diagram showing the rolling shutter effect.

2 is a schematic diagram of an image processing apparatus according to an embodiment.

FIG. 3 is a schematic diagram showing the acquisition of feature point pairing according to an embodiment.

4 is a schematic diagram showing the acquisition of sampling points between adjacent two pictures according to an embodiment.

FIG. 5 is a schematic diagram of calculating an estimated motion vector according to an embodiment.

FIG. 6 is a schematic diagram showing a third matrix of a portion according to an embodiment.

7 is a flow chart illustrating a method of correcting a rolling shutter effect, in accordance with an embodiment.

Steps of S702, S704, S706, S708, S710, S712, S714‧‧‧ rolling shutter correction method

Claims (16)

A method for correcting a rolling shutter effect, which is applicable to an image processing apparatus, comprising: acquiring a plurality of feature point pairs of a plurality of images in a video, wherein each of the feature point pairs corresponds to an action vector, and the video is Obtained by an image capturing unit; between the two temporally adjacent images in the images, a plurality of sampling points are obtained, wherein each of the sampling points corresponds to a column position; the image is captured The at least one moving speed and the at least one angular velocity of the unit at each of the sampling points are set to a plurality of variables; according to the variables, a focal length of the image capturing unit, and the column positions where the feature points are paired, Obtaining a plurality of estimated motion vector pairs of the feature points; performing an optimization algorithm according to a gap between the motion vectors and the estimated motion vectors, to calculate the at least corresponding to the variables a moving speed and the at least one angular velocity; changing the position of the plurality of pixels of the one of the images according to the at least one moving speed and the at least one angular velocity corresponding to the variables Thereby generating a first correction image. The method of claim 1, wherein the feature point pairing comprises a second feature point pairing, the second feature point pairing comprises a second feature point and a third feature point, the second feature The position of the point is (x ₁ , y ₁ ), the position of the third feature point is (x ₂ , y ₂ ), and the action vector corresponding to the pair of second feature points is (x ₂ -x ₁ , y ₂ - y ₁ ), wherein the step of obtaining the estimated motion vectors paired by the feature points according to the variables, the focal length of the image capturing unit, and the column positions where the feature points are paired comprises: Equation (1) calculates an x component of the estimated motion vector corresponding to the second feature point pairing, and calculates one of the estimated motion vectors corresponding to the second feature point pairing according to the following equation (2) y component: Wherein S1 and S2 are real numbers, S1 represents the sampling point corresponding to the column position where the second feature point is located, S2 represents the sampling point corresponding to the column position where the third feature point is located, and f is the image 撷Taking a focal length of the unit, Z is a depth of field of the images, The at least one moving speed of the image capturing unit at the sampling point i in the x direction, The at least one moving speed of the image capturing unit in the y direction at the sampling point i, The at least one angular velocity of the image capturing unit on the x-axis at the sampling point i, The at least one angular velocity of the image capturing unit on the y-axis at the sampling point i, The at least one angular velocity of the image capturing unit at a sampling point i on a z-axis. The method of claim 1, wherein the step of performing the optimization algorithm according to the gap between the motion vectors and the estimated motion vectors comprises: determining an x according to the motion vectors Multiple limits for components and one y component Generating a first matrix according to the estimated motion vectors, wherein the number of columns in the first matrix is greater than the number of rows in the first matrix; and subtracting the first matrix from the variables according to the constraints A result of the generation produces a cost function, and the optimization algorithm is executed according to the cost function to obtain the at least one moving speed and the at least one angular velocity corresponding to the variables. The method of claim 3, wherein the cost function further comprises a result of multiplying the variables by a differential matrix, wherein the value of the jth column in the jth column of the differential matrix is -1, the differential The value of the j+1th row of the jth column in the matrix is 1, and j is a positive integer. The method of claim 3, wherein a multiplication of a second matrix with a third matrix is the first matrix, and in the ith column of the second matrix, only from the 5th (i) -1) The value of the +1 line to the 5th line is not 0, and the ith line of the second matrix corresponds to the ith limit of the restrictions, where i is a positive integer, the ith The restriction is corresponding to the (5(i-1)+1)th column to the 5th column of the third matrix, and the (5(i-1)+1)th column to the 5ith column does not correspond to the first The value of a sampling interval of i limits is zero. The method of claim 1, wherein the image includes a second image, wherein a position of a first pixel in the second image is (x _rs , y _rs ), wherein the variable corresponds to the variable The at least one moving speed and the at least one angular speed change positions of the pixels of the one of the images, wherein the step of generating the first corrected image comprises: calculating the first pixel according to equation (3) a displacement p _{x in the} x direction, and calculating a displacement p _{y of} the first pixel in a y direction according to equation (4), where p _x and p _y are real numbers, Where n is a positive integer, the second image is exposed from the time point nS to the time point (n+1)S, a is a floating point number, f is a focal length of the image capturing unit, and Z is the image a depth of field, v _x (t) is the at least one moving speed of the image capturing unit in the x direction at a time point t, and v _y (t) is the image capturing unit at the time point t in the y direction The at least one moving speed, w _x (t) is the at least one angular velocity of the image capturing unit on an x-axis at a time point t, and w _y (t) is that the image capturing unit is at a time point t The at least one angular velocity on a y-axis, w _z (t), is the at least one angular velocity of the image capturing unit on a z-axis at a time point t. The method of claim 1, further comprising: generating a trajectory according to the at least one moving speed and the at least one angular velocity; performing a filter operation on the captured trajectory; and performing a smooth trajectory Set as a second variable, generating a cost function according to a second difference between the second variable and the captured trajectory calculated by the filter; and performing a second optimization algorithm according to the cost function, Thereby the smooth capture trajectory is obtained. The method of claim 7, further comprising: changing a position of a pixel in the first corrected image according to a gap between the smoothed trajectory and the captured trajectory, thereby generating a second corrected image . An image processing apparatus includes: a memory storing a plurality of instructions; and a processor coupled to the memory for executing the instructions to perform a plurality of steps: obtaining a plurality of images in a video Feature point pairing, wherein each of the feature point pairs corresponds to an action vector, and the video is captured by an image capturing unit; and between the two temporally adjacent images in the images, Obtaining a plurality of sampling points, wherein each of the sampling points corresponds to a column position; at least one moving speed and at least one angular velocity of the image capturing unit at each of the sampling points is set to a plurality of variables; And obtaining, by the plurality of variables, a focal length of the image capturing unit, and the column positions where the feature points are paired, obtaining a plurality of estimated motion vectors paired by the feature points; and determining the motion vector according to the motion vectors Performing an optimization algorithm between the gaps to calculate the at least one moving speed and the at least one angular velocity corresponding to the variables; and the at least one moving speed corresponding to the variables The speed change position at least one corner of one of the plurality of pixels of the plurality of images, thereby generating a first correction image. The image processing device of claim 9, wherein the feature point pairing comprises a second feature point pairing, the second feature point pairing includes a second feature point and a third feature point, the second The position of the feature point is (x ₁ , y ₁ ), the position of the third feature point is (x ₂ , y ₂ ), and the action vector corresponding to the second feature point pairing is (x ₂ -x ₁ , y _{2 -} y ₁ ), wherein the step of obtaining the estimated motion vectors paired by the feature points according to the variables, the focal length of the image capturing unit, and the column positions where the feature points are paired comprises: Calculating an x component of the estimated motion vector corresponding to the second feature point pair according to the following equation (1), and calculating the estimated motion vector corresponding to the second feature point pair according to the following equation (2) One y component: Wherein S1 and S2 are real numbers, S1 represents the sampling point corresponding to the column position where the second feature point is located, S2 represents the sampling point corresponding to the column position where the third feature point is located, and f is the image 撷Taking a focal length of the unit, Z is a depth of field of the images, The at least one moving speed of the image capturing unit at the sampling point i in the x direction, The at least one moving speed of the image capturing unit in the y direction at the sampling point i, The at least one angular velocity of the image capturing unit on the x-axis at the sampling point i, The at least one angular velocity of the image capturing unit on the y-axis at the sampling point i, The at least one angular velocity of the image capturing unit at a sampling point i on a z-axis. The image processing device of claim 9, wherein the step of performing the optimization algorithm according to the difference between the motion vectors and the estimated motion vectors comprises: according to one of the motion vectors The x component and the y component generate a plurality of constraints; generating a first matrix according to the estimated motion vectors, wherein the number of columns in the first matrix is greater than the number of rows in the first matrix; and subtracting the A result of multiplying the first matrix by the variables produces a cost function, and the optimization algorithm is executed in accordance with the cost function to obtain the at least one moving speed and the at least one angular velocity corresponding to the variables. The image processing device of claim 11, wherein the cost function further comprises a result of multiplying the variables by a differential matrix, wherein the value of the jth column in the jth column of the differential matrix is -1, The value of the j+1th row of the jth column in the differential matrix is 1, and j is a positive integer. The image processing device of claim 11, wherein a multiplication of a second matrix with a third matrix is the first matrix, and only the fifth from the ith column of the second matrix I-1) The value of the +1 line to the 5th line is not 0, and the ith line of the second matrix corresponds to the ith limit of the restrictions, where i is a positive integer, the ith The limit is corresponding to the (5(i-1)+1)th column to the 5th column of the third matrix, and the (5(i-1)+1)th column to the 5ith column are not corresponding to the The value of a sampling interval of the i-th limit is zero. The image processing device of claim 9, wherein the images comprise a second image, wherein a position of a first pixel in the second image is (x _rs , y _rs ), wherein the variables are based on the variables Corresponding at least one moving speed and the at least one angular velocity change positions of the pixels of the one of the images, thereby generating the first corrected image comprises: calculating the first pixel according to equation (3) a displacement p _{x in the} x direction, and calculating a displacement p _{y of} the first pixel in a y direction according to equation (4), where p _x and p _y are real numbers, Where n is a positive integer, the second image is exposed from the time point nS to the time point (n+1)S, a is a floating point number, f is a focal length of the image capturing unit, and Z is the image a depth of field, v _x (t) is the at least one moving speed of the image capturing unit in the x direction at a time point t, and v _y (t) is the image capturing unit at the time point t in the y direction The at least one moving speed, w _x (t) is the at least one angular velocity of the image capturing unit on an x-axis at a time point t, and w _y (t) is that the image capturing unit is at a time point t The at least one angular velocity on a y-axis, w _z (t), is the at least one angular velocity of the image capturing unit on a z-axis at a time point t. The image processing device of claim 9, wherein the steps further comprise: generating a capture trajectory according to the at least one moving speed and the at least one angular velocity; performing a filter operation on the captured trajectory; Setting a smoothed trajectory as a second variable, generating a cost function according to a second difference between the second variable and the captured trajectory calculated by the filter; and performing a second according to the cost function The algorithm is optimized to obtain the smoothed trajectory. The image processing device of claim 15, wherein the steps further comprise: changing a position of the pixel in the first corrected image according to a gap between the smoothed captured track and the captured track, thereby generating A second corrected image.