JP2012160887A - Imaging device and motion vector detection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device and a motion vector detection method that are capable of correcting an error in the vertical direction of a motion vector detected from an image captured using a rolling shutter method.SOLUTION: A camera 11 has imaging elements arranged in a matrix state, and extracts signals of the imaging elements using an XY address reading method in which shutter times for adjacent rows from the top row to the bottom row have a predetermined deviation time, thereby obtaining a frame image in a predetermined frame period. A motion vector detection unit 13, for a target object included in two temporally continuous frame images, detects a movement amount of the target object in the row direction and the column direction between the two frame images. A motion vector vertical component correction unit 14 corrects a column direction component of the movement amount detected by the motion vector detection unit 13 using at least one of the predetermined deviation time and the frame period, and outputs a row direction component of the movement amount and the corrected column direction component.

Description

  Embodiments described herein relate generally to an imaging apparatus and a motion vector detection method.

  One of driving methods of image sensors such as CMOS (Complementary Metal Oxide Semiconductor) image sensors is an XY address reading method (hereinafter referred to as a rolling shutter method). The rolling shutter method is a method in which when one frame image is created, a shutter operation is sequentially performed for each pixel row from the upper portion to extract an image signal, and the readout operation for all rows is completed within a frame cycle.

  The image signal is sampled while the shutter is open. For this reason, in the rolling shutter method, the sampling time is delayed by a predetermined shift time for each row from the upper row to the lower row in one frame image. For this reason, when a moving body is imaged using the rolling shutter method, it is preferable to correct an image using a motion vector while taking into account different sampling times for each row.

JP 2009-141717 A

  By the way, in the rolling shutter method, the sampling interval between the same rows of temporally adjacent frame images coincides with the frame period. Therefore, the pixel row direction (horizontal direction) component of the motion vector can be easily obtained by dividing the moving pixel amount between frames by the frame period.

  On the other hand, the sampling interval between different rows of temporally adjacent frame images is different from the frame period. For this reason, if the component in the pixel column direction (vertical direction) of the motion vector is calculated by dividing the amount of moving pixels between frames by the frame period in the same manner as the horizontal component, an error occurs. Therefore, in the conventional technique, for a subject whose motion vector includes a vertical component (a subject moving in the vertical direction), an error is included in the vertical component of the detected motion vector.

  In order to solve the above-described problem, an imaging apparatus according to an embodiment of the present invention includes a camera, a motion vector detection unit, and a motion vector vertical component correction unit. The camera has image pickup devices arranged in a matrix, and takes out signals from the image pickup device by an XY address reading method in which a shutter time between adjacent rows from a top row to a bottom row has a predetermined shift time. Frame images are obtained at a predetermined frame period. The motion vector detection unit detects the amount of movement of the object in the row direction and the column direction between the two frame images for the object included in the two temporally continuous frame images. The motion vector vertical component correction unit corrects the column direction component of the movement amount detected by the motion vector detection unit using at least one of a predetermined shift time and a frame period, and corrects the row direction component of the movement amount and the corrected column. Outputs the direction component.

1 is an overall configuration diagram illustrating an example of an imaging apparatus according to an embodiment of the present invention. FIG. 2A is an explanatory diagram showing an example of the relationship between the sampling plane of the frame image and time in the global shutter method, and FIG. 2B is an explanatory diagram showing an example of the relationship between the sampling plane of the frame image and time in the rolling shutter method. FIG. 3 is an explanatory diagram illustrating an example of a yt plane of a sampling surface of a frame image in the rolling shutter method illustrated in FIG. The CPU of the main control unit shown in FIG. 1 corrects the error in the column direction (vertical direction) of the motion vector detected from the image acquired by the rolling shutter camera, and corrects the image distortion using the corrected motion vector. The flowchart which shows the procedure at the time of correct | amending.

  Embodiments of an imaging apparatus and a motion vector detection method according to the present invention will be described with reference to the accompanying drawings.

  FIG. 1 is an overall configuration diagram illustrating an example of an imaging apparatus 10 according to an embodiment of the present invention.

  The imaging device 10 includes a rolling shutter camera 11 and a main control unit 12.

  The rolling shutter camera 11 includes a CMOS (Complementary Metal Oxide Semiconductor) image sensor that has image sensors arranged in a matrix and outputs an image signal by a rolling shutter method (XY address reading method).

  The main control unit 12 is configured by a storage medium such as a CPU, RAM, and ROM. The CPU of the main control unit 12 loads a motion vector detection program stored in a storage medium such as a ROM and data necessary for executing the program into the RAM, and is acquired by the rolling shutter camera 11 according to the program. An error in the column direction (vertical direction) of the motion vector detected from the image is corrected.

  The RAM of the main control unit 12 provides a work area for temporarily storing programs and data executed by the CPU. The storage medium such as the ROM of the main control unit 12 stores a motion vector detection program and various data necessary for executing these programs.

  A storage medium such as a ROM has a configuration including a recording medium readable by a CPU, such as a magnetic or optical recording medium or a semiconductor memory, and a part of programs and data in the storage medium. Or you may comprise so that all may be downloaded via an electronic network via the network connection part which is not shown in figure.

  In this case, the network connection unit implements various information communication protocols according to the network form, and connects the main control unit 12 and other electric devices via the electronic network according to the various protocols. For this connection, an electrical connection via an electronic network can be applied. Here, an electronic network means an information communication network using telecommunications technology in general, in addition to a wireless / wired LAN (Local Area Network) and the Internet network, a telephone communication line network, an optical fiber communication network, a cable communication network, Includes satellite communications networks.

  As shown in FIG. 1, the CPU of the main control unit 12 functions as at least a motion vector detection unit 13, a motion vector vertical component correction unit 14, and an image distortion correction unit 15 by a motion vector detection program. Each of the units 13 to 15 uses a required work area of the RAM as a temporary storage location for data. In addition, you may comprise these function implementation parts by hardware logics, such as a circuit, without using CPU.

  FIG. 2A is an explanatory diagram showing an example of the relationship between the sampling plane of the frame image and time in the global shutter method, and FIG. 2B is an explanation showing an example of the relationship between the sampling plane of the frame image and time in the rolling shutter method. FIG.

  There are two main methods of driving a camera having image sensors arranged in a matrix, a global shutter method and a rolling shutter method.

  The pixel on the frame corresponding to each element of the imaging elements arranged in the matrix is P (x, y), and the pixel on the frame where sampling starts at time t is P (x, y, t). .

  The global shutter method is a method in which all image sensors are sampled simultaneously. For this reason, in the global shutter method, arbitrary pixels on the same frame are sampled at the same time t. Therefore, in the global shutter method, as shown in FIG. 2A, the x axis, the y axis, and the t axis are orthogonal to each other.

  On the other hand, the rolling shutter method is a method in which when one frame image is created, a shutter operation is sequentially performed for each pixel row from the top to extract an image signal, and the readout operation for all rows is completed within the frame period T. is there. For this reason, the sampling times are the same between the image sensors in the same row, but the sampling time is delayed by a predetermined shift time k for each row from the upper row to the lower row. Therefore, in the rolling shutter method, as shown in FIG. 2B, the x-axis and the y-axis and the x-axis and the t-axis are orthogonal, but the y-axis and the t-axis are not orthogonal.

  Next, a method for correcting a motion vector detected based on a frame image obtained by the rolling shutter method will be described.

  FIG. 3 is an explanatory diagram showing an example of the yt plane of the sampling plane of the frame image in the rolling shutter method shown in FIG. In FIG. 3, a broken line represents a sampling plane of a frame image in the rolling shutter method.

  The motion vector detection unit 13 uses a block matching method, a gradient method, or the like for a target object included in two temporally continuous frame images obtained by the rolling shutter method, between two frame images. A motion vector m (u, v) in the row direction and the column direction of the object is detected.

  For example, let us consider a case where two frames that are temporally continuous are a frame n-1 and a frame n, and an object at the y coordinate y1 in the frame n-1 is observed at the y coordinate y2 in the frame n.

At this time, the motion vector detection unit 13 calculates a motion vector m (u, v) using the frame period T. Therefore, the column direction component v of the motion vector m detected by the motion vector detection unit 13 can be written as follows.
v = (y2−y1) / T (1)

In general, a motion vector is often defined as a movement amount between frames, not per unit time. For this reason, the motion vector detection unit 13 may output the amount of movement between frames. At this time, for example, the column direction component of the movement amount between frames output from the motion vector detection unit 13 can be obtained by the following equation obtained by modifying the equation (1).
vT = y2−y1 (2)

  However, the rolling shutter method sampling plane has a time lag with respect to the global shutter method sampling plane in accordance with the vertical pixel position, so that the motion vector m obtained by the equations (1) and (2) The column direction component v includes an error.

  Therefore, the motion vector vertical component correction unit 14 converts the column direction component v · T of the movement amount detected by the motion vector detection unit 13 into a predetermined shift time (shift time between sampling times in the rolling shutter method) and a frame. Correction is performed using at least one of the periods T, and the row direction component u · T and the column direction component v0 · T after correction are output.

In the rolling shutter method, when the shift time between sampling times is k, the time required to read one frame is τ, and the total number of rows of the image sensor is N, the shift time k between rows is expressed as k = τ / N. Is done. At this time, the time delays Δt1 and Δt2 of the sampling surface of the rolling shutter method with respect to the sampling surface of the global shutter method at the y coordinates y1 and y2 can be expressed by the following equations (3) and (4), respectively.
Δt1 = (τ / N) y1 = ky1 (3)
Δt2 = (τ / N) y1 = ky2 (4)

Therefore, the time from the time when the y coordinate y1 is sampled in the frame n-1 to the time when the y coordinate y2 is sampled in the frame n is shifted by the time Δt compared to the frame period T (see FIG. 3). ).
Δt = Δt2−Δt1 = k (y2−y1) (5)

Therefore, the movement amount v0 per unit time of the column direction component (vertical component) of the motion vector corresponding to the movement from y1 to y2 can be expressed as follows.
v0 = (y2−y1) / (T + Δt) (6)

This expression (5) is an expression indicating the movement amount v0 per unit time of the column direction component (vertical component) of the correct motion vector, and can be considered as the movement amount per unit time converted into the global shutter sampling plane. . By transforming equation (5), the following equation that gives the column direction component of the correct amount of movement between frames can be obtained.
v0 T = (y2−y1) T / (T + Δt)
= (y2−y1) / (1 + Δt / T) (7)

Assuming that the vertical component of the movement amount between frames is v · T = V before correction and v0 · T = V0 after correction, the following equation is derived from Equation (2) and Equation (7).
V0 = V / (1 + Δt / T) (8)

Moreover, since it can be written as V = Δt / k from Expression (2) and Expression (5), Expression (8) can be modified as follows.
V0 = V / (1 + kV / T) (9)

  The motion vector vertical component correction unit 14 corrects the vertical component V of the movement amount output from the motion vector detection unit 13 to V0 using this equation (9) and outputs it.

When kV / T is sufficiently smaller than 1, Equation (9) can be approximated as follows.
V0 ≒ V (1−KV) (10)
However, K = k / T = (τ / T) / N

At this time, if τ / T≈1, then K = (τ / T) / N≈1 / N, so equation (10) can be modified as follows.
V0 ≒ V (1−V / N) (11)

  The motion vector vertical component correction unit 14 may correct the vertical component V of the movement amount output by the motion vector detection unit 13 to V0 and output it using Expression (10) or Expression (11).

  Note that since the row direction component (horizontal component) of the motion vector is on the same row between frames, a sampling time difference due to the rolling shutter method does not occur. For this reason, it is not necessary to correct the horizontal component.

Next, image distortion correction will be described.
The image distortion correction unit 15 uses the row direction component U of the movement amount output by the motion vector vertical component correction unit 14 and the column direction component V0 after correction, and the rolling shutter of the frame image output by the rolling shutter camera 11. Correct distortion caused by the law. Here, it is assumed that U = u · T. Hereinafter, the image distortion correction will be described more specifically.

For the vertical direction, the image distortion correction unit 15 calculates the y coordinate y0 of the pixel position on the sampling surface of the global shutter method using the corrected column direction component V0.
y0 = y2−v0Δt2
= y2 (1−kv0) (Formula (4))
= y2 (1−kV0 / T)
= y2 (1−KV0) (12)

For the horizontal direction, the image distortion correction unit 15 calculates the x coordinate x0 of the pixel position on the sampling plane of the global shutter method using the row direction component U of the movement amount between frames output from the motion vector detection unit 13.
x0 = x2−uΔt2
= x2−uky2 (Formula (4))
= x2−Uky2 / T
= x2−UKy2 (13)

Furthermore, if τ / T≈1, then K = (τ / T) / N≈1 / N, and therefore Expression (12) and Expression (13) can be modified as follows.
y0 = y2 (1−V0 / N) (14)
x0 = x2−U (y2 / N) (15)

  The image distortion correction unit 15 corrects the pixel position of each pixel of the frame image output from the rolling shutter camera 11 using Expression (14) and Expression (15), thereby causing image distortion caused by the rolling shutter method. Correct.

  Next, an example of the operation of the imaging apparatus and the motion vector detection method according to the present embodiment will be described.

  4 corrects an error in the column direction (vertical direction) of a motion vector detected from an image acquired by the rolling shutter camera 11 by the CPU of the main control unit 12 shown in FIG. It is a flowchart which shows the procedure at the time of correcting distortion of an image using it. In FIG. 4, reference numerals with numbers added to S indicate steps in the flowchart.

  First, in step S1, the motion vector detection unit 13 acquires image data of two temporally continuous frame images obtained from the rolling shutter camera 11 by the rolling shutter method.

  Next, in step S2, the motion vector detection unit 13 performs the row direction component U and the column direction component of the interframe movement amount of the object between two temporally continuous frame images based on the acquired image data. V is calculated.

  Next, in step S3, the motion vector vertical component correction unit 14 determines, based on any one of the equations (9) to (11), a time difference between the sampling times in the rolling shutter method (shutter deviation time) k (= Using at least one of τ / N) and the frame period T, the column direction component V0 after the correction of the inter-frame movement amount is calculated.

  Next, in step S4, the image distortion correction unit 15 performs the row direction component U of the movement amount between frames and the post-correction based on the equations (12) and (13) or the equations (14) and (15). By correcting the pixel position of each pixel of the frame image output by the rolling shutter camera 11 using the column direction component V0, the image distortion caused by the rolling shutter method is corrected.

  Through the above procedure, errors in the column direction (vertical direction) of the motion vector detected from the image acquired by the rolling shutter camera 11 can be corrected, and the distortion of the image can be corrected using the corrected motion vector. .

  In the imaging apparatus 10 according to the present embodiment, the motion vector vertical component correction unit 14 calculates the column direction component V0 after correcting the inter-frame movement amount based on any one of Expressions (9) to (11). it can. For this reason, it is possible to easily and accurately correct the error in the column direction component of the motion vector caused by the rolling shutter method. Therefore, the imaging apparatus 10 according to the present embodiment is suitable for a technique that requires an accurate motion vector, such as a driving support technique.

  Further, it is possible to accurately correct image distortion caused by the rolling shutter method based on the column direction component V0 after the correction of the inter-frame movement amount.

  In addition, although some embodiment of this invention was described, these embodiment is shown as an example and is not intending limiting the range of invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  Further, in the embodiment of the present invention, each step of the flowchart shows an example of processing that is performed in time series in the order described. The process to be executed is also included.

DESCRIPTION OF SYMBOLS 10 Imaging device 11 Rolling shutter camera 12 Main control part 13 Motion vector detection part 14 Motion vector vertical component correction | amendment part 15 Image distortion correction part

Claims (6)

The image sensor is arranged in a matrix, and a shutter time between adjacent rows from the upper end row to the lower end row has a predetermined shift time. A camera that obtains a frame image at a frame period;
A motion vector detection unit that detects a movement amount of the object in a row direction and a column direction between the two frame images with respect to an object included in two temporally continuous frame images;
The column direction component of the movement amount detected by the motion vector detection unit is corrected using at least one of the predetermined shift time and the frame period, and the row direction component of the movement amount and the corrected column direction A motion vector vertical component correction unit that outputs a component;
An imaging apparatus comprising: The motion vector vertical component correction unit is
When the column direction component of the amount of movement detected by the motion vector detection unit is V, and the value obtained by dividing the predetermined shift time by the frame period is K, after the correction by V0 = V / (1 + KV) The column direction component V0 of the movement amount is output, and the row direction component of the movement amount and the corrected column direction component V0 are output.
The imaging device according to claim 1. The motion vector vertical component correction unit is
When the column direction component of the amount of movement detected by the motion vector detection unit is V and the value obtained by dividing the predetermined shift time by the frame period is K, the correction is performed by V0 = V (1−KV). Calculating a subsequent column direction component V0, and outputting the row direction component of the movement amount and the corrected column direction component V0;
The imaging device according to claim 1. The motion vector vertical component correction unit is
When the column direction component of the amount of movement detected by the motion vector detection unit is V and the total number of rows of the image sensor is N, V0 = V (1−V / N) and the corrected column direction. Calculating a component V0, and outputting the row direction component of the movement amount and the corrected column direction component V0;
The imaging device according to claim 1. Distortion due to the XY address reading method of the frame image obtained by the camera using the row direction component and the corrected column direction component of the movement amount output by the motion vector vertical component correction unit An image distortion correction unit that corrects
The imaging device according to any one of claims 1 to 4, further comprising: By extracting a signal of the image sensor from the camera having the image sensor arranged in a matrix by an XY address reading method in which the shutter time between adjacent rows from the upper row to the lower row has a predetermined deviation time. Obtaining a frame image at a predetermined frame period;
Detecting a movement amount of the object in a row direction and a column direction between the two frame images for an object included in two temporally continuous frame images;
Correcting the column direction component of the detected movement amount using at least one of the predetermined shift time and the frame period, and outputting the row direction component and the corrected column direction component of the movement amount; ,
A motion vector detection method comprising:

Images