JP2015220564A - Image processing system and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system capable of precisely correcting rolling shutter distortion without increase in scale of correction means.SOLUTION: The image processing system comprises: input means which receives image data generated from an imaging device of a rolling shutter system; acquisition means which acquires first information relating to the travel of an apparatus body; calculation means which uses information on the scanning direction of the imaging device out of the first information to calculate the correction amount of the image data on the basis of second information converted so as to correspond to the coordinates of the image different from the image corresponding to the first information; and correction means which corrects the image data by using the correction amount calculated by the calculation means.

Description

  The present invention relates to an image processing device and an imaging device.

  As imaging devices such as video cameras and digital cameras have become smaller and lighter, opportunities for taking images with portable imaging devices are increasing. In recent years, CMOS (Complementary Metal Oxide Semiconductor) image sensors are increasingly used as image sensors in such image pickup apparatuses. The main factors for using a CMOS image sensor include low cost, easy increase in the number of pixels, and low power consumption.

  In the CMOS image sensor, a rolling shutter type in which image information is sequentially read out in units of one pixel or one line (usually in the horizontal direction and sometimes in the vertical direction) is the mainstream. In the case of such sequential reading, the exposure period varies for each pixel or line. However, even in the case of sequentially reading in units of one pixel, the reading time difference in one line is so small that it can be ignored compared to the reading time difference between lines. In the following description, reading in units of pixels is handled in the same way as sequential reading in units of lines. When taking an image with a rolling shutter sensor, if the subject moves during the exposure period from the top line to the bottom line on the screen or if camera shake occurs, scan lines The image of the subject is deformed by the difference in exposure period. This becomes distortion. Hereinafter, this distortion is referred to as rolling shutter distortion.

  FIG. 9 is a diagram for explaining a rolling shutter distortion generation process. The upper part of each of FIGS. 9A to 9D shows the positional relationship between the camera and the main subject at the moment of exposure and reading of a certain row of sensors. The left side of the lower part is a graph showing the camera moving speed at each point in time during which each row of the image is scanned, in other words, the moving speed on the image. A circle indicates the speed at the time of reading the read line. The lower right side represents the position of the readout line at that moment on the image. FIG. 9E shows an image finally obtained. Here, a case where the camera gradually moves vertically upward will be described.

  FIG. 9A shows the moment of reading the first row when the reference row for rolling shutter distortion correction is the first row (uppermost row in the figure) of image reading of the sensor. FIG. 9B shows the moment when the readout row exposes the top of the main subject and reads out. The uppermost portion of the main subject in FIG. 9B is the uppermost portion of the main subject in the final image in FIG. FIG. 9C shows the moment when the vicinity of the center of the main subject in the height direction is exposed and read. FIG. 9D shows the moment when the vicinity of the lowermost portion in the height direction of the main subject is exposed and read. The lowermost portion of the main subject in FIG. 9D is the lowermost portion of the main subject in the final image in FIG. When all the sensor rows have been exposed and read, an image shown in FIG. 9E is obtained. Since the camera is moved while the exposure and readout lines are sequentially moved, rolling shutter distortion extending in the vertical direction occurs in the image of FIG.

  Rolling shutter distortion is finally corrected by image processing, and various methods have been proposed as rolling shutter distortion correction processing. For example, Patent Document 1 proposes a method of performing correction by combining a signal from an orientation sensor or image analysis with geometric deformation processing of an image. Further, in Non-Patent Document 1, when high-precision correction is performed by obtaining movement for each line in the same manner as the sensor signal, the sensor equivalent is super-resolved by a fitting process between the polynomial interpolation of the motion and the image. A method for obtaining motion information with a resolution of 2 is proposed.

  As described above, the rolling shutter distortion is caused by the shift of the exposure period for each line of the CMOS image sensor. As exemplified in Non-Patent Document 1, rolling shutter distortion can be modeled by the following equation, for example.

Is the image coordinates subjected to rolling shutter distortion,

Is the image coordinates prior to rolling shutter distortion, m () is the spatial motion model function,

Is a parameter related to the movement of the model. The second term on the right side reproduces the movement on the image due to the movement of the imaging device that occurs between the exposure period deviation (y−y ₁ ) τ between the reference row y ₁ and the row y including the target pixel of geometric deformation. Term. t ₁ represents the photographing time of the reference row, and t represents the photographing time of the row including the pixel to be corrected.

  For example, in Patent Document 2, in accordance with Expression (2) obtained by moving the second term on the right side of Expression (1), an image including rolling shutter distortion is obtained by inverse transformation of integral of image movement caused by camera movement. A corrected image is obtained by geometric transformation. In this way, a technique for obtaining a corrected image by geometrically deforming a distorted image by a movement opposite to the integration of the measured camera or image movement is called forward mapping.

Japanese Patent No. 438979 Japanese Patent No. 3925415

Simon Baker et.al., “Removing Rolling Shutter Wobble”, CVPR2010, p. 2392-2399 Won-ho Cho, et.al., “Affine Motion Based CMOS Distortion Analysis CMOS Digital Image Stabilization”, IEEE 2007, Aug. 2007, Volume. 53, Issue 3, p. 833-841

  However, when trying to obtain a corrected image by forward mapping, in order to maintain the image quality, pixel sampling that becomes non-uniform after correction according to the magnitude of the motion change needs to be sufficiently high-definition with respect to the output image. . Therefore, the sampling density of the image before correction or the pixels in the sensor must be sufficiently high. As a result, the scale of the processing circuit and the processing time become long.

  In view of such a problem, the present invention can correct the rolling shutter distortion with high accuracy without increasing the scale of the correction means.

  An image processing apparatus according to one aspect of the present invention includes an input unit that inputs image data generated from a rolling shutter type imaging device, an acquisition unit that acquires first information related to movement of the apparatus body, and the first unit. Of the image data based on the second information converted so as to correspond to the coordinates of the image different from the image corresponding to the first information, using the information related to the scanning direction of the image sensor. The image processing apparatus includes: a calculation unit that calculates a correction amount; and a correction unit that corrects the image data using the correction amount calculated by the calculation unit.

  An imaging apparatus according to another aspect of the present invention acquires an image sensor that captures images by a rolling shutter system, an input unit that inputs image data generated from the image sensor, and first information relating to movement of the apparatus body. Using the acquisition means and the information related to the scanning direction of the image sensor in the first information, the second information converted to correspond to the coordinates of the image different from the image corresponding to the first information. Based on this, the image processing apparatus includes: a calculation unit that calculates a correction amount of the image data; and a correction unit that corrects the image data using the correction amount calculated by the calculation unit.

  According to the present invention, it is possible to correct the rolling shutter distortion with high accuracy without increasing the scale of the correcting means.

It is a block diagram of an image processing device concerning this embodiment. It is a figure showing the relationship of the flame | frame and row | line of an image of an imaging device, and a time direction. It is a figure explaining the calculation procedure of rolling shutter distortion correction amount. It is a figure explaining the relationship with respect to the image height of a forward mapping process, a backward mapping process, and each correction amount. It is a figure explaining the error which arises when each correction amount is used approximately in the backward mapping process. It is a figure explaining the relationship and kind of image motion and rolling shutter distortion. It is a figure explaining reconstruction of motion information. It is a figure explaining equalization by nonlinear interpolation processing of sampling data. It is a figure explaining the generation process of rolling shutter distortion.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and redundant description is omitted.

  FIG. 1 is a block diagram of an image processing apparatus 100 according to the present embodiment. The image processing apparatus 100 includes an image input unit 101, an attitude change acquisition unit 102, a rolling shutter distortion correction amount calculation unit 103, and a rolling shutter distortion correction unit 104. In the present embodiment, a case where the image processing apparatus 100 is mounted on an imaging apparatus will be described.

  The image input unit 101 inputs an image showing a subject. Specifically, the image input unit 101 includes an optical system, an imaging device, and an imaging system image processing circuit. First, an optical image of a subject is formed on an image sensor by an optical system and converted into an electric signal by a photoelectric conversion function of the image sensor. Next, the subject image converted into the electrical signal is AD-converted in the imaging system image processing circuit, and in the case of a single plate sensor, Bayer interpolation processing is performed. Thereafter, signal processing of an imaging system including aperture correction, gamma correction, white balance processing, and the like is performed, and an output image related to the photographed subject is generated. These imaging system image processing circuits are actually configured as one circuit or a plurality of partial circuits. In particular, the image pickup system image processing circuit is actually often divided into two components: an analog front end that handles analog electrical signals, and a digital back end after AD conversion.

  Posture change acquisition means (hereinafter referred to as acquisition means) 102 detects the movement of the apparatus body, and outputs movement information of the imaging apparatus or information obtained by converting the movement information of the imaging apparatus into movement on the image. The acquisition unit includes a sensor that calculates the rotation amount and shift amount of the imaging apparatus such as a gyro sensor and an acceleration sensor. The acquisition means acquires motion information at a frequency higher than the sampling interval corresponding to the frame rate or the sampling rate for each row of the image. The motion information can be handled by Equation (3) as a vector quantity for an arbitrary time together with a parameter such as focal length information.

In the expression (3), each component element of the vector is obtained by multiplying the sensor output signal or output signal by a coefficient to obtain motion information of the imaging apparatus, or a variable parameter such as a focal length that changes due to zooming. Corresponds to the one that contains.

  When motion information corresponding to camera work is converted into motion on the image and handled, it is converted to motion on the image using the spatial motion model function of Equation (4).

Is a parameter vector including parameters such as pixel size and focal length in addition to the motion information of the imaging device as described above. m () is a spatial motion model function that changes motion information of various imaging devices into motion on an image. When the motion information is speed information, the image coordinates are expressed in unit time τ according to the motion information.

Resulting in motion vector

Is calculated.

  In the imaging apparatus according to the present embodiment, it is assumed that a motion sensor is equipped with a rotation sensor composed of a gyro sensor having two axes perpendicular to the optical axis as rotation axes. When the motion function model is a linear approximation model in the vicinity of the optical axis with respect to the biaxial rotation of the image pickup apparatus, it can be expressed as in equation (5).

As expressed in Equation (5), the change on the image that occurs in unit time is the position on the image.

It becomes a uniform change not depending on. Δθ ₁ (t) and Δθ ₂ (t) are motion information of the imaging device regarding the yaw direction and the pitch direction, angular velocity information obtained from the rotation sensor, f (t) is the focal length of the imaging device, and d _{x and dy} are pixels Represents size.

  Naturally, the spatial motion model function of this embodiment can be replaced with an arbitrary one. For example, a roll axis whose rotational axis coincides with the optical axis may be added, or the influence of the rotation of the imaging device on the image motion may be handled by a projection model without being approximated by a linear / affine camera. Further, a sensor capable of detecting the translational movement of the imaging device may be added, and the movement of the imaging device may be handled by distinguishing not only the rotation but also the translation.

  Rolling shutter distortion correction amount calculation means (hereinafter referred to as calculation means) 103 calculates a geometric transformation correction amount necessary for rolling shutter distortion correction. That is, the calculation unit 103 corrects the geometric distortion caused by the movement of the image pickup device during the shift of the exposure period for each row forming the image due to the rolling shutter by the inverse transformation geometric transformation. A correction amount is calculated.

  First, FIG. 2A shows a relationship between an image frame and a row and a time direction when an image is captured by a rolling shutter type imaging apparatus. Each frame starts to be read in accordance with the vertical synchronization signal. Each row of each frame is sequentially exposed and read for each row to form and record image information. As shown in FIG. 2A, if the delay time for reading between rows is τ and the total number of rows in the image is H, the reading time of (H−1) τ will be shifted between the first row and the last row. Become.

  The rolling shutter distortion correction amount for each pixel constituting the image is taken in the same exposure period with reference to one of the lines having different exposure periods as shown in the second term of the equation (2). As described above, it is obtained as a geometric deformation amount that cancels the distortion caused by the movement.

  FIG. 2B is obtained by correcting the photographed image of FIG. 2A as an image photographed with a global shutter using the calculation unit 103 and the rolling shutter distortion correction unit (hereinafter referred to as correction unit) 104. is there. FIG. 2B illustrates a state in which correction is performed using a change in the imaging device or the image as a reference line from the first line to the third line. In FIG. 2B, the exposure period is virtually moved by using a value obtained by integrating the geometric change generated during the deviation of the exposure timing from the reference row for each row. As a result, an image obtained by capturing the reference row and all the rows at the same time can be obtained as shown in FIG.

  The correction amount calculation procedure will be described with reference to FIG. FIG. 3A shows a set of change information of the moving speed of the image pickup apparatus with respect to the vertical direction obtained by the shooting of FIG. 2 and image data including rolling shutter distortion. The moving speed of the imaging device is integrated as shown in the integral expression of the second term on the right side of Expression (2) (FIG. 3B), and is subtracted from the integrated value, for example, with the reference line as the first line. The amount of change in motion from the time of exposure is obtained (FIG. 3C). Then, as described in the equation (4), the motion change amount is converted into the motion amount on the image coordinates using parameters such as the focal length and the physical size of the pixel, and then the inverse transform is used as the correction amount of each row ( FIG. 3 (c) right view). A correction image is obtained by correcting each row using the correction means 104 with the correction amount calculated for each row in this way (FIG. 3D).

  However, in the case of correction processing by forward mapping according to the equation (2) as described above, the pixel distribution of the corrected image becomes non-uniform depending on the intensity of motion change. For this reason, in the present embodiment, in order to realize the rolling shutter distortion correction by the backward mapping process, the device described below is added. Note that backward mapping refers to performing correction by sampling on the input image with reference to the coordinates of the output image.

  FIG. 4 is a diagram for explaining the relationship between the forward mapping process and the backward mapping process and each correction amount with respect to the image height. Each figure is a set of a graph of a speed change when the imaging apparatus is moved vertically downward and an image obtained at that time. FIG. 4A is a diagram illustrating a relationship with a correction result by forward mapping. If correction is performed when the imaging apparatus moves in the vertical direction, the image is geometrically deformed as shown in the right figure of FIG. 4A, and accordingly, the image height is measured based on the image height. The shape of the movement change amount of the image changes. From this result, if the movement of the imaging device includes a movement in the image height direction, it cannot be simply corrected by the backward mapping process. Specifically, as shown in FIG. 4B, when correction is performed by backward mapping based on the corrected image, correction amount information corresponding to the image height of the corrected image is required at that time. .

  Further, when the motion change amount corresponding to the image height of the corrected image is approximated with the motion change amount corresponding to the image height of the distorted image (FIG. 5A), it is actually geometrically contradictory. Due to the relationship, as shown in FIG. 5B, a geometric error occurs in a portion where the motion change is large.

  In practice, the movement of the imaging device and the resulting rolling shutter distortion is shown in FIG. 6 as follows: (b) horizontal movement, (c) vertical movement, (d) rotation about the optical axis, or Since forward movement and the like not shown are included, it is necessary to further consider the correction.

  As described above, if the rolling shutter distortion correction is to be realized by backward mapping, the movement change amount of the imaging apparatus can be measured only when the image height including the rolling shutter distortion is used as a reference. Conventionally, for this problem, a method has been proposed in which the motion between frames is assumed to be constant speed and the correction amount is analytically derived (see, for example, Non-Patent Document 2).

  In this embodiment, in order to maintain both accuracy and rigor of rolling shutter distortion correction, the motion information of the imaging apparatus measured at the image height including rolling shutter distortion is used to calculate the image height of the undistorted image (corrected image). Reconstructed into motion information as an argument.

  FIG. 7 is a diagram for explaining the reconstruction of motion information. In the present embodiment, the correction amount obtained from the integration information of the vertical motion that is the scanning direction of the image sensor is geometrically transformed with respect to the image height associated with the correction amount obtained from the integration information of the vertical motion. ing. By doing so, the posture change information obtained according to the image height including the rolling shutter distortion, for example, the horizontal motion, the vertical motion, and the rotational motion, the motion information using the image height of the undistorted image (corrected image) as an argument. Convert to The correction amount is obtained by subtracting the motion of the reference row by converting into the motion amount on the image according to the parameter from the integral value of the motion information according to the equation (4). By subtracting a correction amount that cancels the amount of motion in the vertical direction from the image height, it can be converted into motion information using the image height of an undistorted image (corrected image) as an argument. However, an interpolation method such as spline interpolation is used at the end to match the sampling points of motion information that have become non-uniform with respect to the image height of the undistorted image due to geometric transformation to each row (image height) of the image. To reconstruct motion information.

  FIG. 8 is a diagram for explaining uniformization of sampling data by nonlinear interpolation processing. Motion information having a uniform distribution with respect to the undistorted image height is reconstructed by a process involving interpolation on the motion information (FIG. 8A) that has become non-uniform with respect to the undistorted image height due to geometric transformation (see FIG. 8 (b)).

  The correcting unit 104 corrects rolling shutter distortion by geometric transformation. An image and parameters of geometric deformation are input, and an image in which rolling shutter distortion is corrected by geometric transformation is output. Based on the correction amount calculated by the calculation means 103, correction is performed by performing scaling, in-plane rotation, and horizontal / vertical translation for each row. The image output from the correcting unit 104 is recorded on a recording medium or the like by a recording unit (not shown) or displayed on a display unit (not shown). Alternatively, it is treated as input image information for another image processing means (not shown), for example, a combination of a plurality of images such as super-resolution or HDR, or recognition processing.

  As described above, by using the present invention, rolling shutter distortion can be corrected efficiently and with high accuracy by backward mapping processing.

  In addition, the present invention is not applied only to an imaging device. For example, a portable information terminal to which an imaging device is attached, an image processing device that performs post-processing on a captured video, and such an image processing device. The present invention can also be applied to an image display apparatus that includes the image display apparatus. For example, an image processing apparatus having the input unit 101 and the acquisition unit 102 as input / output units of a recording medium or a terminal node of a network may be used.

  In addition, the present invention is not limited to an imaging device that is photographed with a hand held by a user, but is fixedly installed like a surveillance camera and photographs mainly moving subjects such as passers-by and cars. It is also applicable to

  Further, the functions of the imaging apparatus described above can be realized by hardware having the respective functions, but can also be realized by software. When executed by software, it is only necessary that various functions can be realized by installing a program constituting the software in dedicated hardware, or by installing various programs from a recording medium or a network. For example, it can be realized using a general-purpose personal computer.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 101 Image input means (input means)
102 Camera posture change acquisition means (acquisition means)
103 Rolling shutter distortion correction amount calculation means (calculation means)
104 Rolling shutter distortion correction means (correction means)

Claims (8)

Input means for inputting image data generated from a rolling shutter image sensor;
Obtaining means for obtaining first information relating to movement of the apparatus body;
Based on the second information converted to correspond to the coordinates of an image different from the image corresponding to the first information, using the information related to the scanning direction of the imaging element among the first information, Calculating means for calculating a correction amount of the image data;
An image processing apparatus comprising: a correction unit that corrects the image data using the correction amount calculated by the calculation unit.   The image processing apparatus according to claim 1, wherein the correction unit corrects the image data by backward mapping that performs sampling on the input image with reference to the coordinates of the output image.   The imaging apparatus according to claim 1, wherein the coordinates are coordinates of an undistorted image that is not affected by rolling shutter distortion.   The image processing apparatus according to claim 1, wherein the image corresponding to the first information is a distorted image affected by rolling shutter distortion.   The calculation means converts the first information having the image height of the distorted image as an argument into the second information using the image height of the undistorted image as an argument, using information regarding the scanning direction of the image sensor. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   The calculation means converts the first information having the image height of the distorted image as an argument into information using the image height of the undistorted image as an argument, using information related to the scanning direction of the image sensor. The image processing apparatus according to claim 1, wherein the second information is acquired by interpolating information.   7. The image processing apparatus according to claim 1, wherein the first information is information related to at least one of an amount of rotation and a shift amount of the apparatus main body. An image sensor for taking an image with a rolling shutter system;
Input means for inputting image data generated from the image sensor;
Obtaining means for obtaining first information relating to movement of the apparatus body;
Based on the second information converted to correspond to the coordinates of an image different from the image corresponding to the first information, using the information related to the scanning direction of the imaging element among the first information, Calculating means for calculating a correction amount of the image data;
An image pickup apparatus comprising: correction means for correcting the image data using the correction amount calculated by the calculation means.