JP2010154390A - Imaging device, imaging method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily correct distortion of a figure of a dynamic body generated in imaging the dynamic body by a rolling shutter style imaging sensor without changing resolution, in an imaging device. <P>SOLUTION: By using an imaging means to image an object using a rolling shutter style imaging sensor, the imaging means is made to execute first imaging to read signals of the imaging sensor by a first scanning method for executing scanning on a line basis from one direction. The imaging means is made to execute second imaging to read signals of the imaging sensor by a second scanning method for executing scanning on a line basis from a direction reverse to the one direction. A plurality of feature points are extracted from a first image acquired by the first imaging, and a corresponding point corresponding to each of them is detected from a second image acquired by the second imaging. Coordinate values of intermediate points located at intermediate locations between the plurality of feature points and the corresponding points each corresponding to each of them are calculated. The first image/the second image are deformed so that the coordinate values of the plurality of feature points/corresponding points coincide with the coordinate values of the intermediate points. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an imaging apparatus such as a digital camera, and more particularly to an imaging apparatus and an imaging method as well as a program including a rolling shutter type imaging sensor.

  In recent years, a CMOS (Complementary Mental Oxide Semiconductor) image sensor (hereinafter referred to as a CMOS sensor) that enables high-speed readout by structurally preparing a plurality of readout lines or skipping unnecessary portions as an imaging sensor of a rolling shutter system Is spreading. By mounting such a CMOS sensor on a digital camera, continuous shooting can be performed at a higher speed than before, and imaging can be performed without missing a photo opportunity.

  By the way, in a CMOS sensor, pixel signals are generally read out by a rolling shutter system in which shutters are sequentially opened for each scanning line. When reading out the pixel signals of all the pixels by this method, for example, in order from the upper left pixel of the screen in order to read out the signal of each pixel toward the lower right, the read start point (upper left) and the read end point (lower right) When there is a moving body D that moves in the right direction in the figure as shown in the left diagram of FIG. 13, the image obtained by imaging is as shown in the right diagram of FIG. The figure of the moving body D is distorted.

Therefore, in a rolling shutter type image sensor, horizontal or vertical scanning lines are read at a predetermined thinning rate, and the thinned scanning lines are corrected using adjacent scanning lines, thereby reducing the distortion of a moving object figure. In a plurality of frame images captured using a method (Patent Document 1) or a rolling shutter type imaging sensor, a motion region and a motion vector of the motion region are acquired, and the motion vector and the plurality of frame images are obtained. A method for correcting a motion region in a frame image to be corrected in a plurality of frame images based on information on an imaging interval, information on an exposure start time difference, and information on an exposure start order (Patent Document 2) is disclosed. Has been.
JP 2006-148861 A JP 2007-142929 A

  However, since the method described in Patent Document 1 thins out the scanning lines, the vertical resolution is obtained when the horizontal scanning lines are thinned out, and the horizontal resolution is obtained when the vertical scanning lines are thinned out. It will decline. Further, the method described in Patent Document 2 requires information such as detection of a motion region and difference in exposure start time. For example, when there are a plurality of motion regions having different motions in an image, each motion is detected. Since it is necessary to detect a region and perform different correction for each region, there is a possibility that the processing becomes complicated.

  The present invention has been made in view of such circumstances, and an imaging apparatus capable of easily correcting distortion of a moving object figure that occurs when a moving object is imaged by a rolling shutter imaging sensor without changing the resolution. An object of the present invention is to provide an imaging method arrangement program.

  An image pickup apparatus according to the present invention includes an image pickup unit that picks up an image of a subject using a rolling shutter type image pickup sensor,

  First imaging and second imaging are continuously performed by the imaging unit, and a signal of the imaging sensor is read by a first scanning method in which scanning for each line is performed from one direction at the time of the first imaging, An imaging control means for controlling the imaging means so as to read a signal of the imaging sensor by a second scanning method in which scanning for each line is performed in a direction opposite to the one direction at the time of the second imaging;

  Feature point extracting means for extracting a plurality of feature points from the first image acquired by the first imaging;

  Corresponding point detecting means for detecting corresponding points corresponding to each of the plurality of feature points extracted by the feature point extracting means from the second image acquired by the second imaging;

  A coordinate value of an intermediate point located between the plurality of feature points and the corresponding point corresponding to each of the plurality of feature points is calculated, and the coordinate value of the feature point matches the coordinate value of the intermediate point And / or image deformation means for deforming the second image so that the coordinate value of the corresponding point matches the coordinate value of the intermediate point. It is characterized by.

In the present invention, the “rolling shutter type imaging sensor” refers to an image sensor of a type in which the pixels that perform the shutter operation shift in units of rows in the vertical direction or the horizontal direction with time, not all the pixels simultaneously. An image sensor can be mentioned. As the MOS type image sensor, for example, a CMOS sensor can be used.
In addition, “perform first imaging and second imaging continuously” may be performed first first imaging or second imaging first. However, “continuous” imaging is performed so that the time interval between the first imaging and the second imaging does not exceed 1/60 seconds, for example.

  The imaging apparatus of the present invention includes a motion vector calculating unit that calculates a motion vector from the plurality of feature points and the corresponding points corresponding to the plurality of feature points,

  Image deformation prohibiting means for preventing the image deforming means from deforming the first image or the second image when the motion vector calculated by the motion vector calculating means is smaller than a predetermined threshold. And may be further provided.

  As the “motion vector” to be compared with the predetermined threshold, a vector having the largest magnitude among the plurality of motion vectors may be selected, or an average value or an intermediate value of the plurality of motion vectors may be used. The selection method can be changed as appropriate.

  Further, in the imaging apparatus of the present invention, the image deformation means deforms the first image and the second image,

  You may further provide the image synthetic | combination means which synthesize | combines this deformed 1st image and 2nd image, and produces | generates one image.

  An image pickup method of the present invention is an image pickup method using an image pickup means for picking up an object using a rolling shutter type image pickup sensor.

  Causing the imaging means to perform a first imaging;

  The signal of the imaging sensor is read out by a first scanning method in which scanning for each line is performed from one direction during the first imaging,

  Causing the imaging means to perform second imaging;

  The signal of the imaging sensor is read by a second scanning method in which scanning for each line is performed from the direction opposite to the one direction during the second imaging,

  Extracting a plurality of feature points from the first image acquired by the first imaging;

  Detecting corresponding points corresponding to each of the extracted plurality of feature points from the second image acquired by the second imaging;

  Calculating a coordinate value of an intermediate point located between the plurality of feature points and a corresponding point corresponding to each of the plurality of feature points;

  The first image is deformed so that the coordinate values of the plurality of feature points match the coordinate value of the intermediate point, and / or the coordinate value of the corresponding point matches the coordinate value of the intermediate point. Further, the second image is deformed.

  The imaging method of the present invention calculates a motion vector from the plurality of feature points and the corresponding points corresponding to each of the plurality of feature points,

  When the calculated motion vector is smaller than a predetermined threshold value, the first image or the second image may not be deformed.

  Further, the imaging method of the present invention deforms the first image and the second image,

  The deformed first image and second image may be combined to generate one image.

  The program of the present invention is a program for causing a computer to execute an imaging method using an imaging means for imaging a subject using a rolling shutter imaging sensor.

Causing the imaging means to perform a first imaging;
The signal of the imaging sensor is read out by a first scanning method in which scanning for each line is performed from one direction during the first imaging,

  Causing the imaging means to perform second imaging;

  The signal of the imaging sensor is read by a second scanning method in which scanning for each line is performed from the direction opposite to the one direction during the second imaging,

  Extracting a plurality of feature points from the first image acquired by the first imaging;

  Detecting corresponding points corresponding to each of the extracted plurality of feature points from the second image acquired by the second imaging;

  Calculating a coordinate value of an intermediate point located between the plurality of feature points and a corresponding point corresponding to each of the plurality of feature points;

  The first image is deformed so that the coordinate values of the plurality of feature points match the coordinate value of the intermediate point, and / or the coordinate value of the corresponding point matches the coordinate value of the intermediate point. And performing the transformation of the second image.

  The program of the present invention calculates a motion vector from the plurality of feature points and the corresponding points corresponding to each of the plurality of feature points on a computer,

  When the calculated motion vector is smaller than a predetermined threshold, the first image or the second image may be prevented from being deformed.

  Further, the program of the present invention transforms the first image and the second image into a computer,

  The deformed first image and second image may be combined to generate one image.

  According to the imaging apparatus, the imaging method, and the program of the present invention, the substantially identical scene is imaged twice, and the readout scanning direction of the signal of the imaging sensor is reversed between the first imaging and the second imaging. Thus, in the first image acquired by the first imaging and the second image acquired by the second imaging, when the moving object moving at a constant speed is included in the subject, the distortion of the moving object is Since there is a symmetric difference in shape, the coordinate values of intermediate points located between the plurality of feature points extracted from the first image and the corresponding points respectively corresponding to the plurality of feature points are calculated. The first image is deformed so that the coordinate value of the feature point matches the coordinate value of the intermediate point, and / or the second image is changed so that the coordinate value of the corresponding point matches the coordinate value of the intermediate point. The deformation of the rolling shutter system The distortion of the shape of the body that occur when imaging a moving object by the sensor can be easily corrected without changing the resolution.

  A motion vector is calculated from a plurality of feature points and corresponding points corresponding to each of the plurality of feature points, and when the calculated motion vector is smaller than a predetermined threshold, the first image or the second image When the deformation is not performed, the distortion of the moving object due to the imaging sensor is more conspicuous as the speed of the moving object is faster, and conversely, when the moving object is slow, the distortion of the moving object figure becomes smaller. By prohibiting deformation of the image when the movement of the moving body is small, useless processing can be eliminated.

  In addition, when the first image and the second image are deformed and one image is generated by combining the deformed first image and the second image, the first image is acquired. By changing the imaging condition between the imaging and the second imaging for acquiring the second image, for example, when the exposure condition is changed, the first image and the second image have different portions with appropriate exposure. Therefore, by synthesizing the appropriate exposure portion, it is possible to correct the distortion of the moving object figure and expand the dynamic range.

  Hereinafter, an embodiment of an imaging apparatus according to the present invention will be described in detail with reference to the drawings. In the following embodiments, a digital camera will be described as an example of the imaging apparatus in the present invention. However, the scope of the present invention is not limited to this, and for example, electronic imaging such as a mobile phone with a camera, a PDA with a camera, etc. The present invention can also be applied to other electronic devices having functions.

  FIG. 1 is a block diagram showing the functional configuration of the digital camera 1. As shown in FIG. 1, the digital camera 1 according to the present embodiment includes a photographing lens 11, a CMOS sensor (imaging sensor) 15, a diaphragm 12 provided between the photographing lens 11 and the CMOS sensor 15, and an infrared cut filter. 13 and an optical low-pass filter 14.

  The photographing lens 11 is movable in the optical axis direction, and is controlled by a CPU 19 described later and moved to a focus position via a lens driving unit 16. The aperture 12 can adjust the aperture diameter, that is, the aperture amount, and is controlled by a CPU 19 described later to adjust the exposure amount to an appropriate exposure amount via the aperture drive unit 17.

  The CMOS sensor 15 is controlled by a CPU 19 which will be described later and is driven via an image sensor driving unit 18 to output a subject image captured through the photographing lens 11 as a color signal. In the CMOS sensor 15, pixel signals are read out by a rolling shutter system in which the shutter is sequentially released for each scanning line. When reading out the pixel signals of all pixels, first scanning is performed in order from the top line of the screen toward the bottom line in order to scan the pixel signals in order from the pixel at the left end of the screen toward the pixel at the right end. Or a second scanning method in which scanning for each horizontal line is performed in order from the pixel at the left end of the screen toward the pixel at the right end of the screen in order from the line at the lower end of the screen toward the line at the upper end. Are read out and this pixel signal is output as an analog image signal.

  In this embodiment, the scanning for each horizontal line is performed in the upward or downward direction of the screen. However, the scanning for each vertical line may be performed in the left or right direction of the screen. Good.

  In the present embodiment, the imaging lens 10, the diaphragm 12, the infrared cut filter 13, the optical low-pass filter 14, the CMOS sensor 15, the lens driving unit 16, the diaphragm driving unit 17, and the image sensor driving unit 18 described above capture an image of the subject. Function as.

  The CPU (imaging control means) 19 controls the entire digital camera 1, and also functions as an imaging control means for controlling the imaging means 10, and the imaging means 10 has the first imaging and the second imaging. The imaging is continuously performed, the signal of the pixel of the imaging sensor 15 is read by the first scanning method described above at the time of the first imaging, and the imaging sensor 15 by the second scanning method described above at the time of the second imaging. The pixel signal is read out.

  In addition, the CPU 19 controls the light emitting unit 20 and the light receiving unit 21 for flash, and a user instruction signal input via an operation unit 22 such as a shutter button (not shown), a menu / OK button, a zoom / up / down arrow button or the like. Various controls are performed according to

The analog image signal output from the CMOS sensor 15 is input to the analog signal processing unit 23. The analog signal processing unit 23 includes a correlated double sampling circuit (CDS) that removes noise from the analog image signal, and an auto gain controller (AGC) that adjusts the gain of the analog image signal.
The A / D 24 is an A / D converter (ADC) that converts the analog image signal processed by the analog signal processing unit 23 into digital image data. The digital image data converted into the digital signal is CCD-RAW data having RGB density values for each pixel.

  The digital image data converted into a digital signal by the A / D 24 is controlled by a command from the CPU 19, and is output to various units connected to each other by the control bus 37 and the data bus 38 as necessary to perform various processes. Is done. Hereinafter, each part will be described.

  A memory control unit 26 connected to the main memory 25 that stores various constants set in the digital camera 1 and programs executed by the CPU 19 reads various data stored in the main memory 25 in response to commands from the CPU 19. .

  The digital signal processing unit 27 performs image quality correction processing such as gamma correction processing, contour enhancement (sharpness) processing, contrast processing, noise reduction processing, and the like on the input image data, and the CCD-RAW data is a luminance signal. YC processing is performed to convert Y data into YC data composed of Cb data that is a blue color difference signal and Cr data that is a red color difference signal.

  The compression / decompression processing unit 28 performs compression processing on the image data that has been subjected to processing such as image quality correction by the digital signal processing unit 27 in a compression format such as JPEG, and generates an image file. Based on the Exif format or the like, a tag that stores additional information such as the shooting date and time and whether the image was shot in the portrait right protection mode is added to the image file. In the reproduction mode, the compression / decompression processor 28 reads a compressed image file from a recording medium 30 (described later) and performs an expansion process. The expanded image data is output to a display control unit 33 (described later), and the display control unit 33 displays an image based on the image data on a display unit 32 (described later).

  The integrating unit 29 integrates the photometric data and adjusts the white balance gain.

  The external memory control unit 31 to which the detachable recording medium 30 is connected corresponds to a media slot that reads / writes data when the recording medium 30 is connected, and reads out image files and the like stored in the recording medium 30. Or writing an image file.

  A display control unit 33 connected to a display unit 32 such as a liquid crystal monitor mounted on the back surface of the digital camera 1 displays an image based on the image data on the display unit 32.

  The feature point extraction unit 34 extracts feature points from the image captured by the imaging unit 10. A feature point is a point having a feature as a corner of an object, an image pattern, or the like, and is extracted based on gradient information of a pixel signal. For example, the Moravec method or the Harris method can be used as the feature point extraction method. However, in the present invention, the extraction method is not limited as long as the feature points can be extracted.

  The corresponding point detection unit 35 detects a corresponding point corresponding to the feature point extracted by the feature point extraction unit 34 from the image captured by the imaging unit 10. Corresponding points are points that substantially match the features of the feature points. For example, the KLT method, block matching, or the like can be used as a method for detecting the corresponding points. The method is not limited.

  The image deforming unit 36 deforms the image in order to correct the distortion of the subject. The image deformation method will be described later in detail.

  The digital camera 1 of the present embodiment is configured as described above. Next, an imaging method using the digital camera 1 having the above configuration will be described in detail with reference to the drawings. 2 is a flowchart of the imaging process of the digital camera 1 in FIG. 1, FIG. 3A is an image P in which the moving body D in the subject is not distorted, and FIG. 3B is acquired by the first scanning method. FIG. 3C is an image P2 obtained by the second scanning method, and FIG. 4 is a diagram for explaining the symmetry of distortion.

  In the digital camera 1 of this embodiment, as shown in FIG. 2, first, the CPU 19 sets the scanning method of the CMOS sensor 15 to the above-described first scanning method (step S1). Then, the CPU 19 determines whether or not the shutter button is fully pressed (step S2). If the shutter button is not fully pressed (step S2; NO), the process of step S2 is repeated until the shutter button is fully pressed. Do.

  On the other hand, if it is determined in step S2 that the shutter button has been fully pressed (step S2; YES), the CPU 19 causes the imaging means 10 to perform the first imaging by the first scanning method to display the first image P1. It is acquired (step S3). For example, as shown in FIG. 3A, when a moving body D that moves in the right direction in the figure exists in the subject, the first image P1 is acquired by the first scanning method, and then FIG. As shown in b), the CMOS sensor 15 scans the horizontal line from the pixel at the left end of the screen in the right direction, that is, reading the pixel signal in the direction of the arrow A1 in the figure, from the upper end of the screen toward the lower end of the screen. Since the steps are sequentially performed in the direction of the middle arrow A2, there is a difference in time for reading out pixel signals between the upper and lower pixels of the moving object D in the subject, and the moving object D is moved by the amount of movement of the moving object D during this time difference. Is distorted, and the moving object in the first image P1 becomes a distorted figure moving object D1.

  Thus, after the imaging means 10 acquires the image P1 in which the figure of the moving body D1 is distorted (step S3), the CPU 19 changes the scanning method from the first scanning method to the second scanning method described above (step S4). . Next, the CPU 19 causes the imaging unit 10 to perform the second imaging by the second scanning method and acquire the first image P1 (step S3). The CPU 19 causes the imaging unit 10 to continuously perform imaging so that the time difference between the first imaging and the second imaging does not exceed 1/60 seconds. In the present embodiment, the first imaging and the second imaging have the same imaging conditions such as brightness, shutter speed, and aperture.

  When the second image P2 is acquired by the second scanning method, as shown in FIG. 3C, the CMOS sensor 15 detects the pixel signals sequentially from the pixel at the left end of the screen in the right direction, that is, in the direction of the arrow B1 in the drawing. Since the scanning of the horizontal line for reading out is sequentially performed from the lower end of the screen toward the upper end of the screen, that is, in the direction of the arrow B2 in the figure, the upper end pixel and the lower end pixel of the moving object D in the subject are the same as in the first scanning method. Thus, a difference in time for reading out the pixel signal occurs, and the moving object D is distorted by the movement of the moving object D during this time difference, and the moving object in the second image P2 becomes a moving object D2 of a distorted figure.

  In the first scanning method, when the pixel signal is read, the upper end side of the moving object D is read first, so that the distorted moving object D1 of the first image P1 becomes a figure whose lower end is moved in the moving direction of the moving object D. In the scanning method, when the image signal is read, the lower end side of the moving object D is read first, so that the distorted moving object D2 of the second image P2 becomes a figure whose upper end has moved in the moving direction of the moving object D. That is, the distortion of the moving body D varies depending on the moving speed of the moving body D and the scanning time and scanning direction of the CMOS sensor 15.

  That is, when the moving speed of the moving body D is constant and the scanning time when the first scanning method and the second scanning method of the CMOS sensor 15 are the same, for example, the moving body D in an undistorted state (FIG. 3A 4), the vertical right end of the first image P1 becomes a distorted moving object D1, and the second image P2 shows a distorted moving object D2, as shown in FIG. When E2 is translated into the first image P1 so as to intersect with the inclination E1 to be the inclination E2 ′, the inclination E1 and the inclination E2 ′ are symmetrical with respect to the axis G passing through each intersection. Therefore, the distortion shape of the moving object D1 in the first image P1 and the distortion shape of the moving object D2 in the second image P2 have symmetry.

  Then, by calculating the shape of the moving body D without distortion from this symmetry, the distortion is corrected by the processing from step S6 to step S11 in FIG. As shown in FIG. 2, when the first image P1 and the second image P2 are acquired (steps S3 and S5), the feature point extraction unit 34 extracts a plurality of feature points from the first image P1 as described above. Extract (step S6). Next, the CPU 19 determines whether or not the number of feature points extracted by the feature point extraction unit 34 is less than N1 (200 in the present embodiment) (step S7).

  When the feature point extraction unit 34 has extracted less than N1 (200 in this embodiment) feature points (step S7; YES), it is unlikely that the number of corresponding points necessary for image deformation is detected. The process is terminated without deforming the figure. The image P1 and / or the image P2 acquired at this time may be recorded on the recording medium 30.

  On the other hand, when the feature point extraction unit 34 has extracted N1 (200 in the present embodiment) or more feature points (step S7; NO), the corresponding point detection unit 35 starts from the second image P2 as described above. Corresponding points corresponding to each of the plurality of feature points extracted by the feature point extraction unit 34 are detected (step S8). Next, the CPU 19 determines whether or not the number of corresponding points detected by the corresponding point detector 35 is less than N2 (100 in this embodiment) (step S9).

  When the corresponding point detection unit 35 detects less than N2 (100 in the present embodiment) corresponding points (step S9; YES), the number of corresponding points necessary for the deformation of the image has not been detected. The process is terminated without performing. The image P1 and / or the image P2 acquired at this time may be recorded on the recording medium 30.

  On the other hand, when the corresponding point detection unit 35 detects the corresponding point N2 (100 in the present embodiment) or more (step S9; NO), the image transformation unit 36 calculates the coordinate value of the intermediate point (step S10). Here, FIG. 5A shows a feature point extraction result, FIG. 5B shows a corresponding point detection result, FIG. 6 shows a feature point and an intermediate point, and FIG. The figure explaining a deformation | transformation method is each shown. 5 (a), 5 (b), and 6 are described below assuming that the number of feature points, corresponding points, and intermediate points is five for the sake of convenience, it is assumed that there are actually more.

上記表１に示すように、特徴点ｆ１〜ｆ５および対応点ｍ１〜ｍ５の座標値に基づいて中間点ｈ１〜ｈ５を算出すると（ステップＳ１０）、画像変形部３６は、第１の画像を変形する（ステップＳ１１）。
上述したように第１の画像Ｐ１の動体Ｄ１と第２の画像Ｐ２の動体Ｄ２とは、歪み形状が対称性を有するので、図６に示す如く、特徴点ｆと対応点ｍの中間点ｈの座標値すなわち対称軸上に、特徴点ｆの座標値が重なるように、第１の画像Ｐ１を変形することにより動体Ｄ１の歪みを補正することができる。 For example, as shown in FIG. 5A, the feature point extraction unit 34 extracts five feature points f1 to f5 in the first image P1, and as shown in FIG. 5B, the corresponding point detection unit 35. Are detected corresponding points m1 to m5 corresponding to the feature points f1 to f5, respectively, in the second image P2. Table 1 below shows the respective intermediate points h1 to h5 calculated based on the coordinate values of the feature points f1 to f5 and the corresponding points m1 to m5 and the coordinate values of the feature points f1 to f5 and the corresponding points m1 to m5. .

As shown in Table 1 above, when the intermediate points h1 to h5 are calculated based on the coordinate values of the feature points f1 to f5 and the corresponding points m1 to m5 (step S10), the image deforming unit 36 deforms the first image. (Step S11).
As described above, since the moving object D1 of the first image P1 and the moving object D2 of the second image P2 have symmetrical distortion shapes, as shown in FIG. 6, the intermediate point h between the feature point f and the corresponding point m. The distortion of the moving object D1 can be corrected by deforming the first image P1 so that the coordinate value of the feature point f overlaps with the coordinate value of i.e., the symmetry axis.

  For the deformation of the first image P1, for example, warping in which the image is locally distorted, that is, warped and deformed, can be used. In general, warping divides an image into a plurality of triangles and performs deformation for each triangle. This will be specifically described below. Note that the extracted feature point f, the detected corresponding point m, and the calculated intermediate point h are each assumed to be ten.

上記式（１）において（ｘ，ｙ）は第１の画像Ｐ１における所定の三角形の中の所定の点の座標値で、（ｘ’，ｙ’）は（ｘ，ｙ）に対応する点の座標値を表す。またａ，ｂ，ｃ，ｄ，ｓ，ｔは未知のパラメータであり、三角形の３つの頂点の座標値から一意に定めることができる。つまり、三角形の３つの頂点をなす特徴点ｆと中間点ｈの組からアフィン変換のパラメータを算出し、算出したパラメータに基づいて三角形内部の点つまり各画素をアフィン変換する。そして全ての三角形をアフィン変換してワーピングを終了する。このようにワーピングを行うことにより第１の画像Ｐ１を変形して動体Ｄ１の歪みを補正することができる。
そして図２に示す如く、画像変形部３６が第１の画像Ｐ１を変形すると（ステップＳ１１）、ＣＰＵ１９は変形された画像Ｐ１’を、外部メモリ制御部３１を介して記録媒体３０に記録する。以上のようにしてデジタルカメラ１による撮影処理を行う。 As shown in the left diagram of FIG. 7, the first image P1 is divided into a plurality of triangles by connecting the four corners of the first image P1 and the extracted feature points f1 to f10 in the first image P1. Then, the image P1 ′ is generated by affine transformation of each triangle of the first image P1 so that the feature points f1 to f10 of the first image P1 overlap the calculated intermediate points h1 to h10. The formula of affine transformation used at this time is shown in the following formula (1).

In the above equation (1), (x, y) is the coordinate value of a predetermined point in the predetermined triangle in the first image P1, and (x ′, y ′) is the point corresponding to (x, y). Represents a coordinate value. Further, a, b, c, d, s, and t are unknown parameters, and can be uniquely determined from the coordinate values of the three vertices of the triangle. That is, an affine transformation parameter is calculated from a set of the feature point f and the intermediate point h forming the three vertices of the triangle, and the point inside the triangle, that is, each pixel, is affine transformed based on the calculated parameter. Then, all triangles are affine transformed and the warping is finished. By warping in this way, the first image P1 can be deformed to correct the distortion of the moving object D1.
As shown in FIG. 2, when the image deforming unit 36 deforms the first image P1 (step S11), the CPU 19 records the deformed image P1 ′ on the recording medium 30 via the external memory control unit 31. The photographing process by the digital camera 1 is performed as described above.

  As described above, according to the digital camera 1 of the present embodiment, the substantially identical scene is imaged twice, and the readout scanning direction of the signal of the CMOS sensor 15 is reversed between the first imaging and the second imaging. By doing so, the first image P1 acquired by the first imaging and the second image P2 acquired by the second imaging have a symmetrical difference in the shape of the distortion of the moving object D. By calculating the coordinate value of the intermediate point h located between the plurality of feature points f extracted from the first image P1 and the corresponding point m corresponding to each of the plurality of feature points f, the coordinates of the feature point f are calculated. By deforming the first image P1 so that the value coincides with the coordinate value of the intermediate point h, the distortion of the figure of the moving object D that occurs when the moving object D is imaged by the CMOS sensor 15 can be easily performed without changing the resolution. Can be corrected.

  In this embodiment, the number N1 of feature points is 200 and the number N2 of corresponding points is 100. However, the present invention is not limited to this, and N1 may be N2 or more. N1 is preferably at least twice as large as N2, and N2 is preferably at least 20 or more, more preferably 100 or more.

  In the present embodiment, the first imaging is performed before the second imaging. However, the present invention is not limited to this, and the second imaging may be performed before the first imaging. Good. In the present embodiment, the first imaging and the second imaging have the same imaging conditions, but the imaging conditions may be changed as long as corresponding points can be detected. In the present embodiment, feature points are extracted from the first image P1 and corresponding points are detected from the second image P2. However, the present invention is not limited to this. Feature points may be extracted from the second image P2 and corresponding points may be detected from the first image P1. In the present embodiment, the image P1 is deformed so that the coordinate value of the feature point of the first image P1 matches the coordinate value of the intermediate point, but the present invention is not limited to this, and the second image The image P2 may be deformed so that the coordinate value of the corresponding point of P2 matches the coordinate value of the intermediate point.

  Next, a digital camera 1-2 according to the second embodiment of the present invention will be described in detail with reference to the drawings. FIG. 8 is a block diagram showing a functional configuration of the digital camera 1-2, FIG. 9 is a flowchart of an imaging process of the digital camera 1-2 in FIG. 8, and FIG. 10 is a diagram showing a motion vector. In FIG. 8, for the sake of convenience, the same parts as those in the digital camera 1 in FIG. For the sake of convenience, FIG. 9 shows the same processes as those in the flowchart of FIG.

  As shown in FIG. 8, the digital camera 1-2 according to the present embodiment further includes a motion vector calculation unit (motion vector calculation unit) 39 and an image deformation prohibition unit (image deformation prohibition unit) 40 in addition to the configuration of the digital camera 1 of FIG. It has. The motion vector calculation unit 39 calculates a motion vector j from the feature point f and the corresponding point m corresponding to the feature point f, and the image deformation prohibition unit 40 prohibits the image deformation unit 36 from deforming the image. To do.

  As illustrated in FIG. 9, the digital camera 1-2 according to the present embodiment has N2 or more corresponding points m detected by the corresponding point detection unit 35 in step S9 (step S9; NO). The motion vector calculation unit 39 calculates the motion vector j (step S20). As illustrated in FIG. 10, the motion vector j includes a plurality of feature points f extracted by the feature point extraction unit 34 and corresponding points m corresponding to the plurality of feature points f detected by the corresponding point detection unit 35. Calculate from the coordinate values.

  Next, the CPU 19 determines whether or not the largest motion vector j (motion vector j1 in the present embodiment) among the calculated motion vectors j1 to j5 is equal to or greater than a predetermined threshold (step S21). In the present embodiment, the predetermined threshold is 3 pixels, but can be changed by the user operating the operation unit 22.

  When the motion vector j1 is less than 3 pixels (step S21; NO), the image deformation prohibiting unit 40 determines that the distortion of the moving object D is small and prohibits the image deforming unit 36 from deforming the first image P1. (Step S22), and the process ends. The image P1 and / or the image P2 acquired at this time may be recorded on the recording medium 30.

  On the other hand, when the motion vector j1 is 3 pixels or more (step S21; YES), the image deforming unit 36 determines that the distortion of the moving object D is large and performs the processing from step S10 onward. The distortion of the moving body D1 is corrected.

  The distortion of the moving body D due to the CMOS sensor becomes more conspicuous as the speed of the moving body D increases. Conversely, when the speed of the moving body D is lower, the distortion of the moving body D becomes smaller. When the movement is small, useless processing can be eliminated by prohibiting the deformation of the image.

  In the present embodiment, the largest motion vector j1 among the calculated motion vectors j1 to j5 is the object of discrimination, but the present invention is not limited to this, and for example, the average of the magnitudes of the motion vectors j1 to j5 A value, an intermediate value, or the like may be a target for determination, or may be changed as appropriate.

  Next, a digital camera 1-3 according to a third embodiment of the present invention will be described in detail with reference to the drawings. FIG. 11 is a block diagram illustrating a functional configuration of the digital camera 1-3, and FIG. 12 is a flowchart of an imaging process of the digital camera 1-2 in FIG. For the sake of convenience, FIG. 11 shows the same parts as those of the digital camera 1 of FIG. For the sake of convenience, FIG. 12 shows the same processes as those in the flowchart of FIG.

  As shown in FIG. 11, the digital camera 1-3 according to the present embodiment further includes an image composition unit (image composition unit) 41 in addition to the configuration of the digital camera of FIG. 1. In addition, the image deforming unit 36-3 according to the present embodiment performs image deformation on both the first image P1 and the second image P2.

  In the digital camera 1-3 according to the present embodiment, as shown in FIG. 12, the exposure conditions among the imaging conditions are different between the first imaging in step S3 'and the second imaging in step S5'.

  In the second imaging (step S5 '), imaging is performed by underexposure with a number of steps corresponding to the dynamic range rather than the exposure conditions of the first imaging.

  In step S11, when the image deforming unit 36-3 deforms the first image P1 (step S11), the image deforming unit 36-3 then deforms the second image P2 (step S30). The deformation of the second image P2 can be performed by warping in the same manner as the deformation of the first image P1, and is deformed by making each corresponding point m coincide with the intermediate point h. Note that the warping procedure is the same as that of the modification of the first image P1 described above, and thus the description thereof is omitted here.

  After the first image P1 and the second image P2 are deformed in this manner (steps S11 and S12), the image composition unit 41 synthesizes the first image P1 and the second image P2 (step S31). ). Note that the composition method described in Japanese Patent Application Laid-Open No. 2001-8104 can be used as the composition method of the first image P1 and the second image P2 having different brightness by the image composition unit 41. The detailed description of is omitted. Then, the CPU 19 records the synthesized image on the recording medium 30 via the external memory control unit 31. The imaging process by the digital camera 1-3 is performed as described above.

  By performing the synthesizing process in this way, it is possible to correct the distortion of the moving object D and expand the dynamic range.

  In the present embodiment, two images having different exposure conditions are combined in order to expand the dynamic range. However, the present invention is not limited to this. For example, two images having different sensitivities are combined. It may be.

  Note that the imaging apparatus of the present invention is not limited to the digital camera 1 of the above-described embodiment, and can be appropriately changed without departing from the spirit of the present invention.

1 is a block diagram showing a functional configuration of a digital camera according to a first embodiment. Flowchart of the imaging process of the digital camera of FIG. (A) An image P in which the moving body D in the subject is not distorted, (b) an image P1 acquired by the first scanning method, (c) an image P2 acquired by the second scanning method. Diagram explaining symmetry of distortion (A) The figure which shows the extraction result of a feature point, (b) The figure which shows the detection result of a corresponding point Diagram showing feature points and midpoints The figure explaining the deformation method of an image The block diagram which shows the function structure of the digital camera of 2nd Embodiment. Flowchart of the imaging process of the digital camera of FIG. Illustration showing motion vector The block diagram which shows the function structure of the digital camera of 3rd Embodiment. The flowchart of the imaging process of the digital camera of FIG. Diagram showing moving object in subject and image obtained by imaging

Explanation of symbols

1 Digital camera (imaging device)
10 imaging means 15 CMOS sensor (imaging sensor)
19 CPU (imaging control means)
34 feature point extraction unit (feature point extraction means)
35 Corresponding point detector (corresponding point detector)
36 Image deformation part (image deformation means)
39 Motion vector calculation unit (motion vector calculation means)
40 Image deformation prohibition section (image deformation prohibition means)
41 Image composition section (image composition means)
P1 First image P2 Second image D, D1, D2 Moving object

Claims (9)

Imaging means for imaging a subject using a rolling shutter imaging sensor;
First imaging and second imaging are continuously performed by the imaging unit, and a signal of the imaging sensor is read by a first scanning method in which scanning for each line is performed from one direction at the time of the first imaging, An imaging control means for controlling the imaging means so as to read a signal of the imaging sensor by a second scanning method in which scanning for each line is performed in a direction opposite to the one direction at the time of the second imaging;
Feature point extracting means for extracting a plurality of feature points from the first image acquired by the first imaging;
Corresponding point detecting means for detecting corresponding points corresponding to each of the plurality of feature points extracted by the feature point extracting means from the second image acquired by the second imaging;
A coordinate value of an intermediate point located between the plurality of feature points and the corresponding point corresponding to each of the plurality of feature points is calculated, and the coordinate value of the feature point matches the coordinate value of the intermediate point And / or image deformation means for deforming the second image so that the coordinate value of the corresponding point matches the coordinate value of the intermediate point. An imaging apparatus characterized by the above. Motion vector calculating means for calculating a motion vector from the plurality of feature points and the corresponding points corresponding to each of the plurality of feature points;
Image deformation prohibiting means for preventing the image deforming means from deforming the first image or the second image when the motion vector calculated by the motion vector calculating means is smaller than a predetermined threshold. The imaging apparatus according to claim 1, further comprising: The image deformation means deforms the first image and the second image;
The imaging apparatus according to claim 1, further comprising an image synthesis unit that synthesizes the deformed first image and the second image to generate one image. In an imaging method using an imaging unit that images a subject using a rolling shutter type imaging sensor,
Causing the imaging means to perform a first imaging;
The signal of the imaging sensor is read out by a first scanning method in which scanning for each line is performed from one direction during the first imaging,
Causing the imaging means to perform second imaging;
The signal of the imaging sensor is read by a second scanning method in which scanning for each line is performed from the direction opposite to the one direction during the second imaging,
Extracting a plurality of feature points from the first image acquired by the first imaging;
Detecting corresponding points corresponding to each of the extracted plurality of feature points from the second image acquired by the second imaging;
Calculating a coordinate value of an intermediate point located between the plurality of feature points and a corresponding point corresponding to each of the plurality of feature points;
The first image is deformed so that the coordinate values of the plurality of feature points match the coordinate value of the intermediate point, and / or the coordinate value of the corresponding point matches the coordinate value of the intermediate point. An imaging method characterized by deforming the second image. Calculating a motion vector from the plurality of feature points and the corresponding points corresponding to each of the plurality of feature points;
The imaging method according to claim 4, wherein the first image or the second image is not deformed when the calculated motion vector is smaller than a predetermined threshold. Transforming the first image and the second image;
6. The imaging method according to claim 4, wherein the deformed first image and second image are combined to generate one image. In a program for causing a computer to execute an imaging method using imaging means for imaging a subject using a rolling shutter type imaging sensor,
Causing the imaging means to perform a first imaging;
The signal of the imaging sensor is read out by a first scanning method in which scanning for each line is performed from one direction during the first imaging,
Causing the imaging means to perform second imaging;
The signal of the imaging sensor is read by a second scanning method in which scanning for each line is performed from the direction opposite to the one direction during the second imaging,
Extracting a plurality of feature points from the first image acquired by the first imaging;
Detecting corresponding points corresponding to each of the extracted plurality of feature points from the second image acquired by the second imaging;
Calculating a coordinate value of an intermediate point located between the plurality of feature points and a corresponding point corresponding to each of the plurality of feature points;
The first image is deformed so that the coordinate values of the plurality of feature points match the coordinate value of the intermediate point, and / or the coordinate value of the corresponding point matches the coordinate value of the intermediate point. A program for causing the second image to be deformed. A computer calculates a motion vector from the plurality of feature points and the corresponding points corresponding to each of the plurality of feature points,
8. The method according to claim 7, wherein when the calculated motion vector is smaller than a predetermined threshold, the first image or the second image is not deformed. Program. A computer that transforms the first image and the second image;
The program according to claim 7 or 8, characterized in that said deformed first image and second image are combined to generate one image.

Images