JP2011087203A - Imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To stably remove focal plane distortion. <P>SOLUTION: An imaging apparatus performs photographing using a solid-state imaging device (CMOS image sensor) having exposure timing different between adjacent horizontal lines. When photographing and recording a moving image at 30 fps, acquisition of high-speed photographing images (I<SB>S</SB>[1], I<SB>S</SB>[2], ...) under a high-speed photographing condition equivalent to 330 fps and acquisition of ordinary photographing images (I<SB>L</SB>[1], I<SB>L</SB>[2], ...) under an ordinary imaging condition equivalent to 33 fps are alternately carried out. Considering that a high-speed photographing image having a small exposure timing difference between adjacent horizontal lines does not almost contain focal plane distortion, focal plane distortion contained in an ordinary photographing image I<SB>L</SB>[i] is removed by image processing based on an optical flow between a high-speed photographing image I<SB>S</SB>[i] and the ordinary photographing image I<SB>L</SB>[i]. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an imaging apparatus such as a digital video camera or a digital still camera, and more particularly to a technique for removing focal plane distortion.

  Electronic shutters used in solid-state imaging devices are roughly classified into a global shutter in which all light receiving pixels on the imaging surface are exposed at a common timing, and a rolling shutter in which exposure timing is different between different horizontal lines. The rolling shutter is used in an XY address type solid-state imaging device, for example, a CMOS image sensor, which can read out an output signal of a light receiving pixel in pixel units.

  In an imaging apparatus employing a rolling shutter, it is known that, in principle, focal plane distortion occurs in a captured image when a positional change occurs in a solid-state imaging device during imaging due to the influence of a camera shake of a photographer. However, the focal plane distortion can be reduced by performing appropriate image processing on the captured image.

  For example, Patent Document 1 below proposes a method for obtaining the distortion amount of each frame from a motion vector between frames. In this method, the amount of distortion (focal plane distortion) of the second frame is detected from the motion vector between the first and second frames that are temporally adjacent, and the focal plane distortion is removed using the detected distortion amount. Yes.

JP 2007-208580 A

  However, this method works effectively only on the assumption that the first frame does not include focal plane distortion. Actually, it is assumed that the first frame includes the same focal plane distortion as the second frame, or the amount of the focal plane distortion included in the first frame is unknown. Therefore, with the method of Patent Document 1, it is difficult to stably remove the focal plane distortion.

  Accordingly, an object of the present invention is to provide an imaging apparatus that contributes to stable reduction of focal plane distortion.

  An image pickup apparatus according to the present invention includes an image pickup device that takes an image while changing exposure timing between first and second horizontal lines included in a plurality of horizontal lines, and exposure timing between the first and second horizontal lines. A first image is obtained by reading out an image signal from the image sensor with an exposure with a relatively short difference, while a second image is obtained by reading out an image signal with an exposure with a relatively long exposure timing difference from the image sensor. An image acquisition unit for performing an operation, a motion detection unit for deriving an optical flow between the first and second images based on the image signals of the first and second images, and correcting the second image based on the optical flow And a correction unit.

  Since the first image is shot in a state where the exposure timing difference is shorter than that of the second image, the focal plane distortion included in the first image is relatively small. Accordingly, by deriving the optical flow between the first and second images, it is possible to accurately estimate the focal plane distortion included in the second image, and if the second image is corrected using the optical flow, It is possible to appropriately reduce the focal plane distortion of the two images.

  The first and second horizontal lines may be two horizontal lines adjacent to each other on the image sensor, or two horizontal lines that are not adjacent to each other on the image sensor. Also good.

  That is, for example, the correction unit can correct distortion generated in the second image due to different exposure timings between the first and second horizontal lines based on the optical flow.

  Specifically, for example, the motion detection unit derives a plurality of motion vectors forming the optical flow based on the image signals of the first and second images, and the correction unit is generated in the second image. The distortion is corrected based on each motion vector.

  More specifically, for example, the correction unit corrects the distortion by applying a geometric transformation based on each motion vector to the second image.

  Further, for example, the image acquisition unit repeatedly executes the alternate shooting of the first and second images, and the imaging device records each correction image based on each second image obtained by the correction of the correction unit in a predetermined recording manner. Recording on the recording medium at the frame rate, the sum of the time required to obtain one first image and the time required to obtain one second image is equal to or less than the reciprocal of the recording frame rate. Is set.

  Thereby, the focal plane distortion of the recorded moving image can be stably reduced while following a predetermined recording frame rate.

  According to the present invention, it is possible to provide an imaging apparatus that contributes to stable reduction of focal plane distortion.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

1 is an overall block diagram of an imaging apparatus according to an embodiment of the present invention. It is an internal block diagram of the imaging part of FIG. It is a figure which shows the horizontal line set on the image pick-up element. It is the figure which showed a mode that exposure timing differed between the horizontal lines on an image pick-up element. It is a block diagram of the site | part which concerns on embodiment of this invention and contributes to removal of focal plane distortion. It is a figure which shows the image sequence imaged at the time of moving image imaging | photography, and the imaging timing of this image sequence. It is a figure which shows a mode that a high-speed picked-up image and a normal picked-up image are obtained from the light receiving pixel in the common area of an imaging surface. It is a figure which shows a mode that each of the high-speed captured image after size normalization and a normal captured image is divided | segmented into several detection blocks. It is a figure which shows the optical flow between the high-speed captured image after size normalization, and a normal captured image. FIG. 6 is a conceptual diagram of generation of a corrected image from which focal plane distortion has been removed. 5 is an operation flowchart for capturing and recording a moving image according to the first embodiment of the present invention. FIG. 10 is a diagram illustrating a state in which a high-speed photographed image and a normal photographed image are continuously photographed in accordance with a second embodiment of the present invention as a shutter button is pressed. FIG. 10 is a diagram illustrating a state in which one high-speed photographed image and three normal photographed images are continuously photographed according to a continuous shooting operation according to the second embodiment of the present invention. FIG. 10 is a diagram illustrating a state in which continuous shooting of one high-speed photographed image and one normal photographed image is repeated three times according to a continuous shooting operation according to the second embodiment of the present invention. FIG. 10 is a diagram illustrating a manner in which an image is divided along horizontal and vertical directions according to the fourth embodiment of the present invention. It is a figure which concerns on 5th Example of this invention and shows a high-speed picked-up image sequence, a normal picked-up image sequence, and those image pick-up timings. FIG. 16 is a diagram illustrating a high-speed captured image after size normalization and a normal captured image according to the fifth embodiment of the present invention. FIG. 18 is a diagram in which a set of a plurality of feature points and a motion vector of each feature point are superimposed on FIG. FIG. 10 is a diagram for describing a method of generating a corrected image from a normal captured image according to the fifth embodiment of the present invention. It is a figure which shows the correction | amendment image which concerns on 5th Example of this invention. It is a figure which concerns on 6th Example of this invention and shows a high-speed picked-up image sequence and a normal picked-up image sequence, and those image pick-up timings. FIG. 20 is a diagram illustrating a high-speed captured image after size normalization and a normal captured image according to the sixth embodiment of the present invention. It is a figure which shows the correction | amendment image which concerns on 6th Example of this invention.

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle.

  FIG. 1 is an overall block diagram of an imaging apparatus 1 according to an embodiment of the present invention. The imaging device 1 has each part referred by the codes | symbols 11-28. The imaging device 1 is a digital video camera, and can capture a moving image and a still image, and also can simultaneously capture a still image during moving image capturing. Each part in the imaging apparatus 1 exchanges signals (data) between the parts via the bus 24 or 25. The display unit 27 and / or the speaker 28 may be interpreted as being provided in an external device (not shown) of the imaging device 1.

  The imaging unit 11 captures a subject using an imaging element. FIG. 2 is an internal configuration diagram of the imaging unit 11. The imaging unit 11 includes an optical system 35, a diaphragm 32, an imaging element 33 that is a solid-state imaging element, and a driver 34 that drives and controls the optical system 35 and the diaphragm 32. The optical system 35 is formed of a plurality of lenses including a zoom lens 30 for adjusting the angle of view of the imaging unit 11 and a focus lens 31 for focusing. The zoom lens 30 and the focus lens 31 are movable in the optical axis direction. Based on the control signal from the CPU 23, the position of the zoom lens 30 and the focus lens 31 in the optical system 35 and the opening of the diaphragm 32 are controlled, so that the focal length (field angle) and the focal position of the imaging unit 11 and the imaging are controlled. The amount of light incident on the element 33 is controlled.

  The image sensor 33 is formed by arranging a plurality of light receiving pixels in the horizontal and vertical directions. Each light receiving pixel of the image sensor 33 photoelectrically converts an optical image of a subject incident through the optical system 35 and the diaphragm 32, and outputs an electric signal obtained by the photoelectric conversion to an AFE 12 (Analog Front End).

  The AFE 12 amplifies the analog signal output from the image sensor 33 (each light receiving pixel), converts the amplified analog signal into a digital signal, and outputs the digital signal to the video signal processing unit 13. The amplification degree of signal amplification in the AFE 12 is controlled by a CPU (Central Processing Unit) 23. The video signal processing unit 13 performs necessary image processing on the image represented by the output signal of the AFE 12, and generates a video signal for the image after the image processing. The microphone 14 converts the ambient sound of the imaging device 1 into an analog audio signal, and the audio signal processing unit 15 converts the analog audio signal into a digital audio signal.

  The compression processing unit 16 compresses the video signal from the video signal processing unit 13 and the audio signal from the audio signal processing unit 15 using a predetermined compression method. The internal memory 17 is composed of a DRAM (Dynamic Random Access Memory) or the like, and temporarily stores various data. The external memory 18 as a recording medium is a non-volatile memory such as a semiconductor memory or a magnetic disk, and records a video signal and an audio signal compressed by the compression processing unit 16.

  The decompression processing unit 19 decompresses the compressed video signal and audio signal read from the external memory 18. The video signal expanded by the expansion processing unit 19 or the video signal from the video signal processing unit 13 is sent to the display unit 27 such as a liquid crystal display via the display processing unit 20 and displayed as an image. Further, the audio signal that has been expanded by the expansion processing unit 19 is sent to the speaker 28 via the audio output circuit 21 and output as sound.

  The TG (timing generator) 22 generates a timing control signal for controlling the timing of each operation in the entire imaging apparatus 1, and gives the generated timing control signal to each unit in the imaging apparatus 1. The timing control signal includes a vertical synchronization signal Vsync and a horizontal synchronization signal Hsync. The TG 22 further generates a drive pulse for the image sensor 33 under the control of the CPU 23 and supplies the drive pulse to the image sensor 33. The CPU 23 comprehensively controls the operation of each part in the imaging apparatus 1. The operation unit 26 includes a recording button 26a for instructing start / end of moving image shooting and recording, a shutter button 26b for instructing shooting and recording of a still image, and a zoom button for designating a zoom magnification. 26c and the like, and accepts various operations by the user. The operation content for the operation unit 26 is transmitted to the CPU 23.

  The image sensor 33 has an electronic shutter function, and performs exposure of each light receiving pixel by a so-called rolling shutter. The imaging element 33 is provided with an imaging surface formed by a plurality of light receiving pixels, and the light receiving pixels are two-dimensionally arrayed along a plurality of horizontal lines and a plurality of vertical lines defined on the imaging surface. . In the rolling shutter, the timing at which each light receiving pixel on the imaging surface is exposed differs for each horizontal line. That is, the exposure timing of the light receiving pixels is different between different horizontal lines on the imaging surface.

  A supplementary description of the rolling shutter of the image sensor 33 will be given with reference to FIGS. 3 and 4. As shown in FIG. 3, on the imaging surface, the direction along the vertical line (that is, the vertical direction of the imaging surface) is the vertical direction, and the horizontal line arranged at the uppermost end of the effective pixel area on the imaging surface is the first horizontal line. And the horizontal line arranged at the lowermost end of the effective pixel area of the imaging surface is the Mth horizontal line (M is an integer equal to or greater than 2, and is generally several hundred to several thousand) ). Then, as shown in FIG. 4, for a certain target frame, the exposure start time at the light receiving pixels on the (i + 1) th horizontal line is Δt seconds from the exposure start time at the light receiving pixels on the i th horizontal line. Slow (Δt> 0 and i is an integer). In one frame, the exposure time of the light receiving pixels on the i-th horizontal line is the same as that of the light receiving pixels on the (i + 1) -th horizontal line, and Δt is constant regardless of the value of i. It is. After the exposure of each light receiving pixel is completed, the output signal of each light receiving pixel can be read, and the read output signal is sent to the AFE 12.

  An integer multiple of Δt and Δt is called an exposure timing difference between different horizontal lines. Δt is an exposure timing difference between adjacent horizontal lines.

  Of the first to Mth horizontal lines, there may be a plurality of horizontal lines having the same exposure timing. For example, the exposure timing may be different for every two horizontal lines. In this case, although the exposure timing of the second horizontal line and the exposure timing of the third horizontal line are different, the exposure timing of the first and second horizontal lines is the same and the exposure timing of the third and fourth horizontal lines is the same. It becomes. Further, although the exposure timing of each light receiving pixel is sequentially shifted in one horizontal line, the difference in exposure timing between different light receiving pixels on one horizontal line is sufficiently smaller than that between different horizontal lines, so The exposure timing of each light receiving pixel on the line can be regarded as the same.

  The image sensor 33 is, for example, an XY address type CMOS (Complementary Metal Oxide Semiconductor) image sensor. A CMOS image sensor is formed by forming an imaging surface composed of a plurality of light receiving pixels, a vertical scanning circuit, a horizontal scanning circuit, a pixel signal output circuit, and the like on a semiconductor substrate capable of forming a CMOS. However, if the image pickup device 33 performs exposure of each light receiving pixel by a rolling shutter, the image pickup device 33 may not be an XY address scanning type CMOS image sensor.

  As described in the description of the background art, focal plane distortion (hereinafter referred to as FP distortion) occurs in an image captured by a rolling shutter. The FP distortion occurs due to Δt being larger than 0, and the FP distortion tends to increase as Δt increases. The imaging device 1 has a function of appropriately removing this FP distortion.

  FIG. 5 shows a block diagram of main parts responsible for this function. The image acquisition unit 51 is formed by the video signal processing unit 13, the TG 22, and the CPU 23 of FIG. The distortion detection unit 52 is formed by the CPU 23 or by the video signal processing unit 13 and the CPU 23. The distortion correction unit 53 is provided in the video signal processing unit 13. The distortion detector 52 includes a motion detector 54. As will be described later, since the FP distortion amount that is the detection target of the distortion detection unit 52 is directly derived from the motion vector that is the detection target of the motion detection unit 54, the distortion detection unit 52 is the motion detection unit 54 itself. It is also possible to think.

  The shooting frame rate (hereinafter referred to as shooting frame rate) using the image sensor 33 is variable, and the image acquisition unit 51 controls the shooting frame rate and exposes the light receiving pixels of the image sensor 33. And the timing for reading out the output signal of each light receiving pixel. Under these controls, the output signal of each light receiving pixel of the image sensor 33 is given to the image acquisition unit 51 as an image signal. However, it is assumed that an image signal obtained by performing necessary signal processing on the output signal of each light receiving pixel is given to the image acquisition unit 51. The signal processing here includes signal amplification in the AFE 12. The image acquisition unit 51 can convert the signal format of the image signal from the RAW data format to the YUV format by using so-called demosaicing processing. The YUV image signal is formed from a luminance signal and a color difference signal. However, this conversion may be performed at the front part or the rear part of the image acquisition unit 51. In this specification, an image signal and a video signal are synonymous.

The image acquisition unit 51 causes the image sensor 33 to perform shooting under a plurality of shooting conditions. The plurality of shooting conditions include high-speed shooting conditions and normal shooting conditions. Under high-speed shooting conditions, the exposure timing difference Δt is set to a relatively short Δt _S , and under normal shooting conditions, the exposure timing difference Δt is set to a relatively long Δt _L. Further, in the high-speed imaging conditions, a relatively short exposure time T _S is the exposure time of the light receiving pixels, the normal imaging conditions, the exposure time of the light receiving pixel is relatively long exposure time T _L. Therefore, the inequalities “0 <Δt _S <Δt _L ” and “0 <T _S <T _L ” hold. The exposure time of the light receiving pixels refers to the time for the light receiving pixels to accumulate signal charges. Although different from the above assumption, the exposure time T _S and the exposure time T _L may be the same.

  An image photographed by the image sensor 33 under the high-speed photographing condition is called a high-speed photographed image, and an image photographed by the image sensor 33 under the normal photographing condition is called a normal photographed image. The image acquisition unit 51 controls the shooting frame rate and the like so that the image signal of the high-speed captured image and the image signal of the normal captured image are sequentially input to the image acquisition unit 51.

  FIG. 6 shows an image sequence captured at the time of moving image shooting and the shooting timing of the image sequence. In this example, it is assumed that the frame rate of moving images recorded in the external memory 18 (hereinafter referred to as recording frame rate) is set to 30 fps (frame per second).

When shooting and recording of a moving image is started by pressing the recording button 26a in FIG. 1, the image acquisition unit 51 controls the image sensor 33 so that high-speed shot images and normal shot images are shot alternately. . Of the high-speed shot images taken one after another, the i-th high-speed shot image is represented by I _S [i], and the normal shot taken i-th among the normal shot images taken one after another The image is represented by I _L [i]. Then, photographing is performed in the order of I _S [1], I _L [1], I _S [2], I _L [2],. In this specification, for simplification of description, the name of an image or the like corresponding to the symbol may be abbreviated by describing the symbol. For example, the high-speed captured image I _S [i] may be simply abbreviated as the image I _S [i], and the normal captured image I _L [i] may be simply abbreviated as the image I _L [i].

  The driving condition of the image sensor 33 when performing moving image shooting at 300 fps is referred to as a 300 fps driving condition, and the driving condition of the image sensor 33 when performing moving image shooting at 33 fps is referred to as a 33 fps driving condition.

Since a normal photographed image is photographed next to the high-speed photographed image, the photographing frame rate of the moving image composed of the high-speed photographed image sequence I _S [1], I _S [2], I _S [3]. Is not 300 fps, but one high-speed photographed image is photographed under a 300 fps drive condition. Therefore, the time required to acquire one high-speed captured image is equivalent to 300 fps, that is, about 3.3 ms (milliseconds).

Since a high-speed photographed image is photographed next to the normal photographed image, the photographing frame rate of the moving image composed of the normal photographed image sequence I _L [1], I _L [2], I _L [3]. Is not 33 fps, but one normal shot image is shot under a 33 fps drive condition. Therefore, the time required to acquire one normal photographed image is that corresponding to 33 fps, that is, about 30 ms.

  As described above, the total time of the time required to acquire one high-speed captured image and the time required to acquire one normal captured image is set to be equal to or less than the reciprocal of the recording frame rate (about 33.3 ms). The In this example, the total time is substantially the same as the reciprocal of the recording frame rate, but the total time can be made much smaller than the reciprocal of the recording frame rate.

  FIG. 6 assumes that the position of the image sensor 33 is changed in the horizontal direction during shooting of a moving image with an upright person as a subject. As a result, each normal captured image having a large exposure timing difference Δt is large. FP distortion has occurred. On the other hand, since the exposure timing difference Δt with respect to the high-speed captured image is sufficiently small, the FP distortion included in each high-speed captured image is small enough to be ignored.

The temporally adjacent images I _S [i] and I _L [i] are obtained by photographing a common subject. The angle of view of the high-speed photographed image I _S [i] and the angle of view of the normal photographed image I _L [i] are equal, and as shown in FIG. 7, the output of the light receiving pixels in the common area 210 belonging to the effective pixel area of the imaging surface. Images I _S [i] and I _L [i] are generated from the signal. Accordingly, the upper left corner, upper right corner, lower left corner, and lower right corner of the high-speed photographed image I _S [i] are respectively obtained from the output signals of the light receiving pixels arranged in the upper left corner, upper right corner, lower left corner, and lower right corner of the common area 210. Are formed, and image signals of pixels arranged in the upper left corner, upper right corner, lower left corner, and lower right corner of the normal captured image I _L [i] are formed.

However, while the normal captured image I _L [i] is obtained by individually reading the output signals of all the light receiving pixels in the common area 210, the high-speed captured image has a short readout time, resulting in high speed. When the captured image I _S [i] is generated, some light receiving pixels in the common area 210 are thinned out (that is, so-called thinning readout is performed), or a plurality of light receiving pixels in the common area 210 is generated. The output signal of the light receiving pixel is read after the addition signal of the output signals is regarded as the output signal of one light receiving pixel (that is, so-called addition reading is performed). For this reason, the image size of the high-speed captured image I _S [i] is smaller than the image size of the normal captured image I _L [i]. In this example, as shown in FIG. 7, for an arbitrary integer i, the number of pixels in the horizontal and vertical directions of the high-speed captured image I _S [i] is 448 and 336, respectively, and the normal captured image I _L [i ] In the horizontal and vertical directions are 1280 and 720, respectively. Hereinafter, the fact that the number of pixels in the horizontal and vertical directions of an image is N _X and N _Y , respectively, is expressed as an image size (N _X × N _Y ). Although different from the above assumption, the image size of the high-speed captured image I _S [i] and the image size of the normal captured image I _L [i] may be the same.

Further, based on the exposure time ratio (T _S : T _L ) of both images so that the brightness of the high-speed captured image I _S [i] and the brightness of the normal captured image I _L [i] are comparable. The signal amplification degree of the AFE 12 at the time of generating the image signal of the captured image I _S [i] is made larger than that of the normal captured image I _L [i]. However, by setting the signal amplification degree of the AFE 12 to be the same between the images I _S [i] and I _L [i], the distortion detection unit 52 increases the luminance level of each pixel of the high-speed captured image I _S [i]. The brightness of the high-speed captured image I _S [i] and the normal captured image I _L [i] may be matched.

The image signal of each high-speed captured image and the image signal of each normal captured image provided to the image processing unit 51 are provided to the distortion detection unit 52 and the motion detection unit 54. First, the distortion detection unit 52 or the motion detection unit 54 performs size normalization so that the image size of the high-speed captured image is the same as the image size of the normal captured image. Although size normalization may be realized by reducing the image size of the normal captured image, in this example, image size enlargement processing for increasing the image size of the high-speed captured image is executed as size normalization. That is, in the image size enlargement process as size normalization, the image size of the high-speed captured image given from the image processing unit 51 is set so that the image size of the high-speed captured image after size normalization becomes (1280 × 720). Increase. The high-speed photographed image I _S [i] after size normalization is represented by I _S '[i]. The image size enlargement process is realized by a known geometric transformation using pixel interpolation or the like.

  The distortion detection unit 52 or the motion detection unit 54 divides the high-speed captured image after size normalization into a plurality of detection blocks, and divides a normal captured image into a plurality of detection blocks under the same division conditions. To do. An image 220 in FIG. 8A represents a high-speed captured image after size normalization, and an image 230 in FIG. 8B represents a normal captured image. By setting (n−1) boundary lines parallel to the horizontal direction of the image 220 to the image 220, the image 220 is divided into n detection blocks (n is an integer of 2 or more). Similarly, by setting (n−1) boundary lines parallel to the horizontal direction of the image 230, the image 230 is divided into n detection blocks. The n detection blocks provided in the image 220 are represented by BL [1] to BL [n], and the n detection blocks provided in the image 230 are also represented by BL [1] to BL [n]. To express. Each detection block is formed of a plurality of pixels.

  A common horizontal line belongs to the detection block BL [j] on the image 220 and the detection block BL [j] on the image 230 (j is an integer). Therefore, assuming that the position of the subject and the image sensor 33 is fixed, ideally, the image in the detection block BL [j] on the image 220 and the detection block BL [j on the image 230 are displayed. The images in] are the same. The number of horizontal lines belonging to each detection block may be one or plural. For example, when a detection block is set for every three horizontal lines, each detection block BL [1] on the images 220 and 230 includes the first to third horizontal lines on the images 220 and 230. Each detection block BL [2] on the images 220 and 230 includes the fourth to sixth horizontal lines on the images 220 and 230.

  The motion detection unit 54 derives an optical flow between the images 220 and 230 based on the image signals of the images 220 and 230 using known block matching, representative point matching, a gradient method, or the like. An optical flow is a bundle of motion vectors. The optical flow between the images 220 and 230 is derived so that one motion vector is obtained for each detection block. The motion vector between the images 220 and 230 for the detection block BL [j] is a vector representing the direction and magnitude of the motion between the images 220 and 230 of the subject in the detection block BL [j]. Here, the movement of the subject is considered with reference to the image 220. Therefore, for example, when the position of the subject of interest in the detection block BL [j] of the image 230 is located on the right side of the position of the subject of interest in the detection block BL [j] of the image 220, the detection is performed. The direction of the motion vector of the block BL [j] is rightward.

FIG. 9 shows optical flows obtained for the high-speed captured image I _S ′ [1] and the normal captured image I _L [1]. Such an optical flow is performed for the high-speed captured image I _S '[2] and the normal captured image I _L [2], and for the high-speed captured image I _S ' [3] and the normal captured image I _L [3]. ... are derived one after another.

The distortion detection unit 52 uses the motion vector for the detection block BL [j] obtained for the high-speed captured image I _S ′ [i] and the normal captured image I _L [i] as the normal captured image I _L [ i] is treated as the FP distortion amount in the detection block BL [j]. The FP distortion amount is an amount representing FP distortion existing in a normal captured image. Similar to the motion vector, the FP distortion amount is an amount having a direction and a magnitude.

The distortion correction unit 53 corrects the image signal of each normal captured image from the image acquisition unit 51 based on the FP distortion amount obtained by the distortion detection unit 52, thereby removing the FP distortion of each normal captured image. To do. An image generated by the distortion correction unit 53 through the removal of the FP distortion is referred to as a corrected image, and a corrected image based on the normal captured image I _L [i] is represented by I _C [i]. Here, the removal means complete removal or partial removal of FP distortion, and can be read as reduction.

Distortion correction unit 53 is normally captured image I _L [i] geometry is translated by the magnitude of the FP distortion amount FP distortion amount of orientation in the opposite direction to the direction of the normal captured image I _L [i] the position of each pixel in the Perform dynamic conversion. This geometric transformation is performed for each detection block. Thereby, the FP distortion included in the normal captured image I _L [i] is removed. Then, a corrected image I _C [i] is generated from the image obtained as a result of this geometric transformation (that is, the normal photographed image I _L [i] after this geometric transformation). FIG. 10 is a conceptual diagram of generation of a corrected image including the state of this geometric transformation.

For example, regarding the detection block BL [j] of the normal captured image I _L [i], when the direction of the FP distortion amount is rightward and the magnitude of the FP distortion amount is 5 pixels, the normal captured image I _L The image in the detection block BL [j] of [i] is shifted from the ideal position by 5 pixels to the right due to FP distortion. Therefore, in this case, the distortion correction unit 53 performs a geometric transformation in which the position of each pixel belonging to the detection block BL [j] of the normal captured image I _L [i] is translated leftward by 5 pixels. . In FIG. 10, the arrow in the normal photographed image I _L [i] indicates the state of this geometric transformation. By performing such geometric conversion on all the detection blocks of the normal captured image I _L [i], a converted image I _CA [i] from which FP distortion has been removed is obtained.

Distortion correction unit 53 sets the rectangular area having a predetermined image size to the transformed image I _CA [i], extracts the image of the rectangular region from the transformed image I _CA [i], the extracted image corrected image It is generated as I _C [i]. If the image size of the corrected image I _C [i] should be (1280 × 720), an image size enlargement process for increasing the image size of the extracted image to (1280 × 720) is performed, and after the image size enlargement process Are extracted as a corrected image I _C [i].

The generation of the corrected image is sequentially executed for each set of the high-speed captured image and the normal captured image. That is, the corrected image I _C [1] is generated from the image I _L [1] using the FP distortion amount based on the images I _S [1] and I _L [1], and the images I _S [2] and I _L [ 2] is used to generate the corrected image I _C [2] from the image I _L [2]. The same applies to the corrected image I _C [3] and the like. Each corrected image is recorded in the external memory 18 as each frame of a moving image with a recording frame rate of 30 fps.

  As described above, since the exposure timing difference Δt with respect to the high-speed captured image is sufficiently small, the FP distortion included in each high-speed captured image is small enough to be ignored. However, if an image having the same image size as the normal captured image is obtained under high-speed shooting conditions, the power consumption increases, or if high-speed shooting is performed while suppressing power consumption, an image that can be read from the image sensor 33 is read. The image size is smaller than usual (see FIG. 7). Therefore, it is not desirable to perform moving image shooting only under high-speed shooting conditions. Taking these into consideration, as described above, the high-speed captured image and the normal captured image are alternately obtained, and the difference between the high-speed captured image and the normal captured image is evaluated with reference to the high-speed captured image, so Is detected. Thereby, the FP distortion of the normal photographed image is accurately detected, and the FP distortion can be removed with high accuracy by correcting the normal photographed image using the detected FP distortion amount.

In the above example, since it is assumed that the image size of the normal captured image I _L [i] is (1280 × 720), the image size of the corrected image I _C [i] should be (1280 × 720). Then, an image size enlargement process is required for the image extracted from the converted image I _CA [i] (see FIG. 10), but the image size of the normal photographed image is set so that this image size enlargement process is unnecessary. It may be larger than the original (1280 × 720). That is, for example, when the image size of the corrected image I _C [i] is determined to be (1280 × 720), the image size of (1600 × 1200) under the condition that the number of pixels of the image sensor 33 is sufficiently large. It is made to acquire the normal picked-up image which has. Then, unless an abnormally large FP distortion occurs, a corrected image having an image size of (1280 × 720) can be directly extracted from the converted image based on the normal captured image.

In addition, the FP distortion amount based on the images I _S [1] and I _L [1] includes the FP distortion included in the normal captured image (for example, I _L [2]) other than the normal captured image I _L [1]. And using the FP distortion amount based on the images I _S [1] and I _L [1], a normal captured image other than the normal captured image I _L [1] (for example, I _L [2]) It is also possible to remove the FP distortion.

  The above operation is called a basic operation. Hereinafter, a plurality of embodiments based on basic operations will be described. In each of the embodiments described later, the description of the basic operation described above is applied to items that are not particularly described unless a contradiction arises. Moreover, as long as there is no contradiction, it is also possible to implement combining the matter described in a certain Example, and the matter described in the other Example.

<< First Example >>
A first embodiment will be described. In the first embodiment, the flow of the above-described moving image shooting and recording operations will be described. FIG. 11 is a flowchart showing the flow of this operation.

  When the start of recording and recording of a moving image is instructed by pressing the recording button 26a in FIG. 1, first, the process of step S11 is executed. In step S11, the image acquisition unit 51 in FIG. 5 acquires an image signal of a high-speed captured image for one frame by driving the image sensor 33 under a 300 fps drive condition. The image signal of the obtained high-speed captured image is temporarily stored in the internal memory 17. In subsequent step S12, the image acquisition unit 51 in FIG. 5 acquires the image signal of the normal captured image for one frame by driving the image sensor 33 under the 33 fps drive condition. The obtained image signal of the normal captured image is temporarily stored in the internal memory 17.

  Thereafter, in step S13, the distortion detection unit 52 or the motion detection unit 54 performs size normalization to match the image size of the high-speed captured image obtained in step S11 with the image size of the normal captured image obtained in step S12. Further, in step S14, the distortion detector 52 uses the motion detector 54 to calculate the FP distortion amount from the high-speed captured image and the normal captured image after size normalization. As described above, the FP distortion amount is calculated for each detection block. In subsequent step S15, the distortion correction unit 53 generates a corrected image from which the FP distortion is removed by correcting the normal captured image based on the FP distortion amount calculated in step S14. In step S <b> 16, the image signal of the generated corrected image is recorded in the external memory 18 under the control of the CPU 23.

  After the processes of steps S11 to S16 described above are sequentially performed, the process proceeds to step S17. In step S <b> 17, the image acquisition unit 51 confirms whether or not the vertical synchronization signal Vsync is output from the TG 22, and waits for the image sensor 33 to be driven until the output of the vertical synchronization signal Vsync is confirmed. Here, the vertical synchronization signal Vsync is a vertical synchronization signal Vsync for 30 fps generated at a period of 1/30 seconds. If the output of the vertical synchronization signal Vsync is confirmed, the process proceeds from step S17 to step S18, and whether the image acquisition unit 51 is instructed to end the shooting and recording of the moving image by pressing the recording button 26a again. Confirm whether or not. When the termination is instructed, the operation of FIG. 11 is terminated, but when the termination is not instructed, the process returns to step S11, and each processing after step S11 is executed again.

<< Second Example >>
A second embodiment will be described. The FP distortion removal method described above can also be applied to still image shooting. That is, when a still image shooting instruction is issued by pressing the shutter button 26b in FIG. 1, the image acquisition unit 51 continuously drives the image sensor 33 under high-speed shooting conditions and the image sensor 33 under normal shooting conditions. Therefore, it should be executed only once. As a result, as shown in FIG. 12, one high-speed photographed image 311 and one normal photographed image 312 continuously photographed are obtained. The high-speed captured image 311 and the normal captured image 312 obtained here correspond to the high-speed captured image I _S [1] and the normal captured image I _L [1] in FIG. Accordingly, the high-speed captured image 311 and the normal captured image 312 are treated as the high-speed captured image I _S [1] and the normal captured image I _L [1], and the normal captured image 312 is used by using the distortion detection unit 52 and the distortion correction unit 53. A corrected image from which the FP distortion is removed can be obtained.

  Furthermore, the FP distortion removal method described above can also be applied to continuous shooting of still images. The user can instruct continuous shooting of still images by performing a predetermined continuous shooting operation on the operation unit 26.

  For example, when it is instructed to continuously shoot three still images by the continuous shooting operation, the image acquisition unit 51 performs driving of the image sensor 33 under the high-speed shooting conditions once, and then performs imaging under the normal shooting conditions. It is preferable to drive the element 33 continuously three times. Thereby, as shown in FIG. 13, one high-speed photographed image 321 and three normal photographed images 322 to 324 continuously photographed are obtained.

Distortion detector 52, the high-speed captured image I _S [i] and the normal shooting image I _L a method similar to the method of deriving from [i] a FP distortion amount with respect to the normal shooting image I _L [i] for each detection block Thus, the FP distortion amount for the image 322 is derived from the images 321 and 322 for each detection block, the FP distortion amount for the image 323 is derived for each detection block from the images 321 and 323, and the image 324 is derived from the images 321 and 324. The FP distortion amount can be derived for each detection block. The distortion correction unit 53 is normally captured image I _L [i] similar to the method of generating a corrected image I _C [i] from the normal captured image I _L [i] based on the FP distortion quantity for each detection block for The method generates a corrected image from which the FP distortion is removed from the image 322 based on the FP distortion amount for each detection block with respect to the image 322, and the FP distortion from the image 323 based on the FP distortion amount for each detection block with respect to the image 323. A corrected corrected image is generated, and a corrected image in which the FP distortion is removed from the image 324 based on the FP distortion amount for each detection block with respect to the image 324 can be generated.

Further, for example, when it is instructed to continuously shoot three still images by the continuous shooting operation, the image acquisition unit 51 drives the image sensor 33 under high-speed shooting conditions and drives the image sensor 33 under normal shooting conditions. The process that is executed once may be repeated three times continuously. Accordingly, as shown in FIG. 14, the high-speed captured image 331, the normal captured image 332, the high-speed captured image 333, the normal captured image 334, the high-speed captured image 335, and the normal captured image 336 are continuously captured in this order. become. The images 331, 332, 333, 334, 335, and 336 obtained here are used as images I _S [1], I _L [1], I _S [2], I _L [2], I _S [3, respectively. ] And I _L [3] and using the distortion detection unit 52 and the distortion correction unit 53, three corrected images from which the FP distortions of the normal captured images 332, 334, and 336 are removed can be obtained.

In each of the above-described operation examples including the examples described with reference to FIGS. 6 and 12 to 14, the high-speed captured image acquired for derivation of the FP distortion amount of the normal captured image is the normal captured image. Although it was shot before, the shooting order of the high-speed shot image and the normal shot image is not limited. That is, for example, in the example of FIG. 6, shooting may be performed in the order of images I _L [1], I _S [1], I _L [2], I _S [2],. In the example of FIG. 12, the image 311 may be taken after the image 312.

<< Third Example >>
A third embodiment will be described. In each of the above-described operation examples, it is assumed that a correction image is generated in real time when a moving image or a still image is captured, but the generation timing of the correction image is arbitrary. That is, for example, in the example shown in FIG. 6, the captured images I _S [1], I _L [1], I _S [2], I _L [2],..., I _S [m] and I _L The image signal of [m] is temporarily recorded in the external memory 18 (m is an integer of 2 or more), and then at any timing as necessary, the images I _S [1], I _L [1], The corrected images I _C [1] to I _L [m] may be generated from I _S [2], I _L [2],..., I _S [m], and I _L [m].

  The distortion detection unit 52 and the distortion correction unit 53 may be provided in an electronic device (not shown) different from the imaging device 1. Electronic devices include a display device such as a television receiver, a personal computer, a mobile phone, and the like, and an imaging device is also a kind of electronic device. If the image signals of the high-speed captured image and the normal captured image obtained using the image acquisition unit 51 of the imaging device 1 are transmitted to the electronic device via the external memory 18 or by communication, In the distortion detection unit 52 and the distortion correction unit 53, it is possible to generate a corrected image from the high-speed captured image and the normal captured image.

<< 4th Example >>
A fourth embodiment will be described. In the above example, a plurality of detection blocks are set by dividing the image 220 along only the horizontal direction (FIG. 8A), but the image 220 is divided along the horizontal and vertical directions. Thus, a plurality of detection blocks may be set (the same applies to the image 230).

That is, for example, as shown in FIG. 15, the image 220 may be divided into (n _A × n _B ) detection blocks by dividing the entire image area of the image 220 along the horizontal and vertical directions. (N _A and n _B are integers of 2 or more). The detection block for the image 230 is set in the same manner as the image 220. Also in this case, as described above, the motion detection unit 54 and the distortion detection unit 52 calculate the motion vector and the FP distortion amount between the high-speed captured image and the normal captured image for each detection block, and the distortion correction unit 53 There is no change in the correction of the normal captured image based on the FP distortion amount for each detection block.

<< 5th Example >>
A fifth embodiment will be described. In the basic operation described above, it is assumed that the position of the image sensor 33 has changed in the horizontal direction corresponding to the horizontal direction of the image during the shooting of a moving image with an upright person as the subject. Although the FP distortion in the direction is removed, in the fifth embodiment and the sixth embodiment, which will be described later, during the shooting of a moving image with an upright person as the subject, the image pickup device 33 corresponds to the vertical direction of the image. Assume that the position has changed in the direction.

In the vertical position change, there are an upward position change and a downward position change. In the fifth embodiment, the imaging element 33 moves upward while shooting a moving image with an upright person as a subject. Consider the case where the position has changed. FIG. 16 is a diagram illustrating a high-speed photographed image sequence and a normal photographed image sequence according to the fifth embodiment, and photographing timings thereof. Images 410 and 430 are high-speed captured images I _S [1] and I _S [2] in the fifth embodiment, respectively, and images 420 and 440 are normal captured images I _L [1] in the fifth embodiment, respectively. ] And I _L [2]. The method for generating the high-speed captured image and the normal captured image according to the fifth embodiment is the same as that of the basic operation. Accordingly, each of the high-speed captured images 410 and 430 has an image size of (448 × 336), and each of the normal captured images 420 and 440 has an image size of (1280 × 720).

  Similar to the basic operation, the distortion detection unit 52 or the motion detection unit 54 performs size normalization so that the image sizes of the high-speed captured images 410 and 430 are the same as the image sizes of the normal captured images 420 and 440, respectively. To do. Size normalization may be realized by reducing the image size of the normal captured image, but it is assumed that the image size enlargement process for increasing the image size of the high-speed captured image is executed as size normalization as in the basic operation. To do. That is, in the image size enlargement process as size normalization, the high-speed captured images 410 and 430 given from the image processing unit 51 are set so that the image size of the high-speed captured image after size normalization becomes (1280 × 720). Increase image size. The high-speed captured images 410 and 430 after size normalization are represented by 410 'and 430', respectively. FIG. 17 shows a high-speed captured image 410 ′ after size normalization together with a normal captured image 420.

  The motion detector 54 derives an optical flow between the images 410 ′ and 420 based on the image signals of the images 410 ′ and 420. Specifically, for example, the following processing may be performed.

  By extracting the first to Nth feature points from the image 410 ′ and performing image matching (for example, block matching) between the image 410 ′ and the image 420, the first to Nth feature points on the image 410 ′ are obtained. N feature points on the image 420 corresponding to the feature points are detected (N is an integer of 2 or more). However, it is assumed that the first to Nth feature points have different vertical positions on the image 410 '. That is, the horizontal lines to which the first to Nth feature points belong are different from each other. Similarly to the case where n detection blocks BL [1] to BL [n] are set for the image 220 in FIG. 8A, the detection blocks BL [1] to BL for the image 410 ′ are also set. [N] may be set, and feature points may be extracted one by one from each detection block on the image 410 ′ (in this case, n = N).

  The feature point is preferably a point that can be distinguished from the surrounding points and can be easily traced. Such feature points can be automatically extracted by using a well-known feature point extractor (not shown) that detects pixels in which the amount of change in shading in the horizontal and vertical directions increases. The feature point extractor is, for example, a Harris corner detector or a SUSAN corner detector. Then, for each feature point on the image 410 ', a vector from the position of the feature point on the image 410' to the position of the corresponding feature point on the image 420 is obtained as a feature point motion vector.

FIG. 18 shows three feature points 411 to 413 extracted from the image 410 ′, and feature points 421 to 423 on the image 420 corresponding to the feature points 411 to 413. Ideally, the image signals of the pixels located at the feature points 411 to 413 on the image 410 ′ match the image signals of the pixels located at the feature points 421 to 423 on the image 420, respectively. The vector VEC ₁ is a feature point motion vector from the position of the feature point 411 on the image 410 ′ to the position of the corresponding feature point 421 on the image 420, and the vector VEC ₂ is the feature point 412 of the image 410 ′. A feature point motion vector from the position to the position of the corresponding feature point 422 on the image 420, and the vector VEC ₃ is moved from the position of the feature point 413 on the image 410 ′ to the position of the corresponding feature point 423 on the image 420. It is a feature point motion vector that is headed. A feature point motion vector for each of the N feature points is obtained by the motion detection unit 54. The optical flow required in the fifth embodiment is a bundle of feature point motion vectors, which is a kind of motion vector. In the fifth embodiment and the sixth embodiment, which will be described later, since it is assumed that the position of the image sensor 33 has changed in the upward direction, the direction of each feature point motion vector is the vertical direction of the normal captured image and the high-speed captured image. Is parallel to the vertical direction.

  In the above example, feature points corresponding to the feature points on the image 410 ′ are detected from the image 420 after the feature points are extracted from the image 410 ′. Then, a feature point corresponding to a feature point on the image 420 may be detected from the image image 410 ′, thereby obtaining a feature point motion vector. Also, optical flow derivation in the basic operation can be realized by the same feature point search as described above.

  The motion detection unit 54 also obtains an optical flow formed from a plurality of feature point motion vectors for the images 430 ′ and 440, and for a set of high-speed captured images and normal captured images obtained after the image 440 is captured. But we will ask for it sequentially.

  The distortion detection unit 52 treats the feature point motion vector obtained for the high-speed captured image 410 ′ and the normal captured image 420 as the FP distortion amount for the normal captured image 420. The same applies to the high-speed photographed image 430 ′ and the normal photographed image 440. The FP distortion amount is an amount representing FP distortion existing in a normal captured image. The FP distortion amount is an amount having a direction and a size, like the motion vector and the feature point motion vector.

The distortion correction unit 53 corrects the image signal of each normal captured image from the image acquisition unit 51 based on the FP distortion amount obtained by the distortion detection unit 52, thereby removing the FP distortion of each normal captured image. To do. Images obtained by removing the FP distortion of the normal captured images 420 and 440 are referred to as corrected images 420 _C and 440 _C , respectively.

An example of a method for generating the corrected image 420 _C from the normal captured image 420 based on the FP distortion amount will be described.

The distortion correction unit 53 sets an inverse vector of the feature point motion vector as the FP distortion amount as a distortion correction vector. The inverse vector of the feature point motion vector refers to a vector having the opposite direction to the feature point motion vector and the same size as the feature point motion vector. First, a distortion correction vector is set for each horizontal line to which a feature point belongs. That is, as shown in FIG. 19, the feature point motion vectors VEC ₁ , VEC ₂ and VEC ₃ are respectively applied to the horizontal lines 451, 452 and 453 on the normal captured image 420 to which the feature points 421, 422 and 423 belong. Distortion correction vectors VEC ₁ ′, VEC ₂ ′ and VEC ₃ ′ which are inverse vectors are set. A distortion correction vector is also set for a horizontal line to which other feature points other than the feature points 421 to 423 belong.

A horizontal line 450 shown in the normal captured image 420 in FIG. 19 is a horizontal line on the normal captured image 420 corresponding to the first horizontal line of the image sensor 33 and is located on the uppermost side in the normal captured image 420. It is a horizontal line. The magnitude of the distortion correction vector VEC _TOP ′ for the horizontal line 450 is zero. For convenience of explanation, it is assumed that the u-th feature point is located above the (u + 1) -th feature point (u is an integer) on the normal captured image 420, and the feature points 421 to 423 are the first. It is assumed to be a third feature point.

The distortion correction unit 53 sets a distortion correction vector for each horizontal line to which the first to Nth feature points belong based on the feature point motion vectors for the first to Nth feature points, and then uses interpolation. A distortion correction vector for each horizontal line is derived. As an interpolation method, any interpolation method represented by linear interpolation can be used. Here, it is assumed that linear interpolation is used. Then, the distortion correction vector for the horizontal line positioned between the horizontal lines 450 and 451 is an average vector of the distortion correction vectors VEC _TOP ′ and VEC ₁ ′, and the horizontal line positioned between the horizontal lines 451 and 452 is used. Is a mean vector of the distortion correction vectors VEC ₁ ′ and VEC ₂ ′. Similarly, the distortion correction vectors for the horizontal line 453 to which the third feature point 423 belongs and the horizontal line located between the horizontal line to which the fourth feature point belongs are the distortion correction vector VEC ₃ ′ and the fourth This is an average vector with a distortion correction vector for the horizontal line to which the feature point belongs. As described above, the distortion correction vector of the horizontal line located between the horizontal line to which the u-th feature point belongs and the horizontal line to which the (u + 1) -th feature point belongs is obtained with respect to the horizontal line to which the u-th feature point belongs. It is derived by linear interpolation using the distortion correction vector and the distortion correction vector for the horizontal line to which the (u + 1) th feature point belongs.

  After deriving the distortion correction vector for each horizontal line of the normal captured image 420, the distortion correction unit 53 determines the horizontal line of the normal captured image 420 and each pixel on the horizontal line for each horizontal line. A geometrical transformation is performed by translating along the line by the magnitude of the distortion correction vector. On the right side of FIG. 19, horizontal lines 451 ′ to 453 ′ obtained by applying the geometric transformation to the horizontal lines 451 to 453 are shown on the common image space together with the horizontal lines 451 to 453.

Image obtained by performing the geometric transformation according to the distortion correction vector to the normal photographic image 420, a corrected image 420 _C. FIG. 20 shows the obtained corrected image 420 _C. The geometric transformation in the assumption of the fifth embodiment corresponds to a scale transformation that enlarges the image size in the vertical direction. For example, when there are five horizontal lines between the horizontal lines 451 and 452 on the normal captured image 420, if the distance between the horizontal lines 451 and 452 is doubled by the geometric transformation, corrected image 420 between horizontal lines 451 'and 452' on _C is inserted 10 horizontal lines, as the image signal of the image signal corrected image 420 _C of each pixel on the horizontal line of the ten, usually It is derived through interpolation (re-sampling of the image signal as necessary) based on the image signal of the captured image 420.

Although the correction operation for the normal captured image 420 has been described, the generation of the corrected image is sequentially executed for each set of the high-speed captured image and the normal captured image. That is, the corrected image 420 _C is generated from the image 420 using the FP distortion amount based on the images 410 and 420, and the corrected image 440 _C is generated from the image 440 using the FP distortion amount based on the images 430 and 440. The same applies to a set of a high-speed captured image and a normal captured image obtained after the image 440 is captured. Each corrected image is recorded in the external memory 18 as each frame of a moving image with a recording frame rate of 30 fps.

  According to the fifth embodiment, the FP distortion of the normal photographed image in the vertical direction is accurately detected, and the FP distortion in the vertical direction is accurately removed by correcting the normal photographed image using the detected FP distortion amount. It becomes possible to do.

<< Sixth Example >>
A sixth embodiment will be described. In the sixth embodiment, contrary to the assumption of the fifth embodiment, a case is considered in which the position of the image sensor 33 changes in the downward direction during shooting of a moving image with an upright person as a subject. The operation of the sixth embodiment is the same as that of the fifth embodiment, except that the contents of scale conversion used when generating the corrected image from the normal photographed image are different.

FIG. 21 is a diagram illustrating a high-speed captured image sequence and a normal captured image sequence according to the sixth embodiment, and their imaging timing. Images 510 and 530 are high-speed captured images I _S [1] and I _S [2] in the sixth embodiment, respectively, and images 520 and 540 are normal captured images I _L [1] in the sixth embodiment, respectively. ] And I _L [2]. The method for generating the high-speed captured image and the normal captured image according to the sixth embodiment is the same as that of the basic operation. Accordingly, the image sizes of the high-speed captured images 510 and 530 are (448 × 336), and the image sizes of the normal captured images 520 and 540 are (1280 × 720).

  Similar to the basic operation, the distortion detection unit 52 or the motion detection unit 54 performs size normalization so that the image sizes of the high-speed captured images 510 and 530 are the same as the image sizes of the normal captured images 520 and 540, respectively. To do. Size normalization may be realized by reducing the image size of the normal captured image, but it is assumed that the image size enlargement process for increasing the image size of the high-speed captured image is executed as size normalization as in the basic operation. To do. That is, in the image size enlargement process as size normalization, the high-speed captured images 510 and 530 given from the image processing unit 51 are set so that the image size of the high-speed captured image after size normalization becomes (1280 × 720). Increase image size. High-speed captured images 510 and 530 after size normalization are denoted by 510 'and 530', respectively. FIG. 22 shows the high-speed captured image 510 ′ after size normalization together with the normal captured image 520.

  The motion detection unit 54 derives an optical flow between the images 510 ′ and 520, which is composed of a plurality of feature point motion vectors, based on the image signals of the images 510 ′ and 520. The method of deriving the optical flow here is the same as that shown in the fifth embodiment. The motion detection unit 54 also obtains an optical flow formed from a plurality of feature point motion vectors for the images 530 ′ and 540, and for a set of high-speed captured images and normal captured images obtained after the image 540 is captured. But we will ask for it sequentially.

  The distortion detection unit 52 treats the feature point motion vector obtained for the high-speed captured image 510 ′ and the normal captured image 520 as an FP distortion amount for the normal captured image 520. The same applies to the high-speed photographed image 530 ′ and the normal photographed image 540.

The distortion correction unit 53 corrects the image signal of each normal captured image from the image acquisition unit 51 based on the FP distortion amount obtained by the distortion detection unit 52, thereby removing the FP distortion of each normal captured image. To do. Images obtained by removing the FP distortion of the normal captured images 520 and 540 are referred to as corrected images 520 _C and 540 _C , respectively.

The distortion correction unit 53 derives a distortion correction vector for each horizontal line of the normal captured image 520 from the FP distortion amount (feature point motion vector) of the normal captured image 520 by the same method as described in the fifth embodiment. Then, the horizontal line of the normal photographed image 520 and each pixel on the horizontal line are subjected to geometric transformation in which the horizontal line is translated by the magnitude of the distortion correction vector along the direction of the distortion correction vector. A corrected image 520 _C is generated.

FIG. 23 shows the obtained corrected image 520 _C. The geometric transformation in the assumption of the sixth embodiment corresponds to a scale transformation that reduces the image size in the vertical direction. For example, when there are ten horizontal lines between the first and second target horizontal lines on the normal captured image 520, the distance between the first and second target horizontal lines is 1 / If it is reduced by a factor of 2, only five horizontal lines are inserted between the two horizontal lines on the corrected image 520 _C corresponding to the first and second horizontal lines of interest. That is, half of the ten horizontal lines are thinned out during the scale conversion process.

Although the correction operation for the normal captured image 520 has been described, the generation of the corrected image is sequentially executed for each set of the high-speed captured image and the normal captured image. That is, the corrected image 520 _C is generated from the image 520 using the FP distortion amount based on the images 510 and 520, and the corrected image 540 _C is generated from the image 540 using the FP distortion amount based on the images 530 and 540. The same applies to a set of high-speed captured images and normal captured images obtained after the image 540 is captured. Each corrected image is recorded in the external memory 18 as each frame of a moving image with a recording frame rate of 30 fps.

<< Deformation, etc. >>
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. As modifications or annotations of the above-described embodiment, notes 1 to 3 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
Although the method for removing horizontal FP distortion and the method for removing vertical FP distortion have been described separately, if the normal photographed image contains horizontal and vertical FP distortion, both can be removed simultaneously. It is. That is, by performing the basic operation in combination with the fifth and sixth embodiments described above, the FP distortion in the horizontal and vertical directions that may be included in the normal photographed image may be simultaneously removed.

[Note 2]
At the time of generating a high-speed captured image, a part of the light receiving pixels in the common area 210 in FIG. 7 is thinned, while at the time of generating a normal captured image, output signals of all the light receiving pixels in the common area 210 are individually read out. As described above, the light receiving pixels may be thinned out even when the normal captured image is generated. Therefore, for example, light receiving pixels on several horizontal lines belonging to the common area 210 of the image sensor 33 may be thinned out not only when generating a high-speed captured image but also when generating a normal captured image. However, the horizontal line to be thinned may be different between when a normal captured image is generated and when a high-speed captured image is generated.

[Note 3]
The imaging apparatus 1 in FIG. 1 can be configured by hardware or a combination of hardware and software. When the imaging apparatus 1 is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part. A function realized using software may be described as a program, and the function may be realized by executing the program on a program execution device (for example, a computer).

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging part 33 Imaging element 51 Image acquisition part 52 Distortion detection part 53 Distortion correction part 54 Motion detection part

Claims (5)

An image sensor that captures an image while varying the exposure timing between the first and second horizontal lines included in the plurality of horizontal lines;
A first image is obtained by reading out an image signal obtained by exposure with a relatively short exposure timing difference between the first and second horizontal lines from the image sensor, and an image signal obtained by exposure with a relatively long exposure timing difference. An image acquisition unit that acquires a second image by reading from the image sensor;
A motion detector for deriving an optical flow between the first and second images based on image signals of the first and second images;
An imaging apparatus comprising: a correction unit that corrects the second image based on the optical flow. 2. The correction unit according to claim 1, wherein the correction unit corrects distortion generated in the second image due to an exposure timing being different between the first and second horizontal lines based on the optical flow. Imaging device. The motion detection unit derives a plurality of motion vectors forming the optical flow based on the image signals of the first and second images;
The imaging apparatus according to claim 2, wherein the correction unit corrects the distortion generated in the second image based on each motion vector. The imaging apparatus according to claim 3, wherein the correction unit corrects the distortion by performing geometric transformation on the second image based on each motion vector. The image acquisition unit repeatedly executes alternate shooting of the first and second images,
The imaging apparatus records each corrected image based on each second image obtained by the correction of the correction unit on a recording medium at a predetermined recording frame rate,
The total of the time required to obtain one first image and the time required to obtain one second image is set to be equal to or less than the reciprocal of the recording frame rate. The imaging device according to claim 1.

Images