JP2008078808A - Imaging apparatus, image distortion correcting method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus, an image distortion correcting method and a program which can improve accuracy of camera shake correction by decreasing the influence of MOS distortion. <P>SOLUTION: The imaging apparatus 100 is provided with an imaging section 101 for reading pixels per line; image storage section 102 for storing image data photographed at a plurality of times; a division number designating section 103 for designating the number of divisions; an image division control section 104 for dividing an image in a pixel reading line direction and controlling each of portions; a movement detecting section 105 for calculating a quantity of movement between images from a plurality of image data in each of the divided images; an image compositing section 106 for compositing the images on the basis of the quantity of movement between the images of the divided images; and an image preserving section 107 for preserving the composited image. The imaging apparatus 100 divides an image in a reading line direction, obtains a camera shake quantity (movement quantity) independently for each of divided portions and composites the images using affine transformation. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an imaging apparatus, an image distortion correction method, and a program that are mounted on an electronic still camera, a video camera, and the like and correct image distortion caused by camera shake.

  Conventionally, a CCD (Charge Coupled Device) sensor has been used as an imaging element of an imaging apparatus such as an electronic still camera or a video camera. Recently, MOS-based image sensors have been mounted for reasons such as low power consumption. The CCD sensor is an image sensor in which the charge accumulation timing of all pixels is the same, and the influence of camera shake on a still image is small. In contrast to a CCD sensor that reads all pixels simultaneously, a MOS sensor that reads pixels for each line causes image distortion due to a difference in reading time, so-called MOS distortion. MOS distortion is a phenomenon in which the charge accumulation timing of the first scanning line and the last scanning line on the imaging surface is shifted by nearly one image scanning period, and figure distortion occurs due to camera shake. In the case of a still image, a straight line is bent and reflected. In the case of a moving image, the straight line fluctuates and the size of the subject image expands or contracts.

  Even in the case of MOS-based image sensors, readout of all lines is fast and elements with a high frame rate are less likely to cause MOS distortion. On the other hand, the number of lines tends to increase as the number of pixels increases. As the number of lines increases, it is difficult to increase the frame rate, and MOS distortion tends to occur.

  As a method for performing still image electronic image stabilization, for example, there is a method described in Patent Document 1.

  FIG. 16 is a block diagram illustrating a configuration of the imaging apparatus described in Patent Document 1. FIGS. 17A and 17B are diagrams for explaining electronic camera shake correction of the image pickup apparatus of FIG. 16. FIG. 17A illustrates normal shooting and FIG. 17B illustrates electronic camera shake correction.

  In FIG. 16, the imaging apparatus 10 includes an imaging unit 11, an image storage unit 12, a motion detection unit 13, an image synthesis unit 14, and an image storage unit 15.

  Camera shake is a phenomenon in which an image blurs due to hand movement during an exposure period. As shown in FIG. 17A, when the exposure surface is blurred due to camera shake from the start of exposure to the end of exposure, the image is blurred due to camera shake.

  The imaging unit 11 captures a plurality of images with a short exposure time, and stores the plurality of captured images in the image storage unit 12. The image composition unit 14 converts the amount of movement (shake amount) between the images based on the motion detection result detected by the motion detection unit 13 and composes the image without blur and having a high S / N. Store in the storage unit 15. In the camera shake correction method described in Patent Document 1, as shown in FIG. 17 (b), images are obtained by reducing the exposure time, continuously shooting, obtaining the amount of movement from the first sheet, and converting this amount of movement. The image is synthesized after correcting the positional deviation.

  Patent Document 2 discloses a camera system that obtains a camera shake amount three-dimensionally from three angular velocity sensors and synthesizes a plurality of images corresponding to rotational shake by affine transformation.

In Patent Document 3, a horizontal position having a high correlation between the current line and the previous line is obtained, and the current line is written at the position to correct the horizontal CMOS distortion. The vertical direction is the angular velocity. An imaging device that corrects using a sensor is disclosed.
JP 2001-045359 A JP 2000-224461 A JP 2004-363869 A

  However, such a conventional imaging apparatus has the following problems.

  (1) In a MOS-type sensor, distortion corresponding to camera shake occurs, and the influence appears remarkably during image composition.

  (2) In order to reduce the influence of distortion, it is necessary to use a camera with a short frame interval.

  (3) Non-linear distortion cannot be corrected by simply applying linear transformation such as affine transformation.

  The above problem will be described in more detail.

  The camera shake correction method described in Patent Document 1 is based on the premise that an image without distortion such as a CCD sensor is used.

  18 and 19 are diagrams for explaining that image composition is not correctly performed due to MOS distortion. FIG. 18 shows camera shake correction by a CCD sensor, and FIG. 19 shows camera shake correction by a MOS sensor.

In FIG. 18 and FIG. 19, for the sake of simplicity, camera shake only in the horizontal direction (X-axis direction) is assumed, and an image output at time t = 0 is used as a position reference. 18 (a) and 19 (a) show the amount of camera shake generated in the images of the nth and n + 1th (n, n + 1) th frames in continuous shooting, and the movement amount x _n determined during camera shake correction. , X _{n + 1} are shown. FIGS. 18B and 19B show the n-th and n + 1-th images in continuous shooting, and the positions of the images after camera shake are deviated from the original positions (t = 0).

Since the CCD sensor shown in FIG. 18 reads all pixels simultaneously, the amount of camera shake at the start time of each frame is the amount of movement of the entire image. On the other hand, since the MOS sensor shown in FIG. 19 reads out line by line, the amount of camera shake at each time is reflected in the image. In the example of FIG. 19, since the camera shake amount is read for each line with respect to the camera shake correction reference (x _n , x _{n + 1} ) of the n, n + 1th frame, the camera shake correction cannot be performed well because it varies for each line. . A hatched portion in FIG. 19A indicates a portion where camera shake correction is not possible. In addition, since nonlinear elements are included, the straight line of the corrected image is bent and synthesized as shown in FIG.

As described above, in the case of a MOS sensor, only a part of an image is correctly synthesized only by obtaining the movement amounts x _n and x _{n + 1} between images using camera shake correction.

  SUMMARY An advantage of some aspects of the invention is that it provides an imaging apparatus, an image distortion correction method, and a program capable of reducing the influence of MOS distortion and improving the accuracy of camera shake correction.

  An image pickup apparatus according to the present invention includes an image pickup device that performs pixel readout for each line, an image storage unit that stores image data captured a plurality of times by the image pickup device, and an image that divides an image in the pixel read line direction of the image pickup device. In each of the divided images, a divided image obtained from the motion detection unit, a motion detection unit that calculates a movement amount between images from a plurality of image data stored in the image storage unit The image synthesizing means for synthesizing the images based on the movement amount between the respective images is adopted.

  An image distortion correction method according to the present invention includes a step of photographing a plurality of times by an image sensor that performs pixel readout for each line, a step of storing image data photographed a plurality of times, and an image being divided in the pixel readout line direction of the image sensor. A step of calculating a movement amount between the images from a plurality of image data stored in the image storage means in each of the divided images, and a movement amount between the divided images. And synthesizing the image.

  From another viewpoint, the present invention is a program for causing a computer to execute the steps of the image distortion correction method.

  According to the present invention, it is possible to reduce the influence of MOS distortion and improve the accuracy of camera shake correction, and to prevent image composition failure due to MOS distortion.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(Embodiment 1)
FIG. 1 is a block diagram showing a configuration of an imaging apparatus according to Embodiment 1 of the present invention.

  In FIG. 1, the imaging apparatus 100 includes an imaging unit 101, an image storage unit 102, a division number designation unit 103, an image division control unit 104, a motion detection unit 105, an image composition unit 106, and an image storage unit 107. The

  The imaging unit 101 is a MOS sensor that performs pixel readout for each line. The image storage unit 102 stores image data captured a plurality of times.

  The division number designating unit 103 designates the number of divisions for dividing the image in the scan direction based on a prescribed value based on the frame rate of the MOS sensor or a user set value.

  The image division control unit 104 changes the data reading start position / end position for a plurality of image data in the image storage unit 102 based on the division number designated by the division number designation unit 103 to change the motion detection unit 105 and the image composition. The image composition unit 106 is notified of the cut start position / end position of the composite image, and the image composition unit 106 is notified of the output image data write start position / end position in the image storage unit 107.

  The motion detection unit 105 calculates the amount of movement between images from a plurality of image data stored in the image storage unit 102. The image synthesis unit 106 synthesizes images based on the movement amount between images obtained from the motion detection unit 105. The image storage unit 107 stores the composition result of the image composition unit 106.

  Hereinafter, the operation of the imaging apparatus 100 configured as described above will be described.

  First, the basic concept of the present invention will be described.

  FIG. 2 is a diagram for explaining the basic concept of the present invention.

  As shown in FIG. 2 (a), in the conventional example, the MOS sensor reads out line by line, so that distortion corresponding to camera shake occurs between the first image and the second image. The effect of distortion was prominent in the synthesized image.

  Therefore, according to the present invention, each continuous shot image is divided along the image reading line direction, and movement amount calculation and screen composition are performed for each divided portion. FIG. 2B illustrates an example in which an image is divided into a plurality (two) in the scanning direction, the amount of camera shake is obtained for each divided portion, and the synthesis process is performed for each divided portion. Thereafter, the divided and combined images are joined together. This eliminates the failure of image composition due to MOS distortion.

  When image synthesis is performed using affine transformation at the time of image synthesis, the accuracy of image synthesis can be further improved. Alternatively, high correction accuracy can be obtained with a smaller number of divisions.

FIG. 3 is a diagram for explaining the principle of affine transformation. Affine transformation is coordinate transformation using a determinant represented by the following equation (1). Moving, rotating, and transforming images is realized by changing matrix parameters.

  FIG. 4 is a diagram for explaining camera shake correction by affine transformation, and corresponds to the camera shake correction explanatory diagram of FIG.

The camera shake correction references _{xn (Ln)} and _{xn + 1 (Ln + 1)} of the nth frame and the _{( n + 1 ) th} frame are expressed by the following equation (2).
_{x n (Ln) = a n} * L n + b n
_{x n + 1 (Ln + 1} ) = a n + 1 * L n + 1 + b n + 1 ... (2)

By making the camera shake correction reference ( _{xn (Ln)} , _{xn + 1 (Ln + 1)} ) close to the inclination of the camera shake correction amount by the linear transformation based on the affine transformation, it is possible to reduce the camera shake correction impossible portion. However, since the affine transformation is a linear transformation, the distortion generated nonlinearly cannot be corrected.

  Next, the overall operation of the imaging apparatus 100 will be described.

  First, in the division number designation unit 103, the number of divisions for dividing an image in the scan direction is designated in advance by the frame rate of the MOS sensor or the user setting value. The imaging unit 101 is a MOS sensor that performs pixel readout for each line. For example, the number of divisions “3” is designated by default according to the frame rate of the MOS sensor. Increasing the number of divisions increases the calculation amount but increases the correction accuracy. The user can also specify the number of divisions. For example, when the user designates “image quality priority mode”, the number of divisions is set large.

  FIG. 5 is a diagram illustrating a change in the total processing time with respect to the number of divisions. FIG. 6 is a diagram illustrating a change in synthesis accuracy with respect to the number of divisions.

  Increasing the number of divisions can increase the correction accuracy, but also increases the amount of processing.

  As shown in FIG. 5, when the number of divisions increases n times, the processing amount required for motion detection increases n times, but the processing amount required for image synthesis does not change because the total number of pixels to be processed does not change (margin is reduced). If not). That is, even if the number of divisions is doubled, the total processing amount is less than doubled.

As a method for determining the optimum number of divisions, the value shown in the following equation (3) may be seen. The value shown in Equation (3) represents a pseudo frame rate at the time of division.
(Sensor frame rate) x (Number of divisions) (3)

  As shown in FIG. 6, it can be seen that the effect is saturated from a certain saturation point even if the above value is increased. The inventors have found that for normal camera shake, the number of divisions may be set so that the above value is about “15”. However, if large blurring is likely to occur due to the weight or shape of the optical system or camera body, a larger value must be set.

  Returning to FIG. 1, an image is captured a plurality of times by the imaging unit 101, and image data captured a plurality of times is stored in the image storage unit 102.

  The image division control unit 104 (1) changes the data read start position / end position for a plurality of image data in the image storage unit 102 based on the division number designated by the division number designation unit 103. (2) Further, based on the number of divisions designated by the number-of-divisions designation unit 103, the cut start position / end position of the composite image is changed. (3) Further, based on the number of divisions designated by the division number designation unit 103, the output image data writing start position and end position in the image storage unit 107 are changed. The read start position / end position of data changed based on the number of divisions, the cut start position / end position of the composite image, and the output image data write start position / end position are stored in the motion detection unit 105 and the image composition unit 106, respectively. Be notified. Details of the operation of the image division control unit 104 will be described later with reference to FIG.

  The motion detection unit 105 calculates the amount of movement between images from a plurality of image data stored in the image storage unit 102 and outputs the movement amount to the image composition unit 106.

  The image composition unit 106 composes an image based on the amount of movement between images obtained from the motion detection unit 105 and outputs it to the image storage unit 107, and the image storage unit 107 stores the composition result of the image composition unit 106. . Details of the operation of the image composition unit 106 will be described later with reference to FIG.

  FIG. 7 is a flowchart showing the operation of the image division control unit 104. This flow is executed by the CPU of the control unit that controls the entire imaging apparatus 100. In the figure, S indicates each step of the flow.

  When the operation of the image division control unit 104 starts, the read start / end positions are input to a plurality of images in the image storage unit 102 in step S11. The image data taken continuously is input from which address in the image storage unit 102 to start reading and from which address. Next, the image division number n determined by the division number designation unit 103 in step S12 is input.

  In step S13, the reading start / end positions in the divided i-th image storage unit 102 are determined and output for all input images. That is, the address at which the i-th location in the n-division starts and ends at the image storage unit 102 is output to the image composition unit 106. Next, in step S14, the cut start / end positions of the divided i-th composite image are determined and output for all input images. That is, the range in which the synthesized image is cut out after the synthesis of the i-th location during n division is completed is output to the image synthesis unit 106.

  In step S15, writing start / end positions in the divided i-th image storage unit 107 are determined and output for all input images. In other words, after completion of cutting out the i-th portion in the n-division, it is determined from which address of the image storage unit 107 the writing is to be started and the address is to be ended, and is output to the image composition unit 106.

  Next, in step S16, it is determined whether or not the processing has been completed in all n areas divided into n. If the processing is not completed, i is incremented (i = i + 1) in step S17 and the process returns to step S13. Then, the process proceeds to the next division point. When the process for n times is completed, this flow is finished.

  FIG. 8 is a flowchart showing the operation of the image composition unit 106. This flow is executed by the CPU of the control unit that controls the entire imaging apparatus 100.

  When the operation of the image composition unit 106 starts, in step S21, the read start / end positions in the image storage unit 102 are input for a plurality of input images. From which address in the image storage unit 102 the image data is read out is input. Next, in step S22, the cut start / end positions of the composite image are input to a plurality of input images, and the range in which the composite image is cut is designated.

  In step S23, the writing start / end positions in the image storage unit 107 are input to a plurality of input images. After the cutting is completed, the address at which the image storage unit 107 starts writing and the address at which the writing ends is input.

  Next, the movement amount of the nth image relative to the first image obtained by the motion detection unit 105 is input in step S24, and the image is synthesized by shifting by the movement amount in step S25. Next, the composite image is cut at the cut position specified in step S26, and the cut image is output to the writing position (address) of the image storage unit 107 specified in step S27, and this flow is finished.

  FIG. 9 is a diagram for explaining the MOS distortion reduction effect by the image dividing operation of the imaging apparatus 100, and is an example in which an image is divided into three in the readout line direction.

For simplicity, camera shake is limited to the horizontal direction (X-axis direction). In this case, three movement amounts _{xn, 1} , _{xn, 2} , _{xn, 3} are applied to the nth image, and three movement amounts _{xn + 1,1} , x are applied to the _{n + 1th} image. _{n + 1} , ₂ and _{xn + 1} , ₃ are obtained. In this way, the image is divided into three in the image reading line direction, and the amount of camera shake (movement amount) is calculated for each of the three divided portions. As can be seen from comparison with FIG. An area that cannot be corrected (see the hatched portion in FIG. 9) can be reduced. Therefore, camera shake correction can be performed while reducing the influence of the distortion of the MOS sensor.

  Although FIG. 9 shows an example in which the image is divided into three in the readout line direction, the correction accuracy can be increased by increasing the number of divisions, although the processing amount increases.

  As described above, by dividing the image in the readout line direction and obtaining the amount of camera shake (movement amount) for each divided portion, the influence due to the distortion of the MOS sensor could be reduced.

  When the affine transformation described above with reference to FIGS. 3 and 4 is applied to this method, a further camera shake correction effect can be obtained. This is because, in addition to the correction effect due to the affine transformation itself, this method employs image readout line direction division, making it possible to apply affine transformation to each division location and correcting non-linear distortion. by.

  Hereinafter, the effect of performing image composition using affine transformation in image synthesis in this method will be described.

  FIG. 10 is a diagram for explaining the limit of camera shake correction by affine transformation.

The camera shake correction references _{xn (Ln)} and _{xn + 1 (Ln + 1)} of the nth frame and the _{( n + 1 ) th} frame are expressed by the above equation (2). As shown in FIG. 10, when the amount of camera shake (distortion amount) generated nonlinearly is large, the generated distortion cannot be corrected by simply applying affine transformation. In the conventional example, this is the limit of linear transformation such as affine transformation.

  FIG. 11 is a diagram for explaining image division and camera shake correction by affine transformation according to the present method, and MOS distortion is a diagram corresponding to FIG. 10. This is an example in which an image is divided into two in the readout line direction.

Since the image is divided into two in the readout line direction and affine transformation is performed for each of the two divided portions, the nth frame and the (n + 1) th frame have two camera shake correction references _{xn, 1 (Ln)} , _{xn, 2 (Ln), respectively.} And x _{n + 1,1 (Ln + 1)} and x _{n + 1,2 (Ln + 1)} . n th frame image stabilization reference _{x n, 1 (Ln),} x n, 2 (Ln) is represented by the following formula (4), n + 1 th frame image stabilization reference _{x n + 1,1 (Ln + 1} ), x n + 1 _{, 2 (Ln + 1)} is expressed by the following equation (5).
_{xn, 1 (Ln)} = an _{, 1} * _{Ln , 1} + b _{n, 1}
x _{n, 2 (Ln)} = an _{, 2} * L _{n, 2} + b _{n, 2} (4)
x _{n + 1,1 (Ln + 1)} = an _{, 1} * L _{n, 1} + b _{n, 1
xn + 1,2 (Ln + 1)} = an _{+ 1,2} * _{Ln + 1,2} + b _{n + 1,2} (5)

  As shown in FIG. 11, although the affine transformation itself is a linear transformation, by using two affine transformations for a non-linearly distorted portion with respect to a non-linearly distorted portion, an image stabilization portion (hatching portion in FIG. 11) is used. Reference) can be greatly reduced. Compared with FIG. 10, the effect is clear.

  Thus, even when camera shake occurs non-linearly, if an image is divided in the readout line direction and affine transformation is used for each divided portion, high-precision correction can be performed.

  As described above, according to the present embodiment, the imaging apparatus 100 includes the imaging unit 101 that performs pixel readout for each line, the image storage unit 102 that stores image data captured a plurality of times, and the number of divisions that specifies the number of divisions. A designation unit 103, an image division control unit 104 that divides an image in the pixel readout line direction and controls each unit, and a motion detection unit 105 that calculates a movement amount between images from a plurality of image data in each of the divided images; An image combining unit 106 that combines the images based on the movement amount between the divided images and an image storage unit 107 that stores the combined result are divided into the readout line direction, and each divided portion is divided. Since each camera shake amount (movement amount) is obtained independently and image synthesis is performed using affine transformation, it is possible to reduce the influence of MOS distortion and improve the accuracy of camera shake correction. It is possible to prevent the failure of the image synthesis by S distortion.

  The present invention having such excellent features is preferably applied to an electronic still camera such as a digital camera, an imaging device in a video camera having a camera shake correction function, a mobile phone with a camera, and the like.

(Embodiment 2)
FIG. 12 is a block diagram showing a configuration of an imaging apparatus according to Embodiment 2 of the present invention. The same components as those in FIG. 1 are denoted by the same reference numerals, and description of overlapping portions is omitted.

  In FIG. 12, the imaging apparatus 200 is further provided with a detection accuracy determination unit 210. The determination output from the detection accuracy determination unit 210 is input to the division number specifying unit 103.

  The detection accuracy determination unit 210 determines that the accuracy of the movement amount calculated by the motion detection unit 105 is equal to or less than a predetermined threshold value.

  Hereinafter, the operation of the imaging apparatus 200 configured as described above will be described.

  There may be only a small amount of blur when shooting, and distortion may be small. In this case, the smaller the number of divisions, the smaller the processing amount. On the other hand, when large blurring occurs, it is preferable to increase the number of divisions.

  Therefore, in the present embodiment, the detection accuracy determination unit 210 determines whether the accuracy of the movement amount obtained by the motion detection unit 105 is equal to or less than a predetermined threshold value, and outputs the determination result to the division number designation unit 103. . The division number designation unit 103 designates the division number based on the determination result. The following two applications are possible for specifying the number of divisions.

  (1) The process is executed with a certain number of divisions to determine how accurately the image is synthesized, and the accuracy of motion detection is determined. If the accuracy of the movement amount obtained by the motion detection unit 105 is equal to or less than a reference value, the division is performed. Increase the number and perform synthesis again. In this case, according to the difference from the reference value of the accuracy of motion detection, the number of divisions may be increased if the difference is large.

  (2) The process is executed with a certain number of divisions, and the user confirms the result on the screen, and selects whether to save the image as it is or to increase the number of divisions and perform the synthesis process again.

  As described above, according to the present embodiment, when the correction accuracy obtained when calculating the amount of camera shake is equal to or less than the reference value, the correction accuracy is improved with a smaller processing amount by increasing the number of divisions and performing the synthesis process again. be able to.

  If the user sets the reference value in advance or the user confirms the result on the screen, it is possible to realize a synthesis process that matches the user's intention.

(Embodiment 3)
According to the first and second embodiments, it is possible to synthesize an image without MOS distortion by dividing the image in the reading line direction and performing movement amount calculation and screen composition for each divided part. In the present embodiment, an example in which a combined trace when a divided image is combined will be described.

  FIG. 13 and FIG. 14 are diagrams for explaining a method for dealing with a combined trace. As shown in FIG. 13, when two images are combined using affine transformation, a second frame 301 appears at the time of combining and becomes a combined mark. In this case, the frame 301 at the time of synthesis can be removed by cutting and outputting in the center area 302. However, in the first and second embodiments, since the images are divided and the screens are combined, as shown in FIG. 14, for example, if two images are combined in two parts, they are cut out in the center area. However, the frame 310 at the time of synthesis remains in the center as a synthesis trace.

  FIG. 15 is a diagram for explaining the removal of a combined trace when combining divided images. Take the case of two divisions as an example.

  In the present embodiment, a margin is given to the divided area so as not to leave a combined trace in each divided image.

  15A and 15B, the upper half divided area 401a and the lower half divided area 401b are divided with a margin with respect to the captured image 400. The output area 402 is equal between the upper half output area 402a and the lower half output area 402b.

  As shown in FIGS. 15 (a) and 15 (b), when the image is divided into two parts, the upper half divided area 401a and the lower half divided area 401b are not divided into two equal parts. The output area 402 in the center area is cut out and output at the time of output. FIG. 15C shows an output image output by cutting out the output area 402 in the center area. This eliminates the problem that the frame during synthesis remains in the center.

  In FIG. 15, if the cutting positions of the upper half and the lower half are shared by BB ′, it is possible to prevent a discontinuous image at the division boundary. If the output image 402 is stored in a memory area different from that of the input image, the upper half synthesis result does not affect the lower half.

  The above description is an illustration of a preferred embodiment of the present invention, and the scope of the present invention is not limited to this.

  Further, the present invention can be applied to any device as long as it is an electronic device having an imaging device and an image distortion correction method. For example, the present invention can be applied not only to an electronic still camera and a video camera, but also to an information processing apparatus such as a mobile phone with a camera, a portable information terminal such as a PDA (Personal Digital Assistants), and a personal computer equipped with an imaging device.

Further, any image sensor such as a CMOS sensor may be used as long as the image sensor performs pixel readout for each line. In CCD sensors where the charge accumulation timing of all pixels is the same,
Although there is no influence of MOS distortion, it can be applied to improve the accuracy of camera shake correction.

  In the above-described embodiments, the names of the imaging device and the image distortion correction method are used. However, this is for convenience of description, and it is needless to say that a camera system, a camera shake correction method, or the like may be used.

  Further, the type, number, connection method, and the like of each circuit unit constituting the imaging apparatus are not limited to the above-described embodiments.

  The imaging apparatus and the image distortion correction method described above are also realized by a program for causing the imaging apparatus and the image distortion correction method to function. This program is stored in a computer-readable recording medium.

  INDUSTRIAL APPLICABILITY The imaging apparatus and the image distortion correction method according to the present invention are useful as an imaging apparatus and an image distortion correction method that are mounted on an electronic still camera, a video camera, or the like that includes an imaging apparatus that corrects image distortion caused by camera shake.

1 is a block diagram showing a configuration of an imaging apparatus according to Embodiment 1 of the present invention. Diagram explaining the basic concept of the present invention Diagram explaining the principle of affine transformation FIG. 6 is a diagram for explaining camera shake correction by affine transformation of the imaging apparatus according to the first embodiment. The figure which shows the change of the total processing time with respect to the division | segmentation number of the imaging device which concerns on the said Embodiment 1. FIG. The figure which shows the change of the synthetic | combination precision with respect to the division | segmentation number of the imaging device which concerns on the said Embodiment 1. FIG. The flowchart which shows operation | movement of the image division control part of the imaging device which concerns on the said Embodiment 1. FIG. Flow chart showing the operation of the image composition unit of the imaging apparatus according to the first embodiment. The figure explaining the MOS distortion reduction effect by the image division | segmentation operation | movement of the imaging device which concerns on the said Embodiment 1. FIG. The figure explaining the limit of the camera-shake correction by the affine transformation of the imaging device according to the first embodiment The figure explaining the image division of the imaging device which concerns on the said Embodiment 1, and the camera shake correction by affine transformation FIG. 3 is a block diagram showing a configuration of an imaging apparatus according to Embodiment 2 of the present invention. The figure explaining the coping method with respect to the composite mark of the imaging device which concerns on the said Embodiment 2. The figure explaining the coping method with respect to the composite mark of the imaging device which concerns on the said Embodiment 2. The figure explaining the removal of the synthetic | combination trace when the divided image of the imaging device which concerns on the said Embodiment 2 is synthesize | combined. Block diagram showing the configuration of a conventional imaging device The figure explaining the electronic camera-shake correction of the conventional imaging device The figure explaining camera shake correction by a CCD sensor Diagram explaining camera shake correction by MOS sensor

Explanation of symbols

DESCRIPTION OF SYMBOLS 100,200 Image pick-up device 101 Image pick-up part 102 Image memory | storage part 103 Division number designation | designated part 104 Image division | segmentation control part 105 Motion detection part 106 Image composition part 107 Image preservation | save part 210 Detection accuracy determination part

Claims (13)

An image sensor that performs pixel readout for each line;
Image storage means for storing image data photographed a plurality of times by the imaging device;
Image dividing means for dividing an image in a pixel readout line direction of the image sensor;
In each of the divided images, motion detection means for calculating a movement amount between images from a plurality of image data stored in the image storage means;
An image pickup apparatus comprising: an image composition unit configured to compose an image based on a movement amount between each of the divided images obtained from the motion detection unit.   The imaging apparatus according to claim 1, wherein the image dividing unit divides the image with a margin so that the divided images overlap a predetermined area.   The imaging apparatus according to claim 1, wherein the image composition unit performs image composition using affine transformation.   The imaging apparatus according to claim 1, further comprising division number designation means for designating a division number for dividing the image in the pixel readout line direction.   The imaging apparatus according to claim 3, wherein the division number designating unit designates the division number by a specified value based on a frame rate of the imaging element or a user set value.   The imaging apparatus according to claim 1, further comprising an image synthesis accuracy determination unit that determines the accuracy of image synthesis based on a movement amount between the images.   The imaging apparatus according to claim 6, wherein the image dividing unit determines the number of divisions based on determination accuracy by the image synthesis accuracy determination unit.   The imaging apparatus according to claim 6, wherein the image dividing unit increases the number of divisions when the determination accuracy by the image synthesis accuracy determination unit is a predetermined reference value or less. A step of photographing a plurality of times by an image sensor that performs pixel readout for each line;
Storing image data taken multiple times;
Dividing an image in a pixel readout line direction of the image sensor;
Calculating the amount of movement between images from a plurality of image data stored in the image storage means in each of the divided images;
And a step of synthesizing the images based on the movement amount between the divided images.   The image distortion correction method according to claim 9, wherein in the dividing step, the divided image is divided with a margin so as to overlap a predetermined area.   The image distortion correction method for an imaging apparatus according to claim 9, wherein the image composition step performs image composition using affine transformation. Furthermore, it has a step of determining the accuracy of image composition based on the amount of movement between images,
The image distortion correction method according to claim 9, wherein in the division step, the number of divisions is increased when determination accuracy is equal to or lower than a predetermined reference value.   A program for causing a computer to execute the steps according to claim 9.

Images