JP2015154334A - Imaging apparatus, control method thereof and control program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To effectively correct blur in an image by accurately detecting a motion vector.SOLUTION: Image reduction circuits 107, 108 respectively adopt one of images as an original image, use images photographed before the original image as reference images and respectively perform resize processing of the original image and the reference images to obtain a first image and a second image. Under control of a microcomputer 112, a reduction rate determination circuit 106 controls a resize rate in the image reduction circuits in accordance with a focal distance of an optical system. A motion vector detection circuit 109 detects a motion vector between the first image and the second image, and a geometrical deformation parameter estimation circuit 110 and a geometrical deformation circuit 111 correct blur in the original image in accordance with the motion vector.

Description

  The present invention relates to an imaging apparatus, a control method thereof, and a control program, and more particularly, to a technique for correcting image quality degradation caused by blurring of the imaging apparatus.

  In recent years, in an imaging apparatus such as a digital video camera or a digital camera, it has become possible to obtain a high-quality image by improving an imaging sensor (imaging element) and signal processing technology. On the other hand, the image quality may deteriorate in an image obtained as a result of imaging due to a change in posture of the imaging device (for example, blurring). In order to prevent such deterioration of image quality, it is necessary to correct the posture change (that is, shake) of the imaging apparatus in the image obtained as a result of imaging.

  With the downsizing of imaging devices, the need to correct posture changes has become extremely high. When acquiring stable images and combining multiple images, regardless of whether they are movies or still images It is indispensable to correct the posture change.

  By the way, when obtaining a posture change of the image pickup apparatus, a sensing device such as an angular velocity sensor (for example, a gyro sensor) or an acceleration sensor is generally used. Furthermore, instead of using a sensing device, a so-called motion vector detection method that detects a motion of a subject according to an input image and a past reference image to obtain a posture change may be used.

  When performing motion vector detection processing in real time in a shooting situation with high time constraints such as video shooting or continuous shooting of still images, processing time, signal processing circuit scale, memory, memory bus bandwidth, etc. There are some restrictions. Under such restrictions, it is difficult to stably and sequentially acquire a large number of motion vectors with high accuracy by processing with a high resource load.

  In order to deal with such a problem, for example, an imaging apparatus is known that reduces the processing load by controlling the reduction rate of image reduction processing, which is preprocessing for motion vector detection (Patent Document 1). reference).

JP 2010-252259 A

  In the above-mentioned Patent Document 1, when performing inter-frame compression encoding such as MPEG or AVCHD, the degree of influence of image quality degradation at the time of encoding due to a motion vector error is estimated based on the size of a prediction vector, etc. Controls the reduction ratio in motion vector search.

  However, when correcting a motion blur by detecting a plurality of motion vectors from a plurality of areas in an image, it is necessary to control a reduction rate of the image used for motion vector search. For example, it is necessary to control the reduction ratio so as to satisfy the error level of the motion vector and the number of detected vectors in accordance with the posture change component of the imaging device and the shooting situation.

  If the reduction rate is not appropriately controlled according to the posture change component and the shooting situation, it is difficult to accurately detect the motion vector, and as a result, it is not possible to correct the blur in the image satisfactorily.

  Accordingly, an object of the present invention is to provide an imaging apparatus, a control method thereof, and a control program capable of accurately detecting a motion vector in an image and correcting a blur in the image satisfactorily.

  In order to achieve the above object, an image pickup apparatus according to the present invention is an image pickup apparatus that includes an optical system having a variable focal length and that captures a plurality of temporally continuous images via the optical system. Resizing means that uses one of the original image as an original image, an image taken before the original image as a reference image, and resizes the original image and the reference image to form a first image and a second image, respectively. Control means for controlling the resizing rate in the resizing means in accordance with the focal length of the optical system, and motion vector detecting means for detecting a motion vector between the first image and the second image. And blur correction means for correcting blur of the original image in accordance with the motion vector.

  A control method according to the present invention is a control method for an imaging apparatus that includes an optical system having a variable focal length, and that captures a plurality of temporally continuous images via the optical system. A resizing step to resize the original image and the reference image into a first image and a second image, respectively, using the image taken before the original image as a reference image, and the optical system; A control step for controlling a resizing rate in the resizing process performed in the resizing step according to a focal length of the image, and a motion vector detecting step for detecting a motion vector between the first image and the second image, And a blur correction step of correcting blur of the original image in accordance with the motion vector.

  A control program according to the present invention is a control program used in an imaging apparatus that includes an optical system having a variable focal length and captures a plurality of temporally continuous images via the optical system. The computer uses one of the images as an original image, an image taken before the original image as a reference image, and resizes the original image and the reference image, respectively. An image resizing step, a control step of controlling a resizing rate in the resizing process performed in the resizing step in accordance with a focal length of the optical system, and between the first image and the second image. A motion vector detection step for detecting a motion vector and a blur correction step for correcting blur of the original image according to the motion vector are executed. And characterized in that.

  According to the present invention, since the resizing rate of the original image and the reference image, which are images for motion vector detection, is controlled according to the focal length of the optical system, the motion vector can be detected with high accuracy. As a result, it is possible to generate an image in which blurring is corrected favorably.

1 is a block diagram illustrating a configuration of an example of an imaging apparatus according to a first embodiment of the present invention. 3 is a flowchart for explaining a photographing operation in the camera shown in FIG. 1. It is a figure which shows typically the relationship between the reduction rate set by the reduction rate determination circuit shown in FIG. 2, and the number of detected motion vectors. It is a figure which shows typically the relationship between the reduction rate set by the reduction rate determination circuit shown in FIG. 2, and the precision of the detected motion vector. It is a figure which shows collectively the relationship between a reduction rate, the number of remaining vectors, and vector accuracy based on the relationship shown in FIG. 3 and FIG. It is a figure for demonstrating the relationship between the focal distance of a lens, and a reduction rate, (a) is a figure which shows an example of the relationship between the focal distance of a lens, and a reduction rate, (b), the focal length of a lens, and a reduction rate It is a figure which shows the other example of a relationship. It is a figure for demonstrating an example of the relationship between the image acquired as a result of imaging, and a motion vector, (a) is a figure which shows the relationship between the captured image and motion vector when a focal distance is short (wide end), (B) is a figure which shows the relationship between the captured image and motion vector when a focal distance is long (tele end). It is a figure for demonstrating an example of the relationship between the focal distance at the time of changing the aperture value (F value) or tilting coefficient of a lens, and a reduction ratio, (a) is changing the aperture value (F value) of a lens. FIG. 5B is a diagram showing the relationship between the focal length and the reduction ratio when changing the tilt coefficient. FIG. FIG. 2 is a diagram for explaining detection of a motion vector by template matching performed by the motion vector detection circuit shown in FIG. 1, (a) is a diagram showing an original image, and (b) is a diagram showing a reference image. It is a block diagram which shows the structure about an example of the camera by the 2nd Embodiment of this invention. 11 is a flowchart for explaining an example of a photographing operation performed by the camera shown in FIG. It is a block diagram which shows the structure about an example of the camera by the 3rd Embodiment of this invention. It is a figure which shows the relationship between the focal distance at the time of changing the amount of angular blurring in the camera shown in FIG. 12, and a reduction rate. It is a block diagram which shows the structure about an example of the camera by the 4th Embodiment of this invention.

  Hereinafter, an example of an imaging apparatus according to an embodiment of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram showing the configuration of an example of an imaging apparatus according to the first embodiment of the present invention.

  The illustrated imaging apparatus is, for example, a digital camera (hereinafter simply referred to as a camera), and the camera includes an imaging optical system (hereinafter simply referred to as an optical system) 101. The optical system 101 is an optical system having a variable focal length, and the camera captures a plurality of temporally continuous images via the optical system 101.

  A subject image (optical image) that has passed through the optical system 101 is formed on the image sensor 102. The image sensor 102 is a CCD sensor or a CMOS sensor, and outputs an electrical signal (analog signal) corresponding to the optical image to the camera signal processing circuit 103.

  The camera signal processing circuit 103 includes, for example, an A / D conversion circuit, an auto gain control circuit (AGC), and an auto white balance circuit (AWB). The camera signal processing circuit 103 is an analog signal (that is, a camera Signal) is converted into a digital signal by A / D conversion, and then a predetermined image process is performed on the digital signal to generate a video signal (also referred to as image data). The optical system 101, the image sensor 102, and the camera signal processing circuit 103 constitute an image capturing system that acquires image data.

  The video signal output from the camera signal processing circuit 103 is sent to the memory 104, and the memory 104 temporarily holds or stores one frame or a plurality of frames (that is, frame images) of the video signal. The reduction rate determination circuit 106 determines a reduction rate when reducing the frame image as described later. The image reduction circuits 107 and 108 perform a reduction process on the frame image input from the memory 104 based on the reduction rate determined by the reduction rate determination circuit 106, respectively, so that the first reduced image and the second reduced image are respectively received. Output a reduced image.

  The motion vector detection circuit 109 detects a motion vector between the two frame images (that is, the first reduced image and the second reduced image) input from the image reduction circuits 107 and 108. Based on the motion vector detected by the motion vector detection circuit 109, the geometric deformation parameter estimation circuit 110 outputs a motion correction amount of blur between frame images as a geometric deformation parameter.

  The microcomputer 112 controls the geometric deformation circuit 111 according to the geometric deformation parameter, and performs a geometric deformation process for correcting blurring on the frame image input from the memory 104 by the geometric deformation circuit 111. The microcomputer 112 controls the entire camera.

  FIG. 2 is a flowchart for explaining a photographing operation in the camera shown in FIG. Note that the processing according to the illustrated flowchart is performed under the control of the microcomputer 112.

  When the photographing operation is started, a subject image is formed on the image sensor 102 by the optical system 101 under the control of the microcomputer 112. Then, an analog signal that is an output of the image sensor 102 is supplied to the camera signal processing circuit 103, and the camera signal processing circuit 103 performs predetermined image processing on the digital signal obtained by A / D converting the analog signal. To generate a video signal (step S201: image input).

  The camera signal processing circuit 103 converts an analog signal into, for example, a 12-bit digital signal by an A / D conversion circuit. The camera signal processing circuit 103 corrects the signal level using AGC and AWB and also performs white level correction to generate a video signal. This video signal is held in the memory 104.

  In the illustrated camera, frame images are sequentially generated at a predetermined frame rate, and the frame images are held in the memory 104. Then, the same size frame image held in the memory 104 is input to the image reduction circuits 107 and 108. Note that the frame images held in the memory 104 are sequentially updated.

  Subsequently, under the control of the microcomputer 112, the reduction ratio determination circuit 106 sets the determined image reduction ratio in the image reduction circuits 107 and 108 (step S202). The reduction rate determination method performed by the reduction rate determination circuit 106 will be described with reference to FIGS. 3 to 6 described later. Here, the relationship between the reduction rate and the accuracy and number of detected motion vectors will be described. To do.

  FIG. 3 is a diagram schematically showing the relationship between the reduction rate set by the reduction rate determination circuit 106 shown in FIG. 2 and the number of detected motion vectors.

  In FIG. 3, the horizontal axis indicates the reduction ratio, and the larger the reduction ratio, the larger the size of the image after reduction. When the reduction ratio = 1, the image is not reduced. The vertical axis represents the number of motion vectors determined to have high reliability in the detected motion vectors (this is the number of remaining vectors). Here, as the reliability of the motion vector, for example, there is one determined depending on what kind of texture the motion vector is detected from.

  For example, a low-contrast region or a region including a repetitive pattern is a region where it is difficult to detect a motion vector by template matching, and therefore there is a low possibility that a motion vector is correctly detected. In order to perform such texture determination, here, an average value and a variance value are obtained for the luminance values of the pixels in the template, and it is determined whether or not the area is a low contrast area or a repeated pattern area.

  If it is determined that the region is a low-contrast region or a repetitive pattern region, motion vectors detected in the region are excluded as having low reliability. Such a determination is performed on all the motion vectors, and the number of highly reliable motion vectors remaining finally becomes the number of remaining vectors.

  Referring to FIG. 3, when the reduction ratio is close to 1, that is, when the image size used for motion vector detection is large, a tendency to generate a plurality of portions having high correlation with the template is increased, resulting in erroneous motion. Vectors are easier to detect. These erroneous motion vectors cannot be used because they cause errors. Furthermore, the number of remaining vectors decreases because the texture in the template becomes poor and the number of motion vectors that are determined to be low-contrast regions and are excluded increases.

  On the other hand, when the reduction ratio is close to 0, that is, when the image size used for motion vector detection is small, the texture in a wide area of the image enters the template. For this reason, it becomes difficult to be excluded by low contrast or repeated determination, and the number of remaining vectors increases.

  FIG. 4 is a diagram schematically showing the relationship between the reduction rate set by the reduction rate determination circuit 106 shown in FIG. 2 and the accuracy of the detected motion vector.

  In FIG. 4, the horizontal axis represents the reduction ratio, and the vertical axis represents the motion vector detection accuracy. Here, when the reduction ratio is close to 1, that is, when the image size used for motion vector detection is large, the accuracy of each motion vector is improved since the image is clear. On the other hand, when the reduction ratio is close to 0, the image becomes unclear due to the low-pass filter effect by the reduction process, and the detection accuracy of each motion vector is lowered.

  It should be noted that the image size is not limited to any extent, and is limited to the reduction rate as shown by the reduction rate limit 501 in FIG. 4 so that the accuracy below the detection limit can be maintained even if the detection accuracy is lowered. It is necessary to provide. Here, the detection limit is a limit of accuracy that can be regarded as the same as a correct motion vector. For example, when a geometric deformation of an image is performed using a motion vector, the geometric deformation is performed using the correct motion vector. The detection accuracy is such that the difference from the observed image cannot be visually confirmed.

  Further, when calculating the geometric deformation parameter, it is necessary to estimate not only the movement of the so-called translational component but also the movement of the rotation component and the so-called tilting component. For this reason, it is necessary to detect many motion vectors from the entire image, and a state in which the reduction rate is close to 0 is a standard state.

  FIG. 5 is a diagram collectively showing the relationship between the reduction ratio, the number of remaining vectors, and the vector accuracy based on the relationships shown in FIGS. 3 and 4.

  From FIG. 5, when properly using a case where a large number of motion vectors are detected with standard accuracy and a case where a motion vector is detected with high accuracy even with a small number, the size of an image used for detecting a motion vector may be changed. I understand that.

  Up to this point, it has been described how the accuracy and remaining number of motion vectors detected by changing the reduction ratio of the image change. Subsequently, the focal length of the lens provided in the optical system 101 is changed. A description will be given of how the reduction ratio should be changed accordingly.

  FIG. 6 is a diagram for explaining the relationship between the focal length of the lens and the reduction ratio. 6A shows an example of the relationship between the focal length of the lens and the reduction rate, and FIG. 6B shows another example of the relationship between the focal length of the lens and the reduction rate. FIG.

  In FIG. 6, the horizontal axis indicates the focal length, and the vertical axis indicates the reduction ratio. As the reduction ratio increases, the image size increases. In FIG. 6, as the focal length becomes longer, the reduction ratio is increased to increase the image size. In the example shown in FIG. 6A, the relationship between the focal length and the reduction rate is changed according to a linear function. In the example shown in FIG. 6B, the relationship between the focal length and the reduction rate is 2. It is changed according to the next function.

  In consideration of the relationship shown in FIG. 6, when the reduction ratio is close to 1, that is, when the image size used for motion vector detection is large, the motion vector is detected with a clear image. The accuracy of is improved. In addition, there is a strong tendency to generate a plurality of locations having high correlation with the template, and erroneous motion vectors are easily detected. Incorrect motion vectors cannot be used because they cause errors. Furthermore, the number of remaining vectors decreases because the texture in the template becomes poor and the number of motion vectors that are determined to be low-contrast regions and are excluded increases.

  On the other hand, when the reduction ratio is close to 0, that is, when the image size used for detecting the motion vector is small, the motion vector is detected in an unclear image, so the accuracy of the detected motion vector is lowered. In addition, since a texture of a wide area of the image enters in the template, it becomes difficult to be excluded by low contrast or repeated determination, and the number of remaining vectors increases.

  FIG. 7 is a diagram for explaining an example of a relationship between an image obtained as a result of imaging and a motion vector. FIG. 7A is a diagram showing the relationship between the captured image and the motion vector when the focal length is short (wide end), and FIG. 7B is the captured image when the focal length is long (tele end). It is a figure which shows the relationship between a motion vector.

  In FIGS. 7A and 7B, it is assumed that the movement of the camera at the time of shooting is the same. Here, it is assumed that the camera moves from the bottom to the top. As shown in FIG. 7A, when the image is taken at the wide end, the motion vector increases at the top of the screen, and the motion vector appears as a motion that decreases toward the lower side of the screen. On the other hand, as shown in FIG. 7B, when the image is taken at the tele end, the detected motion vector appears as a translational component motion even for the same camera motion.

  Therefore, when the focal length is short as shown in FIG. 7A, not only the translation but also the tilt appears remarkably. Therefore, when estimating the motion, the motion vector in a partial region of the image. It is necessary to detect a large number of motion vectors from the entire image without detecting.

  For this reason, when the image is taken at the wide end, the size of the image used for detecting the motion vector is increased by increasing the image reduction ratio, for example, a size of 1/8 size, and a large number of motion vectors are obtained from the entire image. To detect. In the illustrated example, the case of tilt movement has been described, but the same applies to rotation or enlargement / reduction movement, and it is necessary to detect a large number of motion vectors from the entire image.

  On the other hand, as shown in FIG. 7B, when the focal length is long, the translational movement is dominant in the movement, and even if the movement of the camera includes rotation and tilt movement, the result of imaging is obtained. The motion that occurs in the captured image can be approximated as translation. In this case, since it is not necessary to detect a large number of motion vectors from the entire image, the number of motion vectors to be detected may be small.

  However, since the amount of pixel movement that occurs at the tele end is larger than that at the wide end, blurring must be corrected with high accuracy. Therefore, when the image is taken at the tele end, the image size used for motion vector detection with a reduced image reduction rate is reduced to a high number of vectors with a small number of vectors, for example, the same size or 1/2 size. What is necessary is just to detect a motion vector accurately.

  Note that the change in the reduction ratio with respect to the focal length is not limited to the example shown in FIG. 7. For example, the relationship between the focal length and the reduction ratio may be changed according to the aperture value (F value) or tilt coefficient of the lens. Good.

  FIG. 8 is a diagram for explaining an example of the relationship between the focal length and the reduction ratio when the aperture value (F value) or tilt coefficient of the lens is changed. FIG. 8A is a diagram showing the relationship between the focal length and the reduction ratio when the lens aperture value (F value) is changed, and FIG. 8B is the graph when the tilt coefficient is changed. It is a figure which shows the relationship between a focal distance and a reduction rate.

  First, referring to FIG. 8A, here, the control diagram which is the relationship between the focal length and the reduction ratio is changed according to the aperture value (F value) of the lens. When the F value is large, the depth of field becomes deep. Therefore, if the entire image is shaken, it can contribute to the vibration-proofing effect. In this case, not only the translation component but also the rotation component and the tilt component are emphasized, and the reduction rate is increased so that these motions can be detected, that is, the image size is reduced and a large number of motions from the entire image. Detect vectors.

  On the other hand, when the F value is small, the depth of field becomes shallow, so if the subject that is in focus than the entire image is shaken, it can contribute to the anti-shake effect. In this case, considering a local region in the imaging scene, it can be regarded as a translational motion. Furthermore, when the depth of field is reduced, the photographer uses the focused area as the area of interest, so it is necessary to improve the blur correction accuracy. Therefore, the reduction rate is reduced, that is, the image size is increased, and the motion vector is detected with high accuracy even when the number of motion vectors is small.

  Next, referring to FIG. 8B, here, the control diagram showing the relationship between the focal length and the reduction ratio is changed according to the tilt coefficient. When the deformation of the image is expressed by a homography matrix described later, the movement of the tilt component has a correlation with the magnitude of the tilt coefficient described later. If only the amount of translational deformation is used, the tilt coefficient becomes a value close to 0, and if the tilt component increases, the tilt coefficient becomes larger than 0.

  Therefore, when the tilt coefficient is large, the reduction rate is increased so that the motion can be detected, that is, the motion vector is detected by reducing the image size. On the other hand, when the tilt coefficient is small, it is only necessary to detect the translational motion, so the motion vector is detected by reducing the reduction rate, that is, increasing the image size.

  As described above, under the control of the microcomputer 112, that is, according to the focal length of the optical system 101 given from the microcomputer 112, the reduction ratio determination circuit 106 determines the reduction ratio of the image, and determines the determined reduction ratio. This is given to the image reduction circuits 107 and 108.

  Subsequently, each of the image reduction circuits 107 and 108 reduces the frame image held in the memory 104 (step S203). Here, frame images used for motion vector detection from a plurality of frame images held at the same size in the memory 104 in step S201 are input to the image reduction circuits 107 and 108 as original images and reference images, respectively. The image reduction circuits 107 and 108 reduce the original image and the reference image, respectively, based on the set reduction rate in the reduction rate setting circuit 106, and output a first image and a second image.

  The method for reducing the image is not particularly limited. For example, in the case of 1/8 reduction processing, a pixel average method for calculating an average value of pixels in an 8 × 8 rectangular area or a central pixel is used. A method using a reduction filter that performs weighting is used.

  Subsequently, the motion vector detection circuit 109 detects a motion vector between two frame images (that is, the first image and the second image) (step S204). For example, the motion vector detection circuit 109 detects a motion vector using template matching.

  FIG. 9 is a diagram for explaining the detection of a motion vector by template matching performed by the motion vector detection circuit 109 shown in FIG. FIG. 9A shows an original image, and FIG. 9B shows a reference image.

  The illustrated original image 901 and reference image 902 are images obtained by reducing the same-size frame image held in the memory 104. When performing template matching, a template block 903 is arranged at an arbitrary position in the original image 901. Then, a correlation value between the template block 903 and the reference image 902 is calculated.

  In this case, if the correlation value is calculated for all the regions of the reference image 902, the amount of calculation becomes enormous. Therefore, a rectangular area is set as the search range 904 in the reference image 902, and a correlation value is calculated for the search range 904. Although the position and size of the search range 904 are not particularly limited, the motion vector cannot be correctly detected unless the search range 904 includes an area corresponding to the destination of the template block 903.

  In the illustrated example, when the correlation value is calculated, for example, a correlation value S_SAD represented by Expression (1) is obtained by using a sum of absolute difference (hereinafter referred to as SAD).

In equation (1), f (i, j) indicates the luminance value at the coordinates (i, j) of the template block 903, and g (i, j) is the correlation value calculation target of the block 905 in the search range 904. Indicates the luminance value. Then, the absolute value of the difference is calculated for the luminance values f (i, j) and g (i, j) of the blocks 903 and 905, and the sum is obtained to obtain the correlation value S_SAD.

  The smaller the correlation value S_SAD, the smaller the difference in luminance value between the blocks 903 and 905. That is, the textures of the template block 903 and the correlation value calculation block 905 are similar.

  Here, the SAD is used when obtaining the correlation value. However, for example, the correlation value may be obtained using a sum of squared differences (SSD) or a normalized cross-correlation (NCC). When calculating a correlation value using SAD, there are cases where the similarity is higher as the correlation value is smaller depending on the characteristic, and the case where the similarity is higher as the correlation value is larger. Also need to be changed.

  The correlation value target block 905 is moved for the entire region of the search range 904, and the correlation value is calculated as described above. Then, by calculating a correlation value between the template block 903 and the search range 904 and determining a position where the correlation value is the smallest, the template block 903 of the original image 901 is located at any position in the reference image 902. The movement vector, that is, the motion vector between images is detected.

  The motion vector detection circuit 109 performs the above-described motion vector detection processing for a plurality of regions between the frame images, and sends the detected motion vector group to the geometric deformation parameter estimation circuit 110.

  Referring to FIG. 2 again, subsequently, the geometric deformation parameter estimation circuit 110 estimates the geometric deformation parameters between the frame images (that is, here, the original image and the reference image) based on the motion vector group (step S205). Here, an example in which a homography matrix is used as an image deformation amount will be described as a geometric deformation model.

  Now, it is assumed that a certain point a on the image represented by Expression (2) has moved to a point a ′ represented by Expression (3) in the next frame.

Here, the subscript T indicates a transposed matrix.

  By using the homography matrix H, the correspondence relationship between the point a shown in Expression (2) and the point a ′ shown in Expression (3) can be expressed by Expression (4).

The homography matrix H is a determinant representing the amount of deformation caused by translation, rotation, scaling, shearing, and tilting between images, and is represented by Expression (5).

Note that h ₃₁ and h ₃₂ are tilt coefficients.

  Each element of the homography matrix H can be calculated using a statistical process such as a least square method using the motion vector group obtained in step S204, that is, the correspondence of representative points between frame images.

  However, since the motion vector detected using the reduced frame image is a motion vector related to the reduced image, it is necessary to convert the motion vector to the same size when estimating the geometric deformation parameter. .

The homography matrix H obtained as described above indicates the amount of deformation of the image due to camera shake. Therefore, in order to correct image blurring, it is necessary to convert the homography matrix H so as to obtain an image deformation amount that cancels the deformation caused by the blurring. That is, by converting the homography matrix H to the inverse matrix H- ¹ , the correspondence between the point a 'and the point a can be obtained by Expression (6).

Using the equation (6), the point a ′ after the blurring can be returned to the same coordinates as the point a before the blurring.

  Here, the homography matrix has been described as an example of the model representing the blurring amount between the frame images. However, instead of the homography matrix H, another model such as a Helmat matrix or an affine matrix is used. Also good. Furthermore, when the main subject is set as the image stabilization target, it is only necessary to obtain the blur correction amount in translation, and if the average value of the motion vector group is simply obtained, the calculation amount can be reduced without performing statistical estimation. be able to.

  After estimating the geometric deformation parameter as described above, the geometric deformation parameter estimation circuit 110 sends the geometric deformation parameter to the microcomputer 112. Then, according to the geometric deformation parameter given from the microcomputer 112, the geometric deformation circuit 111 performs a geometric transformation process on the frame image input from the memory 104 to obtain an image subjected to the image stabilization process (step S206). . The geometric deformation circuit 111 stores the image subjected to the image stabilization processing in a storage device (not shown) and displays it on a display device (not shown) (step S207). Thereafter, the microcomputer 112 ends the photographing operation.

  As described above, in the first embodiment of the present invention, the reduction ratio of an image used for detecting a motion vector is changed according to an object for which blurring is to be corrected in a shooting scene. As a result, it is possible to generate a video in which the blur correction process has been satisfactorily performed on a specified target even with limited resources.

  As described above, the reduction rate may be set according to the focal length of the currently captured frame, but the reduction rate set in the past frame, that is, before the original image is captured. A reduction rate obtained by smoothing a plurality of set reduction rates in the time direction may be set.

  Furthermore, the reduction ratio may be changed in accordance with the aperture value obtained by smoothing a plurality of aperture values set in the time direction before capturing the original image. The reduction ratio may be changed according to the geometric deformation parameter obtained by smoothing a plurality of set geometric deformation parameters in the time direction.

[Second Embodiment]
Next, an example of a camera according to the second embodiment of the present invention will be described.

  FIG. 10 is a block diagram showing the configuration of an example of a camera according to the second embodiment of the present invention. In FIG. 10, the same components as those of the camera shown in FIG.

  In the camera according to the second embodiment, a frame image subjected to the reduction process is also stored in the memory 104 so as to reduce a bandwidth when the image is transmitted to the memory 104. The illustrated camera includes an image resizing circuit 1001 instead of the image reducing circuit 108, and a frame image is given to the image reducing circuit 107 from the camera signal processing circuit 103.

  FIG. 11 is a flowchart for explaining an example of a photographing operation performed by the camera shown in FIG.

  Note that the processing according to the illustrated flowchart is performed under the control of the microcomputer 112. In the illustrated flowchart, the same steps as those in the flowchart shown in FIG.

  When the photographing operation is started, a frame image that is an output of the camera signal processing circuit 103 is sent to the memory 104 and also sent to the image reduction circuit 107 in step S201. In step S202, the reduction rate determination circuit 106 sets the reduction rate in the image reduction circuit 107 and the image resizing circuit 1001.

  Thereafter, in the process of step S203, the image reduction circuit 107 reduces the frame image according to the set reduction rate to obtain a reduced image. Then, the reduced image is sent from the image reduction circuit 107 to the memory 104 and stored in the memory 104.

  Subsequently, the image resizing circuit 1001 performs resizing processing on the reduced image stored in the memory 104 (step S1101). The size of the reduced image stored in the memory 104 corresponds to the reduction rate set by the reduction rate determination circuit 106 when the frame image is input to the image reduction circuit 107.

  On the other hand, when the reduction ratio is changed according to the temporal variation of the shooting scene as described above, between the reduced image already stored in the memory 104 and the image that has been reduced after the reduction ratio is changed. Differences in image size occur. For this reason, the image size may differ between the original image and the reference image, and the motion vector cannot be detected if the image sizes differ. For this reason, in order to match the image sizes of both (that is, the original image and the reference image), the image resizing circuit 1001 applies the reduced image input from the memory 104 in accordance with the reduction ratio set by the reduction ratio determination circuit 106. Resizing processing is performed on the image.

  For example, it is assumed that the set reduction ratio at the time when a frame image is input to the image reduction circuit 107 is 1/8 reduction. When the reduction ratio is changed to 1/4 reduction according to the temporal variation of the shooting scene, the image resizing circuit 1001 performs a double enlargement process on the reduced image stored in the memory 104, and The image processed outside the cabinet is sent to the motion vector detection circuit 109.

  As a result, the image output from the image reduction circuit 107 and the image output from the image resizing circuit 1001, that is, the original image and the reference image are both reduced to ¼, and the motion vector detection circuit 109 performs the above-described operation. In this way, motion vector detection can be performed.

  Note that the method of resizing processing performed by the image resizing circuit 1001 is not particularly limited, and for example, bilinear or bicubic interpolation processing is used.

  Thereafter, in step S204, a motion vector is detected by the motion vector detection circuit 109, and in step S205, the geometric deformation parameter estimation circuit 110 estimates the geometric deformation parameter as described above. In step S <b> 206, the geometric deformation circuit 111 generates an image subjected to blur correction by performing image geometric deformation processing according to the geometric deformation parameter on the frame image of the same size stored in the memory 104. Thereafter, in step S207, an image is output.

  In the first embodiment described above, only the same size frame image is stored in the memory 104. As a result, the image reduction process is performed on the same size frame image every time the set reduction rate changes. Become. For this reason, a frame image of the same size is always sent in the memory 104, and the required bandwidth becomes large.

  On the other hand, in the second embodiment, the image is reduced and stored in the memory 104 at a set reduction rate when the frame image is input to the image reduction circuit 107. Then, the reduced image stored in the memory 104 is resized according to the change in the reduction ratio. As a result, a reduced image is sent to the memo 104, and the band (transmission band) can be reduced.

[Third Embodiment]
Next, an example of a camera according to the third embodiment of the present invention will be described.

  FIG. 12 is a block diagram showing the configuration of an example of a camera according to the third embodiment of the present invention. In FIG. 12, the same components as those of the camera shown in FIG. 1 are denoted by the same reference numerals and description thereof is omitted.

  The illustrated camera has posture change detection units 120 to 122, detects the posture change of the camera in three axis directions, and sends the detection result (hereinafter referred to as posture change information) to the microcomputer 112. The illustrated posture change detection units 120 to 122 are, for example, gyro sensors. Here, the gyro sensor 120 detects a posture change in the yaw direction, and the gyro sensor 121 detects a posture change in the pitch direction. The gyro sensor 122 detects a change in posture in the roll direction.

  The microcomputer 112 detects the amount of camera shake based on the posture change information. Then, the microcomputer 112 changes the control diagram indicating the relationship between the focal length and the reduction ratio in accordance with the amount of angular blur.

  FIG. 13 is a diagram showing the relationship between the focal length and the reduction ratio when the amount of angular blur is changed in the camera shown in FIG.

  Here, the control diagram which is the relationship between the focal length and the reduction ratio is changed in accordance with the amount of camera shake. The outputs of the gyro sensors 120 to 122 indicate angular changes per unit time (that is, angular velocities). Accordingly, the microcomputer 112 integrates each of these angular velocities to obtain an angular blur amount.

  When the amount of angular blur is large, the motion vector to be detected becomes large. For this reason, it is necessary to widen the search range for motion vector detection. However, since the amount of calculation increases when the search range is expanded, the motion vector is detected by increasing the reduction ratio, that is, reducing the image size.

  On the other hand, when the amount of angular blur is small, the motion vector to be detected is small. For this reason, the search range for motion vector detection may be narrowed. In this case, the motion vector is detected by reducing the reduction ratio, that is, increasing the image size.

  As described above, the reduction rate may be set for the focal length of the frame image obtained by the current imaging, but the angle blur detected in the past frame image is smoothed in the time direction to reduce the reduction rate. The rate may be set.

  Since the outputs of the gyro sensors 120 to 122 include changes due to camera work such as pan and tilt, high-pass filter processing is performed on the outputs of the gyro sensors 120 to 122 in order to remove these components. Also good.

  As described above, in the third embodiment of the present invention, a posture change detection unit such as a gyro sensor is provided, and the control diagram indicating the relationship between the focal length and the reduction ratio is changed according to the amount of camera shake. To do. As a result, in addition to the effects of the first embodiment described above, it is possible to detect the motion vector by accurately selecting the relationship between the focal length and the reduction ratio according to the amount of camera shake.

[Fourth Embodiment]
Next, an example of a camera according to the fourth embodiment of the present invention will be described.

  FIG. 14 is a block diagram showing the configuration of an example of a camera according to the fourth embodiment of the present invention. In FIG. 14, the same components as those of the camera shown in FIG. 10 are denoted by the same reference numerals and description thereof is omitted.

  The illustrated camera has posture change detection units 120 to 122, detects the posture change of the camera in three axis directions, and sends the detection result (hereinafter referred to as posture change information) to the microcomputer 112. The illustrated posture change detection units 120 to 122 are, for example, gyro sensors. Here, the gyro sensor 120 detects a posture change in the yaw direction, and the gyro sensor 121 detects a posture change in the pitch direction. The gyro sensor 122 detects a change in posture in the roll direction.

  The microcomputer 112 detects the amount of camera shake based on the posture change information. Then, the microcomputer 112 changes the control diagram indicating the relationship between the focal length and the reduction ratio in accordance with the amount of angular blur.

  As described above, the fourth embodiment of the present invention includes a posture change detection unit such as a gyro sensor, and changes the control diagram indicating the relationship between the focal length and the reduction ratio according to the amount of camera shake. To do. As a result, in addition to the effects of the second embodiment described above, the motion vector can be detected by accurately selecting the relationship between the focal length and the reduction ratio according to the amount of camera shake.

  As described above, in the embodiment of the present invention, the resizing rate of the original image and the reference image, which are images for motion vector detection, is controlled according to the focal length of the optical system. As a result, it is possible to accurately detect a motion vector for tilt correction on the wide side and for translation correction on the tele side, and to generate an image in which blurring is well corrected.

  As is clear from the above description, in the example shown in FIG. 1, the image reduction circuits 107 and 108 function as resizing means, and the motion vector detection circuit 109 functions as motion vector detection means. In addition, the microcomputer 112 and the reduction ratio determination circuit 106 function as control means, and the microcomputer 112, the geometric deformation parameter estimation circuit 110, and the geometric deformation circuit 111 function as blur correction means.

  As mentioned above, although this invention was demonstrated based on embodiment, this invention is not limited to these embodiment, Various forms of the range which does not deviate from the summary of this invention are also contained in this invention. .

  For example, the function of the above embodiment may be used as a control method, and this control method may be executed by the imaging apparatus. Further, a program having the functions of the above-described embodiments may be used as a control program, and the control program may be executed by a computer included in the imaging apparatus. The control program is recorded on a computer-readable recording medium, for example.

  Each of the above control method and control program has at least a resizing step, a control step, a motion vector detection step, and a shake correction step.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various recording media, and the computer (or CPU, MPU, etc.) of the system or apparatus reads the program. To be executed.

DESCRIPTION OF SYMBOLS 101 Optical system 102 Image pick-up element 103 Camera signal processing circuit 104 Memory 106 Reduction rate determination circuit 107,108 Image reduction circuit 109 Motion vector detection circuit 110 Geometric deformation parameter estimation circuit 111 Geometric deformation circuit 112 Microcomputer

Claims (15)

An imaging apparatus that includes an optical system having a variable focal length, and that captures a plurality of temporally continuous images via the optical system,
One of the images is used as an original image, an image taken before the original image is used as a reference image, and the original image and the reference image are resized to be a first image and a second image, respectively. Resizing means;
Control means for controlling a resizing rate in the resizing means in accordance with a focal length of the optical system;
Motion vector detection means for detecting a motion vector between the first image and the second image;
Blur correction means for correcting blur of the original image according to the motion vector;
An imaging device comprising: Memory means for holding at least the reference image;
2. The resizing means comprises: a first reducing means for reducing the original image as the resizing process; and a second reducing means for reducing the reference image as the resizing process. The imaging device described in 1. Memory means for holding at least the reference image;
The resizing means includes a reducing means for reducing the original image as the resizing process, and a resizing processing means for reducing or enlarging the reference image as the resizing process,
The imaging apparatus according to claim 1, wherein the reduction unit stores a first image obtained by reduction processing in the memory unit as the reference image.   The said control means changes the said resizing rate so that the size of the said original image and the said reference image may become large as the said focal distance becomes large, The any one of Claims 1-3 characterized by the above-mentioned. Imaging device. The blur correction unit estimates a geometric deformation parameter indicating a deformation amount of an image caused by the blur of the imaging device according to the motion vector;
5. The apparatus according to claim 1, further comprising a geometric deformation unit that performs a geometric transformation process on the original image in accordance with the geometric deformation parameter to correct blur of the original image. 6. Imaging device.   6. The imaging according to claim 4, wherein the control unit changes the resizing rate so that the sizes of the original image and the reference image become smaller as the aperture value of the optical system becomes larger. apparatus.   6. The imaging apparatus according to claim 5, wherein the control unit changes the resizing ratio so that the sizes of the original image and the reference image become smaller as the tilt component of the geometric deformation parameter becomes larger. . Furthermore, it comprises posture change detection means for detecting posture change of the imaging device and obtaining posture change information,
The imaging apparatus according to claim 1, wherein the control unit controls a resizing rate in the resizing unit according to the posture change information in addition to the focal length.   9. The imaging according to claim 8, wherein the control unit changes the resizing rate so that the sizes of the original image and the reference image become smaller as the amount of blur indicated by the posture change information becomes larger. apparatus.   10. The control unit according to claim 1, wherein the resizing rate obtained by smoothing a plurality of resizing rates set before photographing the original image is used as a resizing rate in the resizing unit. The imaging apparatus of Claim 1.   The said control means changes the resizing rate in the said resizing means according to the aperture value obtained by smoothing the several aperture value set before image | photographing the said original image, It is characterized by the above-mentioned. The imaging device described.   The control means changes a resizing rate in the resizing means according to a geometric deformation parameter obtained by smoothing a plurality of geometric deformation parameters set before photographing the original image. 8. The imaging device according to 7.   The control means sets the resizing rate in the resizing means according to the shake amount obtained by smoothing the shake amount indicated by the plurality of posture change information obtained by the posture change detection means before photographing the original image. The imaging apparatus according to claim 8, wherein the imaging apparatus is changed. A control method of an imaging apparatus that includes an optical system having a variable focal length and that captures a plurality of temporally continuous images via the optical system,
One of the images is used as an original image, an image taken before the original image is used as a reference image, and the original image and the reference image are resized to be a first image and a second image, respectively. Resize step,
A control step for controlling a resizing rate in the resizing process performed in the resizing step in accordance with a focal length of the optical system;
A motion vector detection step of detecting a motion vector between the first image and the second image;
A blur correction step of correcting blur of the original image according to the motion vector;
A control method characterized by comprising: A control program used in an imaging apparatus that includes an optical system having a variable focal length and captures a plurality of temporally continuous images through the optical system,
In the computer provided in the imaging device,
One of the images is used as an original image, an image taken before the original image is used as a reference image, and the original image and the reference image are resized to be a first image and a second image, respectively. Resize step,
A control step for controlling a resizing rate in the resizing process performed in the resizing step in accordance with a focal length of the optical system;
A motion vector detection step of detecting a motion vector between the first image and the second image;
A blur correction step of correcting blur of the original image according to the motion vector;
A control program characterized by causing