JP2016163257A - Image processor, control method thereof, and control program, and imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To satisfactorily correct blurring in an image by detecting a motion vector with high accuracy regardless of a shutter speed.SOLUTION: Image reduction circuits 107 and 108 regard one image as an original image, regard an image obtained by a result of photographing after the original image as a reference image, and perform reduction processing of the original image and the reference image. A reduction rate determination circuit 106 determines a reduction rate when the image reduction circuits perform reduction processing in accordance with a shutter speed during photographing, and a motion vector detection circuit 109 detects a motion vector between the original image and the reference image on the basis of the original image and the reference image subjected to the reduction processing.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing device, a control method thereof, a control program, and an imaging device, and more particularly, to an image stabilization processing technique for reducing image blur due to motion (shake) of the imaging device.

  In general, electronic image stabilization processing is performed to reduce image blurring caused by image blurring of an imaging apparatus such as a digital still camera or a digital video camera. In order to perform the image stabilization process, it is necessary to detect the amount of motion between frames and perform the alignment process for a plurality of images.

  In the technique for detecting the amount of motion between frames, for example, shake information obtained by an external device such as a gyro sensor is used. Further, the amount of motion is estimated from an image obtained as a result of shooting. For example, as a technique for estimating a motion amount from an image, there is a technique for detecting a motion vector using template matching.

  In template matching, for example, one of the two images is used as an original image, and the other image is used as a reference image. Using a rectangular area of a predetermined size arranged in the original image as a template block, the correlation with the luminance value distribution of the template block is obtained in the reference image. Then, the region having the highest correlation value in the reference image is set as the movement destination of the template block, and the direction and movement amount with respect to the movement destination when the position of the template block in the original image is used as a reference is set as the motion vector.

  Next, statistical processing or the like is performed using the plurality of motion vectors obtained as described above, and a motion between images is obtained as a geometric deformation amount. At this time, if a large number of motion vectors can be accurately obtained from the entire image, the amount of geometric deformation between the images can be accurately obtained.

  However, it is difficult to obtain a large number of motion vectors with high accuracy in order to reduce processing time and resources.

  On the other hand, if the images are unclear (that is, unclear), it is difficult to obtain the correlation between the images, and the motion vector accuracy decreases. Here, the unclear image refers to an image in which high-frequency components such as edges are attenuated.

  For example, when shooting in a dark place such as at night, in order to increase the amount of light incident on the image sensor, shooting may be performed with a slow shutter speed. In such a case, motion blur occurs in an image obtained as a result of shooting, resulting in an unclear image. This reduces the accuracy of the motion vector. Therefore, it is necessary to control shooting so as to guarantee the accuracy of the motion vector even if the shutter speed is slowed down.

  In order to guarantee the accuracy of the motion vector, for example, there is a method of changing a parameter used when detecting the motion vector according to the accuracy of the motion vector. For example, an influence parameter indicating how an erroneous detection of a motion vector affects an image is generated, and the reduction ratio of the original image and the reference image is controlled according to the influence parameter ( Patent Document 1).

JP 2010-252259 A

  However, in the method described in Patent Document 1 described above, the image reduction rate is changed based on the variance value and the average value of the luminance values in the template block, and the image reduction rate is adjusted according to the shutter speed. It is not done. For this reason, the technique described in Patent Document 1 may not be able to detect a motion vector with high accuracy depending on the shutter speed.

  Therefore, an object of the present invention is to provide an image processing apparatus capable of detecting a motion vector with high precision regardless of the shutter speed, a control method thereof, a control program, and an imaging apparatus.

  In order to achieve the above object, an image processing apparatus according to the present invention uses one image as an original image, an image obtained as a result of shooting after the original image as a reference image, and the original image and the reference image. Reduction means for reduction processing, determination means for determining a reduction rate when the reduction processing is performed by the reduction means in accordance with a shutter speed at the time of photographing the original image and the reference image, and the original on which the reduction processing has been performed And detecting means for detecting a motion vector between the original image and the reference image based on the image and the reference image.

  The control method of the image processing apparatus according to the present invention includes a reduction step of reducing the original image and the reference image using one image as an original image and an image obtained as a result of photographing after the original image as a reference image. A determination step of determining a reduction ratio when performing the reduction process in the reduction step according to a shutter speed at the time of shooting the original image and the reference image, and the original image and the reference image subjected to the reduction process And a detection step of detecting a motion vector between the original image and the reference image.

  A control program according to the present invention is a control program used in an image processing apparatus, and is obtained as a result of photographing after making an original image one image on a computer provided in the image processing apparatus. A reduction ratio for reducing the original image and the reference image as a reference image, and performing a reduction process in the reduction step according to a shutter speed at the time of shooting the original image and the reference image. A determination step of determining and a detection step of detecting a motion vector between the original image and the reference image based on the original image and the reference image on which the reduction process has been performed are executed.

  According to the present invention, since the reduction ratio of the image used for detecting the motion vector is determined according to the shutter speed, the motion vector is detected accurately regardless of the shutter speed, and the blur in the image is improved. It can be corrected.

1 is a block diagram illustrating a configuration of an example of an imaging apparatus including an image processing apparatus according to a first embodiment of the present invention. 3 is a flowchart for explaining an example of a photographing process performed by the camera shown in FIG. 1. It is a figure which shows the example about the relationship between the reduction rate of an image, and the detection number of a motion vector. It is a figure which shows the example about the relationship between the reduction ratio of an image, and the precision of a motion vector. It is a figure which shows the relationship between the reduction rate based on FIG. 3 and FIG. 4, the number of remaining motion vectors, and vector accuracy. It is a figure for demonstrating an example of the relationship between shutter speed and a reduction rate, (a) is a figure which shows a 1st example, (b) is a figure which shows a 2nd example. It is a figure for demonstrating an example of an imaging | photography scene, (a) is a figure which shows an image when shutter speed is fast, (b) is a figure which shows an image when shutter speed is slow. It is a figure for demonstrating the other example of the relationship between shutter speed and a reduction rate. 2A and 2B are diagrams for explaining template matching performed by the motion vector detection circuit illustrated in FIG. 1. FIG. 2A is a diagram illustrating an example of an original image, and FIG. 2B is a diagram illustrating an example of a reference image. It is a block diagram which shows the structure about an example of the camera by the 2nd Embodiment of this invention. It is a flowchart for demonstrating an example of the imaging | photography process performed with the camera shown in FIG.

  Hereinafter, an example of an image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram showing a configuration of an example of an imaging apparatus including an image processing apparatus according to the first embodiment of the present invention.

  The illustrated imaging apparatus is, for example, a digital camera (hereinafter simply referred to as a camera) and has an imaging optical system (hereinafter simply referred to as an optical system). A subject image (optical image) is formed on the image sensor 102 via the optical system 101. The image sensor 102 is a CCD sensor or a CMOS sensor, and outputs an electrical signal (analog signal) corresponding to an optical image by photoelectric conversion.

  The camera signal processing circuit 103 generates an image signal (also referred to as a camera signal) corresponding to an analog signal that is an output of the image sensor 102. The camera signal processing circuit 103 includes, for example, an A / D conversion circuit, an auto gain control circuit (AGC), and an auto white balance circuit (AWB), and performs predetermined image processing on the analog signal to perform digital processing. The camera signal which is a signal is output.

  The camera signal that is the output of the camera signal processing circuit 103 is temporarily recorded in the memory 104. The memory 104 is a memory for recording one frame or a plurality of frames of the camera signal. In the following description, an image of one frame is called a frame image. In the illustrated example, the imaging device 102 and the camera signal processing circuit 103 constitute an imaging system for acquiring an image.

  The reduction rate determination circuit 106 determines a reduction rate when reducing the frame image according to the shutter speed at the time of shooting under the control of the microcomputer 112. Each of the image reduction circuits 107 and 108 performs a reduction process on the frame image recorded in the memory 104 based on the reduction rate determined by the reduction rate determination circuit 106.

  A motion vector detection circuit 109 detects a motion vector between two reduced frame images, which is an output of the image reduction circuits 107 and 108. The geometric deformation parameter estimation circuit 110 uses the motion vector obtained by the motion vector detection circuit 109 to output a blur correction amount between frame images as a geometric deformation parameter. The geometric deformation circuit 111 performs a geometric deformation process for correcting blurring of the frame image based on the geometric deformation parameters. The microcomputer 112 controls the entire camera.

  FIG. 2 is a flowchart for explaining an example of a photographing process performed by the camera shown in FIG. Note that the processing according to the illustrated flowchart is performed under the control of the microcomputer 112.

  When shooting is started, under the control of the microcomputer 112, the camera signal processing circuit 103 performs processing image processing on the analog signal that is the output of the image sensor 102 to generate a camera signal (image input: step). S201). Here, the camera signal processing circuit 103 outputs, for example, a 12-bit digital signal as a camera signal. At this time, the camera signal processing circuit 103 performs signal level correction and white level correction by AGC and AWB, and records a camera signal (that is, a frame image) in the memory 104.

  In the illustrated camera, frame images are sequentially generated at a predetermined frame rate, and the frame images are recorded in the memory 104. Then, the same size frame image recorded in the memory 104 is input to the image reduction circuits 107 and 108. Note that the frame images recorded in the memory 104 are sequentially updated.

  Subsequently, the reduction rate determination circuit 106 determines the reduction rate of the frame image under the control of the microcomputer 112 as described later, and sets the reduction rate in the image reduction circuits 107 and 108 (step S202). .

  FIG. 3 is a diagram illustrating an example of the relationship between the image reduction rate and the number of detected motion vectors.

  In FIG. 3, the horizontal axis indicates the reduction ratio. The larger the reduction ratio, the larger the image size after reduction, and the smaller the reduction ratio, the smaller the image size after reduction. When the reduction ratio = 1, the image is not reduced. The vertical axis represents the number of motion vectors (the number of remaining motion vectors) determined to have high reliability in the detected motion vectors.

  Here, the reliability of the motion vector is determined in accordance with, for example, what kind of texture the motion vector is detected from. That is, the low-contrast region and the region including the repetitive pattern are regions in which it is difficult to detect a motion vector by template matching, and therefore there is a low possibility that a motion vector is accurately detected in the region.

  In order to perform texture determination, the average value and dispersion value of the luminance values of the pixels in the template are obtained, and it is determined whether the area is a low contrast area or a repeated pattern area according to the average value and dispersion value. To do. If it is determined that the region is a low-contrast region or a repeated pattern region, the motion vector detected in that region is excluded as having low reliability. Such a determination is performed for all the motion vectors, and the number of motion vectors with high reliability remaining finally is set as the number of remaining motion vectors.

  As shown in FIG. 4, the closer the reduction ratio is to 1, that is, the larger the image size used for motion vector detection, the more likely that multiple locations with high correlation with the template are generated, resulting in erroneous motion vectors. More are detected. The erroneously detected motion vector cannot be used because it causes an error.

  Furthermore, as the reduction ratio is closer to 1, the texture in the template becomes poor, and as a result of being determined as a low contrast region, the number of motion vectors to be excluded increases. For this reason, the number of remaining motion vectors decreases as the reduction ratio approaches 1.

  On the other hand, the closer the reduction ratio is to 0, that is, the smaller the image size used for motion vector detection, the more texture in the wide area of the image will be in the template. As a result, the number of motion vectors excluded due to low contrast and repeated determination decreases, and the number of remaining motion vectors increases.

  FIG. 4 is a diagram illustrating an example of the relationship between the image reduction ratio and the accuracy of the motion vector.

  In FIG. 4, the horizontal axis represents the reduction ratio, and the vertical axis represents the motion vector detection accuracy. As shown in the figure, the closer the reduction ratio is to 1, that is, the larger the image size used for motion vector detection, the clearer the image, the more accurate the motion vector. On the other hand, as the reduction ratio is closer to 0, the image becomes unclear due to the low-pass filter effect by the reduction processing, and the motion vector detection accuracy decreases.

  Therefore, when the image is reduced, there is a limit to making the reduction rate close to 0, and even if the detection accuracy of the motion vector is lowered, the reduction rate limit 501 is maintained so that the accuracy beyond the detection limit can be maintained. Is set. The detection limit is a limit of accuracy that can be regarded as the same as an accurate motion vector. For example, the detection limit is detection accuracy that does not allow visual recognition of the difference from an image that has been geometrically deformed using an accurate motion vector when the image is geometrically deformed using a certain motion vector. Say.

  Further, when obtaining the geometric deformation parameter, it is necessary to estimate not only the movement of the translation component but also the movement of the rotation component and the tilt component. For this purpose, it is necessary to detect many motion vectors from the entire image. Therefore, here, a state in which the reduction ratio is close to 0 is set as a standard state.

  FIG. 5 is a diagram showing the relationship between the reduction rate, the number of remaining motion vectors, and the vector accuracy based on FIGS.

  As shown in FIG. 5, when it is desired to detect a large number of motion vectors with standard accuracy (that is, when the number of remaining motion vectors is increased), the reduction rate is brought close to zero. On the other hand, when it is desired to detect a highly accurate motion vector even if it is a small number (that is, when the number of remaining motion vectors is at least good), the reduction ratio is brought close to 1. That is, the image size is changed according to the detection accuracy of the motion vector.

  FIG. 6 is a diagram for explaining an example of the relationship between the shutter speed and the reduction ratio. FIG. 6A is a diagram illustrating a first example, and FIG. 6B is a diagram illustrating a second example.

  In FIG. 6, the horizontal axis indicates the shutter speed, and when the shutter speed is slowed down, camera shake (ie, shake) increases. The vertical axis indicates the reduction ratio, and the image size increases as the reduction ratio increases.

  In the example shown in FIG. 6A, the relationship between the shutter speed and the reduction rate changes linearly (linear function), and the reduction rate approaches 1 as the shutter speed decreases. That is, the image size is increased as the shutter speed is decreased. In the example shown in FIG. 6B, the relationship between the shutter speed and the reduction rate changes exponentially, and here, the reduction rate approaches 1 as the shutter speed decreases.

  When the relationship shown in FIG. 6 is used, if the reduction ratio is close to 1, that is, the size of the image used for detecting the motion vector is large, the motion vector is detected with a clear image. For this reason, while the accuracy of the detected motion vector is improved, there is a strong tendency to generate a plurality of portions having a high correlation with the template, and erroneous detection of the motion vector is likely to occur. Incorrect motion vectors cannot be used because they cause errors.

  Further, when motion vectors are detected with a clear image, the texture in the template becomes poor, and the number of motion vectors that are excluded because they are determined to be low-contrast regions increases. For this reason, the number of remaining motion vectors decreases.

  On the other hand, when the reduction ratio is close to 0, that is, when the size of the image used for detecting the motion vector is small, vector detection is performed with an unclear image. Therefore, while the accuracy of the detected motion vector is reduced, a texture in a wide area of the image is included in the template, so that it is difficult to be excluded by low contrast and repeated pattern determination. As a result, the number of remaining motion vectors increases.

  FIG. 7 is a diagram for explaining an example of a shooting scene. FIG. 7A shows an image when the shutter speed is fast, and FIG. 7B shows an image when the shutter speed is slow.

  Here, it is assumed that the movement (shake) of the object to be photographed and the camera are the same regardless of the shutter speed. In the example shown in FIG. 7B, since the shutter speed is slower than in the example shown in FIG. 7A, even if the movement of the camera is the same, the image is likely to be blurred.

  Since the shutter speed when the image shown in FIG. 7A is obtained is fast, the image becomes clear. For this reason, there is a strong tendency that a plurality of locations having a high template correlation are generated. Therefore, when detecting a motion vector in the image shown in FIG. 7A, the number of remaining motion vectors can be improved by reducing the reduction rate to 0, that is, by reducing the image size used for detecting the motion vector. , It can perform the anti-vibration treatment well (standard condition).

  On the other hand, since the shutter speed when the image shown in FIG. 7B is obtained is slow, the image becomes unclear due to the movement (shake) of the camera. Therefore, if the image is reduced at the same reduction ratio as in the standard state, the image becomes more unclear and the accuracy of the motion vector is less than the accuracy of the motion vector obtained in the standard state. Therefore, for the image shown in FIG. 7B, the motion vector is set in a state in which the reduction ratio is brought close to 1, that is, the image size used for detecting the motion vector is increased to prevent the image from becoming unclear. Is detected. As a result, the motion vector can be detected with the same accuracy as the standard state.

  However, as described with reference to FIG. 3, if the image size is increased, the number of remaining motion vectors decreases, so that the reduction rate needs to be changed according to the shutter speed as described above.

  By using the motion vector obtained as described above, it is possible to estimate the geometric deformation parameter that accurately indicates the blurring that occurs in the image obtained as a result of shooting, and the image stabilization process is performed with high accuracy on the image. Can be applied.

  Note that the shutter speed differs for each mode such as the program AE mode, aperture priority AE mode, shutter priority AE mode, and manual mode. For example, in the case of the program AE mode and the aperture priority AE mode, an appropriate shutter speed is set by the camera. In the case of the program AE mode, the shutter speed is automatically set together with an aperture corresponding to the brightness of the subject by the camera. In the case of the aperture priority AE mode, after the aperture is determined, the shutter speed is automatically set so that proper exposure is obtained.

  On the other hand, in the manual mode or the shutter priority AE mode, the shutter speed is arbitrarily set by the photographer. In the manual mode, the photographer determines both the aperture and the shutter speed. However, in the shutter priority AE mode, after the photographer determines the shutter speed, the aperture is set so that a proper exposure is obtained by the camera.

  The relationship between the shutter speed and the reduction ratio is not limited to the example shown in FIG. For example, if the shutter speed is set slower than “1 / focal length” seconds, the image tends to be blurred.

  FIG. 8 is a diagram for explaining another example of the relationship between the shutter speed and the reduction ratio.

  In FIG. 8, when the shutter speed is “1 / focal length” seconds or less, an image obtained as a result of shooting is less likely to be blurred due to camera movement. Therefore, in this case, the motion vector is detected by reducing the image at the reduction ratio of the standard state described above. On the other hand, when the shutter speed exceeds “1 / focal length” seconds, the reduction rate is increased, that is, the image size is increased to detect the motion vector.

  In this way, the reduction rate determination circuit 106 determines the image reduction rate according to the shutter speed under the control of the microcomputer 112, and sends the reduction rate to the image reduction circuits 107 and 108.

  Referring again to FIG. 2, each of the image reduction circuits 107 and 108 reduces the image according to the reduction ratio determined by the reduction ratio determination circuit 106 (step S203). Here, an original image and a reference image used for motion vector detection are selected from a plurality of frame images stored in the memory 104 at the same size in step S201, and are input to the image reduction circuits 107 and 108. Then, the image reduction circuits 107 and 108 reduce the original image and the reference image based on the image reduction rate determined by the reduction rate determination circuit 106, respectively.

  As for the image reduction method, for example, when the reduction ratio is 1/8, an average value of pixel values in an 8 × 8 rectangular area is calculated, and a pixel average method for reducing the image is used. Also good. Alternatively, the image may be reduced using a so-called reduction filter that weights the pixel value located in the center.

  Subsequently, the motion vector detection circuit 109 detects a motion vector in the two frame images (images subjected to reduction processing) input from the image reduction circuits 107 and 108 (step S204). For example, the motion vector detection circuit 109 detects a motion vector using template matching.

  FIG. 9 is a diagram for explaining template matching performed by the motion vector detection circuit 109 shown in FIG. FIG. 9A shows an example of the original image, and FIG. 9B shows an example of the reference image. In FIG. 9, an image obtained by reducing the frame image of the same size stored in the memory 104 is shown.

  The motion vector detection circuit 109 arranges the template block 901 at an arbitrary position in the original image. The motion vector detection circuit 109 performs template matching on the reference image using the template block 901, and calculates a correlation value in each block region of the reference image. At this time, if the correlation values are calculated for all the block regions of the reference image, the amount of calculation becomes enormous. Therefore, the motion vector detection circuit 109 sets a rectangular area for calculating a correlation value in the reference image as the search range 902. Here, the position and size of the search range 902 are not particularly limited, but if the block range corresponding to the destination of the template block 901 is not included in the search range 902, a motion vector can be detected correctly. Can not.

  In the illustrated example, a sum of absolute difference (hereinafter referred to as SAD) technique is used as a correlation value calculation technique.

  When obtaining the correlation value S_SAD, the motion vector detection circuit 109 uses the following equation (1).

  In equation (1), f (i, j) indicates the luminance value at the coordinates (i, j) of the template block 901, and g (i, j) is the block area 903 that is the target of correlation value calculation in the search range 902. Indicates the luminance value. In the SAD technique, the absolute value of the difference between the luminance value f (i, j) of the template block 901 and the luminance value g (i, j) of the block region 903 is obtained, and the correlation is obtained by calculating the sum thereof. Get the value S_SAD.

  Therefore, the smaller the correlation value S_SAD, the smaller the difference in luminance value between the template block 901 and the block area 903. That is, the textures of the template block 901 and the block area 903 are similar.

  Here, the correlation value is obtained by the SAD method, but, for example, a sum of squared differences (SSD) or a normalized cross-correlation (NCC) may be used. However, when using a technique other than SAD, there are cases where the similarity is higher as the correlation value is smaller, and the similarity is higher as the correlation value is larger, and so on depending on the relationship between the correlation value and the similarity. It is also necessary to change the process.

  The motion vector detection circuit 109 moves the block area 903 for the entire area of the search range 902 and calculates the correlation value with the template block 901 for the entire area of the search range 902. Then, the motion vector detection circuit 109 determines a position (that is, a block region) where the correlation value is the smallest, and to which position the template block on the original image has moved, that is, the original image and the reference The motion vector between the images is detected.

  In this way, the motion vector detection circuit 109 detects motion vectors for a plurality of regions between frame images. That is, a template block is set at another position in the original image, and similarly, a motion vector is detected to obtain a motion vector group. Then, the motion vector detection circuit 109 sends the motion vector group to the geometric deformation parameter estimation circuit 110.

  Subsequently, the geometric deformation parameter estimation circuit 110 estimates a geometric deformation parameter between frame images according to the motion vector group (step S205). Here, a case where a homography matrix is used as an image deformation amount will be described as an example of a geometric deformation model.

  Now, it is assumed that a point a on one frame (image) is represented by Expression (2), and the point a has moved to a point a ′ represented by Expression (3) in the next frame.

  Note that in the expressions (2) and (3), the subscript T indicates a transposed matrix.

  The correspondence relationship between the point a shown in Expression (2) and the point a ′ shown in Expression (3) is expressed by Expression (4) using the homography matrix H.

  The homography matrix H is a determinant indicating the amount of deformation caused by translation, rotation, scaling, shear, and tilt between images, and is expressed by the following expression (5).

  Each element of the homography matrix H is calculated by performing a statistical process such as a least square method in accordance with the motion vector group obtained by the process of step S204, that is, the correspondence of representative points between frame images. The However, since the motion vector detected from the reduced frame image is a motion vector in the reduced image, it is necessary to convert the motion vector to one of the same size when estimating the geometric deformation parameter. is there.

  The homography matrix H obtained as described above indicates the amount of image deformation caused by camera shake. Therefore, when correcting image blur, it is necessary to convert the homography matrix H so as to obtain an image deformation amount that cancels the deformation caused by the blur. That is, by converting the homography matrix H to the inverse matrix H, the correspondence between the point a ′ and the point a is expressed by the following equation (6).

  Expression (6) makes it possible to return the point a ′ after the blurring to the same coordinates as the point a before the blurring.

  In the illustrated example, a homography matrix is used as a model representing the amount of blur between frame images, but other models such as a Helmat matrix or an affine matrix may be used. Furthermore, when the main subject is set as an image stabilization target, only the translational blur correction amount needs to be obtained. In this case, the average value of the motion vector group may be obtained, so that statistical estimation is performed. There is no need, and the amount of calculation can be reduced.

  Next, the microcomputer 112 gives the geometric deformation parameter obtained as described above to the geometric deformation circuit 111. The geometric deformation circuit 111 performs a vibration reduction process by applying a geometric transformation process to the frame image stored in the memory 104 using the geometric deformation parameter (step S206). The geometric deformation circuit 111 stores the image subjected to the image stabilization process in a storage device (not shown) and displays it on a display device (not shown). Thereafter, the microcomputer 112 ends the photographing process.

  Thus, in the first embodiment of the present invention, the reduction ratio of the image used for motion vector detection is changed according to the shutter speed. As a result, it is possible to detect the motion vector with high accuracy regardless of the shutter speed and to correct the blurring in the image satisfactorily.

[Second Embodiment]
Next, an example of a camera according to the second embodiment of the present invention will be described.

  FIG. 10 is a block diagram showing the configuration of an example of a camera according to the second embodiment of the present invention. In FIG. 10, the same components as those of the camera shown in FIG.

  The illustrated camera includes an image resizing circuit 1001 instead of the image reduction circuit 108.

  FIG. 11 is a flowchart for explaining an example of a photographing process performed by the camera shown in FIG. In the illustrated flowchart, the same steps as those in the flowchart shown in FIG.

  When shooting is started, the camera signal processing circuit 103 records the frame image in the memory 104 and sends the frame image to the image reduction circuit 107 under the control of the microcomputer 112 (step S1101). In step S202, the reduction rate determination circuit 106 determines the reduction rate of the image as described above, and sets the reduction rate in the image reduction circuit 107 and the image resizing circuit 1001.

  Subsequently, the image reduction circuit 107 reduces the frame image sent from the camera signal processing circuit 103 based on the reduction rate set by the reduction rate setting circuit 106 (step S1103). Then, the image reduction circuit 107 stores the reduced frame image in the memory 104.

  The image resizing circuit 1001 resizes the reduced frame image stored in the memory 104 (step S1104).

  By the way, the reduced frame image stored in the memory 104 is reduced according to the reduction rate determined by the reduction rate determination circuit 106 at the time of shooting. However, if the reduction ratio is changed according to the temporal variation of the shooting scene, the frame image that has already been reduced and that has already been stored in the memory 104 and the frame image that has been reduced after the reduction ratio has been changed. Differences in image size occur between the two. If the image size is different between the original image and the reference image, the motion vector cannot be detected. Therefore, the image resizing circuit 1001 resizes the reduced image stored in the memory 104 in order to match the image sizes of both the original image and the reference image.

  For example, it is assumed that the reduction ratio at a photographing time point (that is, an input time point) for a certain frame image is 1/8. Then, when the reduction ratio is changed to ¼ according to the temporal variation of the shooting scene, the image resizing circuit 1001 enlarges the ８ reduced frame image stored in the memory 104 twice and performs original processing. The image is sent to the motion vector detection circuit 109 as an image. On the other hand, the 1/4 reduced frame image is sent as a reference image from the image reduction circuit 107 to the motion vector detection circuit 109 and stored in the memory 104.

  As a result, the original image and the reference image are both reduced to ¼, and the motion vector detection circuit 109 can detect the motion vector using the original image and the reference image having the same image size.

  Note that the resizing process performed by the image resizing circuit 1001 may be any technique, for example, an interpolation process such as bilinear or bicubic may be used.

  Thereafter, after performing the processing of steps S204 to S207 described in FIG. 2, the microcomputer 112 ends the photographing process.

  Thus, in the second embodiment of the present invention, the frame image is reduced and stored in the memory 104 at the reduction rate at the time of shooting. Then, the reduced frame image stored in the memory 104 is resized according to the change in the reduction rate. In the first embodiment described above, only the frame image of the same size is stored in the memory 104. Therefore, if there is a change in the reduction ratio, the image reduction circuit reads the frame image of the same size from the memory 104. Thus, the image reduction process is performed. Therefore, the bandwidth for transmission becomes large. On the other hand, in the second embodiment, the frame image is reduced and stored in the memory 104 at the reduction rate at the time of shooting, and the reduced frame image stored in the memory 104 according to the change in the reduction rate is resized. As a result, the bandwidth for transmission can be reduced.

  As is apparent from the above description, in the example shown in FIGS. 1 and 10, the image reduction circuits 107 and 108 function as reduction means, and the microcomputer 112 and the reduction rate determination circuit 106 function as determination means. The motion vector detection circuit 109, the geometric deformation parameter estimation circuit 110, the geometric deformation circuit 111, and the image resizing circuit 1001 function as a detection unit, a calculation unit, a correction unit, and a resizing unit, respectively.

  In the above-described embodiment, an example in which the image stabilization process is performed inside the imaging apparatus has been described. However, the present invention is not limited to this. The image capturing apparatus may output the shutter speed corresponding to the captured image in association with the image, and may perform image reduction and motion vector detection using a personal computer or the like that has received the shutter speed. That is, any image processing apparatus having a function of reducing a plurality of images according to the shutter speed and detecting a motion vector using these images may be used.

  As mentioned above, although this invention was demonstrated based on embodiment, this invention is not limited to these embodiment, Various forms of the range which does not deviate from the summary of this invention are also contained in this invention. .

  For example, the function of the above embodiment may be used as a control method, and this control method may be executed by the image processing apparatus. In addition, a program having the functions of the above-described embodiments may be used as a control program, and the control program may be executed by a computer included in the image processing apparatus. The control program is recorded on a computer-readable recording medium, for example.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 102 Image pick-up element 103 Camera signal processing circuit 104 Memory 106 Reduction rate determination circuit 107,108 Image reduction circuit 109 Motion vector detection circuit 110 Geometric deformation parameter estimation circuit 111 Geometric deformation circuit 112 Microcomputer 1001 Image resizing circuit

Claims (8)

One image as an original image, an image obtained as a result of photographing after the original image as a reference image, a reduction means for reducing the original image and the reference image,
Determining means for determining a reduction ratio when performing reduction processing by the reduction means according to a shutter speed at the time of photographing the original image and the reference image;
Detecting means for detecting a motion vector between the original image and the reference image based on the original image and the reference image on which the reduction processing has been performed;
An image processing apparatus comprising: Memory means for storing the original image and the reference image;
The image processing apparatus according to claim 1, wherein the reduction unit reads the original image and the reference image from the memory unit and performs the reduction process. Memory means for storing the original image and the reference image subjected to the reduction processing;
Resizing means for performing resizing processing to make the same reduction ratio of the original image and the reference image subjected to the reduction processing when the reduction rate according to the shutter speed changes during the photographing. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 1, wherein the determination unit determines the reduction ratio so that an image size increases as the shutter speed decreases. Calculating means for obtaining a blur between the original image and the reference image as a geometric deformation parameter based on the motion vector;
5. The image according to claim 1, further comprising a correcting unit that geometrically deforms an image obtained as a result of photographing based on the geometric deformation parameter and corrects blurring of the image. Processing equipment. Imaging means for imaging the original image and the reference image;
An image pickup apparatus comprising: the image processing apparatus according to claim 1. A reduction step of reducing the original image and the reference image using one image as an original image, and an image obtained as a result of shooting after the original image as a reference image;
A determination step of determining a reduction ratio when performing the reduction process in the reduction step according to the shutter speed at the time of shooting the original image and the reference image;
A detection step of detecting a motion vector between the original image and the reference image based on the original image and the reference image on which the reduction processing has been performed;
A control method for an image processing apparatus, comprising: A control program used in an image processing apparatus,
In the computer provided in the image processing apparatus,
A reduction step of reducing the original image and the reference image using one image as an original image, and an image obtained as a result of shooting after the original image as a reference image;
A determination step of determining a reduction ratio when performing the reduction process in the reduction step according to the shutter speed at the time of shooting the original image and the reference image;
A detection step of detecting a motion vector between the original image and the reference image based on the original image and the reference image on which the reduction processing has been performed;
A control program characterized by causing execution.