JP6381212B2 - Imaging apparatus and control method thereof - Google Patents

Description

The present invention relates to an imaging apparatus and a control method therefor, and more particularly to an image stabilization technique for reducing image blurring caused by movement of the imaging apparatus.

  In order to perform image stabilization processing (blur correction) on images shot using an imaging device such as a digital still camera or digital video camera, the amount of motion between frame images is detected, and alignment for multiple images is performed. There is a need to do. As a method for detecting the motion amount between frame images, there are a method using information of an accessory device such as a gyro sensor, a method for estimating a motion amount from a captured frame image, and the like.

  There are various methods for estimating the amount of motion from the frame image. A typical example is a motion vector detection method based on template matching. In template matching, first, one of two arbitrary frame images in a video is defined as an original image, and the other is defined as a reference image. Then, a rectangular area having a predetermined size arranged on the original image is used as a template block, and a correlation with the luminance value distribution in the template block is obtained for each position of the reference image. At this time, the position where the correlation is highest in the reference image is the destination of the template block, and the direction to the destination and the amount of movement based on the position of the template block on the original image are the motion vector. . By performing statistical processing using the plurality of motion vectors thus obtained, the motion between the frame images is calculated as a geometric deformation amount. At this time, if a large number of motion vectors can be calculated from the entire screen with high accuracy, an accurate geometric deformation amount between frame images can be obtained.

  However, when performing an imaging operation and an image stabilization operation in real time in the imaging apparatus, it is difficult to obtain a large number of motion vectors with high accuracy in consideration of processing time and resources. Therefore, depending on the situation, it is conceivable to use a detection method for detecting a large number of vectors with standard accuracy and a detection method for detecting a small number of vectors with high accuracy. As an example, it is conceivable to use different detection methods depending on which subject in the image is subjected to image stabilization processing.

  For example, when image stabilization processing is performed on a main subject such as a person in the vicinity of an imaging device, it is sufficient to detect only the movement of the main subject area, so that it is sufficient to detect a small number of vectors. A certain number of motion vectors may be detected. On the other hand, when the image stabilization process is performed on the background area, it is necessary to detect a large number of motion vectors from the entire screen and to obtain a geometric deformation amount representing a tilt or rotation motion. In this way, the subject that can detect the motion vector differs depending on whether the accuracy or the number of the motion vector is important. Therefore, if only one of the accuracy and the number of the motion vector is important, a specific subject For, a good motion vector cannot be obtained.

  Therefore, a method has been proposed in which a parameter for detecting a motion vector is changed according to the accuracy of the detected motion vector. For example, a technique has been proposed in which an influence parameter representing how an erroneous detection of a motion vector affects an image is generated, and the reduction ratio of the original image and the reference image is controlled according to the generated influence parameter ( Patent Document 1).

JP 2010-252259 A

  However, the technique described in Patent Document 1 changes the reduction ratio of an image by paying attention to the variance value and average value of luminance values in a template block, and the amount of geometric deformation between frame images to be calculated. Not paying attention to the nature of.

  An object of the present invention is to provide a technique that makes it possible to detect a motion vector suitable for shake correction in accordance with an object to be subjected to image stabilization processing.

Imaging device according to the present invention includes an imaging means for obtaining an object image formed by the optical system as an image, a storage means for storing image data of an image to the imaging unit has acquired, images that the imaging unit has acquired Determination means for determining the image stabilization target from the information of the above, a setting means for setting a reduction ratio for the image acquired by the imaging means, and an image input through the storage means based on the reduction ratio set by the setting means Reduction means for performing reduction processing on the image, detection means for detecting a plurality of motion vectors of the image stabilization target using the image reduced by the reduction means, and determination means for determining the reliability of the motion vector the a removing means reliability of removing the motion vector of a part that is determined to be low, wherein the determination means, the area of the image which the motion vector is detected It determines the reliability based on Kusucha, the setting means, the Ri by the changing the reduction ratio in accordance with the vibration-proof object, the reliability of the part which is removed is determined to be low the It is characterized by controlling the number and accuracy of motion vectors.

In the present invention, a part of motion vectors determined to be low in reliability is removed so that an optimal geometric deformation amount can be estimated according to the subject to be subjected to the image stabilization process. By changing the reduction ratio according to the object to be transferred, the number and accuracy of some motion vectors that are judged to have low reliability and are removed are controlled. This makes it possible to detect motion vectors that distinguish between subjects that are effective in detecting a large number of motion vectors with standard accuracy and subjects that require a small number of motion vectors with high accuracy. It is possible to obtain an image in which the generated blur is well corrected.

1 is a block diagram illustrating a schematic configuration of an imaging apparatus according to a first embodiment of the present invention. 3 is a flowchart for explaining the operation of the imaging apparatus in FIG. 1. It is a figure which shows typically an example of the imaging | photography scene by the imaging device of FIG. 1 schematically shows a relationship between the image reduction rate and the number of detected motion vectors in the image pickup apparatus of FIG. 1, and schematically shows a relationship between the image reduction rate and the detected motion vector accuracy. It is the table | surface which put together the relationship between a graph and the reduction rate, the number of remaining vectors, and vector accuracy. It is a figure which shows an example of the motion vector detected from each of the main to-be-photographed object and background area | region of FIG. FIG. 4 is a diagram illustrating a shooting scene when the distance between the main subject and the imaging apparatus is increased from the shooting scene of FIG. 3 and motion vectors detected at that time. It is a figure which shows the original image and reference image for demonstrating the outline | summary of template matching. It is a block diagram which shows schematic structure of the imaging device which concerns on 2nd Embodiment of this invention. It is a flowchart explaining operation | movement of the imaging device of FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<First Embodiment>
[Schematic configuration of imaging device]
FIG. 1 is a block diagram illustrating a schematic configuration of an imaging apparatus according to an embodiment of the present invention. The imaging apparatus according to the first embodiment includes an optical system 101, an imaging element 102, a camera signal processing circuit 103, a memory 104, an image stabilization target determination circuit 105, a reduction ratio setting circuit 106, a first image reduction circuit 107, and a second image reduction circuit. The image reduction circuit 108 is provided. The imaging apparatus according to the first embodiment includes a motion vector detection circuit 109, a geometric deformation parameter estimation circuit 110, a geometric deformation circuit 111, and a microcomputer (control unit) 112.

  The optical system 101 forms a subject image. The image sensor 102 is an image sensor such as a CCD sensor or a CMOS sensor that photoelectrically converts a subject image formed by the optical system 101. The camera signal processing circuit 103 includes an A / D conversion circuit, an auto gain control circuit (AGC), and an auto white balance circuit, and forms a digital image signal (image data) from an analog electric signal output from the image sensor 102. . In the imaging apparatus according to the first embodiment, the optical system 101, the imaging element 102, and the camera signal processing circuit 103 constitute an imaging unit that acquires an image.

  The memory 104 temporarily stores image data of one or a plurality of frame images generated by the camera signal processing circuit 103. The image stabilization target determination circuit 105 determines an object to be subjected to image stabilization processing (hereinafter referred to as “image stabilization target”) using a frame image (image data) input from the camera signal processing circuit 103 via the memory 104. To do. The reduction ratio setting circuit 106 sets a reduction ratio (hereinafter simply referred to as “reduction ratio”) when reducing the frame image in accordance with the image stabilization target determined by the image stabilization target determination circuit 105. Each of the first image reduction circuit 107 and the second image reduction circuit 108 performs a reduction process on the frame image input from the memory 104 based on the reduction rate determined by the reduction rate setting circuit 106.

  The motion vector detection circuit 109 detects a motion vector between two frame images input from the first image reduction circuit 107 and the second image reduction circuit 108. The geometric deformation parameter estimation circuit 110 uses the motion vector detected by the motion vector detection circuit 109 to output a correction amount of blur motion between frame images as a geometric deformation parameter. Based on the geometric deformation parameter calculated by the geometric deformation parameter estimation circuit 110, the geometric deformation circuit 111 performs a geometric deformation process for correcting the shake on the frame image.

  The microcomputer 112 temporarily stores a CPU, a ROM that stores a program executed by the CPU to control the operation of the imaging apparatus, a parameter used by the CPU and a calculated value, and a work for developing the program. A RAM is provided as an area. The microcomputer 112 controls the operation of each circuit such as the optical system 101, the image stabilization target determination circuit, the geometric deformation circuit 111, and performs overall operation control of the imaging apparatus.

[Imaging operation of the imaging device]
FIG. 2 is a flowchart of the operation (imaging and image processing) of the imaging apparatus according to the first embodiment, and each process shown in FIG. 2 is executed under the control of the microcomputer 112. In step S201, an image input process is performed. That is, the subject image formed by the optical system 101 is converted into an analog electrical signal corresponding to the subject brightness by the image sensor 102. The analog electrical signal output from the image sensor 102 is converted into a digital signal (image data) by the camera signal processing circuit 103 and stored in the memory 104.

  The camera signal processing circuit 103 converts an analog electric signal into, for example, a 12-bit digital signal by an A / D conversion circuit, and performs signal level correction and white level correction by AGC and AWB. In the imaging apparatus according to the first embodiment, image data of frame images is sequentially generated at a predetermined frame rate and stored in the memory 104. Thus, the frame image of the same size stored in the memory 104 is transmitted to the first image reduction circuit 107 and the second image reduction circuit 108 in step S203 described later. Along with this, the frame images stored in the memory 104 are sequentially updated.

  Subsequently, in step S202, the image stabilization target determination circuit 105 reads the frame image from the memory 104 to determine an object to be subjected to image stabilization processing, and the reduction rate setting circuit 106 sets the reduction rate based on the determined image stabilization target. Set. Details of the processing in step S202 will be described later.

  The reduction ratio set in step S202 is transmitted from the image stabilization target determination circuit 105 to the first image reduction circuit 107 and the second image reduction circuit 108. In step S203, the first image reduction circuit 107 and the second image reduction circuit 108 perform image reduction processing at the reduction rate set in step S202.

  Specifically, in step S203, an original image and a reference image used for motion vector detection are selected from a plurality of frame images of the same size stored in the memory 104 in step S201. Then, the selected original image and reference image are input to the first image reduction circuit 107 and the second image reduction circuit 108, respectively. Each of the first image reduction circuit 107 and the second image reduction circuit 108 performs a reduction process on the acquired frame image (original image or reference image) based on the reduction rate set in the reduction rate setting circuit 106. Apply. The image reduction method is not particularly limited. For example, in the case of 1/8 reduction, a pixel average method for calculating an average value of pixels in a neighboring 8 × 8 rectangular area can be used. Also, a method using a reduction filter that weights the central pixel can be used.

  The image data reduced in step S203 is transmitted to the motion vector detection circuit 109. In subsequent step S204, the motion vector detection circuit 109 detects a motion vector (motion vector group) between the two frame images acquired from the first image reduction circuit 107 and the second image reduction circuit 108. The motion vector (motion vector group) detected by the motion vector detection circuit 109 is transmitted to the geometric deformation parameter estimation circuit 110. Specific processing in step S204 will be described later.

  In the subsequent step S205, the geometric deformation parameter estimation circuit 110 estimates the geometric deformation parameter between the frame images using the motion vector group detected in step S204. Specific processing in step S205 will be described later.

  In subsequent step S206, the geometric transformation circuit 111 performs a geometric transformation process for image stabilization on the frame image stored in the memory 104 using the geometric transformation parameter obtained in step S205. In step S207, an output process of the image subjected to the image stabilization process is performed. For example, the image subjected to the image stabilization process is displayed on a display device (not shown), or the image data of the image is stored. It is stored in a storage device (not shown).

[Details of processing in step S202]
Details of the processing in step S202 described above will be described. First, a method for determining the image stabilization target by the image stabilization target determination circuit 105 will be described. FIG. 3 is a diagram schematically illustrating an example of a shooting scene by the imaging apparatus according to the first embodiment. In this imaging scene, a person is shown as a main subject 301 in the vicinity of the imaging device, and a background 302 is shown at a position far away from the imaging device.

  As an example of a method for determining an image stabilization target in an image (photographing scene) as shown in FIG. 3, there is a method in which a user (photographer) manually sets. For example, the scene of FIG. 3 is displayed on a display device (not shown) such as a liquid crystal monitor included in the imaging device. Therefore, the user refers to the display device and selects the region of the main subject 301 as the image stabilization target. As a method for selecting the image stabilization target, for example, a method of moving a cursor key to an area to be selected or a method of touching an image of the main subject 301 when the display device has a touch panel function can be cited.

  The image stabilization target determination circuit 105 determines whether the selected area is the main subject or the background. In this determination, a subject region is extracted by performing region division based on color information or the like around the selected position, and the subject region is the main subject based on the position and size of the extracted subject region. A method for determining whether there is a background or a background can be used.

  For example, if the extracted subject area exists near the center of the screen and is smaller than the other areas, the selected area is determined to be the main subject. In addition, referring to the distance information obtained as the AF evaluation value in the selected area and the distance information obtained from other distance measuring means, the selected area exists closer to the imaging device than the other areas. In this case, it can be determined that the selected area is the main subject area. In the case where the person is the main subject 301 as shown in FIG. 3, a method of determining the person as the main subject using the face recognition function can be used.

  On the other hand, when the area of the background 302 is selected, it can be determined that the extracted area exists in the periphery of the screen and further occupies most of the screen. it can. Further, if it is known that the selected area exists farther than other areas based on the distance information of the selected area, it can be determined that the selected area is the background area.

  In contrast to the method in which the user selects the image stabilization target in this way, there is a method in which the imaging apparatus automatically determines the image stabilization target (by program control). For example, the microcomputer 112 divides the screen area to extract the main subject area and the background area, and when the main subject area exists at the center of the screen, the main subject is determined as the image stabilization target. Accordingly, it is possible to generate a vibration-proof image (a vibration-proof image) that reflects the intention of the user (photographer). On the other hand, if the main subject area is small or exists at the edge of the screen, an image (video) that has been subjected to image stabilization processing for the majority of the screen by targeting the background area for image stabilization. ) Can be generated. Thus, the image pickup apparatus automatically determines the image stabilization target based on the position and size of the subject in the shooting scene, so that a good image (video) can always be obtained without bothering the user. it can.

  Based on the image stabilization target determined as described above, the reduction rate setting circuit 106 sets the reduction rate of the reduction processing to be performed on the image used for motion vector detection. First, a description will be given of the relationship between the characteristics of apparent blur occurring on the image stabilization target on the image and the reduction ratio.

  FIG. 4A is a graph schematically showing the relationship between the reduction rate and the number of detected motion vectors. The horizontal axis of the graph of FIG. 4A shows the reduction rate, and the image size after reduction increases as the value of the reduction rate increases. For example, when the reduction ratio is 1, the image size becomes the same size and is not reduced as a result. That is, the image size after multiplying the image size before reduction by the reduction ratio becomes the image size after reduction. In addition, the vertical axis of the graph of FIG. 4A represents the number of motion vectors determined to have high reliability among the detected motion vectors.

  The high reliability of the motion vector depends on the texture vector detected from the region having the texture. That is, it is unlikely that a correct motion vector is detected from a low-contrast region or a region including a repetitive pattern, which is a region that is difficult to detect a motion vector by template matching. Therefore, in order to perform such texture determination, an average value and a variance value are obtained for the luminance values of the pixels in the template block, and it is determined whether the area is a low contrast area or a repetitive pattern area. If it is determined that the region is a low-contrast region or a repeated pattern region, motion vectors detected from the region are excluded as those with low reliability. Such a determination is performed for all the motion vectors, and the number of motion vectors having high reliability that is finally left is the number of remaining vectors represented by the vertical axis in FIG.

  As shown in FIG. 4A, when the reduction ratio is close to 1, that is, when the image size used for motion vector detection is large, there is a strong tendency to generate a plurality of portions having high correlation with the template block. This makes it easier to detect erroneous motion vectors. Since these erroneous motion vectors cause errors, they cannot be used. In addition, the texture in the template block becomes scarce, and the motion vector that is determined to be a low-contrast region and excluded is also increased. Therefore, the number of remaining vectors is reduced.

  On the other hand, when the reduction ratio is close to 0, that is, when the image size used for motion vector detection is small, the texture of a wide area in the image enters the template block. The number of motion vectors excluded by As a result, the number of remaining vectors increases.

  FIG. 4B is a graph schematically showing the relationship between the reduction ratio and the accuracy (detection accuracy) of the detected motion vector. The horizontal axis of the graph of FIG. 4B shows the reduction ratio, as in the horizontal axis of FIG. Also, the vertical axis of the graph of FIG. 4B represents the motion vector detection accuracy. Here, when the reduction ratio is close to 1, that is, when the image size used for motion vector detection is large, since the image is clear, the accuracy of each motion vector is improved. On the other hand, when the reduction ratio is close to 0, the image becomes unclear due to the low-pass filter effect by the reduction process, and the accuracy of each motion vector is lowered.

The image size does not have to be as small as possible. As shown in FIG. 4B, the reduction rate is limited so that the accuracy that does not fall below the detection limit can be maintained even if the detection accuracy is reduced. It is necessary to provide a threshold value S _L ). The detection limit in the first embodiment refers to a limit of accuracy that can be regarded as the same as a correct motion vector. For example, the detection limit is defined as the lower limit of detection accuracy when the difference between an image that has undergone geometric deformation using the motion vector and an image that has undergone geometric deformation using the correct motion vector cannot be visually confirmed. Can do.

  FIG. 4C is a table summarizing the relationship among the reduction ratio, the number of remaining vectors, and the vector accuracy based on the graphs of FIGS. 4A and 4B. From the table of FIG. 4 (c), in order to distinguish between the case where it is desired to detect a large number of motion vectors with standard accuracy and the case where it is desired to detect a high-precision motion vector even with a small number, an image used for motion vector detection It can be seen that the size of should be changed. Next, how to reduce the reduction ratio according to the image stabilization target in order to obtain a good image stabilization will be described.

  Consider a case where the main subject 301 is near the imaging apparatus and the background 302 is farther than the main subject 301 as in the above-described shooting scene shown in FIG. FIG. 5A is a diagram illustrating an example of a motion vector detected from the main subject 301, and FIG. 5B is a diagram illustrating an example of a motion vector detected from the background 302.

  Since the main subject 301 exists in the vicinity of the imaging apparatus, the movement that occurs in the main subject 301 is dominated by the translational movement as indicated by the arrow 501 in FIG. Further, since the main subject 301 exists only in a part of the area (central part) in the screen, the rotation and the movement of the tilt can be approximated as a translational movement. In this case, it is not necessary to detect a large number of motion vectors from the entire screen, and it is only necessary to detect a motion vector only from the area of the main subject 301. Therefore, the number of motion vectors to be detected may be small.

  However, since the main subject 301 is an area that is easily noticed at the time of reproduction, it is necessary to correct the blur occurring in the main subject 301 with high accuracy. For this reason, when the main subject 301 is set as an image stabilization target, a large reduction ratio is set, and an image size used for motion vector detection is set to a size such as an equal size or a half size, for example. A highly accurate motion vector may be detected with a small number of vectors. For the main subject 301, by using the motion vector obtained in this way, it is possible to estimate a highly accurate geometric deformation parameter that accurately represents the motion occurring in the main subject 301. On the other hand, it is possible to perform a favorable vibration isolation treatment.

  On the other hand, since the background 302 exists at a long distance, not only the translational motion but also the tilting motion appears remarkably in the motion generated in the background 302 as indicated by the arrow 502 in FIG. In order to estimate the movement of such a tilt, for example, as shown by a broken line in FIG. 5B, even if a motion vector is detected while paying attention to only a partial region 503 in the image, It is misjudged as a mere upward movement. Therefore, it is impossible to know that the movement of the tilt is occurring. Therefore, in order to estimate the movement of the tilt, it is necessary to detect a large number of motion vectors from the entire screen.

  Therefore, when the background 302 is set as an image stabilization target, the size of an image used for motion vector detection is reduced to a size such as 1/8 size by reducing the reduction ratio, and a large number of images are displayed from the entire screen. It is only necessary to detect the motion vector. Here, the case of tilt movement has been described, but in the case of detecting rotation or enlargement / reduction movement, it is necessary to detect a large number of motion vectors from the entire screen.

  Up to this point, the method of setting the reduction ratio based on the size of the image stabilization target and the distance from the imaging device has been described, but in addition to this, a better motion vector can be obtained by taking the motion of the imaging device into account. You may be able to get it. For example, when the imaging apparatus is largely tilted near the main subject 301, the tilt movement may be dominant over the translation movement. On the other hand, even when the background 302 is set as the image stabilization target, if the magnitude of the blur generated in the imaging apparatus is very small, it is possible to approximate the movement of the tilt or rotation as the translational movement.

  As a method for determining the motion of the imaging device, a method for determining from a temporal variation of a motion vector between frame images may be used, or motion information obtained from a gyro sensor (not shown) mounted on the imaging device. The method of determining from the above may be used, and is not particularly limited.

  In addition, it may be possible to generate a better vibration-proof image by changing the reduction ratio according to the temporal variation of the shooting scene due to the movement of the vibration-proof object or camera work. FIG. 6 is a diagram illustrating a shooting scene and a motion vector detected at that time when the distance between the main subject 301 and the imaging apparatus is increased from the shooting scene of FIG.

  For example, as shown in FIG. 3, when the main subject 301 is present near the imaging device, the translational movement is dominant in the main subject region as described above, and therefore, a small number of images from the main subject region. It is necessary to detect a highly accurate motion vector. At this time, when the main subject 601 moves away from the imaging device, as in the main subject 601 in FIG. 6 corresponding to the main subject 301 in FIG. 3, the movement of the main subject area (arrow shown in FIG. 6) In addition to translational movements, the movement of the cage becomes conspicuous. Further, the proportion of the main subject area in the image is also reduced. For this reason, when image stabilization processing is performed on the main subject 601, uncorrected movement becomes significant in most areas of the image other than the main subject 601, and the image is not good as a whole. .

  In such a case, it is necessary to change the reduction ratio in accordance with temporal changes such as the distance to the main subject and the proportion of the image in the image. As an example of a method for changing the reduction ratio, there is a method in which a threshold is set in advance for the distance to the main subject and the ratio of the area occupied in the image, and the reduction ratio is switched using the threshold as a boundary. Thereby, it is possible to generate an image (video) in which the main area in the shooting scene is always well-vibrated.

  As another example of the method for changing the reduction ratio, there is a method of switching the reduction ratio in stages in accordance with the temporal change in the distance to the main subject and the proportion of the image in the image. By using this method, it is possible to generate a vibration proof image in which the vibration proof object moves smoothly. Here, when the main subject changes in the image in time, the reduction ratio is changed based on the distance to the main subject and the size of the main subject. However, the present invention is not limited to this. For example, the image stabilization target and the reduction ratio may be switched based on the entry / exit of the main subject into the screen, the distance from the background area, and the like.

  Up to this point, the method for analyzing the frame image has been described as a method for setting the image stabilization target and the reduction ratio. However, as another method, there is a method using an imaging parameter. There is focal length information as an example of imaging parameters that can be used to set the image stabilization target and the reduction ratio. In other words, if the focal length at the time of imaging is short, the imaging scene will be imaged with a wide angle of view, and since the movement appearing on the image will be noticeable in the tilt and rotation, switch the image stabilization target to the background area. The reduction ratio is set so that the image size becomes small. As a result, a large number of motion vectors can be detected from the entire image, and an anti-vibration image in which the tilt and rotation motions are corrected well can be generated.

  On the other hand, when the focal length at the time of imaging is long, a telephoto image (an image obtained by cutting out and expanding a local area in the imaging scene) is generated. The movement becomes inconspicuous, and the movement appearing on the image is dominated by the translational movement. Therefore, when the focal length at the time of imaging is long, the reduction ratio is set so that the image size becomes large, as in the case where the image stabilization target is determined as the main subject. As a result, since a small number of highly accurate motion vectors can be detected, it is possible to obtain a vibration-proof image in which the translational motion is well corrected.

  Another example of the shooting parameter used for determining the image stabilization target is an aperture value (F value). For example, as the aperture value is decreased to approach the open state, the depth of field for the captured scene becomes narrower. At this time, since an amount of blur (blur) increases in an out-of-focus area of the imaging scene, it is difficult to detect a correct motion vector. In this case, the in-focus area is set as the image stabilization target, and the reduction ratio is set according to whether the area is the main subject area or the background area. As a result, it is possible to generate a vibration-proof image in which the blurring occurring in the in-focus area is well corrected. On the other hand, when the aperture value is increased, the depth of field with respect to the captured scene becomes deeper, so that any region in the captured scene is in focus. Therefore, in this case, the image stabilization target is determined based on the image of the imaging scene, and the reduction ratio may be set according to whether the image stabilization target is the main subject area or the background area.

  Note that the above-described shooting parameters (focal length, aperture value) are determined by the microcomputer 112 included in the imaging apparatus, and the microcomputer 112 transmits the determined shooting parameters to the optical system 101 and the image stabilization target determination circuit 105. To control the operation of the imaging apparatus.

  In step S202, the target to be subjected to the image stabilization process is determined as described above, and the reduction rate is determined so that a motion vector that can estimate the motion of the shake occurring in the determined image stabilization target with high accuracy can be detected. Set.

[Details of Step S204]
Details of the processing in step S204 described above will be described. Here, a method using template matching will be described as an example of a motion vector detection method. 7A and 7B are diagrams for explaining the outline of template matching. FIG. 7A shows an original image 701 and FIG. 7B shows a reference image 702. The original image 701 and the reference image 702 are images based on the image data obtained by performing the reduction process in step S203 on the frame image of the same size stored in the memory 104 at the reduction rate set in step S202. is there.

  A template block 703 is arranged at an arbitrary position in the original image 701, and a correlation value between the template block 703 and each region of the reference image 702 is calculated. At this time, if the correlation value is calculated for the entire region of the reference image 702, the amount of calculation becomes enormous. Therefore, in practice, a rectangular region for calculating the correlation value is searched in the reference image 702. Set as range 704. The position and size of the search range 704 are not particularly limited, but in order to detect a correct motion vector, it is necessary to include an area corresponding to the destination of the template block 703 within the search range 704. In the first embodiment, a sum of absolute differences (SAD) is used as an example of a correlation value calculation method. The calculation formula of SAD is as the following formula 1.

  In the following equation 1, f (i, j) represents the luminance value at the coordinates (i, j) in the template block 703, and g (i, j) is the correlation for which the correlation value is calculated in the search range 704. Each luminance value in the value calculation block 705 is represented. In SAD, the absolute value of the difference is calculated for these luminance values f (i, j) and g (i, j), and the sum is obtained to obtain the correlation value S_SAD. Therefore, the smaller the correlation value S_SAD value, the smaller the difference in luminance value between the template block 703 and the correlation value calculation block 705, that is, the texture in each block of the template block 703 and the correlation value calculation block 705 is similar. It will be.

  Correlation values are calculated by moving the correlation value calculation block 705 for all regions in the search range 704. Then, a correlation value is calculated between the template block 703 and the search range 704, and a position where the value becomes the smallest is determined, whereby the template block 703 on the original image 701 has moved to which position in the reference image 702. To detect. In this way, a motion vector between the original image 701 and the reference image 702 can be detected.

  The motion vector detection circuit 109 performs the above-described motion vector detection processing in a plurality of regions between two frame images acquired from the first image reduction circuit 107 and the second image reduction circuit 108.

  In the first embodiment, SAD is used as an example of the correlation value. However, the present invention is not limited to this, and other correlation values such as sum of squares of differences (SSD) and normalized cross correlation (NCC) are used. May be. However, when a correlation value other than SAD is used, there are two cases depending on the characteristics: a case where the degree of similarity increases as the correlation value decreases and a case where the degree of similarity increases as the correlation value increases. The subsequent processing needs to be changed according to each tendency.

[Details of processing in step S205]
Details of the above-described processing of step S205 (geometric deformation parameter estimation processing) will be described. Here, a case will be described in which a homography matrix is used as a geometric deformation parameter indicating an amount for deforming an image as a geometric deformation model.

  When a point a on the image represented by the following equation 2 moves to a point a ′ represented by the following equation 3 in the next frame, the correspondence between the point a and the point a ′ is as follows using the homography matrix H: It can be expressed by Equation 4. Note that the subscript T shown in Equation 2 and Equation 3 indicates a transposed matrix.

  The homography matrix H is a determinant representing the amount of deformation caused by translation, rotation, scaling, shearing, and tilting between images, and is expressed by the following formula 5. Here, each element of the homography matrix H is calculated by performing statistical processing such as a least-squares method using the motion vector group obtained in step S204, that is, the correspondence of representative points between frame images. be able to. However, since the motion vector detected using the reduced frame image is a value in the reduced image, when estimating the geometric deformation parameter, the motion vector value is the same size. It is necessary to convert to.

  The homography matrix H obtained in this way represents the amount of deformation of the image due to camera shake. Therefore, in order to correct image blurring, it is necessary to convert the homography matrix H so as to obtain an image deformation amount that cancels deformation due to blurring. In other words, by converting the homography matrix H to the inverse matrix H, the correspondence between the points a ′ and a can be expressed by the following equation (6). It becomes possible to return to the same coordinates as the point a before the shake occurs.

  Here, the case where a homography matrix is used as a model representing the amount of blur between frame images has been described. However, the present invention is not limited to this, and other models such as a Helmat matrix and an affine matrix may be used. Further, when the main subject is set as an image stabilization target, it is only necessary to obtain the translational blur correction amount, and therefore it is possible to simply obtain the average value of the motion vector group. This eliminates the need for statistical estimation, thereby reducing the amount of calculation.

  As described above, in the first embodiment, the motion to be detected is detected by changing the reduction ratio used for detecting the motion vector in accordance with the target (anti-shake target) to be subjected to the anti-shake process in the shooting scene. Control the properties (precision, number) of vector groups. As a result, it is possible to generate an image in which the image stabilization process is performed satisfactorily for the specified image stabilization target with limited resources.

Second Embodiment
[Schematic configuration of imaging device]
FIG. 8 is a block diagram showing a schematic configuration of an imaging apparatus according to the second embodiment of the present invention. Among the components of the imaging device shown in the block diagram of FIG. 8, the same components as those of the imaging device shown in the block diagram of FIG. 1 are denoted by the same reference numerals and description thereof is omitted here. The imaging apparatus of FIG. 8 according to the second embodiment does not include the second image reduction circuit 108 included in the imaging apparatus of FIG. 1 according to the first embodiment, and includes an image resizing circuit 801 instead. Yes.

  In the second embodiment, the image data of the frame image obtained from the imaging unit is input to the memory 104 and stored in the memory 104, and is input to the first image reduction circuit 107 to perform the reduction process and the reduction process is performed. The frame image is stored in the memory 104. Thereby, the bandwidth of the memory 104 to be used is reduced.

[Imaging operation of the imaging device]
FIG. 9 is a flowchart of the operation of the imaging apparatus according to the second embodiment, and each process shown in FIG. 9 is executed under the control of the microcomputer 112. Among the processes in the flowchart of FIG. 9, the same processes as those in the flowchart of FIG. 2 are denoted by the same step numbers (S202, S204 to S207), description thereof is omitted, and the processes of the flowchart in FIG. Only the processing different from that will be described.

  In step S901, an image input process is performed. Here, the frame image output from the imaging unit (the camera signal processing circuit 103 in the subsequent stage) is input and stored in the memory 104 in the same size and input to the first image reduction circuit 107.

  In step S902, the first image reduction circuit 107 performs a reduction process on the input frame image at a reduction rate set by the reduction rate setting circuit 106 when the frame image is input. The reduced image generated in this way is stored in the memory 104.

  In step S903, the image resizing circuit 801 performs resizing processing on the reduced frame image stored in the memory 104 (the frame image reduced in step S902) as necessary. This is due to the following reason.

  That is, as described above, the size of the frame image subjected to the reduction process stored in the memory 104 is reduced by the reduction rate set by the reduction rate setting circuit 106 when input to the first image reduction circuit 107. It has been made. In this case, when the reduction ratio is changed in accordance with the temporal variation of the scene being shot, the reduction process is performed after the reduction process and the reduced frame image already stored in the memory 104 are changed. There is a difference in image size between frame images. That is, the image size differs between the original image and the reference image, and the motion vector cannot be detected as it is. Therefore, in order to match the image sizes of both the original image and the reference image, the image resizing circuit 801 performs image resizing processing.

  For example, it is assumed that the reduction ratio at the time of input for a certain frame image is 1/8, and then the reduction ratio is changed to 1/4 according to the temporal variation of the shooting scene. In this case, the image resizing circuit 801 performs a double enlargement process on the frame image reduced to 1/8 stored in the memory 104, and the enlarged frame image is a motion vector detection circuit. 109 is input. Accordingly, both the original image and the reference image have the same ¼ size image size, so that the motion vector detection circuit 109 can detect a motion vector. Any resizing method may be used. For example, bilinear or bicubic interpolation processing may be used.

  In step S206 of the second embodiment, as in the first embodiment, the geometric deformation circuit 111 performs the geometric deformation calculated in step S205 on the frame image of the same size that is stored in the memory 104. Perform geometric deformation processing using parameters. Thereby, a vibration-proof image is generated.

  In the method of storing only the same size frame image in the memory 104 as in the first embodiment, it is only necessary to perform the reduction process on the same size frame image in accordance with the fluctuation of the reduction rate. Since a frame image is transmitted, a necessary band becomes large. On the other hand, in the second embodiment, at the time of image input, the image is reduced and stored in the memory 104 at a predetermined reduction rate, and the image size in the memory 104 is set so as to correspond to the change in the reduction rate. Resize processing is performed on the reduced image. Accordingly, since it is sufficient to transmit a reduced image, it is possible to reduce the transmission band.

<Other embodiments>
Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program code. It is a process to be executed. In this case, the program and the storage medium storing the program constitute the present invention.

DESCRIPTION OF SYMBOLS 105 Anti-vibration object determination circuit 106 Reduction ratio setting circuit 107 1st image reduction circuit 108 1st image reduction circuit 109 Motion vector detection circuit 110 Geometric deformation parameter estimation circuit 111 Geometric deformation circuit 112 Microcomputer 801 Image resizing circuit

Claims (12)

Imaging means for acquiring a subject image formed by the optical system as an image;
Storage means for storing image data of an image acquired by the imaging means;
Determining means for determining the vibration damping target from information of images that the imaging unit has acquired,
Setting means for setting a reduction ratio for the image acquired by the imaging means;
Reduction means for performing reduction processing on an image input through the storage means based on the reduction ratio set by the setting means;
Detecting means for detecting a plurality of motion vectors of the image stabilization target using the image reduced by the reducing means ;
Determining means for determining the reliability of the motion vector;
Removing means for removing a part of the motion vectors determined to have low reliability ,
The determination means determines the reliability based on a texture of the region of the image in which the motion vector is detected;
The setting means, the controller controls the Ri by the changing the vibration isolating said reduction ratio in response to the target, the number and accuracy of the motion vector of the part that reliability is low and the removal is determined An imaging device that is characterized.   The imaging apparatus according to claim 1, wherein the determination unit determines the image stabilization target based on at least one of a size, a position, and a distance of a subject in an image acquired by the imaging unit. Imaging means for acquiring a subject image formed by the optical system as an image;
Storage means for storing image data of an image acquired by the imaging means;
Determining means for determining an image stabilization target from information on the image acquired by the imaging means and at least one of a focal length and an aperture value of the optical system;
Setting means for setting a reduction ratio for the image acquired by the imaging means;
Reduction means for performing reduction processing on an image input through the storage means based on the reduction ratio set by the setting means;
Detecting means for detecting a plurality of motion vectors of the image stabilization target using the image reduced by the reducing means;
Determining means for determining the reliability of the motion vector;
Removing means for removing a part of the motion vectors determined to have low reliability ,
The determination means determines the reliability based on a texture of the region of the image in which the motion vector is detected;
The setting means controls the number and accuracy of a part of the motion vectors to be removed when the reliability is determined to be low by changing the reduction ratio according to the vibration isolation target. An imaging device. Display means for displaying an image input to the reduction means;
2. The determination unit according to claim 1, wherein the determination unit determines whether an image stabilization target region determined by a photographer on the image displayed on the display unit is a main subject region or a background region. 4. The imaging device according to any one of 3.   The setting means sets the reduction ratio so that the image size is increased when the image stabilization target is a main subject, and the image size is decreased when the image stabilization target is a background area. The imaging apparatus according to claim 1, wherein the reduction ratio is set.   The said setting means changes the said reduction ratio according to at least 1 time fluctuation | variation of the magnitude | size of the said vibration-proof object, a position, and distance, The Claim 1 characterized by the above-mentioned. Imaging device. The storage means stores the image input from the imaging means in the same size size,
The imaging apparatus according to claim 1, wherein the reduction unit performs a reduction process on the same-size image. The storage means stores the image reduced by the reduction means,
When the reduction ratio set by the setting means is changed from the reduction ratio of the image reduced by the reduction means and stored in the storage means, the reduction ratio is reduced by the reduction means and stored in the storage means The image pickup apparatus according to claim 1, further comprising a resizing unit that resizes an image to an image size having the changed reduction ratio. Estimating means for estimating blur between images input to the reducing means using the motion vector detected by the detecting means as a geometric deformation parameter;
The imaging according to any one of claims 1 to 8, further comprising a deformation unit that performs geometric deformation on the image input to the reduction unit using the geometric deformation parameter estimated by the estimation unit. apparatus. 2. The determination unit according to claim 1, wherein the determination unit determines that the reliability is low as the region of the image in which the motion vector is detected has a low contrast or as a repetitive pattern exists in the region. The imaging apparatus according to any one of 9. An image pickup apparatus control method comprising an image pickup means for acquiring a subject image formed by an optical system as an image,
A storage step of storing image data of an image acquired by the imaging unit in a storage unit;
A determination step of determining an image stabilization target from information of an image acquired by the imaging unit;
A setting step for setting a reduction ratio for the image acquired by the imaging means;
A reduction step of performing a reduction process on the image input through the storage unit based on the reduction rate set in the setting step;
A detecting step of detecting a plurality of motion vectors of the image stabilization target by using the image reduced in the reducing step;
A determination step of determining a reliability of the motion vector;
Removing a part of the motion vectors determined to be low in reliability, and
In the determination step, the reliability is determined based on a texture of the area of the image in which the motion vector is detected,
In the setting step, the number and accuracy of a part of the motion vectors to be removed when the reliability is determined to be low are controlled by changing the reduction ratio according to the image stabilization target. Control method to do. An image pickup apparatus control method comprising an image pickup means for acquiring a subject image formed by an optical system as an image,
A storage step of storing image data of an image acquired by the imaging unit in a storage unit;
A determination step of determining an image stabilization target from the information of the image acquired by the imaging unit and at least one of a focal length and an aperture value of the optical system;
A setting step for setting a reduction ratio for the image acquired by the imaging means;
A reduction step of performing a reduction process on the image input through the storage unit based on the reduction rate set in the setting step;
A detecting step of detecting a plurality of motion vectors of the image stabilization target by using the image reduced in the reducing step;
A determination step of determining a reliability of the motion vector;
Removing a part of the motion vectors determined to be low in reliability, and
In the determination step, the reliability is determined based on a texture of the area of the image in which the motion vector is detected,
In the setting step, the number and accuracy of a part of the motion vectors to be removed when the reliability is determined to be low are controlled by changing the reduction ratio according to the image stabilization target. Control method to do.