JP2008160274A - Motion vector detection method, its apparatus and its program, electronic hand-blur correction method, its apparatus and its program, as well as imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately calculate a motion vector needed for electronic hand-blur correction, even when an optical zoom magnification is changed. <P>SOLUTION: Comparison operation processing is carried out for a reference image (n frame image) and a compared image (n+1 frame image), a segment position of a compared image having a high correlation to the reference image is determined, and a motion vector of the compare image to the reference image is computed using the position. In this occasion, if an optical zoom magnification of the compared image to the reference image is changed, electronic zoom correction is carrying out for the reference image and/or the compared image based on information on the optical zoom magnification so that magnification of the reference image is the same as the compared image, and the motion vector is computed. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an imaging apparatus such as a digital camera and an electronic camera shake correction apparatus mounted on the imaging apparatus, and more particularly to an imaging apparatus equipped with an optical zoom lens, a moving vector detection method and apparatus, and a program thereof, electronic The present invention relates to a camera shake correction method, an apparatus thereof, and a program thereof.

  For example, when shooting a video with a digital still camera or video camera with a video shooting function in hand, the position of the still object image shown in the image of a certain frame is the position of the next frame when the hand holding the camera shakes. It moves on the screen, making it difficult to see the video. Therefore, a movement vector of an image of the next frame is detected with respect to an image of a certain frame, and camera shake correction is performed so that the image does not shake.

  FIG. 6 is an explanatory diagram illustrating the principle of detecting a movement vector when performing camera shake correction. If the image in the block 2 indicated by the predetermined address in the image 1 of the Nth frame shown in FIG. 6A is a reference image, the block 4 indicated by the same predetermined address in the image 3 of the (N + 1) th frame is used. If the cut out image is the same as the reference image, the image 3 is not shaken with respect to the image 1.

  However, if camera shake has occurred, the reference image in block 2 and the image cut out in block 4 do not match. Therefore, the block 4 in the image 3 is shifted by 4 pixels in the X direction (horizontal direction) and the Y direction (vertical direction) from 4a, 4b, 4c,. Each image is compared with the reference image, and the block position where the comparison image having the highest correlation with the reference image is cut out is obtained.

  In the example illustrated in FIG. 6B, if the comparison image having the highest correlation with the reference image of the block 2 is cut out at the block 4 c, the difference between the image 3 and the image 1 is a vector “k”. This vector k is a movement vector between two screens. If image 1 is displayed and then image 3 is moved in the direction opposite to that of movement vector k and displayed, an image without camera shake can be displayed.

  The calculation for obtaining the correlation between the reference image and the comparison image is performed by comparing one pixel per pixel of the reference image and the comparison image to obtain the minimum value of the total sum of the pixels within each block of each difference. The time required for the calculation also becomes long, and the movement vector calculation is not in time until the next frame is displayed. Therefore, normally, the blocks 4a, 4b, 4c,... Are not shifted by one pixel, but are shifted by, for example, eight pixels (equivalent to performing a comparison operation with a reduced image obtained by thinning eight pixels into one pixel). In addition to reducing the load, the calculation time is shortened.

  As a conventional technique related to the movement vector detection technique, there is Patent Document 1 below.

JP-A 61-201581

  An optical zoom lens is usually mounted on a digital camera or the like, and a photographer zooms in or out while photographing a subject image. When this optical zoom operation is performed, the size of the captured image of the same subject is different between the image 1 in FIG. 6A and the image 3 in FIG. 6B.

  For this reason, even if the movement vector is calculated by comparing the reference image and the comparison image during the optical zoom operation, the detection accuracy of the movement vector is deteriorated. If camera shake correction is performed with an inaccurate movement vector, the screen becomes difficult to see. Conventionally, the camera shake correction process is usually stopped while the optical zoom operation is being performed.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a movement vector detection method and apparatus capable of accurately detecting a movement vector even when an optical zoom operation is performed, a program thereof, an electronic camera shake correction method and apparatus thereof, a program thereof, and an imaging apparatus. There is to do.

  In the movement vector detection method of the present invention, a reference image and a comparison image are subjected to a comparison calculation process, a cutout position of a comparison image having a high correlation with the reference image is obtained, and a movement vector of the comparison image with respect to the reference image is determined based on the position. In the movement vector detection method to be calculated, when the optical zoom magnification of the comparison image with respect to the reference image varies, the reference image and the comparison image are set to have the same magnification based on the information of the optical zoom magnification. The movement vector is calculated by electronic zoom correction of the image and / or the comparison image.

  The movement vector detection apparatus according to the present invention performs a comparison calculation process on the reference image and the comparison image, obtains a cutout position of the comparison image having a high correlation with the reference image, and determines a movement vector of the comparison image with respect to the reference image based on the position. In the movement vector detection device to be calculated, when the optical zoom magnification of the comparison image with respect to the reference image varies, the reference image and the comparison image are set to have the same magnification based on the information of the optical zoom magnification. Electronic zoom processing means for correcting electronic zoom of the image and / or the comparison image is provided.

  The movement vector detection program of the present invention performs a comparison calculation process on a reference image and a comparison image, obtains a cutout position of the comparison image having a high correlation with the reference image, and determines a movement vector of the comparison image with respect to the reference image based on the position. In the movement vector detection program to be calculated, the step of detecting a change in the optical zoom magnification of the comparison image with respect to the reference image, and the reference image and the comparison image are set to have the same magnification based on the information of the optical zoom magnification. And calculating the movement vector by performing electronic zoom correction on the reference image and / or the comparison image.

  The electronic camera shake correction method of the present invention is an electronic camera shake correction method for moving a cutout range of captured image data output from a solid-state imaging device based on a motion vector detected according to a camera shake amount. It detects by the movement vector detection method of description.

  An electronic camera shake correction apparatus according to the present invention is an electronic camera shake correction apparatus that moves a cutout range of captured image data output from a solid-state imaging device based on a motion vector detected according to a camera shake amount. It detects by the movement vector detection apparatus of description.

  The electronic camera shake correction program of the present invention is an electronic camera shake correction program for moving a cutout range of captured image data output from a solid-state imaging device based on a motion vector detected according to a camera shake amount. It is detected by the described movement vector detection program.

  An image pickup apparatus according to the present invention includes a solid-state image pickup device, an optical zoom lens system provided at a front portion of the solid-state image pickup device, and the electronic image stabilization device described above.

  According to the present invention, when the optical zoom magnification of the image data from which the comparison image is cut out with respect to the image data from which the reference image is cut out, the electronic zoom correction is performed so that the reference image and the comparison image have the same magnification. Even if the optical zoom magnification changes, the movement vector can be calculated with high accuracy, and when electronic camera shake correction is performed, this correction can be performed with high accuracy.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a configuration diagram of a digital camera according to an embodiment of the present invention. The digital camera includes an imaging unit 21, an analog signal processing unit 22 that performs analog processing such as automatic gain adjustment (AGC) and correlated double sampling processing on analog image data output from the imaging unit 21, and an analog signal processing unit. An analog / digital conversion unit (A / D) 23 that converts analog image data output from the digital image data into digital image data, and an A / D 23, an analog signal processing unit 22, according to instructions from a system control unit (CPU) 29 described later, A drive control unit (including a timing generator) 24 that performs drive control of the imaging unit 21 and a flash 25 that emits light in response to an instruction from the CPU 29 are provided.

  The imaging unit 21 collects light from the object scene and zooms in and out of the subject, and an optical magnification of the optical zoom lens system 21a is instructed from the drive control unit 24. And a solid-state imaging device 100 that receives the light condensed by the optical zoom lens system 21a and narrowed down by the diaphragm and outputs captured image data (analog image data).

  The solid-state imaging device 100 may be a CCD type, a CMOS type, or another type of solid-state imaging device, and an effective pixel region is wider than the imaging range in order to perform electronic camera shake correction processing.

  The digital camera of the present embodiment further includes a digital signal processing unit 26 that takes in digital image data output from the A / D 23 and performs interpolation processing, white balance correction, RGB / YC conversion processing, and the like. A compression / decompression processing unit 27 that compresses or decompresses the image data, a display unit 28 that displays a menu or the like, displays a through image or a captured image, and a system control unit that performs overall control of the entire digital camera ( CPU) 29, an internal memory 30 such as a frame memory, and a media interface (I / F) unit 31 that performs interface processing between a recording medium 32 that stores JPEG image data and the like, and a bus that interconnects them 40, and an operation unit 33 for inputting an instruction from the user is connected to the system control unit 29. It has been.

  FIG. 2 is a functional configuration diagram of an electronic camera shake correction processing device 50 configured when electronic camera shake correction processing is performed with the digital camera shown in FIG. In the electronic camera shake correction processing device 50, when the user inputs an instruction to perform electronic camera shake correction processing from the operation unit 33 shown in FIG. 1, the system control unit 29 includes an electronic camera shake including a movement vector detection program. By starting the correction processing program, the system control unit 29 is configured using the subordinate digital signal processing unit 26, the internal memory 30, and the like.

  The electronic camera shake correction processing device 50 performs a thinning processing unit 51 that thins out image data output from the solid-state imaging device 100 to generate a reduced image, and thinned reduced image data of the nth frame (even frame). Electronic zoom processing is applied to the first memory 52 to be stored, the second memory 53 to store the thinned reduced image data of the (n + 1) th frame (odd frame), and the reduced image data read from the first memory 52. A first electronic zoom processing unit 54 and a second electronic zoom processing unit 55 that performs electronic zoom processing on the reduced image data read from the second memory 53 are provided.

  In this embodiment, in order to speed up the movement vector calculation process, the thinning processing unit 51 generates reduced images of the reference image and the comparison image. If a high-performance arithmetic processing device can be used and the memory capacity is sufficient, the movement vector can be calculated using the original image data without performing the thinning process.

  The electronic camera shake correction processing device 50 further acquires the optical zoom magnification information instructed to the zoom drive unit 21b shown in FIG. 1, and sends the information to the first and second electronic zoom processing units 54 and 55 based on the optical zoom magnification. On the other hand, a movement vector for calculating a movement vector by comparing two pieces of reduced image data captured from the electronic zoom magnification control section 56 for instructing the electronic zoom magnification and the first and second electronic zoom processing sections 54 and 55, respectively. A detection processing unit 57 and a read control processing unit 58 that determines a read range (cutout range) of captured image data in the internal memory 30 based on the detected movement vector.

  An operation when the electronic camera shake correction processing apparatus 50 performs motion camera shake correction of the moving image will be described. The solid-state imaging device 100 alternately outputs captured image data of even frames and captured image data of odd frames, and each captured image data is stored in the internal memory 30.

  In parallel with the storage in the internal memory 30, the captured image data output from the solid-state imaging device 100 is reduced image data through the thinning processing unit 51 and stored in the memories 52 and 53.

  Assume that the movement vector of the image data of the (n + 1) th frame is detected with respect to the image data of the nth frame. That is, the reference image is cut out from the nth frame image data (reduced image data), the comparison image is cut out from the nth frame image data (reduced image data), and the correlation between both is obtained.

  In this case, the electronic zoom magnification control unit 56 obtains the imaging magnification M of the image data of the (n + 1) th frame from the optical zoom magnification information when the imaging magnification of the image data of the nth frame is “1”. Then, an electronic zoom magnification “1” is instructed to the electronic zoom processing unit 54, and an electronic zoom magnification “1 / M” is instructed to the electronic zoom processing unit 55.

  If the photographer does not perform the optical zoom operation and the imaging magnification of the image data of the (n + 1) th frame is the same as the imaging magnification of the image data of the nth frame, both the electronic zoom processing units 54 and 55 The electronic zoom magnification “1” is instructed.

  The movement vector detection processing unit 57 calculates a movement vector by cutting out a reference image from the reduced image of the nth frame, cutting out a comparison image from the reduced image of the (n + 1) th frame, and obtaining the correlation between them. The method of obtaining the movement vector is the same as that described with reference to FIG.

  In this movement vector detection process, in this embodiment, as shown in FIG. 3A, one screen is divided into a total of 16 divided screens of 4 rows and 4 columns, and each divided screen # 1 to # 16 is divided. The movement vector is calculated. That is, the reference image is cut out in the central rectangular frame 5 in the # 1 divided screen of the reference image, and the same frame as the rectangular frame 5 is moved in the # 1 divided screen in the # 1 divided screen of the comparative image, and the correlation is high. The cutout position of the comparative image is obtained.

  When the movement vectors are obtained for each divided screen in this way, a total of 16 partial movement vectors K1 to K16 are calculated as shown in FIG. Then, an average value of 16 partial movement vectors is calculated, and this average value is set as a movement vector between two screens.

  Since the partial movement vectors shown in FIG. 3B all face the same direction within a predetermined threshold range, the average value of 16 partial movement vectors is used as a movement vector between two screens. However, for example, if there is a small animal that moves around in the # 16 divided screen, the partial movement vector K16 indicates an abnormal value that is outside the predetermined threshold range as compared with other partial movement vectors. When such a partial movement vector indicating an abnormal value exists, the partial movement vector indicating the abnormal value is excluded, and the average value of the partial movement vectors within a predetermined threshold range is set as a movement vector between two screens.

  After the movement vector K between the two screens is calculated, the read control processing unit 58 instructs the internal memory 30 about the cutout range. As a result, after the image of the nth frame is cut out and output in a certain cutout range, the image of the (n + 1) th frame is cut out and output in the cutout range based on the movement vector, so that a moving image corrected for camera shake is output. Will be.

  When the image data of the next frame (n + 2 frame) is output from the solid-state imaging device 100, the image data is stored in the internal memory 30, and is thinned out by the thinning processing unit 51, and the reduced image data is overwritten in the memory 52. Is done. This time, the reduced image of the (n + 1) th frame becomes the reference image, the comparison image is cut out from the reduced image of the (n + 2) th frame, and the movement vector of the imaging layer of the (n + 2) th frame with respect to the captured image of the (n + 1) th frame is the same as described above. Based on this movement vector, the captured image data of the (n + 2) th frame is cut out and output.

  While such electronic image stabilization processing is being repeated one after another, the optical zoom magnification is changed. For example, for the nth frame image data shown in FIG. Assume that the image is enlarged.

  In this case, the electronic zoom magnification control unit 56 instructs the electronic zoom processing unit 55 to set the electronic zoom magnification “1 / M” (the instruction magnification for the electronic zoom processing unit 54 is “1”). As a result, the M-magnified enlarged image is reduced to 1 / M, so that the reference image and the comparison image have the same magnification, and a comparison operation is possible.

  Therefore, the movement vector detection processing unit 57 calculates the movement vector between the two screens by the comparison operation between the reference image (n frame image) and the comparison image (the image after the electronic zoom processing of the n + 1 frame image). . At the time of calculating the movement vector, in the example shown in FIG. 4, there is no image data in the divided screens # 1 to # 4, # 5, # 8, # 9, # 12, and # 13 to # 16. ing. Therefore, the movement vector detection processing unit 57 obtains only the partial movement vectors on the divided screens # 6, # 7, # 10, and # 11 and calculates the movement vector between the two screens.

  If the image of the (n + 2) th frame captured from the solid-state imaging device 100 is an image reduced to 1 / M as shown in FIG. The magnification control unit 56 instructs the electronic zoom processing unit 54 to set the electronic zoom magnification 1 / (1 / M) = M (the instruction magnification for the electronic zoom processing unit 55 is “1”). As a result, the reduced image of 1 / M times is enlarged to M times, so that the reference image and the comparison image have the same magnification, and comparison operation is possible.

  Therefore, the movement vector detection processing unit 57 calculates a movement vector between the two screens by the comparison operation described above. In this case, since image data exists on all the divided screens # 1 to # 16, partial movement vectors K1 to K16 are obtained and a movement vector between the two screens is calculated.

  As described above, even when an operation for changing the optical zoom magnification is performed, in this embodiment, since the movement vector can be calculated with high accuracy, it is possible to perform electronic camera shake correction even when the optical zoom magnification is changed. An easy-to-view moving image can be obtained.

  In the above-described embodiment, the electronic zoom magnification of the reference image is always “1” and the electronic zoom magnification of the comparison image is the reverse magnification of the optical zoom magnification. However, if the reference image and the comparison image have the same magnification, Since the comparison operation is possible, it is not necessary to always set the electronic zoom magnification of the reference image to “1”. For example, in FIG. 4, the n-frame image may be enlarged and electronically zoomed and compared with the n + 1 frame image. It goes without saying that both the reference image and the comparative image may be corrected to the same magnification by electronic zoom processing.

  In the above-described embodiment, the electronic camera shake correction has been described. However, the above-described embodiment can be similarly applied to the calculation of a movement vector used for moving object detection in the screen. In the above-described embodiment, the processing is described as processing inside the digital camera. However, all the image data output from the solid-state imaging device 100 is recorded, and this is loaded into an external personal computer or the like, and the personal computer moves. The same processing can be performed by executing a vector detection program or an electronic camera shake correction program.

  The imaging apparatus and program according to the present invention can be used as a digital camera or an image processing program because the movement vector can be accurately calculated even during optical zoom.

It is a digital camera block block diagram concerning one Embodiment of this invention. It is a functional block diagram of the electronic camera-shake correction processing apparatus comprised by the program in the digital camera of FIG. It is explanatory drawing of the partial movement vector used for a camera shake correction process. It is explanatory drawing when performing the comparison process of a reference | standard image and an expansion comparison image. It is explanatory drawing when performing the comparison process of a reference | standard image and a reduction | restoration comparison image. It is explanatory drawing of a movement vector calculation process.

Explanation of symbols

21 Image pickup unit 21a Optical zoom lens system 21b Zoom drive unit 29 System control unit 30 Internal memory 50 Electronic camera shake correction processing unit 51 Thinning-out processing unit 52, 53 Memory 54, 55 Electronic zoom processing unit 56 Electronic zoom magnification control unit 57 Movement vector Detection processing unit 58 Read control processing unit 100 Solid-state imaging device

Claims (7)

  In the movement vector detection method for performing a comparison operation on a reference image and a comparison image, obtaining a cutout position of a comparison image having a high correlation with the reference image, and calculating a movement vector of the comparison image with respect to the reference image based on the position, When the optical zoom magnification of the comparison image with respect to the reference image varies, the reference image and / or the comparison image are electronically controlled so that the reference image and the comparison image have the same magnification based on the information of the optical zoom magnification. A movement vector detection method comprising calculating the movement vector by performing zoom correction.   In the movement vector detection device for performing a comparison operation on the reference image and the comparison image, obtaining a cutout position of the comparison image having a high correlation with the reference image, and calculating a movement vector of the comparison image with respect to the reference image based on the position, When the optical zoom magnification of the comparison image with respect to the reference image varies, the reference image and / or the comparison image are electronically controlled so that the reference image and the comparison image have the same magnification based on the information of the optical zoom magnification. An electronic zoom processing means for performing zoom correction is provided.   In a movement vector detection program for performing a comparison operation on a reference image and a comparison image, obtaining a cutout position of a comparison image having a high correlation with the reference image, and calculating a movement vector of the comparison image with respect to the reference image based on the position, Detecting the fluctuation of the optical zoom magnification of the comparison image with respect to the reference image; and the reference image and / or the comparison image so that the reference image and the comparison image have the same magnification based on the information of the optical zoom magnification And a step of calculating the movement vector by performing electronic zoom correction on the movement vector.   The movement vector detection method according to claim 1, wherein the movement vector is a movement vector detection method according to claim 1, wherein the cutout range of the captured image data output from the solid-state imaging device is moved based on a movement vector detected according to a shake amount. An electronic camera shake correction method characterized by detecting.   The movement vector detecting device according to claim 2, wherein the movement vector is moved based on a movement vector detected in accordance with the amount of camera shake in the cutout range of the captured image data output from the solid-state imaging device. An electronic camera shake correction device characterized by detecting.   An electronic camera shake correction program for moving a cutout range of captured image data output from a solid-state imaging device based on a motion vector detected according to a camera shake amount. An electronic camera shake correction program characterized by detecting.   An imaging apparatus comprising: a solid-state imaging device; an optical zoom lens system provided at a front portion of the solid-state imaging device; and the electronic camera shake correction device according to claim 5.

Images