JP2007259103A - Method of calculating motion vector between two screens and apparatus thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable highly accurate calculation of a motion vector between two screens. <P>SOLUTION: Images of the first and second screens are each divided into a plurality of same regions, correlation values between a reference image segmented from the first screen and each of comparing images segmented from the second screen while shifting its position is obtained for each of the divided regions, the maximum values Mh, ..., Mk, Mj, ... of the correlation values are calculated for each of the divided regions, and a motion vector of the second screen for the first screen is calculated using a motion vector for each of the divided regions in which the remaining maximum values Mk, Mj, ... other than the maximum value Mh, ... below a predetermined threshold are calculated, among the maximum values. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a method and apparatus for calculating a movement vector between two screens for detecting a movement vector (movement amount and movement direction) such as camera shake.

  For example, when shooting a movie with a digital still camera or digital video camera with a movie shooting function in hand, the position of the still object image in the image of a certain frame is the next frame when the hand holding the camera shakes. It moves on the screen and becomes a movie that is difficult to see. Therefore, a movement vector of an image of the next frame is detected for an image of a certain frame, and camera shake correction is performed so that the image does not shake.

  FIG. 6 is an explanatory diagram illustrating the principle of detecting a movement vector when performing camera shake correction. If the image in the block 2 indicated by the predetermined address in the image 1 of the Nth frame shown in FIG. 6A is a reference image, the block 4 indicated by the same predetermined address in the image 3 of the (N + 1) th frame is used. If the cut out image is the same as the reference image, the image 3 is not shaken with respect to the image 1.

  However, if camera shake has occurred, the reference image in block 2 and the image cut out in block 4 do not match. Therefore, the block 4 in the image 3 is shifted in the X direction (horizontal direction) and the Y direction (vertical direction) from 4a, 4b, 4c,..., And the respective images in the blocks 4a, 4b, 4c,. Are compared with the reference image, and the block position where the comparison image having the highest correlation with the reference image is cut out is obtained. The calculation for obtaining the correlation between the reference image and the comparison image is performed by comparing one pixel per pixel of the reference image and the comparison image to obtain the minimum value of the total sum (integrated value) of the total pixels in each block of each difference.

  In the example shown in FIG. 6B, if the comparison image having the highest correlation with the reference image of block 2 is cut out at block 4c, the difference between image 3 and image 1 is “K”. This K is a movement vector between two screens. After the image 1 is displayed, if the image 3 is moved and displayed in the same direction by the movement vector K, an image without camera shake can be displayed.

  Note that, as conventional techniques related to camera shake correction, there are Patent Documents 1 and 2 below.

JP 2003-331292 A JP 2004-14695 A

  When actually obtaining a movement vector between two screens, for example, as shown in FIG. 7A, an image 1 of one screen is divided into a plurality of regions (16 regions in the illustrated example) 5 and each divided region 5 is divided. Every time, a movement vector for the next frame image of the block 2 determined in the divided area 5 (the search range is in each divided area) is calculated.

  Then, an average value K [= (k1 + k2 + k3 +... + K16) / 16] of 16 movement vectors k1, k2, k3,..., K16 calculated as shown in FIG. It is set as the movement vector between.

  As a result, even when a partial movement in the captured image, for example, a small animal subject moves in a part of the image, the movement vector between the two screens can be calculated without being confused by this partial movement. become.

  However, even if an image with no partial movement is captured on the screen, depending on the type of image, the movement vector cannot be calculated with high accuracy, and a moving image with large shaking is recorded on the medium. Sometimes.

  An object of the present invention is to provide a method and apparatus for calculating a movement vector between two screens, which can calculate a movement vector between the two screens with high accuracy.

  The method and apparatus for calculating a movement vector between two screens according to the present invention divides the images of the first screen and the second screen into the same plurality of regions, and the reference image cut out from the first screen for each divided region. And the respective correlation images cut out while shifting the position from the second screen, the highest correlation value is calculated for each divided region, and the predetermined threshold is reached among the highest values. The movement vector of the second screen with respect to the first screen is calculated using the movement vector for each divided region for which the remaining maximum value excluding the highest value that is not calculated is calculated.

  The method and apparatus for calculating a movement vector between two screens according to the present invention provides a comparison that gives the highest value for each divided region among the remaining highest values excluding the highest value that does not reach the predetermined threshold among the highest values. A difference between the correlation value of the comparative image shifted from the image with respect to the image and the highest value is obtained, and the remaining highest value excluding the highest value at which the difference does not reach a predetermined value is calculated for each divided region. A movement vector of the second screen relative to the first screen is calculated using a movement vector.

  The method and apparatus for calculating a movement vector between two screens according to the present invention divides the images of the first screen and the second screen into the same plurality of regions, and the reference image cut out from the first screen for each divided region. And the respective correlation images cut out from the second screen while shifting the position, the highest correlation value is calculated for each divided region, and the highest value is calculated for each divided region. A divided region in which the difference between the correlation value of the comparison image shifted in position relative to the given comparison image and the highest value is obtained, and the remaining highest value is calculated excluding the highest value at which the difference does not reach a predetermined value The movement vector of the second screen relative to the first screen is calculated using each movement vector.

  The method and apparatus for calculating a movement vector between two screens according to the present invention is configured such that the correlation value indicates a difference indicating an amount of mismatch between pixel data between the reference image and the comparison image compared in units of one pixel. The minimum value of the integrated values of the reference image and the comparative image is the maximum value of the correlation value.

  In the method and apparatus for calculating a movement vector between two screens according to the present invention, the pixel data is one or a combination of luminance data, color difference data, and color data.

  In the method and apparatus for calculating a movement vector between two screens according to the present invention, the reference image and the comparison image are reduced images obtained by reducing the original image data at the same reduction magnification.

  According to the present invention, out of the movement vectors for each divided area, the movement vectors inappropriate for the calculation of the movement vector between the screens are excluded and not used, that is, only the movement vector for each reliable divided area is used. Therefore, it is possible to calculate a movement vector between two screens with high accuracy.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a configuration diagram of a main part of a digital still camera with a moving image shooting function equipped with a movement vector calculation device between two screens according to an embodiment of the present invention. Image data output from a solid-state imaging device (not shown) is stored in a frame memory 10, subjected to various image processing by a signal processing unit (not shown), compressed by an image compression circuit 11, and stored in a recording medium 12. .

  When the image data is compressed by the image compression circuit 11, the movement vector calculated by the movement vector calculation device between the two screens is used, and the compressed image data subjected to camera shake correction is stored in the recording medium 12.

  The movement vector calculation apparatus between two screens includes a pre-processing unit 21 that captures and thins out image data output from a solid-state imaging device, and two memories that store reduced image data generated by the thinning-out processing by the pre-processing unit 21. 22 and 23, and a switch 24 that switches and connects one of the two memories 22 and 23 to the preprocessing unit 21. The image data captured by the preprocessing unit 21 may be image data temporarily stored in the frame memory 10 or image data after image processing.

  This movement vector calculation device is further provided between the reference image cutting circuit 26, the comparative image cutting circuit 27, the reference image cutting circuit 26 and the comparative image cutting circuit 27, and the memories 22, 23. , 23 is connected to the reference image cut-out circuit 26, and the connection switching circuit 28 that connects the other of the memories 22, 23 to the comparison image cut-out circuit 27 is compared with the output image data of both the cut-out circuits 26, 27. And a comparison calculation processing unit 29 that performs correlation calculation processing and the like to obtain a movement vector between the two screens and outputs it to the image compression circuit 11.

  In such a movement vector calculation apparatus between two screens, image data of the Nth frame, image data of the (N + 1) th frame, image data of the (N + 2) th frame,... The preprocessing unit 21 thins out the image data of each frame one after another to generate and output reduced image data, and the switch 24 is controlled to be switched by a frame switching signal.

  Thereby, the reduced image data of the Nth frame is stored in the memory 22, the reduced image data of the (N + 1) th frame is stored in the memory 23, and the stored image in the memory 23 with respect to the stored image data (current reference image data) in the memory 22 is stored. The movement vector of the data (current comparison image data) is calculated as described later.

  Next, when the reduced image data of the (N + 2) th frame is overwritten in the memory 22, the stored image data (the next comparison) of the image stored in the memory 23 with respect to the stored image data (the reduced image data of the (N + 1) th frame: the next reference image data). The movement vector of (image data) is calculated as described later, and the same processing is repeated thereafter. Thus, when calculating the movement vector, it is preferable to use the reduced image data in order to reduce the processing load of the comparison calculation processing unit 29 and increase the processing speed.

  The connection switching circuit 28 selects one of the memories 22 and 23 storing the reduced image data to be used as the reference image by the frame switching signal and connects it to the reference image cutting circuit 26, and the other of the memories 22 and 23 is connected. The comparison image cutting circuit 27 is connected.

  First, the reference image cutting circuit 26 selects the divided region 5 in the first row and the first column shown in FIG. 7A in the reference image (in this case, reduced image data of N frames). The reference image is cut out by block 2 and output to the comparison calculation processing unit 29.

  Similarly, the comparative image cutout circuit 27 selects the divided region 5 in the first row and first column from the comparative image (in this case, the reduced image data of N + 1 frame) as the search range, and this search range. The comparison image is cut out from the image by the comparison image cutout block having the same size as that of the block 2, and is output to the comparison calculation processing unit 29.

  The comparison calculation processing unit 29 compares both images captured from the cut-out circuits 26 and 27 on a pixel-by-pixel basis, and compares each pixel with a difference (brightness difference, color difference difference, or RGB color in the case of a primary color filter). In the case of a difference or complementary color filter, a difference for each complementary color is obtained, and an integrated value of the total pixels in each block of each difference is calculated and stored in an internal memory (not shown).

  After the integrated value is calculated, the comparison calculation processing unit 29 outputs a movement command to the comparison image cutting circuit 27. As a result, the comparison image cutout block moves to the next position within the search range (because the reduced image data is used, the movement is in units of one pixel on the reduced image), and the next comparison image is cut out. The comparison calculation processing unit 29 performs a comparison calculation between the comparison image and the reference image, and similarly obtains the integrated value of the difference between the pixels.

  In the same manner, the comparison images in the search range (division region 5 in the first row and first column) are cut out one after another, the integrated value for each comparative image is obtained, and the minimum value (correlation value) of each integrated value is obtained. Find the highest value.

  After the processing in the divided region of the first row and the first column is completed, a search range switching instruction is output to the reference image cutting circuit 26 and the comparative image cutting circuit 27. As a result, the divided region 5 in the first row and the second column shown in FIG. 7A becomes the search range, and the above processing is repeated.

  Similarly, the minimum values Mi (i = 1 to 16) in the 16 search ranges i (i = 1 to 16) divided into 4 rows and 4 columns shown in FIG. 7A are obtained.

  FIG. 2 is a flowchart showing a processing procedure executed after the comparison calculation processing unit 29 obtains the minimum value Mi of the mismatch amount in the 16 search ranges i.

  First, in step S1, the variable i is set to i = 1. In the next step S2, the minimum value Mi of the search range i indicated by the variable i is extracted. Then, the minimum value Mi is compared with the threshold value α (step S3).

  If it is determined in step S3 that Mi ≧ α, the minimum value Mi is discarded in step S4, and the process proceeds to step S6. If it is determined in step S3 that Mi <α, the process proceeds to step S5, the minimum value Mi is selected and held, and the process proceeds to step S6.

  In step S6, it is determined whether i = 16. If i ≠ 16, i = i + 1 is set in step S7, and the process returns to step S2. That is, the above processing is repeated for the search range i = 1 to 16, and the minimum value Mi smaller than the threshold value α among the 16 minimum values Mi is selected and held in step S5.

  When i = 16 in step S6, the process proceeds to next step S8. In step S8, the minimum value Mi selected in step S5 is extracted. Then, the address of the comparison image cutout block in each search range i indicating each minimum value Mi is obtained, and the difference between each address and the reference image cutout block 2, that is, the movement amount and the movement direction, are calculated in the search range i. Calculated as a movement vector.

  In the next step S9, the average value of the movement vectors for each search range i obtained in step S8 is calculated and output to the image compression circuit 11 in FIG. 1 as a movement vector between two screens, and this process is terminated. .

  When the minimum value of the integrated value of each of the 16 search ranges, that is, the minimum value of the mismatch amount is shown in the graph, for example, a case as shown in FIG. 3 occurs. That is, there may be a search range in which the minimum value is larger than the predetermined threshold value α. In the illustrated example, the minimum values Mh,... Of the five search ranges are larger than the threshold value α. That is, the correlation is low.

  For example, when the position of the comparison image that matches the reference image cannot be detected in a certain search range, the amount of mismatch between the comparison image cut out from any location in the search range and the reference image becomes large. However, the “minimum value” of the mismatch amount can be obtained from the inside by calculation processing, and this is calculated as the minimum value Mi of the search range i. However, when the movement vector between the two screens is obtained based on such a minimum value Mi, the accuracy is deteriorated.

  Therefore, the comparison calculation processing unit 29 of the apparatus for calculating a motion vector between two screens according to the present embodiment eliminates processing results in the search range having a larger mismatch amount than the threshold value α shown in FIG. 3 by performing the processing shown in FIG. Thus, only the highly reliable data is left, and the calculation accuracy of the movement vector between the two screens is improved.

  Even a minimum value smaller than the threshold value α shown in FIG. 3 may not be suitable for use when calculating a movement vector between two screens. For example, the correlation between the reference image and the comparison image may be high (the amount of mismatch is small) in the entire search range. Excluding such a case further improves the calculation accuracy of the movement vector between the two screens.

  FIG. 4 is a flowchart showing a processing procedure of a movement vector calculation process between two screens. This process may be performed independently or may be performed in place of steps S8 and S9 in FIG. In the following description, a case where it is combined with the processing of FIG. 2 will be described.

  First, in step S11, one minimum value Mi is extracted from the minimum values Mi selected in step S5 in FIG. In the next step S12, the comparison image processing unit 29 cuts out the comparison image from which the minimum value Mi has been obtained from the integrated values previously calculated and stored (which are stored in the internal memory). An integrated value is obtained by comparing the comparison images of the cut-out position immediately before and the cut-out position immediately after the position with the reference image.

  FIG. 5 illustrates this, and the integrated value Mk−1 at the previous cutting position and the integrated value Mk + 1 at the next cutting position with respect to the minimum value Mk are obtained. Further, an integrated value Mj-1 at the previous cut position and an integrated value Mj + 1 at the next cut position with respect to the minimum value Mj are obtained.

  Then, the difference between the minimum value Mi (Mk, Mj in FIG. 5) and the previous integrated value Mi-1 (Mk-1, Mj-1 in FIG. 5) or the subsequent integrated value Mi + 1 (in FIG. 5). The difference from Mk + 1, Mj + 1) is obtained (step S12).

  After the difference is obtained, the difference is compared with the threshold value β in the next step S13. If the difference is less than β, the minimum value Mi is discarded (step S14). In the example of FIG. 5, the minimum value Mk is discarded. If the difference is greater than or equal to β, the minimum value Mi is selected (step S15). In the example of FIG. 5, the minimum value Mj is selected.

  In step S16 after steps S14 and S15, it is determined whether or not the above processing has been completed for all the minimum values selected in step S5 in FIG. 2, and if completed, the process proceeds to step S17, and in step S15. Using only the selected minimum value Mi, a movement vector between two screens is calculated in the same manner as in steps S8 and S9 in FIG.

  With the above processing, as shown in FIG. 5, it is possible to determine whether or not the minimum value of the integrated values obtained while moving the comparison image cut block is truly “minimum value”. It is possible to calculate the movement vector between the two screens with high accuracy using only the movement vector having high reliability among the movement vectors.

  In the above-described embodiment, the difference between the integrated values to be compared with the threshold value β is set to be one before or one after the minimum value, but may be determined based on both the one before and one after. good. In the embodiment described above, an example in which one screen is divided into 16 has been described, but the number of divisions is not limited thereto.

  The method and apparatus for calculating the movement vector between two screens according to the present invention can increase the calculation accuracy of the movement vector, and is useful when applied to a digital still camera, a digital video camera or the like having a moving image shooting function. It is.

It is a principal part block diagram of the digital still camera with a video recording function carrying the movement vector calculation apparatus between two screens concerning one Embodiment of this invention. It is a flowchart which shows the movement vector calculation process procedure which the comparison calculation process part shown in FIG. 1 performs. It is explanatory drawing of the process of FIG. It is a flowchart which shows the movement vector calculation process procedure which the comparison calculation process part shown in FIG. 1 performs in combination with the process of FIG. 2, or independently. It is explanatory drawing of the process of FIG. It is explanatory drawing of the movement vector calculation principle between 2 screens. It is a figure which shows the example of a screen division at the time of the movement vector calculation between 2 screens.

Explanation of symbols

10 frame memory 11 image compression circuit 22, 23 memory for storing reduced image 26 reference image extraction circuit 27 comparison image extraction circuit 28 connection switching circuit 29 comparison operation processing unit

Claims (12)

  Each image of the first screen and the second screen is divided into the same plurality of regions, and for each divided region, a reference image cut out from the first screen and each comparative image cut out while shifting the position from the second screen, For each of the divided regions, the highest value of the correlation value is calculated for each divided region, and the remaining highest value excluding the highest value that does not reach the predetermined threshold among the highest values is calculated. A movement vector calculation method between two screens, wherein a movement vector of the second screen relative to the first screen is calculated using a movement vector for each.   Among the remaining highest values excluding the highest value that does not reach the predetermined threshold among the highest values, the correlation value of the comparative image in which the position is shifted with respect to the comparative image that gives the highest value for each divided region; Movement of the second screen relative to the first screen is obtained using a movement vector for each divided area in which the difference between the highest value and the highest value that does not reach the predetermined value is calculated and the remaining highest value is calculated. The method of calculating a movement vector between two screens according to claim 1, wherein the vector is calculated.   Each image of the first screen and the second screen is divided into the same plurality of regions, and for each divided region, a reference image cut out from the first screen and each comparative image cut out while shifting the position from the second screen, The correlation value of each of the comparison images is calculated, the highest correlation value is calculated for each divided region, and the position of the comparison image is shifted with respect to the comparison image that gives the highest value for each divided region. The second screen relative to the first screen is obtained using a movement vector for each divided region from which the difference between the value and the maximum value is obtained and the remaining maximum value excluding the maximum value at which the difference does not reach a predetermined value is calculated. A movement vector calculation method between two screens, characterized by calculating a movement vector of.   The correlation value is an integrated value of a difference indicating an amount of mismatch between pixel data between the reference image and the comparative image compared in units of one pixel, and the integrated value of the reference image and the comparative image The method of calculating a movement vector between two screens according to any one of claims 1 to 3, wherein the minimum value is a maximum value of the correlation value.   5. The method of calculating a movement vector between two screens according to claim 4, wherein the pixel data is one or a combination of luminance data, color difference data, and color data.   6. The method for calculating a movement vector between two screens according to claim 1, wherein the reference image and the comparison image are reduced images obtained by reducing the original image data at the same reduction magnification.   Each of the images on the first screen and the second screen is divided into the same plurality of regions, and a reference image cut out from the first screen for each divided region and each comparative image cut out while shifting the position from the second screen. Means for calculating the correlation value, means for calculating the maximum value of the correlation value for each divided region, and the remaining maximum value excluding the maximum value that does not reach the predetermined threshold value among the maximum values was calculated. A movement vector calculation device between two screens, comprising means for calculating a movement vector of the second screen relative to the first screen using a movement vector for each divided region.   Among the remaining highest values excluding the highest value that does not reach the predetermined threshold among the highest values, the correlation value of the comparative image in which the position is shifted with respect to the comparative image that gives the highest value for each divided region; Movement of the second screen relative to the first screen is obtained using a movement vector for each divided area in which the difference between the highest value and the highest value that does not reach the predetermined value is calculated and the remaining highest value is calculated. The apparatus for calculating a movement vector between two screens according to claim 7, wherein the vector is calculated.   Each of the images on the first screen and the second screen is divided into the same plurality of regions, and a reference image cut out from the first screen for each divided region and each comparative image cut out while shifting the position from the second screen. Means for calculating the correlation value, means for calculating the highest value of the correlation value for each divided region, and the comparison image shifted in position relative to the comparison image that gives the highest value for each divided region. A means for obtaining a difference between the correlation value and the highest value, and a moving vector for each divided area from which the highest value other than the highest value at which the difference does not reach a predetermined value is calculated. A movement vector calculation device between two screens, comprising: means for calculating a movement vector of the second screen.   The correlation value is an integrated value of a difference indicating an amount of mismatch between pixel data between the reference image and the comparative image compared in units of one pixel, and the integrated value of the reference image and the comparative image The movement vector calculation device between two screens according to any one of claims 7 to 9, wherein the minimum value is a maximum value of the correlation value.   The apparatus for calculating a movement vector between two screens according to claim 10, wherein the pixel data is one or a combination of luminance data, color difference data, and color data.   The apparatus for calculating a movement vector between two screens according to any one of claims 7 to 11, wherein the reference image and the comparison image are reduced images obtained by reducing the original image data at the same reduction magnification.

Images