JP4696055B2 - Camera shake vector detection method and apparatus, detection program, electronic camera shake correction method and apparatus, correction program, and imaging apparatus - Google Patents

Description

  The present invention relates to a camera shake vector detection method and apparatus and detection program for detecting a camera shake vector with high accuracy in order to perform electronic camera shake correction with high accuracy in a digital camera or the like, an electronic camera shake correction method and apparatus and correction program therefor, and imaging Relates to the device.

  For example, when shooting a video with a digital still camera or video camera with a video shooting function in hand, the position of the still object image shown in the image of a certain frame is the position of the next frame when the hand holding the camera shakes. It moves on the screen, making it difficult to see the video. Therefore, a movement vector (camera shake vector) of an image of the next frame is detected with respect to an image of a certain frame, and camera shake correction is performed so that the image does not shake.

  FIG. 6 is an explanatory diagram illustrating the principle of detecting a movement vector when performing camera shake correction. If the image in the block 2 indicated by the predetermined address in the image 1 of the Nth frame shown in FIG. 6A is a reference image, the same predetermined in the image 3 of the (N + 1) th frame shown in FIG. 6B. If the image cut out by the block 4 indicated by the address is the same as the reference image, the image 3 is not shaken with respect to the image 1.

  However, if camera shake has occurred, the reference image in block 2 and the image cut out in block 4 do not match. Therefore, the block 4 in the image 3 is shifted in the X direction (horizontal direction) and the Y direction (vertical direction) by 4 pixels from 4a, 4b, 4c,..., And the blocks 4a, 4b, 4c,. Each image is compared with a reference image, and a block position where a comparison image having the highest correlation with the reference image is cut out is obtained.

  In the example illustrated in FIG. 6B, if the comparison image having the highest correlation with the reference image of the block 2 is cut out at the block 4 c, the difference between the image 3 and the image 1 is a vector “k”. This vector k is a movement vector between two screens. If image 1 is displayed and then image 3 is moved in the direction opposite to that of movement vector k and displayed, an image without camera shake can be displayed.

  The movement vector (shake vector) k is obtained based on such a principle. Normally, as shown in FIG. 7A, one screen is divided into a plurality of areas (in the example shown, a total of 16 in 4 rows and 4 columns). The movement vectors k1 to k16 are obtained for each divided screen as shown in FIG. 7B, and these are averaged to obtain the camera shake vector k.

  In the prior art described in Patent Document 1, the screen is divided into a peripheral area and a central area, and the motion vector detected in the peripheral area is determined to have a high probability of being a motion vector due to camera shake. .

  As another conventional technique related to the movement vector detection technique, there is Patent Document 2 below.

JP-A-8-251473 JP-A-6-46316

  As described above, one screen is divided into a plurality of areas, a movement vector is obtained for each divided screen, and the average value is used as a movement vector for the entire screen, that is, a movement vector caused by camera shake. Thus, the hand shake vector can be calculated with higher accuracy than the movement vector obtained from the above.

  However, we do not know what is in the actual scene image, and some of the divided screens show the head and hands of the person walking, and the divided screen below shows the person's feet. In other split screens, birds are flying and dogs and cats are moving around. There is also a possibility that a moving object exists in the entire peripheral area.

  Therefore, the movement vectors k1 to k16 shown in FIG. 7B are combined vectors of the movement vector of the moving object in the divided screen and the true camera shake vector, and have various directions and lengths. Therefore, even if each of the movement vectors k1 to k16 is simply averaged or the average value is obtained by excluding the movement vector indicating the abnormal value, the camera shake vector cannot be calculated with high accuracy.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a camera shake vector detection method and apparatus, a detection program, an electronic camera shake correction method and apparatus, a correction program, and an imaging apparatus that can detect a camera shake vector with high accuracy.

  The hand movement vector detection method, detection program, and detection apparatus of the present invention divide an imaging screen into a plurality of regions, calculate a movement vector for each divided screen, and calculate a hand movement vector for the entire screen from each movement vector. In the method, a combination of movement vectors to be paired among the plurality of movement vectors is obtained, and for each pair, a vector value obtained by performing a predetermined operation on two movement vectors constituting the pair is calculated, and 2 of each pair is calculated. The hand movement vector is calculated by calculating an average of the vector values for each pair using a distance between two divided screens for which one movement vector is obtained as a weight.

  The shake vector detection method, detection program, and detection apparatus according to the present invention provide the shake that is obtained by the addition average calculation when the denominator value of the arithmetic expression of the addition average calculation performed by weighting is less than a certain threshold value. It is characterized by determining that the vector is not reliable and stopping the detection of the shake vector.

  The shake vector detection method, the detection program, and the detection apparatus according to the present invention set the calculation contribution ratio of the pair to the shake vector to be “0” when the difference between the two movement vectors forming the pair differs by a predetermined threshold or more. It is characterized by that.

  The camera shake vector detection method, the detection program, and the detection apparatus according to the present invention are characterized in that the pair whose distance is equal to or less than a predetermined threshold is excluded from the arithmetic mean calculation of the camera shake vectors.

  An electronic camera shake correction method, a correction program, and a correction device according to the present invention are characterized by cutting out an output range from captured image data based on the above-described camera shake vector detection method, detection program, and detection device, and the detected camera shake vector. .

  An imaging apparatus according to the present invention includes a solid-state imaging device that captures a subject image, and the above-described electronic camera shake correction device that takes in output data of the solid-state imaging device and performs electronic camera shake correction processing.

  According to the present invention, since the hand shake vector can be calculated with high accuracy, a hand shake correction value with little error and high accuracy can be obtained, and natural electronic hand shake correction can be performed.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram of a digital camera according to an embodiment of the present invention. The digital camera includes an imaging unit 21, an analog signal processing unit 22 that performs analog processing such as automatic gain adjustment (AGC) and correlated double sampling processing on analog image data output from the imaging unit 21, and an analog signal processing unit. An analog / digital conversion unit (A / D) 23 that converts analog image data output from the digital image data into digital image data, and an A / D 23, an analog signal processing unit 22, according to instructions from a system control unit (CPU) 29 described later, A driving unit (including a timing generator TG) 24 that controls driving of the imaging unit 21 and a flash 25 that emits light in response to an instruction from the CPU 29 are provided.

  The imaging unit 21 collects light from the object field, a diaphragm or a mechanical shutter 21b that condenses the light that has passed through the optical lens system 21a, and a diaphragm that is condensed by the optical lens system 21a. A solid-state image sensor 21c that receives the received light and outputs captured image data (analog image data).

  The digital camera of this embodiment further includes a digital signal processing unit 26 that takes in digital image data output from the A / D 23 and performs white balance correction, RGB / YC conversion processing, and the like, and converts the image data into image data such as JPEG format. A compression / expansion processing unit 27 that compresses or reversely expands, a display unit 28 that displays a menu, displays a through image or a captured image, and a system control unit (CPU) 29 that performs overall control of the entire digital camera A media interface (I / F) unit 31 that performs an interface process between an internal memory 30 such as a frame memory and a recording medium 32 that stores JPEG image data, and a bus 40 that interconnects these. In addition, the system control unit 29 is connected to an operation unit 23 for inputting instructions from the user.

  FIG. 2 is a functional configuration diagram of an electronic camera shake correction processing device 50 configured when electronic camera shake correction processing is performed by the digital camera shown in FIG. In the electronic camera shake correction processing device 50, when the user inputs an instruction to perform electronic camera shake correction processing from the operation unit 33 shown in FIG. 1, the system control unit 29 includes an electronic camera shake including a camera shake vector detection program. By starting the correction processing program, the system control unit 29 is configured using the subordinate digital signal processing unit 26, the internal memory 30, and the like.

  This electronic camera shake correction processing device 50 performs decimation processing on the image data output from the solid-state imaging device 21c to generate a reduced image, and the nth frame (even frame) decimation reduced image data. A first memory 52 to store, a second memory 53 to store the thinned reduced image data of the (n + 1) th frame (odd frame), and a hand shake vector detection process for detecting the hand shake vector by fetching the reduced image data from both memories 52 and 53 And a read control processing unit 56 that determines a read range (cutout range) of the captured image data in the internal memory 30 based on the detected camera shake vector.

  In the present embodiment, in order to increase the speed of the camera shake vector detection process, the thinning processing unit 51 generates reduced images of the reference image and the comparison image. If a high-performance arithmetic processing device can be used and the memory capacity is sufficient, it is also possible to calculate a camera shake vector using original image data without performing thinning processing.

  FIG. 3 is a detailed configuration diagram of the camera shake vector detection processing unit 55 shown in FIG. The hand movement vector detecting unit 55 connects the reference image cutting circuit 61, the comparison image cutting circuit 62, the both cutting circuits 61 and 62, and the first and second memories 52 and 53 based on the frame switching signal. A changeover switch 63 for switching, and a comparison operation unit for performing a comparison operation between the reference image cut out from the reference image cut-out circuit 61 and the comparison image cut out from the comparison image cut-out circuit 62 to calculate a movement vector for each divided screen. 64 and a weighting calculator 65 for calculating a hand movement vector from each movement vector as will be described in detail later.

  In this embodiment, as shown in FIG. 7A, one screen is divided into a total of 16 divided screens of 4 rows and 4 columns, and movement vectors k1 to k16 for each divided screen shown in FIG. calculate. It is also possible to calculate a total of 16 movement vectors simultaneously by parallel operation.

  However, in this embodiment, since the respective movement vectors in the divided screens from # 1 to # 16 are calculated in order, the comparison arithmetic unit 64 causes the reference image cutting circuit 61 and the comparative image cutting circuit 62 to search the search range #. 1 to # 16 are output in order, and a movement instruction for the extracted blocks (4a, 4b, 4c, etc. in FIG. 6B) within the specified search range is output to the comparative image extraction circuit 62. It is like.

  Next, moving image shake correction processing in the electronic shake correction processing device 50 having the above-described configuration will be described. From the solid-state imaging device 21 c, even-frame captured image data and odd-frame captured image data are alternately output, and each captured image data is stored in the internal memory 30.

  In parallel with the storage in the internal memory 30, the captured image data output from the solid-state imaging device 21 c passes through the thinning processing unit 51 and becomes reduced image data, and is stored in the memories 52 and 53.

  Assume that a camera shake vector of image data in the (n + 1) th frame is detected with respect to image data in the nth frame. That is, the reference image is cut out from the image data (reduced image data) of the nth frame, the comparison image is cut out from the image data (reduced image data) of the (n + 1) th frame, and the correlation between both is obtained.

  First, the cutout circuit 61 cuts out the reference image in the # 1 divided screen by a predetermined block frame and outputs it to the comparison computing unit 64, and also compares the comparison image in the # 1 divided screen by a cutout block having the same size as the predetermined block frame. The cut-out circuit 62 outputs the cut-out comparison calculator 64.

  The comparison computing unit 64 obtains a difference between the pixel data of the reference image and the pixel data of the comparison image in units of one pixel of the reduced image and obtains an integrated value of the differences for all the pixels of the reduced image. Each time the integrated value is obtained, a movement instruction is output to the comparative image cutting circuit 62 to move the cut block position in units of one pixel of the reduced image, and the integrated value of the comparative image after the movement and the reference image is moved. The operation of asking for is repeated.

  It can be said that the comparison image showing the minimum value among the respective integrated values thus obtained has the highest correlation with the reference image, and the difference between the cut-out position of the comparison image and the cut-out position of the reference image Becomes the movement vector in the # 1 divided screen.

  Such an operation is also performed on the divided screens # 2 to # 16, and for example, 16 movement vectors k1 to k16 obtained as shown in FIG. The weighting calculator 65 calculates the movement vector k for the entire screen by calculating the 16 movement vectors k1 to k16 passed from the comparison calculator 64 based on the following equation.

  Here, the denominator “f” is a value that is “1” when the pair is established and “0” when the pair is not established. A pair refers to any two sets in the movement vectors k1 to k16.

  For example, it indicates a set of (k2, k11), for example, among the 16 movement vectors k1 to k16 shown in FIG. When the difference between the vector k2 and the vector k11 is equal to or less than the predetermined threshold A, the pair is established, and when the difference exceeds the predetermined threshold A, the pair is not established.

  The “Pair value” is “0” when the pair is not established, and is a vector value obtained by performing a predetermined operation on the vector k2 and the vector k11, for example, an addition average vector value when the pair is established.

  As shown in FIG. 5, “L (n)” is the distance between both vectors (between divided screens # 2 and # 11) when the pair is k2 and k11.

  That is, in Equation 1, the weighted average value of the sum of all pairs is calculated as a shake vector. There are a total of 15 pairs that are paired with k1, 15 pairs that are paired with k2,..., 15 pairs that are paired with k16. It is possible to calculate a hand shake vector with high accuracy by averaging the distances as weights for each pair.

  As described above, in the present embodiment, when a movement vector having the same direction and length is detected at a distance away from each other regardless of the location on the screen, the movement vector is the “shake vector”. If the probability is high, the contribution ratio is increased, and even if the movement vectors have the same direction and length, the contribution ratio is reduced because the probability that the movement vector is a movement vector is high. For this reason, it is possible to improve the calculation accuracy of the hand shake vector regardless of where the moving object is present on the screen.

  The calculated camera shake vector is output to the read control processing unit 56 in FIG. 2, and the cutout range (output range) of the n + 1 frame image cut out and output from the internal memory 30 is controlled.

  Next, when the captured image data of the (n + 2) th frame is output from the solid-state imaging device 21c and stored in the internal memory 30, and the reduced image data is generated by the thinning processing unit 51, the reduced image data is overwritten in the memory 52. The Then, the changeover switch 63 is switched, and this time, the reference image is cut out from the n + 1 frame reduced image, the comparison image is cut out from the n + 2 frame reduced image, and the camera shake vector is detected in the same manner as described above.

  In the above-described embodiment, the screen is divided into 16 to obtain 16 movement vectors. However, it is also possible to further subdivide the screen to obtain the movement vector of each divided screen and calculate the hand movement vector. However, in this case, a new problem arises that the number of pairs increases and the calculation load increases. Therefore, in such a case, in Equation 1, when the distance L (n) is a short distance equal to or smaller than the threshold (distance L (n) is known), the number of pairs to be actually calculated is excluded from the calculation. It is effective to reduce

  Since the denominator exists in Equation 1, the denominator may become “0”. That is, there are cases where the movement vectors in all the divided screens are disjoint and all pairs are not established. For example, when a dark night sky is photographed or when a wall of the same color is photographed, the movement vectors in the divided screens vary. In such a case, the camera shake correction process is stopped.

  Alternatively, when the denominator of Formula 1 is calculated, it is determined whether or not the calculated value is smaller than a predetermined threshold value. If the calculated value is small, it can be determined that the shake vector obtained in Formula 1 is not reliable. Is stopped to suspend the camera shake correction, or the camera shake vector is calculated by another method.

  In the above-described embodiment, the processing is described as the processing inside the digital camera. However, all the image data output from the solid-state imaging device 21c is recorded, and this is captured in an external personal computer or the like. The same processing can be performed by executing a program or an electronic image stabilization program.

  Since the camera shake vector detection method and the like according to the present invention can detect a camera shake vector with high accuracy, it is useful to apply this to a digital camera with an electronic camera shake correction function.

It is a block block diagram of the digital camera which concerns on one Embodiment of this invention. 1 is a functional configuration diagram of an electronic camera shake correction processing apparatus according to an embodiment of the present invention. FIG. 3 is a detailed configuration diagram of a camera shake vector detection processing unit illustrated in FIG. 2. It is an illustration figure of the movement vector calculated | required for every divided screen. It is explanatory drawing of distance L (n) between the movement vectors used as a pair. It is explanatory drawing of the calculation principle of a movement vector. It is explanatory drawing of a division | segmentation screen.

Explanation of symbols

21 Imaging unit 21c Solid-state imaging device 26 Digital signal processing unit 29 System control unit 50 Electronic camera shake correction processing unit 55 Camera shake vector detection processing unit 61 Reference image extraction circuit 62 Comparative image extraction circuit 64 Comparison calculator 65 Weighting calculator

Claims (16)

  In the hand movement vector detection method of dividing the imaging screen into a plurality of areas, calculating a movement vector for each divided screen, and calculating a hand movement vector of the entire screen from each movement vector, the pair of movement vectors among the plurality of movement vectors For each pair, a vector value obtained by performing a predetermined operation on the two movement vectors constituting the pair is calculated, and the distance between the two divided screens for obtaining the two movement vectors of each pair is calculated. A camera shake vector detection method, wherein the camera shake vector is calculated by calculating an average of the vector values for each pair as a weight.   When the denominator value of the arithmetic expression of the weighted average calculation performed by the weighting is less than a certain threshold value, it is determined that there is no reliability of the shake vector obtained by the average calculation, and the shake vector is detected. The camera shake vector detection method according to claim 1, wherein the camera shake vector detection method is stopped.   The calculation contribution rate to the camera shake vector of the pair is set to “0” when the difference between the two movement vectors that form the pair differs by a predetermined threshold value or more. Camera shake vector detection method.   4. The camera shake vector detection method according to claim 1, wherein the pair whose distance is equal to or less than a predetermined threshold is excluded from the arithmetic mean calculation of the camera shake vectors. 5.   5. An electronic camera shake correction method, wherein the output range is determined based on a camera shake vector detected by the camera shake vector detection method according to claim 1 when the output range is cut out from captured image data. .   In a hand shake vector detection program that divides an imaging screen into a plurality of areas, calculates a motion vector for each divided screen, and calculates a hand shake vector of the entire screen from each motion vector, a pair of motion vectors among the plurality of motion vectors And calculating a vector value obtained by performing a predetermined operation on the two movement vectors constituting the pair for each pair, and the distance between the two divided screens for obtaining the two movement vectors of each pair. And a step of calculating the camera shake vector by calculating an average of the vector values for each pair as weights.   When the denominator value of the arithmetic expression of the weighted average calculation performed by the weighting is less than a certain threshold value, it is determined that there is no reliability of the shake vector obtained by the average calculation, and the shake vector is detected. The program according to claim 6, which stops.   8. The calculation contribution rate of the pair to the camera shake vector is set to “0” when the difference between the pair of two movement vectors differs by a predetermined threshold value or more. 8. Camera shake vector detection program. 9. The camera shake vector detection program according to claim 6, wherein the pair whose distance is equal to or less than a predetermined threshold is excluded from the arithmetic mean calculation of the camera shake vectors.   A camera shake vector detection program according to any one of claims 6 to 9, and a step of determining the output range based on a camera shake vector detected by the camera shake vector detection program when the output range is cut out from captured image data. An electronic camera shake correction program characterized by comprising:   In a camera shake vector detection device that divides an imaging screen into a plurality of areas, calculates a motion vector for each divided screen, and calculates a camera shake vector of the entire screen from each of the motion vectors, a pair of motion vectors among the plurality of motion vectors And calculating a vector value obtained by performing a predetermined operation on the two movement vectors constituting the pair for each pair, and the distance between the two divided screens for obtaining the two movement vectors of each pair And a means for calculating the hand shake vector by calculating an average of the vector values for each pair as a weight.   When the denominator value of the arithmetic expression of the weighted average calculation performed by the weighting is less than a certain threshold value, it is determined that there is no reliability of the shake vector obtained by the average calculation, and the shake vector is detected. The hand movement vector detection device according to claim 11, wherein the hand movement vector detection device is stopped.   13. The calculation contribution ratio of the pair to the hand movement vector is set to “0” when the difference between the two movement vectors forming the pair differs by a predetermined threshold or more. 13. Hand shake vector detection device.   The camera shake vector detection device according to claim 11, wherein the pair whose distance is equal to or less than a predetermined threshold is excluded from the arithmetic mean calculation of the camera shake vectors.   15. The shake vector detecting device according to claim 11, and a read control process for determining the output range based on a shake vector detected by the shake vector detecting device when an output range is cut out from captured image data. And an electronic camera shake correction device.   An image pickup apparatus comprising: a solid-state image pickup device that picks up a subject image; and an electronic camera shake correction device according to claim 15 that takes in output data of the solid-state image pickup device and performs electronic camera shake correction processing.

Images