JP2007243332A - Motion vector calculation method, motion vector calculation apparatus, and imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a motion vector calculation method capable of efficiently calculating a motion vector. <P>SOLUTION: The motion vector calculation method includes the steps of: segmenting reference image data from a reference block set to first image data; setting a prescribed range of second image data and segmenting comparison image data from each of comparison blocks of each moving position when the comparison blocks of the same size as that of the reference block are moved within the prescribed range; calculating the correlativity between the reference image data and the comparison image data segmented from each comparison block at each moving position; calculating a motion vector from a position of the reference block and a position of the comparison block from which the comparison image data with the highest correlativity are segmented; and determining the prescribed range in response to a photographing magnification set to an imaging element. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a movement vector calculation method for obtaining a movement vector between first image data and second image data obtained from an image sensor, a movement vector calculation apparatus, and an imaging apparatus including the same.

  For example, when shooting a video with an imaging device such as a digital still camera or video camera having a video shooting function in hand, if the hand holding the camera shakes, the position of the still image displayed in the image of a certain frame It moves on the screen of the next frame, and it becomes a hard-to-see video. Therefore, a movement vector of an image of the next frame is detected with respect to an image of a certain frame, and camera shake correction is performed so that the image does not shake.

FIG. 5 is an explanatory diagram illustrating the principle of calculating a movement vector when performing camera shake correction.
In a digital camera, image data with the maximum resolution (hereinafter referred to as maximum image data) obtained from an image sensor is sequentially input to the camera shake correction apparatus at a predetermined frame rate during moving image shooting. In the camera shake correction apparatus, as shown in FIG. 5A, in the maximum image data 61 of the nth frame obtained from the image sensor, image data of a predetermined size that is cut out to be actually recorded as one frame of a moving image ( Hereinafter, a cutout position of recording image data) is determined, and the image data 66 at the cutout position is cut out and recorded on a recording medium or the like. In this case, if the image data in the block 62 indicated by the predetermined address in the entire image data 61 of the nth frame shown in FIG. 5A is the reference image data, the entire n + 1th frame shown in FIG. If the comparison image data cut out by the block 64 indicated by the same predetermined address as the block 62 in the image data 63 is the same as the reference image data, the whole image data 63 is not shaken with respect to the whole image data 61. Become.

  However, if camera shake has occurred, the reference image data in the block 62 and the comparison image data cut out in the block 64 do not match. Therefore, in the camera shake correction apparatus, the block 64 in the entire image data 63 is represented as blocks 64a, 64b, 64c,... Within a predetermined range 65 (hereinafter referred to as a search range) set in the entire image data 63. , Each of the comparison image data in the blocks 64a, 64b, 64c,... Moved within the search range 65 while shifting by one pixel in the X direction (horizontal direction) and the Y direction (vertical direction). Compared with the data, the correlation between each comparison image data and the reference image data is calculated, and the block position where the comparison image data having the highest correlation with the reference image data is cut out is obtained.

  In the example shown in FIG. 5B, if the comparison image data having the highest correlation with the reference image data of the block 62 is cut out by the block 64 c in the search range 65, the entire image data 63 with respect to the entire image data 61 is extracted. The difference is “k”, and k is a movement vector of the entire image data 63 with respect to the entire image data 61. Therefore, in the camera shake correction apparatus, as shown in FIG. 5C, the recording image data cutout position set in the entire image data 63 is moved by the movement vector k from the position where the image data 66 is cut out. If the image data 67 is cut out from this cut-out position and recorded on a recording medium or the like, image data with corrected camera shake can be obtained. Since the cutout position can be shifted within the range between the end of the recording image data cut out from the image data of the first frame and the end of the entire image data, this range performs camera shake correction. This is the range of camera shake correction that can be performed.

  The maximum value of the movement vector that can be calculated by the camera shake correction apparatus varies depending on the shooting magnification (optical zoom magnification or electronic zoom magnification) set in the image sensor. For example, when the shooting magnification is small, the maximum value of the movement vector is small, and when the shooting magnification is large, the maximum value of the movement vector is also large. When the maximum value of the movement vector is small, the camera shake correction range may be small. However, when the maximum value of the movement vector is large, it is necessary to increase the camera shake correction range. In general, the camera shake correction range is constant regardless of the shooting magnification, so if the shooting magnification is low, the camera shake correction range is too wide, and this requires extra image processing. End up. Therefore, conventionally, an electronic camera has been proposed in which the camera shake correction range is optimized by changing the camera shake correction range in accordance with the shooting magnification (see Patent Document 1).

JP 2001-285698 A

  When the maximum value of the movement vector that can be calculated by the camera shake correction apparatus is small, the movement vector can be calculated even if the search range is narrower than when the maximum value of the movement vector is large. However, in the electronic camera described in Patent Document 1, since the search range is constant regardless of the shooting magnification, even when the maximum value of the movement vector is small, it is necessary to perform the same calculation as when the maximum value of the movement vector is large. Therefore, the calculation efficiency of the movement vector was bad.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide a movement vector calculation method capable of efficiently calculating a movement vector.

  The movement vector calculation method of the present invention is a movement vector calculation method for obtaining a movement vector of the second image data with respect to the first image data obtained from the image sensor, wherein the reference block is set in the first image data. A reference image data cutout step for cutting out reference image data from the image data, and each movement position when a predetermined range is set in the second image data and a comparison block having the same size as the reference block is moved within the predetermined range A comparison image data cutout step for cutting out comparison image data from each of the comparison blocks, and a correlation between the comparison image data cut out from each of the comparison blocks at the respective movement positions and the reference image data. A correlation calculation step, and a position of the comparison block obtained by cutting out the comparison image data having the highest correlation. And a and a position of the reference block includes a motion vector calculation step of calculating the movement vector, and a predetermined range determining step of determining the predetermined range in accordance with the set photographing magnification to the image sensor.

  In the movement vector calculation method of the present invention, in the predetermined range determining step, the predetermined range is relatively wide when the shooting magnification is relatively large, and the predetermined range is set when the shooting magnification is relatively small. Make it relatively narrow.

  A movement vector calculation apparatus according to the present invention is a movement vector calculation apparatus for obtaining a movement vector of second image data with respect to first image data obtained from an image sensor, wherein the reference block is set in the first image data. Reference image data cutout means for cutting out the reference image data from the image, and each movement position when a predetermined range is set in the second image data and the comparison block having the same size as the reference block is moved within the predetermined range The comparison image data cutout means for cutting out the comparison image data from each of the comparison blocks, the correlation between the comparison image data cut out from each of the comparison blocks at the respective movement positions, and the reference image data is calculated. Correlation calculation means, the position of the comparison block from which the comparison image data having the highest correlation is cut out, and the reference And a locking position, including a motion vector calculation means for calculating the movement vector, and a predetermined range determining means for determining the predetermined range in accordance with the set photographing magnification to the image sensor.

  In the movement vector calculation device according to the present invention, the predetermined range determining means relatively widens the predetermined range when the shooting magnification is relatively large, and sets the predetermined range when the shooting magnification is relatively small. Make it relatively narrow.

  The imaging apparatus according to the present invention is an imaging apparatus having a moving image shooting function, wherein the movement vector calculation device and recording image data cut out from the second image data according to the movement vector calculated by the movement vector calculation device A camera shake correction apparatus including a cutout position setting unit for setting the cutout position of the image pickup device, and the image pickup device.

  ADVANTAGE OF THE INVENTION According to this invention, the movement vector calculation method which can calculate a movement vector efficiently can be provided.

FIG. 1 is a block diagram showing a schematic configuration of a digital camera as an example of an imaging apparatus for explaining an embodiment of the present invention.
A digital camera 100 shown in FIG. 1 includes an imaging unit 1 including a photographing lens 1a and an imaging element 1b, an analog signal processing unit 2, an A / D conversion unit 3, a driving unit 4, a lens driving unit 5, and a digital camera. Signal processing unit 6, compression / decompression processing unit 7, display unit 8, system control unit (CPU) 9, internal memory 10, media interface 11, recording medium 12, operation unit 13, camera shake correction Device 15. The digital signal processing unit 6, the compression / decompression processing unit 7, the display unit 8, the system control unit 9, the internal memory 10, the media interface 11, and the camera shake correction device 15 are connected to the system bus 14.

  The imaging unit 1 images a subject with an optical system 1a including a zoom lens and a photographing lens, and an imaging element 1b such as a CCD type or CMOS type solid-state imaging element, and outputs an analog imaging signal. The analog signal processing unit 2 performs predetermined analog signal processing on the imaging signal obtained by the imaging unit 1. The A / D conversion unit 3 converts the analog signal processed by the analog signal processing unit 2 into a digital signal.

  When the digital camera 100 is set to a photographing mode (a mode in which a subject can be photographed and photographed image data can be recorded), the driving unit 4 is driven by a solid-state imaging device 1b, The analog signal processing unit 2 and the A / D conversion unit 3 are driven. The digital camera 100 can set a moving image shooting mode and a still image shooting mode.

  When an instruction to change the optical zoom magnification is given from the operation unit 13, the lens driving unit 5 moves the zoom lens included in the optical system 1a in accordance with the instruction and changes the photographing magnification set in the image sensor 1b. The lens driving unit 5 is controlled by the system control unit 9.

  The digital signal processing unit 6 performs digital signal processing corresponding to the operation mode set by the operation unit 13 on the digital signal from the A / D conversion unit 3 to generate photographed image data. The processing performed by the digital signal processing unit 6 includes black level correction processing (OB processing), linear matrix correction processing, white balance adjustment processing, gamma correction processing, and synchronization processing. The digital signal processing unit 6 is configured by a DSP, for example. When the moving image shooting mode is set, the digital signal processing unit 6 sequentially outputs, for example, maximum resolution image data at a predetermined frame rate, and stores these in the internal memory 10. When an instruction to change the electronic zoom magnification is given from the operation unit 13, the digital signal processing unit 9 performs image processing according to the instruction and changes the photographing magnification set in the image sensor 1 b.

  The compression / decompression processing unit 7 performs compression processing on the captured image data generated by the digital signal processing unit 6 and performs expansion processing on the compressed image data obtained from the recording medium 12.

  The display unit 8 includes an LCD display device, for example, and displays an image based on photographed image data that has been photographed and subjected to digital signal processing. An image is also displayed based on the image data obtained by decompressing the compressed image data recorded on the recording medium 12. It is also possible to display a through image in the shooting mode, various states of the digital camera, information on operations, and the like.

  The system control unit 9 is mainly configured by a processor that operates according to a predetermined program, and performs overall control of the entire digital camera 100 including shooting operations. When there is an instruction to change the photographing magnification from the operation unit 13, the system control unit 9 notifies the lens driving unit 5 and the digital signal processing unit 6 of this instruction, and performs control to change the photographing magnification.

  The internal memory 10 is, for example, a DRAM, and is used as a work memory for the camera shake correction device 15, the digital signal processing unit 6, and the system control unit 9, and temporarily stores captured image data recorded on the recording medium 12. It is also used as a buffer memory for storing and a buffer memory for image data for display on the display unit 8. The media interface 11 inputs / outputs data to / from a recording medium 12 such as a memory card.

  The operation unit 13 performs various operations when using the digital camera, and includes a release button (not shown) for instructing photographing.

FIG. 2 is a block diagram showing a schematic configuration of the camera shake correction apparatus 15 shown in FIG.
The camera shake correction device 15 includes an imaging magnification acquisition unit 21 that acquires an imaging magnification specified by the system control unit 9, a search range determination unit 22 that determines a search range according to the imaging magnification acquired by the imaging magnification acquisition unit 21, and Based on the search range determined by the search range determination unit 22, at time t _{n + 1} (n is a natural number), the maximum image data obtained from the image sensor 1b at the time t _n and the maximum image data D _n (in the claims) The movement vector calculation unit for obtaining the movement vector B _{n + 1} of the maximum image data D _{n + 1} (corresponding to the second image data in the claims) obtained from the image sensor 1b at time t _{n + 1} with respect to the first image data) 23, in accordance with the movement moving vector B _{n + 1} calculated by the vector calculating unit 23, the maximum image de to offset movement based on the movement vector B _{n + 1} Set the cut-out position of the recording image data to be cut out from the data D _{n + 1,} and a cutout position setting section 24 for outputting a clipping position signal to the internal memory 10.

FIG. 3 is a block diagram showing a schematic configuration of the movement vector calculation unit 23 shown in FIG.
Motion vector calculation unit 23 shown in FIG. 3 includes a memory 31 for storing capture the maximum image data D _n, and sets a reference block in a predetermined address position of the maximum image data D _n stored in the memory 31, the reference set the reference image data cutout section 32 cuts out reference image data from the block, and a memory 33 for storing capture the maximum image data D _{n + 1,} which is determined in the maximum image data D _{n + 1} stored in the memory 33 in the search range determining section 22 A comparison image data cutout unit 34 that sets a search range and cuts out comparison image data from each comparison block at each movement position when the comparison block having the same size as the reference block is moved within the search range; The comparison image data at each moving position cut out by the comparison image data cutout unit 34 and the reference image data cutout unit 32 A correlation calculation unit 35 that calculates the correlation with the extracted reference image data, and the position of the comparison block and the position of the reference block from which the comparison image data calculated by the correlation calculation unit 35 has the highest correlation are extracted. To a shift amount calculation unit 36 for calculating a shift amount of the maximum image data D _{n + 1} with respect to the maximum image data D _n .

  The search range determination unit 22 determines a relatively wide search range when the set shooting magnification is relatively large, and searches relatively narrow when the set shooting magnification is relatively small. Determine the range.

  Information of the search range determined by the search range determination unit 22 is input to the comparison image data cutout unit 34, and the comparison image data cutout unit 34 sets a search range according to the input information.

  The correlation calculation unit 35 uses, as a value indicating the correlation between the reference image data and the comparison image data, for example, pixel data at an arbitrary position of the reference image data and pixels of the comparison image data at the same position as the pixel data. A value obtained by adding the absolute value of the difference from the data for all positions (hereinafter referred to as a mismatch value) is obtained. Then, the correlation calculation unit 35 determines that the comparison image data from which the smallest mismatch value is obtained is comparison image data having the highest correlation with the reference image data, and the comparison block data obtained by cutting out the comparison image data. The cutout position is notified to the deviation amount calculation unit 36.

  The shift amount calculation unit 36 calculates the shift amount of the comparison block cut-out position notified from the correlation calculation unit 35 with respect to the position where the reference block is cut out, and outputs the shift amount as a movement vector.

Next, the operation of the camera shake correction device 15 will be described.
First, when the user operates the operation unit 13 to set the shooting magnification, the set shooting magnification is acquired by the shooting magnification acquisition unit 21, and this is input to the search range determination unit 22, according to the set shooting magnification. A search range is determined. Then, imaging is started by the imaging device 1b at the set imaging magnification.

When the maximum image data D ₁ obtained from the image sensor ₁ b at time t ₁ is stored in the internal memory 10, the maximum image data D ₁ is stored in the memory 31, and the maximum image data D is stored by the cutout position setting unit 24. _One cutout position is set. Accordingly, the recording image data from the set cut position to the maximum image data D ₁ is cut out, recording image data cut out is recorded in the recording medium 12. The recording image data may be cut out and recorded by the cut-out position setting unit 24, the system control unit 9, or the like.

Next, when the maximum image data D ₂ obtained from the image sensor 1 b is stored in the internal memory 10 at time t ₂ , this is stored in the memory 33.

Next, the reference image data cutout unit 32 cuts out the reference image data from the maximum image data D ₁ and inputs this to the correlation calculation unit 35.

Then, by comparing the image data cutting unit 34, the search range input from the search range determining section 22 is set to the maximum image data D _2, comparison of the movement position when moving the comparison block within the search range The comparison image data is cut out from the block and sequentially input to the correlation calculation unit 35.

  Next, the correlation calculation unit 35 calculates the mismatch value indicating the correlation between the comparison image data and the reference image data, and the cut-out position of the comparison block from which the comparison image data from which the smallest mismatch value is obtained is obtained as follows: The deviation amount calculation unit 36 is notified.

  Next, a deviation amount between the comparison block position and the reference block position notified from the correlation calculation unit 35 is calculated by the deviation amount calculation unit 36, and this deviation amount is input to the cutout position setting unit 24 as a movement vector. Is done.

Then, the cutout position setting unit 24, the extraction position according to the movement vector is set to the maximum image data D _2. Accordingly, the recording image data from the set cut position to the maximum image data D ₂ is cut out, recording image data cut out is recorded in the recording medium 12.

  As described above, according to the camera shake correction device 15, the correlation between the comparison image data at each moving position and the reference image data when the comparison block is moved within the search range determined according to the shooting magnification. Is calculated, a movement vector is calculated based on this correlation, and camera shake correction is realized.

  As described above, the maximum value of the movement vector that can be calculated by the camera shake correction device 15 varies depending on the shooting magnification (optical zoom magnification or electronic zoom magnification) set in the image sensor 1b. Therefore, the search range can also be changed according to the shooting magnification. The reason for this will be described with reference to FIG. In FIG. 4, reference numeral 40 indicates the maximum image data obtained from the image sensor 1b, reference numeral 41 indicates a search range, and reference numeral 42 indicates a comparison block set at the same position as the reference block.

  For example, consider a case where the maximum value of the movement vector that can be calculated by the movement vector calculation unit 23 is one pixel data in the horizontal direction and one pixel data in the vertical direction. In this case, the range between the search range 41 and the comparison block 42 (hatched range) set in the maximum image data 40 shown in FIG. 4 is that the comparison block 42 is at least one pixel data vertically and horizontally. Any range that can be moved is acceptable. On the other hand, when the maximum value of the movement vector that can be calculated by the movement vector calculation unit 23 is 10 pixel data in the horizontal direction and 10 pixel data in the vertical direction, the search range 41 and the comparison block 42 shown in FIG. The range between (a hatched range) needs to be a range in which the comparison block 42 can be moved by at least 10 pixels in the vertical and horizontal directions.

  Thus, it can be seen that the minimum size of the search range 41 is determined by the maximum value of the movement vector. According to the camera shake correction apparatus 15 of the present embodiment, the size of the search range 41 is large when the shooting magnification is large, that is, when shooting is performed such that the camera shake is relatively large and the maximum value of the movement vector is large. Is set to a size that is at least necessary for the calculation of the movement vector, for example, and the shooting magnification is small, that is, when the camera shake is relatively small and the maximum value of the movement vector is small. In some cases, the size of the search range 41 can be set small, for example, so as to be the minimum size necessary for calculating the movement vector. As a result, it is possible to optimize the processing amount in the comparison image data cutout unit 34 and the correlation calculation unit 35, and to enable efficient camera shake correction.

1 is a block diagram showing a schematic configuration of a digital camera as an example of an imaging apparatus for describing an embodiment of the present invention. The block diagram which shows schematic structure of the camera-shake correction apparatus shown in FIG. The block diagram which shows schematic structure of the movement vector calculation part shown in FIG. The figure for demonstrating that the magnitude | size of a search range may be changed according to imaging magnification. Explanatory drawing which shows the principle which calculates the movement vector when performing camera shake correction

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image pick-up part 1a Optical system 1b Solid-state image sensor 2 Analog signal processing part 3 A / D conversion part 4 Drive part 5 Lens drive part 6 Digital signal processing part 7 Compression / decompression process part 8 Display part 9 System control part 10 Internal memory 11 Media interface 12 Recording medium 13 Operation unit 14 System bus 15 Camera shake correction device 21 Shooting magnification calculation unit 22 Search range determination unit 23 Movement vector calculation unit 24 Cutout position setting units 31 and 33 Memory 32 Reference image data cutout unit 33 Comparative image data Cutout unit 35 Correlation calculation unit 36 Deviation amount calculation unit

Claims (5)

A movement vector calculation method for obtaining a movement vector of second image data with respect to first image data obtained from an image sensor,
A reference image data cutout step of cutting out reference image data from a reference block set in the first image data;
A predetermined range is set in the second image data, and the comparison image data is cut out from each of the comparison blocks at each movement position when the comparison block having the same size as the reference block is moved within the predetermined range. A comparison image data cutting step;
A correlation calculation step of calculating a correlation between the comparison image data cut out from each of the comparison blocks at the respective movement positions and the reference image data;
A movement vector calculation step of calculating the movement vector from the position of the comparison block obtained by cutting out the comparison image data having the highest correlation and the position of the reference block;
And a predetermined range determining step for determining the predetermined range in accordance with a photographing magnification set in the image sensor. The movement vector calculation method according to claim 1, comprising:
In the predetermined range determining step, the movement vector calculation method for relatively widening the predetermined range when the photographing magnification is relatively large and relatively narrowing the predetermined range when the photographing magnification is relatively small. . A movement vector calculation device for obtaining a movement vector of second image data with respect to first image data obtained from an image sensor,
Reference image data cutout means for cutting out reference image data from a reference block set in the first image data;
A predetermined range is set in the second image data, and the comparison image data is cut out from each of the comparison blocks at each movement position when the comparison block having the same size as the reference block is moved within the predetermined range. Comparison image data cutout means;
Correlation calculation means for calculating the correlation between the comparison image data cut out from each of the comparison blocks at each moving position and the reference image data;
A movement vector calculation means for calculating the movement vector from the position of the comparison block obtained by cutting out the comparison image data having the highest correlation and the position of the reference block;
A movement vector calculation device including predetermined range determining means for determining the predetermined range in accordance with a photographing magnification set in the image sensor. The movement vector calculation device according to claim 3, wherein
The predetermined range determining means relatively widens the predetermined range when the photographing magnification is relatively large, and relatively moves the predetermined range when the photographing magnification is relatively small. . An imaging device having a video shooting function,
5. The movement vector calculation device according to claim 3, and cutout position setting means for setting a cutout position of recording image data cut out from the second image data according to the movement vector calculated by the movement vector calculation device. Including an image stabilization device,
An imaging apparatus comprising the imaging element.

Images