JP4904925B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention provides, for example, an image processing device that detects a motion vector in one screen unit of an image composed of a so-called camera shake component included in image information obtained by imaging with an imaging device such as a digital still camera or a video camera. The present invention relates to an image processing method.

  In general, when taking an image by holding an image pickup apparatus such as a digital still camera or a video camera by hand, the vibration of the image pickup apparatus due to camera shake at the time of image pickup appears as a vibration for each screen of a picked-up image.

  As a method of correcting vibrations in captured images due to such camera shake, optical camera shake correction using a gyro (angular velocity) sensor has become available in recent markets as the price, performance, and size of gyro sensors have decreased. The method is dominant.

  However, in the last few years, the rapid spread of digital still cameras and the rapid trend toward higher pixel counts, which are the same as those, have begun to create new problems. Even in still images at low illuminance (long exposure time), camera shake correction is strongly required, but there are only solutions using sensors such as gyro sensors, and the detection accuracy of the gyro sensor itself is high. The weak point of the gyro sensor such as sweetness and other problems are being exposed.

  All of the image stabilization for still images in consumer electronics currently on the market is to measure the camera shake vector using a gyro sensor or accelerometer and feed it back to the mechanical system, CCD (Charge Coupled Device). An image projected on an image sensor such as an imager or a CMOS (Complementary Metal Oxide Semiconductor) imager is controlled at high speed so as not to cause blurring.

  As the mechanism system here, a lens, a prism, and an imager (or a module integrated with the imager) have been proposed, which are called a lens shift, a prism shift, and an imager shift, respectively.

  As camera shake correction is performed in this way, in addition to the accuracy error of the gyro sensor itself mentioned above, feedback delay to the mechanical system or prediction error to avoid feedback delay, and control of the mechanical system Errors are also superimposed and it is impossible to correct with very high pixel accuracy.

  As mentioned above, although there is a major problem that camera shake correction using current sensors cannot be pursued in principle in principle, camera shake is the reason why it is highly evaluated in the market. This is because it can be reduced even if it cannot be corrected.

  However, it is also a matter of time for the market to realize the fact that as the pixel size becomes smaller, the correction limit must be increased with the pixel accuracy as the pixel size becomes smaller.

  On the other hand, as another method for correcting vibration of a captured image due to camera shake, a motion vector in a screen unit of the captured image is detected, and captured image data stored in the image memory is read based on the motion vector. A sensorless camera shake correction method that shifts the position and performs camera shake correction is known.

  As a method for detecting a motion vector of a captured image in screen units from captured image information itself, block matching for obtaining a correlation between captured images for two screens is known. In principle, this sensorless image stabilization method using block matching can realize pixel-accuracy image stabilization vector including rotation component in the roll axis direction, and no mechanical parts such as a gyro sensor are required. It is also advantageous in that the imaging device can be reduced in size and weight.

  51 and 52 illustrate the outline of block matching. FIG. 53 shows a general example of the processing flowchart.

  Block matching is a unit of one screen between a reference screen that is a screen of interest and an original screen that is an imaging screen that is one screen before the reference screen, for example, for the captured image from the imaging device unit. This is a method of calculating a motion vector by calculating a correlation between a reference screen and an original screen for a block of a rectangular area having a predetermined size.

  Here, the screen means an image composed of image data of one frame or one field. However, in this specification, for convenience of explanation, the screen is composed of one frame and the screen is referred to as a frame. And Therefore, the reference screen is referred to as a reference frame, and the original screen is referred to as an original frame (target frame).

  For example, the image data of the reference frame is the current frame image data from the imaging device unit or the current frame image data stored in the frame memory and delayed by one frame. The image data of the original frame is obtained by further storing the image data of the reference frame in the frame memory and delaying by one frame.

  In block matching, as shown in FIG. 51, a target block 103 having a rectangular area of a predetermined size composed of a plurality of pixels in the horizontal direction and a plurality of lines in the vertical direction is set at an arbitrary predetermined position of the original frame 101. Is done.

  On the other hand, in the reference frame 102, a projected image block 104 of the target block (see the dotted line in FIG. 51) is assumed at the same position as the target block 103 of the original frame, and the projected image block 104 of the target block is A search range 105 (see the one-dot chain line in FIG. 51) is set as the center, and a reference block 106 having the same size as the target block 103 is considered.

  Then, the position of the reference block 106 is moved within the search range 105 of the reference frame 102, and the correlation between the image content included in the reference block 106 and the image content of the target block 103 is obtained at each position, and the strongest correlation is obtained. Is detected as the position where the target block 103 of the original frame has moved in the reference frame 102. Then, the displacement amount between the detected position of the reference frame 106 and the position of the target block is detected as a motion vector as an amount including a direction component.

  In this case, the reference block 106 moves the search range 105 in units of one pixel or a plurality of pixels, for example, in the horizontal direction and the vertical direction. Therefore, a plurality of reference blocks are set in the search range 105.

  Here, the correlation between the target block 103 and each reference block 16 that moves within the search range 105 is the luminance value of each pixel in the target block 103 and the luminance value of each corresponding pixel in the reference block 106. Sum of absolute values of differences for all pixels in the block (the sum of absolute values of the differences is referred to as difference absolute value sum. Hereinafter, this difference absolute value sum is referred to as SAD (Sum of Absolute Difference) value. )). That is, the reference block 106 at the position where the SAD value is minimum is detected as the reference block having the strongest correlation, and the amount of displacement of the detected reference block 106 with respect to the position of the target block 103 is detected as a motion vector.

  In block matching, the amount of positional deviation of each of the plurality of reference blocks 106 set in the search range 105 with respect to the position of the target block 103 is expressed by a reference vector 107 (see FIG. 51) as an amount including a direction component. Is done. The reference vector 107 of each reference block 106 has a value corresponding to the position of the reference block 106 on the reference frame 102. In the conventional block matching, the reference vector of the reference block 106 having the minimum SAD value is used as the target. This is detected as a motion vector for the block 103.

  Therefore, in block matching, generally, as shown in FIG. 52, SAD values between each of a plurality of reference blocks 106 set in the search range 105 and the target block 103 (hereinafter referred to for the sake of simplicity). (Referred to as the SAD value for the block) is stored in the memory in correspondence with each reference vector 107 corresponding to the position of each reference block 106 in the search range 105, and all the references stored in the memory are stored. The motion vector 110 for the target block 103 is detected by detecting the reference block 106 having the smallest SAD value from the SAD values for the block 106.

  A difference absolute value sum table (SAD value stored for each reference block 106 corresponding to each of the reference vectors 107 corresponding to the positions of the plurality of reference blocks 106 set in the search range 105 is stored. Hereinafter, this is referred to as an SAD table). This is shown in the SAD table 108 of FIG. 46. In this SAD table 108, the SAD value for each reference block 106 is referred to as an SAD table element 109.

  In the above description, the positions of the target block 103 and the reference block 106 mean any specific position of these blocks, for example, the center position, and the reference vector 107 is the target block in the reference frame 102. A shift amount (including direction) between the position of the projected image block 104 of 103 and the position of the reference block 106 is shown. 51 and 52, the target block 103 is assumed to be at the center position of the frame.

  Then, since the reference vector 107 corresponding to each reference block 106 is misaligned with respect to the position corresponding to the target block 103 on the reference frame 102, the position of the reference block 106 is specified. Then, the value of the reference vector is also specified corresponding to the position. Therefore, when the address of the SAD table element of the reference block in the memory of the SAD table 108 is specified, the corresponding reference vector is specified.

  The conventional block matching process described above will be described with reference to the flowchart of FIG.

  First, one reference block Ii in the search range 105 is designated, which is equivalent to designating a reference vector corresponding to the reference block Ii (step S1). Here, in FIG. 52, (vx, vy) indicates the position indicated by the specified reference vector when the position of the target block on the frame is the reference position (0, 0), and vx is specified. The reference vector is a deviation component in the horizontal direction from the reference position, and vy is a deviation component in the vertical direction component from the reference position due to the designated reference vector.

  Here, the shift amounts vx and vy are values in units of pixels. For example, vx = + 1 indicates a position shifted by one pixel in the right direction in the horizontal direction with respect to the reference position (0, 0). In addition, vx = −1 indicates a position shifted by one pixel in the horizontal left direction with respect to the reference position (0, 0). For example, vy = + 1 indicates a position shifted by one pixel in the downward direction in the vertical direction with respect to the reference position (0, 0), and vy = −1 indicates the reference position (0, 0). On the other hand, a position shifted by one pixel upward in the vertical direction is shown.

  As described above, (vx, vy) indicates a position with respect to the reference position indicated by the reference vector (hereinafter referred to as a position indicated by the reference vector for simplicity), and corresponds to each of the reference vectors. . That is, when vx and vy are integers, (vx, vy) represents each of the reference vectors. Therefore, in the following description, a reference vector indicating the position of (vx, vy) may be described as a reference vector (vx, vy).

Here, when the center position of the search range is the target block position, that is, the reference position (0, 0), and the search range is ± Rx in the horizontal direction and ± Ry in the vertical direction,
-Rx≤vx≤ + Rx, -Ry≤vy≤ + Ry
It is supposed to be.

Next, the coordinates (x, y) of one pixel in the target block Io are designated (step S2). Next, the pixel value Io (x, y) of one designated coordinate (x, y) in the target block Io and the pixel value Ii (x + vx, y + vy) of the corresponding pixel position in the reference block Ii The absolute value α of the difference is calculated (step S3). That is, the difference absolute value α is
α = | Io (x, y) −Ii (x + vx, y + vy) | (Formula 1)
Is calculated as

Then, the calculated absolute difference value α is added to the SAD value so far of the address (table element) indicated by the reference vector (vx, vy) of the reference block Ii, and the SAD value that is the addition is added to the address (Step S4). That is, when the SAD value corresponding to the reference vector (vx, vy) is expressed as SAD (vx, vy),
SAD (vx, vy) = Σα = Σ | Io (x, y) −Ii (x + vx, y + vy) |
... (Formula 2)
And written to the address indicated by the reference vector (vx, vy).

  Next, it is determined whether or not the above calculation has been performed on the pixels of all coordinates (x, y) in the target block Io (step S5), and all the coordinates (x, y) in the target block Io are determined. For the pixel, when it is determined that the calculation has not been completed yet, the process returns to step S2, the pixel position of the next coordinate (x, y) in the target block Io is designated, and the processes after step S2 are repeated. .

  In step S5, when it is determined that the above calculation has been performed for all the pixels of the coordinates (x, y) in the target block Io, it is determined that the calculation of the SAD value for the reference block has ended. It is determined whether or not the above arithmetic processing has been completed for all reference blocks in the search range, that is, all reference vectors (vx, vy) (step S6).

  If it is determined in step S6 that there is a reference vector (vx, vy) for which the above arithmetic processing has not yet been completed, the process returns to step S1, and the next reference vector (vx, vy) for which the above arithmetic processing has not been completed. ) Is set, and the processing after step S1 is repeated.

  If it is determined in step S6 that the reference vector (vx, vy) for which the above arithmetic processing has not been completed is no longer in the search range, the SAD table is completed, and the minimum value is obtained in the completed SAD table. The detected SAD value is detected (step S7). Then, a reference vector corresponding to the address of the SAD value that is the minimum value is detected as a motion vector for the target block Io (step S8). Here, when the minimum value of SAD is written as SAD (mx, my), the target motion vector is calculated as a vector (mx, my) indicating the position (mx, my).

  This completes the motion vector detection process by block matching for one target block.

  Actually, it is difficult to obtain an accurate camera shake vector for the reference frame with respect to the original frame with the motion vector for one target block. Therefore, in the original frame, a plurality of target blocks are set so as to cover the entire range of the original frame, while in the reference frame, projections of the plurality of target blocks are provided as shown in FIG. The search ranges 105, 105,... Are set for the images 104, 104,..., And the motion vectors 110, 110,. To.

  Then, a camera shake vector (global motion vector) for a reference frame with respect to the original frame is detected from the detected plurality of motion vectors 110, 110,.

  As a method of detecting a hand movement vector (global motion vector) from the plurality of motion vectors 110, a method based on majority decision of a plurality of motion vectors, that is, the plurality of motion vectors 110 have the same direction and size. A method of using a motion vector having the largest number of motion vectors as a global motion vector has been mainly proposed. In addition, a method using both the method based on the majority vote and the reliability evaluation based on the change amount (frequency) of the motion vector in the time axis direction has been proposed.

  Most sensorless camera shake correction as a conventional technique is targeted for moving images, as represented by Patent Document 1 (Japanese Patent Laid-Open No. 2003-78807). Further, as a technique for realizing sensorless camera shake correction with a still image, several proposals have been made including Patent Document 2 (Japanese Patent Laid-Open No. 7-283999). In this Patent Document 2, several still images are continuously shot with a short exposure time that does not cause camera shake, a camera shake vector between the still images is obtained, and a plurality of the consecutive shots are obtained according to the camera shake vector. The algorithm is such that the still image is added (or averaged) while being translated (and rotated in the roll axis direction), thereby finally obtaining a high-quality still image free from camera shake and low illuminance noise.

  Patent Document 3 (Japanese Patent Laid-Open No. 2005-38396) can be cited as a realistic proposal that can be realized. The device disclosed in Patent Document 3 includes means for obtaining a motion vector with a reduced size of an image and means for sharing the same SAD table among a plurality of blocks. Image reduction conversion and sharing of the SAD table among a plurality of blocks is a very good technique for reducing the size of the SAD table. Motion vector detection in the MPEG (Moving Picture Experts Group) image compression method, It is also used in other fields such as scene change detection.

  However, as problems of the algorithm of Patent Document 3, time and memory capacity are consumed for image reduction conversion and memory (DRAM (Random Access Memory)) access at that time, and the SAD table. As a result of the method of performing time-division access on a plurality of blocks, memory access increases greatly, and this processing also takes time. In motion blur correction of moving images, a reduction in system delay time is required simultaneously with real-time performance, and this problem of processing time becomes a problem.

  Further, when the original image is reduced and converted, it is necessary to implement aliasing (folding distortion) and a low-pass filter for removing low illuminance noise as pre-processing of the reduction process. However, the characteristics of the low-pass filter change according to the reduction ratio. In particular, in the case of a low-pass filter in the vertical direction, a multi-tap digital filter requires a lot of line memory and arithmetic logic. Therefore, the problem of circuit scale increase arises.

The above prior art documents are as follows.
JP 2003-78807 A JP-A-7-283999 JP 2005-38396 A

  In video stabilization systems, real-time detection of rough camera shake vectors that emphasizes processing time rather than accuracy is required, and even the conventional sensorless image stabilization methods can provide satisfactory results in most situations. .

  However, for example, when the user suddenly changes the brightness by one frame (blinks the entire screen) when another cameraman flashes the flash while the user is shooting a movie, the motion vector is erroneously detected. Has been pointed out.

  In order to avoid this, conventionally, the minimum SAD value for each motion vector in the frame is used as reliability, or the camera shake frequency is calculated from the amount of change in the motion vector in the time axis direction, and the motion of the next frame is calculated. A technique is adopted in which a vector is predicted to eliminate a vector that is significantly different from the prediction.

  However, for example, when shooting a moving water surface with a fine wavefront, the motion vectors detected for a plurality of target blocks set in a frame vary, and the minimum SAD value over a plurality of frames is relatively low. In order to show a small good value, the method as in the screen blinking described above cannot be applied. In addition, as a common subject, there is a scene where trees and grass flutter in the wind. In this case as well, an erroneous camera shake vector is often detected.

  Among the reasons why such a problem occurs in the conventional sensorless image stabilization, most of the conventional sensorless image stabilization methods are detected for one frame as a method for detecting a shake vector as a global motion vector. This is based on the fact that a majority vote is taken for a plurality of motion vectors and a majority motion vector having the largest direction and size is detected as a global motion vector.

  When the above-described erroneous detection is performed, camera shake correction is performed according to a vector different from the actual camera shake, and thus the user feels a very uncomfortable feeling as a resultant image. One of the reasons why electronic camera shake correction using a gyro sensor is currently being considered is that this problem has not been solved.

  However, as described above, in imaging devices such as digital cameras, it is expected that high-density pixels will continue to increase in the future and higher performance is required. In such situations, camera shake correction during still image shooting is expected. It is very significant to realize sensorless without using a gyro (angular velocity) sensor.

  Therefore, as described above, it is promising to detect a shake motion vector sensorlessly using block matching, and to perform shake correction using the detected motion vector, and it is important to solve the above-described problems. .

  In view of the above points, an object of the present invention is to provide an image processing method and apparatus capable of solving the problems of the conventional sensorless camera shake correction method described above.

In order to solve the above problems, the invention of claim 1
An image processing apparatus that calculates a motion vector in units of one screen between a reference screen that is an attention screen and an original screen that is a screen before the reference screen for images that are sequentially input in units of screens. There,
In the original screen, a target block having a predetermined size composed of a plurality of pixels is set at a plurality of predetermined positions, and a reference block having the same size as the target block is set on the reference screen. A plurality of search ranges set in accordance with the position of the target block are set, and in each of the search ranges, each pixel in the reference block is set for each of the set reference blocks. A difference absolute value sum calculating means for obtaining a difference absolute value sum that is a sum of absolute values of differences between a value and a pixel value of each pixel at a corresponding position in the target block;
For each of the plurality of search ranges, the difference absolute value sum calculated by the difference absolute value sum calculation means is added to the sum of the reference block positions corresponding to each of the plurality of search ranges to add up the difference absolute value sum. And a sum difference absolute value sum table generating means for generating a sum difference absolute value sum table for storing the sum of sum difference absolute value sums for a plurality of reference block positions within one search range;
A motion vector detecting means for detecting a motion vector of the reference screen relative to the original screen from the summed absolute difference sum table generated by the summed absolute difference sum table generating means;
An image processing apparatus is provided.

  In the image processing apparatus according to the first aspect of the present invention, the difference absolute value sum (SAD value) between each reference block within the search range of the reference screen and the target block of the original screen is calculated as the difference absolute value. The value sum calculation means is the same as in the prior art.

  In the present invention, the SAD values for each of the plurality of target blocks and search ranges set on the original screen are obtained, and the SAD values of the corresponding reference block positions in each of the plurality of search ranges are added together to obtain the sum difference absolute A value sum (referred to as a combined SAD value) is obtained. As a result, a combined SAD value for a plurality of reference block positions within one search range is obtained, and a combined difference absolute value sum table for storing the combined SAD values is generated. Then, a motion vector (global motion vector) of the reference screen with respect to the original screen is detected from the total SAD table.

  Conventionally, a motion vector (motion vector for each target block) is obtained for each of a plurality of target blocks set on the original screen, and a global motion vector is calculated by majority vote for the obtained plurality of motion vectors. However, unlike this, the sum SAD table composed of the sum SAD values in the present invention is equivalent to the result of block matching the entire frame together.

  For this reason, when camera shake correction is performed using the global motion vector obtained from the combined SAD table, the conventional method is applied to subjects such as the blinking of the entire frame and the wavefront of the water surface when the flash is applied. Such a large and incorrect correction can be avoided, and even when the subject is not such, correction with higher accuracy than before can be performed. That is, it is possible to provide camera shake correction that achieves both robustness and high accuracy.

According to a second aspect of the present invention, in the image processing apparatus according to the first aspect,
A difference absolute value sum table generating means for generating a plurality of difference absolute value sum tables for storing the difference absolute value sums for a plurality of reference block positions in the search range in correspondence with the plurality of search ranges;
The minimum value of the difference absolute value sum in each of the plurality of difference absolute value sum tables generated by the difference absolute value sum table generation means, and the sum difference absolute value calculated by the sum difference absolute value sum calculation means Means for detecting the difference absolute value sum table for the target block considered to be a moving subject part from the minimum value of the sum;
The motion vector detection means recalculates the sum of absolute differences, excluding the sum of absolute differences of the difference absolute value sum table for the target block considered to be the moving subject part, and adds the sum of differences There is provided an image processing apparatus characterized by recalculating an absolute value sum table and detecting the motion vector from the recalculated summed absolute difference sum table.

  According to the second aspect of the present invention, it is a movement part of a moving subject, not a movement due to camera shake or the like, from the minimum value of the combined SAD value and the minimum value of the SAD table of each of the plurality of target blocks. The SAD table of the target block to be accompanied is detected, and the total SAD value is calculated again, excluding the SAD value for the SAD table. Then, a motion vector (global motion vector) is detected from the recalculated sum of absolute differences.

The invention of claim 3
An image processing apparatus that calculates a motion vector in units of one screen between a reference screen that is an attention screen and an original screen that is a screen before the reference screen for images that are sequentially input in units of screens. There,
In the original screen, a target block having a predetermined size composed of a plurality of pixels is set at a plurality of predetermined positions, and a reference block having the same size as the target block is set on the reference screen. A plurality of search ranges set in accordance with the position of the target block are set, and in each of the search ranges, each pixel in the reference block is set for each of the set reference blocks. A difference absolute value sum calculating means for obtaining a difference absolute value sum that is a sum of absolute values of differences between a value and a pixel value of each pixel at a corresponding position in the target block;
A reference reduction including a reference vector including a directional component is used as a positional deviation amount of each reference block on the reference screen relative to the target block on the screen, and the reference vector is reduced at a predetermined reduction ratio. A reference reduced vector obtaining means for obtaining a vector;
For each target block in the plurality of search ranges, the reference vector having a size corresponding to the reference reduction vector is used as the positional shift amount, and a plurality of the plurality of the reductions corresponding to the predetermined reduction ratio are performed. A reduced difference absolute value sum table generating means for generating a reduced difference absolute value sum table for storing the difference absolute value sum for each of the reference blocks;
For the plurality of reduced difference absolute value sum tables generated by the reduced difference absolute value sum table generating means, the difference absolute value sums of the reference block positions corresponding to each of the plurality of search ranges are added together to add up the difference. A summed reduced difference absolute value sum table generating means for obtaining an absolute value sum and generating a summed reduced difference absolute value sum table;
A motion vector calculating unit that calculates a motion vector of the reference screen with respect to the original screen from the combined reduced difference absolute value sum table generated by the combined reduced difference absolute value sum table generating unit;
With
The reduced difference absolute value sum table generating means includes:
Neighborhood reference vector detection means for detecting a plurality of the reference vectors that are neighborhood values of the reference reduction vector acquired by the reference reduction vector acquisition means;
Corresponding to each of the plurality of neighboring reference vectors detected by the neighboring reference vector detecting means from the sum of the absolute values of the differences for each of the reference blocks calculated by the difference absolute value sum calculating means. A variance difference absolute value sum calculating means for calculating each of the difference absolute value sums;
The difference absolute value sum corresponding to each of the plurality of neighboring reference vectors calculated by the variance difference absolute value sum calculating means is used as the difference absolute value corresponding to each of the plurality of neighboring reference vectors so far. Distributed addition means for adding to the sum of values;
An image processing apparatus is provided.

  In the invention of claim 3, as in the case of claim 1 described above, the SAD calculation means obtains the SAD value between each reference block within the search range of the reference screen and the target block of the original screen. Is the same as before.

  In this invention, the SAD value of the obtained reference block is not stored corresponding to the reference vector of the reference block, but is stored in correspondence with the reference reduced vector obtained by reducing the reference vector. Table generating means is provided.

  In this case, since the reference reduced vector does not match the reference vector corresponding to each of the reference blocks, the reduced difference absolute value sum table generation means first selects a plurality of reference vectors in the vicinity of the reference reduced vector. To detect. Then, the SAD value corresponding to each of the plurality of reference vectors in the vicinity thereof is calculated from the SAD value calculated by the SAD calculating means.

  Then, the calculated SAD value corresponding to each of the plurality of reference vectors in the vicinity is added to the SAD value corresponding to each of the plurality of reference vectors in the vicinity so far.

  The SAD table thus obtained is a table of SAD values for only a plurality of reference vectors in the vicinity of each reference reduced vector, and is reduced according to the reduction rate of the reference vector.

  In other words, the generated SAD value table corresponds to the SAD table for the frame image reduced in accordance with the reduction ratio of the reference vector. In this case, since the sizes of the target block and the reference block are not reduced, the size of the generated SAD table (reduced difference absolute value sum table) is reduced.

  In the invention of claim 3, such a reduced difference absolute value sum table is obtained for each of a plurality of target blocks. Then, in the reduced difference absolute value sum table of the plurality of target blocks obtained, the SAD values at the corresponding reference block positions in each of the plurality of search ranges are added together to obtain the sum of absolute differences. Thereby, a combined SAD value (total reduced difference absolute value sum table) for a plurality of reference block positions within one search range is obtained. Then, a motion vector (global motion vector) of the reference screen with respect to the original screen is detected from the summed reduced difference absolute value sum table.

  As a result, a motion vector is calculated from the reduced sum SAD table that is equivalent to the result of block matching for the entire frame. In the invention of claim 3, the reduced SAD table and the combined reduced SAD table have a smaller capacity than the SAD table in the conventional block matching as described above, and the realization becomes realistic.

  In the present invention, SAD values at corresponding reference block positions in each of a plurality of search ranges are added together to obtain a sum of absolute differences (referred to as a combined SAD value), and a plurality of reference block positions within one search range are obtained. The motion vector (global motion vector) of the reference screen relative to the original screen is detected from the total SAD value. The total SAD value in the present invention is equivalent to the result of block matching of the entire frame together, and can avoid a large and erroneous correction as in the conventional method. Can be corrected with high accuracy. That is, it is possible to provide camera shake correction that achieves both robustness and high accuracy.

  Hereinafter, an embodiment of an image processing method and an image processing apparatus according to the present invention will be described with reference to the drawings, taking as an example the case of applying to an imaging apparatus and an imaging method.

[Outline of Embodiment of Image Processing Method According to the Present Invention]
Also in the embodiment of the image processing method according to the present invention, when detecting a motion vector between two frames using the block matching described above, as described above, a plurality of target frames are set for the original frame, Block matching is performed for each of the plurality of target frames.

  In the embodiment described below, for example, 16 target blocks TGi (i = 1, 2,..., 16) are set in the original frame, and in the reference frame 102, as shown in FIG. Sixteen projected images 104i (i = 1, 2,..., 16) corresponding to the 16 target blocks of the original frame are set. Then, a search range 105i (i = 1, 2,..., 16) is set for each projection image, and the corresponding target is set in each search range 105i (i = 1, 2,..., 16). A SAD table TBLi (i = 1, 2,..., 16) for the block is created.

  As shown in FIG. 1, when the 16 SAD tables TBLi are arranged in the vertical direction so as to overlap each other, the search for obtaining each of the SAD tables TBLi for the created 16 target blocks is performed. The SAD values at the reference block positions corresponding to each other within the range are added together to obtain a sum of absolute differences (referred to as a combined SAD value). Then, a sum SAD table SUM_TBL for a plurality of reference block positions within one search range is generated as a SAD table composed of the sum SAD values.

Here, the sum SAD value SUM_TBL (x, y) of the coordinate position (x, y) of the sum SAD table SUM_TBL is the TBAD (x, y) of the SAD value of the corresponding coordinate position (x, y) of each SAD table TBLi. )
SUM_TBL (x, y) = TBL1 (x, y) + TBL2 (x, y) +.
+ TBL16 (x, y)
= ΣTBLi (x, y)
(See (Formula 3) in FIG. 2).

  In this embodiment, the motion vector of the reference screen with respect to the original screen (global motion vector; a camera shake vector in the imaging apparatus) is detected from the total SAD table SUM_TBL.

  As a method for calculating the global motion vector from the summed SAD table SUM_TBL, the position of the minimum value of the summed SAD value is detected in the summed SAD table SUM_TBL, and the detected reference vector corresponding to the position of the minimum value of the summed SAD value is determined globally. A conventional method of detecting as a motion vector can be used.

  However, since the method using the minimum total SAD value can only obtain a motion vector with an accuracy of one pixel unit, in this embodiment, as described later, the total SAD value at the minimum value position of the total SAD value and A global motion vector is detected by performing approximate curved surface interpolation using a plurality of summed SAD values in the vicinity. That is, by generating an approximate higher-order curved surface using the combined SAD value of the minimum value position of the combined SAD value and a plurality of combined SAD values in the vicinity thereof, and detecting the minimum value position of the approximate higher-order curved surface, A global motion vector can be detected with a decimal point precision of one pixel unit or less.

  In the above description, the SAD table TBLi is obtained for each of the plurality of target blocks, and the total SAD table SUM_TBL is obtained from the obtained plurality of SAD tables TBLi. There is no need to obtain the SAD table TBLi for. This is because the sum SAD table SUM_TBL only needs to be obtained, and the calculation of the motion vector corresponding to the minimum SAD value coordinate in the SAD table TBLi for each target block is not necessary in principle.

  For example, the processing for obtaining the SAD value between the target block and the reference block is performed simultaneously in parallel for a plurality of target blocks, and the corresponding reference block positions in the plurality of search ranges performed simultaneously in parallel, that is, the same It is also possible to obtain the sum SAD table SUM_TBL by summing up the SAD values at the coordinate position (x, y).

  However, in this embodiment, in order to increase the reliability of the detected camera shake motion vector (global motion vector), the SAD table TBLi is always obtained for each target block in consideration of the following points, and each target block Motion vector (hereinafter referred to as a block-by-block motion vector) BLK_Vi is detected.

  In other words, in this embodiment, a motion vector component due to a moving subject that is not a motion vector of the entire screen due to camera shake is eliminated as much as possible when detecting a global motion vector due to camera shake, thereby detecting a more accurate global motion vector. I can do it.

  Therefore, in this embodiment, a global motion vector (hereinafter referred to as a combined motion vector) SUM_V obtained from the combined SAD table, and a motion vector (a motion vector for each block) BLK_Vi determined from the SAD table TBLi for each target block And the total SAD table is recalculated using only the SAD table TBLi of the target block that is the same or approximate. Then, a global motion vector serving as a shake vector is detected using the recalculated total SAD table (resumed SAD table) RSUM_TBL.

  However, in this case, whether a plurality of motion vectors for one frame, in this example, 16 block-by-block motion vectors are reliable in order to obtain a shake vector (global motion vector) for the frame. Only when it is determined that the reliability is high, the re-summation SAD table RSUM_TBL is calculated and the global motion vector is calculated.

  The determination of the reliability of each frame, that is, the determination of the reliability of the motion vector for each of a plurality of blocks in the frame is performed as follows.

  First, a SAD table TBLi is obtained for each of a plurality of target blocks TGi (i = 1, 2,..., 16) set in the original frame, and in this example, the coordinate position of the minimum SAD value MINi. From each block, a motion vector BLK_Vi for each block (see FIG. 3A) is obtained. Next, the total SAD table SUM_TBL is obtained from the 16 SAD tables TBLi according to (Equation 3) described above, and the total motion vector SUM_V (see FIG. 3B) is obtained from the coordinate position of the minimum SAD value MINs.

  Next, in this embodiment, the motion vector BLK_Vi (that is, the summed SAD table SUM_TBL) of each of the 16 target blocks with reference to the coordinate position of the summed motion vector SUM_V, that is, the minimum SAD value MINs of the summed SAD table SUM_TBL. 4) and the respective SAD values, the conditions as shown in FIG. 4 are determined for each of the 16 target blocks, and the labeling and score as shown in FIG. Is calculated.

  FIG. 5 shows a flowchart of an example of the labeling and score calculation process. The process of FIG. 5 is a process for one reference frame, and the process of FIG. 5 is repeated for each frame.

  First, it is determined that the first condition is whether the motion vector BLK_Vi obtained for the target block for labeling and score calculation is equal to the combined motion vector SUM_V (step S11). This is equivalent to determining whether the coordinate position of the minimum SAD value MINi of the SAD table TBLi of the target block is equal to the coordinate position of the minimum SAD value MINs of the combined SAD table SUM_TBL.

  When it is determined that the target target block satisfies the first condition, the target target block is labeled “TOP”, and in this example, “4” is assigned as the maximum score value (step S12). ).

  When it is determined that the target target block does not meet the first condition, it is determined whether or not the second condition is met (step S13). The second condition is whether or not the motion vector BLK_Vi obtained for the target target block and the combined motion vector SUM_V are not the same, but are most adjacent on the SAD table. The coordinate position of the minimum SAD value MINi of the SAD table TBLi of the target block and the coordinate position of the minimum SAD value MINs of the combined SAD table SUM_TBL are adjacent to each other, differing by one coordinate value in the vertical, horizontal, and diagonal directions. Whether or not.

  If it is determined that the second target condition is met for the target target block, a label “NEXT_TOP” is assigned to the target target block, and “2” is assigned as a score value in this example (step S14).

  When it is determined that the target target block does not meet the second condition, it is determined whether or not the third condition is met (step S15). The third condition is that, in the SAD table of the target block, the SAD value (minimum SAD value MINi) of the coordinate position serving as the motion vector BLK_Vi and the coordinate position (minimum SAD value) of the combined SAD table serving as the combined motion vector SUM_V. Whether the difference from the SAD value of the coordinate position corresponding to (MINs coordinate position) is equal to or less than a predetermined threshold value. Here, it is desirable that the predetermined threshold is a threshold when converted per pixel. This is because, in this embodiment, camera shake correction with one pixel accuracy is assumed.

  If it is determined that the target target block satisfies the third condition, the target target block is assigned a label “NEAR_TOP”, and in this example, “1” is assigned as the score value (step S16).

  If it is determined that the third condition is not met for the target target block, the target target block is labeled “OTHERS”, and “0” is assigned as the score value in this example (step S17). .

  After the labeling and the score assignment in step S12, step S14, step S16, and step S17 are completed, the assigned scores are cumulatively added to calculate a total score sum_score (step S18).

  Next, it is determined whether or not the above processing has been completed for all 16 target blocks in one frame (step S19). If not, instructions are given for labeling and score calculation of the next target block. Then (step S20), the process returns to step S11, and the above-described processing is repeated.

  When it is determined that the above processing has been completed for all 16 target blocks in one frame, the labeling and score calculation processing routines for 16 target blocks in one frame are terminated. At this time, the total score sum_score calculated in step S18 is the sum of all the scores of the 16 target blocks.

  Note that the flowchart of FIG. 5 is an example, and the match determination of the first condition, the second condition, and the third condition is not fixed, and any of them may be performed first.

  When the labeling and score calculation processing routines for 16 target blocks in one frame as described above are completed, the calculated total score sum_score is compared with a threshold value for reliability. At this time, when the total score sum_score is smaller than the threshold, it can be determined that the motion vector obtained in the frame is low in reliability for the detection of the global motion vector.

  On the other hand, if the total score sum_score is equal to or greater than the threshold value, the detection of the global motion vector using the combined motion vector obtained in the frame can provide a certain level of reliability. Therefore, when it is determined that the total score sum_score is equal to or greater than the threshold value, the SAD value of the SAD table of the target block (label “TOP” and label “NEXT_TOP”) satisfying the first condition and the second condition having a high score It is also possible to generate a re-summation SAD table using only these and calculate a global motion vector based on the re-summation SAD table.

  However, in this embodiment, in order to obtain higher reliability, the following processing is further performed.

  The total SAD table used in this embodiment is not the SAD table for each block, but is almost equivalent to the result of block matching for the entire frame. In a normal subject, the motion vector that has been won by the majority vote described in the prior art, that is, the motion vector at the top of the majority vote, and the sum motion vector obtained from the sum SAD table are equal. However, if the entire frame blinks or is a poor subject such as a wavefront of the water surface, the result of the majority decision is low in reliability and the motion vector is almost random, whereas the combined motion vector is relatively close to the correct answer There is a high probability of deriving results.

  Therefore, it is possible to quantitatively determine at least the reliability of the result of the current frame by comparing the result of both the sum motion vector obtained from the sum SAD table and the global motion vector determined by majority vote. It becomes possible. In the conventional proposal, the main focus was on determining the reliability of the motion vector of each block, but in this embodiment, based on the policy of focusing on the reliability of the entire frame and not correcting suspicion, It is characterized by realizing a stable image stabilization system with little discomfort.

  In consideration of this point, in this embodiment, as in the case of the conventional block matching, a majority vote is taken for each block motion block BLK_Vi detected for 16 target blocks, and the majority vote top (the same size and direction) is taken. Alternatively, the motion vectors having the maximum number of motion vectors per block that are equivalent) are detected.

  Then, using the detected majority motion vector as a reference, the block-specific motion block BLK_Vi detected for the 16 target blocks, and the respective SAD values, as in the case of using the combined motion vector in FIG. Then, labeling and score assignment as shown in FIG. 4 are performed.

  This is equivalent to using the majority motion vector instead of the combined motion vector SUM_V in FIG.

  That is, the first condition is whether or not the motion vector BLK_Vi obtained for the target block for which labeling and score calculation are performed is equal to the motion vector of the majority vote top. That is, it is determined whether or not the coordinate position of the minimum SAD value MINi in the SAD table TBLi of the target block is equal to the coordinate position that becomes the motion vector of the majority vote top.

  The second condition is whether the motion vector BLK_Vi obtained for the target target block and the motion vector of the majority vote top are not the same, but are the most adjacent on the SAD table. The coordinate position of the minimum SAD value MINi in the SAD table TBLi of the target target block is adjacent to the coordinate position corresponding to the motion vector of the majority vote, which differs by one coordinate value in the vertical, horizontal, and diagonal directions. Whether there is.

  The third condition is that, in the SAD table of the target block, the SAD value (minimum SAD value MINi) of the coordinate position that becomes the motion vector BLK_Vi and the SAD of the coordinate position on the SAD table corresponding to the majority motion vector Whether the difference from the value is equal to or less than a predetermined threshold value.

  As described above, labeling and score assignment are performed on the motion vectors for 16 target blocks for one frame based on the motion vector at the top of the majority vote. Then, a total score many_score of the assigned scores is calculated.

  In this embodiment, the difference between the coordinate position of the minimum SAD value corresponding to the combined motion vector SUM_V and the coordinate position of the SAD value corresponding to the majority motion vector is within a predetermined range, for example, the difference is within one adjacent pixel. When the total score sum_score is equal to or greater than a predetermined threshold and the total score many_score is equal to or greater than a predetermined threshold, it is determined that the motion vector obtained for the frame is highly reliable.

  Then, only when it is determined that the reliability is high in this way, in this example, “TOP” and “NEXT_Top” are given among the labels of the target blocks labeled based on the combined motion vector. The re-summation SAD table RSUM_TBL is generated using only the SAD value of the SAD table for the target block.

  Then, a global motion vector is calculated by applying approximate curved surface interpolation to the minimum SAD value of the re-summation SAD table RSUM_TBL and the SAD value of the neighboring coordinate position.

  It should be noted that further reliability can be obtained by combining a conventionally proposed method for predicting a motion vector (global motion vector) from the frequency of a motion vector in the time axis direction and the above-described method according to the embodiment of the present invention. It is also possible to improve the performance and accuracy.

  As described above, in this embodiment, an SAD table is generated for each of a plurality of target blocks in one frame, and a motion vector for each block is detected. In this case, in this embodiment, since the scale of the SAD table increases in proportion to the number of pixels for one screen when it is applied to an imaging apparatus that uses the current imaging device with over 5 million pixels, it is realistic. There is a problem that it is difficult to realize on a circuit scale.

  As a realistic proposal of a level that can be realized, the above-mentioned Patent Document 3 (Japanese Patent Laid-Open No. 2005-38396) can be cited. The device disclosed in Patent Document 3 includes means for obtaining a motion vector with a reduced size of an image and means for sharing the same SAD table among a plurality of blocks. Image reduction conversion and sharing of the SAD table among a plurality of blocks is a very good technique for reducing the size of the SAD table. Motion vector detection in the MPEG (Moving Picture Experts Group) image compression method, It is also used in other fields such as scene change detection.

  However, as problems of the algorithm of Patent Document 3, time and memory capacity are consumed for image reduction conversion and memory (DRAM (Random Access Memory)) access at that time, and the SAD table. As a result of the method of performing time-division access on a plurality of blocks, memory access increases greatly, and this processing also takes time. In motion blur correction of moving images, a reduction in system delay time is required simultaneously with real-time performance, and this problem of processing time becomes a problem.

  As a result of multi-person evaluation, it has been found that, for example, the camera shake range in the case of a still image of 3 frames / second (3 fps) is about ± 10% when all frames are 100%. Assuming the 12 million pixels that are already in the market for high-end machines, the required SAD table size is estimated to be about 80 megabits with the currently proposed technology. Moreover, in order to satisfy a practical processing speed, the memory for storing the SAD table information is required to be a built-in SRAM (Static RAM (Random Access Memory)). Even though the semiconductor process rules have advanced, this size is about three orders of magnitude far from a realistic level.

  Further, when the original image is reduced and converted, it is necessary to implement aliasing (folding distortion) and a low-pass filter for removing low illuminance noise as pre-processing of the reduction process. However, the characteristics of the low-pass filter change according to the reduction ratio. In particular, in the case of a low-pass filter in the vertical direction, a multi-tap digital filter requires a lot of line memory and arithmetic logic. Therefore, the problem of circuit scale increase arises.

  In view of the above, this embodiment uses an image processing method and apparatus that can greatly reduce the SAD table size when detecting a global motion vector between two frames using block matching.

  In addition, regarding the SAD table reduction method by the image reduction conversion described in Patent Document 3, an increase in processing time associated with the image reduction conversion, consumption of memory capacity, and appropriate prevention of aliasing associated with the image reduction conversion Two problems were raised, the increase in circuit accompanying the implementation of a simple low-pass filter. In this embodiment, these problems can be solved.

  That is, in this embodiment, the SAD value obtained between the target block and the reference block is not stored in correspondence with the reference vector of the reference block, but the reference vector is reduced and the reduced reference reduced vector is supported. In addition, it is distributed and added to a plurality of reference vectors in the vicinity of the reference reduced vector and stored.

As a result, the size of the SAD table is greatly reduced as compared with the conventional SAD table, the processing time associated with the image reduction conversion is increased, the memory capacity is consumed, and the aliasing associated with the image reduction conversion is avoided. In order to solve the two problems of circuit increase accompanying the implementation of the appropriate low-pass filter,
6 to 8 are diagrams for explaining an outline of a novel block matching method used in this embodiment. FIG. 6 shows the relationship between the conventional SAD table TBLo and the reduced SAD table TBLs generated by the image processing method of the embodiment.

  Also in this embodiment, as shown in FIG. 54, in the reference frame as in the prior art, a plurality of searches centered on each of the positions of a plurality of original frames, in this example 16 target blocks, in the reference frame. A range is set. Then, in each of the plurality of search ranges, a plurality of reference blocks as described above are set, and the sum of absolute values of differences in luminance values of pixels in each reference block and corresponding pixels in the target block, that is, , The SAD value is determined.

  Conventionally, as shown in FIG. 6, the obtained SAD value is written in the SAD table TBLo as a table element tbl having an address corresponding to the reference vector RV of the target reference block.

  Therefore, in the conventional block matching, the reference vector RV representing the amount of positional deviation between the target block and the reference block on the frame image and the SAD value of the reference block which is each table element of the SAD table TBLo are in a one-to-one relationship. It corresponds. That is, the conventional SAD table TBLo has the same number of SAD value table elements as the reference vector RV that can be taken in the search range.

  On the other hand, in the block matching in this embodiment, as shown in FIG. 6 and FIGS. 7A and 7B, the reference vector RV of the target reference block has a reduction ratio of 1 / n (n Is reduced to a reference reduced vector CV.

  Here, in the following description, the horizontal reduction magnification and the vertical reduction magnification are the same for convenience of explanation, but the horizontal reduction magnification and the vertical reduction magnification may be different values. As will be described later, it is more flexible and convenient that the horizontal reduction ratio and the vertical reduction ratio can be set independently as an arbitrary natural fraction.

  Also in this embodiment, as described in the above-described conventional example, the position of the target block that is the center of the search range is set as the reference position (0, 0), and the reference vector is a pixel unit from the reference position. Of the horizontal direction and the vertical direction (vx, vy) (vx, vy are integers), and each of the reference vectors RV is represented by a reference vector (vx, vy).

  The position (vx / n, vy / n) indicated by the reference reduced vector (vx / n, vy / n) obtained by reducing the reference vector (vx, vy) to 1 / n in each of the horizontal direction and the vertical direction is In some cases, a fractional component is generated instead of an integer. Therefore, in this embodiment, if the SAD value obtained corresponding to the original reference vector RV before reduction is stored as a table element corresponding to one reference vector closest to the reference reduction vector CV, an error occurs. Will occur.

  Therefore, in this embodiment, first, a plurality of positions (table elements) indicated by a plurality of reference vectors in the vicinity of the position (vx / n, vy / n) indicated by the reference reduced vector CV are detected. Then, the SAD values obtained for the reference block of the reference vector RV are distributed and added to SAD values corresponding to the detected plurality of nearby reference vectors.

  In this case, in this embodiment, the value to be distributed and added as a component to be written to the table element tbl corresponding to the position indicated by the plurality of reference vectors around the position indicated by the reference reduced vector CV is the value before reduction. From the SAD value obtained corresponding to the original reference vector RV of the reference, using the positional relationship indicated by each of the reference reduced vector and the reference vector in the vicinity thereof, variance corresponding to each of the reference vectors in the vicinity The SAD value to be added is calculated, and each of the calculated SAD values is added as a table element component of the corresponding reference vector.

  Here, not only dispersion but addition is performed because a plurality of reference vectors in the vicinity of the reference reduced vector are detected redundantly with respect to a plurality of different reference reduced vectors. It means that duplicate SAD values are added.

  When the position (vx / n, vy / n) indicated by the reference reduced vector CV matches the position indicated by the reference vector, that is, when the values of vx / n and vy / n are integers, It is not necessary to detect the reference vector, and the SAD value obtained corresponding to the original reference vector RV before reduction is stored corresponding to the reference vector indicating the position (vx / n, vy / n). It is what you want to do.

  Next, the above process will be described with a specific example. For example, when the position of the target block is set as the reference (0, 0), as shown in FIG. 7A, the reference block RV indicating the position (−3, −5) is set in the horizontal direction and the vertical direction. , 1 / n = 1/4 times, the position indicated by the reference reduced vector CV is (−0.75, −1.25) as shown in FIG.

  Accordingly, a decimal component is generated at the position indicated by the reference reduced vector CV and does not coincide with the position indicated by the reference vector.

  Therefore, in this case, as shown in FIG. 8, a plurality of neighboring reference vectors indicating positions near the position indicated by the reference reduced vector CV are detected. In the example of FIG. 8, four neighboring reference vectors NV1, NV2, NV3, and NV4 are detected for one reference reduced vector CV.

  As described above, in this embodiment, the SAD values obtained for the reference block of the reference vector RV are distributed and added as SAD values corresponding to these four neighboring reference vectors NV1, NV2, NV3, NV4. .

  In this case, in this embodiment, the SAD value to be distributed and added to each of the four neighboring reference vectors NV1, NV2, NV3, and NV4 is the position P0 indicated by the reference reduced vector CV (shown as x in FIG. 8). Is calculated as a linearly weighted variance using the positional relationship between the positions P1, P2, P3, and P4 (shown as circles in FIG. 8) indicated by four neighboring reference vectors NV1, NV2, NV3, and NV4. To do.

  In the case of the example in FIG. 8, the position P0 indicated by the reference reduced vector CV is the positions P1, P2, P3, and P4 indicated by the four reference vectors NV1, NV2, NV3, and NV4 in the vicinity. It is a position that internally divides 1: 3 in the horizontal direction and 3: 1 in the vertical direction.

Therefore, when the SAD value obtained corresponding to the original reference vector RV before reduction is Sα, positions P1, P2, indicated by four reference vectors NV1, NV2, NV3, NV4 in the vicinity of the periphery, respectively. Each of the values SADp1, SADp2, SADp3, SADp4 to be distributed and added to the SAD table elements corresponding to P3, P4 is
SADp1 = Sα × 9/16
SADp2 = Sα × 3/16
SADp3 = Sα × 3/16
SADp4 = Sα × 1/16
It becomes.

  In this embodiment, the obtained values SADp1, SADp2, SADp3, SADp4 are respectively transferred to positions P1, P2, P3, P4 indicated by the four neighboring reference vectors NV1, NV2, NV3, NV4. Each is added to the corresponding SAD table element.

  In this embodiment, the above processing is performed for all reference blocks in the search range.

  From the above, in this embodiment, when the reference vector RV is reduced to 1 / n, the SAD table TBLo of the conventional size corresponding to all the reference vectors on a one-to-one basis is 1 / in the horizontal direction. n, a reduced SAD table TBLs reduced to 1 / n in the vertical direction is prepared, and a SAD value corresponding to a reference vector in the vicinity of the reference reduced vector CV is obtained as a table element of the reduced SAD table TBLs. (See FIG. 6).

Therefore, in the case of this embodiment, the number of table elements of the reduced SAD table TBLs is 1 / n ^{2 of the} number of table elements of the conventional SAD table TBLo, and the table size can be significantly reduced.

  In the description of the above-described embodiment, four reference vectors in the vicinity of the reference reduced vector CV are detected, and the reference reference block (reference) is detected for the SAD table element corresponding to the four neighboring reference vectors. The SAD value calculated for the vector RV) is linearly weighted distributed and added. The method of selecting a plurality of reference vectors near the reference reduced vector CV and the method of distributed addition for the SAD table elements corresponding to the neighboring reference vectors are as follows: It is not limited to the above example.

  For example, nine or sixteen reference vectors in the vicinity of the reference reduced vector CV are detected, and dispersion addition by so-called cubic interpolation is performed on the SAD table elements corresponding to the nine or sixteen neighboring reference vectors. If the operation is performed, the accuracy becomes higher. However, when emphasizing the real-time property and the reduction of arithmetic circuits, the above-described linear weighted dispersion addition to the table elements corresponding to the four reference vectors in the vicinity is more effective.

  Also in this embodiment, the reference block is moved uniformly within the search range, and the SAD values of all the reference blocks are substituted into the SAD table (in this embodiment, the reduced SAD table), which is the same as the conventional method. It is.

  However, conventionally, since the addresses of the reference vector and the SAD table element correspond to each other on a one-to-one basis, the assignment to the SAD table is simple. However, in the method according to this embodiment, the SAD value calculated for the reference block is distributed and added. Therefore, in the reduced SAD table, the reference vector (reduced reference vector) and the table address are not one to one. Therefore, in the case of the method of this embodiment, it is necessary not to simply assign the SAD value to the table address, but to perform so-called substitution addition in which addition is performed. For this reason, each table element of the SAD table (reduced SAD table) must first be initialized (cleared to 0).

  By the way, in the conventional block matching, in the SAD table completed as described above, the table element having the minimum SAD value is searched, and the table address of the table element having the minimum value is converted into a reference vector. In this case, the motion vector detection is completed.

  On the other hand, in the method according to this embodiment, since the SAD table is a reduced SAD table corresponding to a reduced reference vector obtained by reducing the reference vector, the minimum value of the reduced SAD table directly corresponds to an accurate motion vector. Not done.

  However, in the case of an apparatus that allows a certain amount of error, the table element address that is the minimum value of the reduced SAD table is converted to a reference vector, and the reciprocal multiple of the reduction ratio 1 / n, that is, n times. By doing so, it is possible to detect a motion vector.

  However, in order to detect a more accurate motion vector, as described below, an interpolation process is performed on the table element values of the reduced SAD table, so that an accurate motion can be achieved with the original vector accuracy. A vector (motion vector for each block) is detected.

  In the above description, SAD tables for a plurality of target blocks are obtained using a block matching method using a conventional reference vector that does not use a reduced reference vector, and the corresponding coordinates of the obtained plurality of SAD tables are obtained. The sum SAD value is added to obtain the sum SAD table, and the sum SAD table is described to detect the global motion vector by performing approximate curve interpolation. In this case, the interpolation processing described below is performed in this case. It can also be used as an approximate curve interpolation.

[First example of interpolation processing for detecting a more accurate motion vector]
A first example of interpolation processing for detecting a more accurate motion vector is a method of approximating a plurality of SAD table element values (SAD values) in a reduced SAD table with one quadric surface.

  That is, in the reduced SAD table, a table element having a minimum SAD value (integer precision minimum value table element (integer precision table address)), and a plurality of integer precision table elements centered on the integer precision minimum value table element, And using the SAD values of these table elements, the quadratic surface of the SAD value is determined by the least square method, the SAD value that is the minimum value of this quadric surface is detected, and the detected minimum value is obtained. A position corresponding to the SAD value (a position shifted from the reference position on the reference frame) is detected, and the detected position is a decimal precision minimum value table address (a vector in which the SAD value becomes the minimum value in the reduced SAD table). (Corresponding to the minimum value vector)).

  In this case, in order to define a unique quadric surface, as shown in FIG. 9 (A) or (B), the integer precision minimum value table element tm and the position at which the table element tm is sandwiched from both sides At least four integer precision table elements t1, t2, t3, and t4 in the vicinity of the table element tm are necessary.

  Then, as shown in FIG. 10, in the range of the reference reduced vector corresponding to the reduced SAD table within the search range of the reference frame, the position of the target frame is set as the reference position (0, 0), and the horizontal and vertical directions are set. Considering the axis vx / n and the axis vy / n of the deviation amount (corresponding to the reference reduction vector), the axis of the SAD value is considered as an axis perpendicular to the axis vx / n and the axis vy / n, and these three axes Assume a coordinate space consisting of

  Then, for example, a quadratic curve is obtained in the coordinate space of FIG. 10 from the SAD value of the integer precision minimum value table element tm and the SAD values of the two table elements t1 and t3 sandwiching the integer precision minimum value table element tm. Generate. Further, from the SAD value of the integer precision minimum value table element tm and the SAD values of the other two table elements t2 and t4 sandwiching the minimum value table element tm, another quadratic curve is obtained in the coordinate space of FIG. Is generated. Then, a quadric surface 201 including these two quadratic curves is obtained by the method of least squares, and the quadric surface 201 is generated in a coordinate space as shown in FIG.

  Then, the minimum value 202 of the generated quadratic surface 201 of the SAD value is detected, and the position (vx / n, vy / n) (position 203 in FIG. 10) corresponding to the SAD value taking the minimum value is detected. The detected position (vx / n, vy / n) is detected as a table element (table address) with decimal precision. Then, the vector (minimum value vector) 204 corresponding to the detected decimal precision table element is multiplied by n as shown in FIG. 11 to obtain a motion vector 205 with the original magnitude accuracy.

  For example, as shown in FIG. 12, the minimum value vector 204 obtained from the minimum value address of the decimal precision table element of the reduced SAD table TBLs when the reference vector is reduced to ¼ is (−0.777, − In the case of 1.492), the motion vector 205 is obtained by multiplying these by four (-3.108, -5.968). This motion vector 205 is a reproduction of a motion vector on the scale of the original image.

  In the above description, the integer precision minimum value table element tm and the four neighboring table elements are used. However, in order to obtain a quadratic surface of the SAD value by the least square method, a larger number of multiple neighbors are used. It is better to use table elements. Therefore, generally, the table elements of the rectangular area of the horizontal direction × vertical direction = m × m (m is an integer of 3 or more) around the integer precision minimum value table element tm are used.

  However, the number of multiple neighboring table elements is not necessarily large, and using a wide range of table elements leads to an increase in the amount of calculation and using a false value of a local minimum that depends on the image pattern. Therefore, a rectangular area table element having an appropriate number of neighboring table elements is used.

  As an example of a table element in a rectangular area composed of an appropriate number of neighboring table elements, in this embodiment, horizontal integer × vertical direction = 3 × 3 rectangles around the integer precision minimum value table element tm. An example of using a table element of an area and an example of using table elements of a horizontal area × vertical direction = 3 × 3 rectangular areas around the integer precision minimum value table element tm explain.

[Example using 3 × 3 rectangular area table elements]
FIG. 13 uses the table elements of the horizontal direction × vertical direction = 3 × 3 rectangular areas (shown with a fill in FIG. 13) around the integer precision minimum value table element tm. Here is an example.

  In the case of the example of FIG. 13, as shown in FIG. 13A, using the SAD values of the integer precision minimum value table element tm and eight neighboring table elements, FIG. A quadric surface 201 as shown in FIG. 2 is generated by the method of least squares. Then, the minimum value 202 of the quadratic surface 201 of the generated SAD value is detected, and the position (vx / n, vy / n) corresponding to the SAD value taking the minimum value (position 203 in FIG. 13B). , And the detected position 203 is detected as a decimal value minimum value table element position (decimal accuracy minimum value table address).

  Then, the vector (minimum value vector) 204 corresponding to the detected decimal precision table element position 203 is multiplied by n as shown in FIG. 11 to obtain the motion vector 205 with the original magnitude accuracy.

  Here, the calculation method of the position 203 corresponding to the minimum value 202 of the quadratic surface 201 of the SAD value is as follows. That is, as shown in FIG. 14, consider the (x, y) coordinates where the position of the integer precision minimum value table element tm is the origin (0, 0). In this case, the positions of the surrounding eight table elements are three positions in the x-axis direction, that is, x = -1, x = 0, x = 1, and three positions in the Y-axis direction, that is, y = -1, y = 0, y = 1, and (-1, -1), (0, -1), (1, -1), (-1, 0), ( 0, 1), (-1, 1), (0, 1), (1, 1).

And let the SAD value of each table element in the table of FIG. 14 be Sxy. Therefore, for example, the SAD value of the integer precision minimum value table element tm (position (0, 0)) is represented as S _00, and the SAD value of the table element value at the lower right position (1, 1) is S _11. It is expressed.

  Then, the decimal precision position (dx, dy) in the (x, y) coordinates where the position of the integer precision minimum value table element tm is the origin (0, 0) is shown in (Expression A) and (Expression B) shown in FIG. ).

In (Formula A) and (Formula B) of FIG.
When x = -1, Kx = -1
Kx = 0 when x = 0
When x = 0, Kx = 1
It becomes. Also,
When y = -1, Ky = -1
When y = 0, Ky = 0
When y = 0, Ky = 1
It becomes.

  Since the decimal precision position (dx, dy) thus obtained is a position having the position of the integer precision minimum value table element tm as the origin (0, 0), the decimal precision position (dx, dy) and the integer The position 203 from the center position of the search range to be obtained can be detected from the position of the minimum accuracy table element tm.

[Example using table elements of 4 × 4 rectangular areas]
In FIG. 16, the table elements (shown with a fill in FIG. 16) of the rectangular area of the horizontal direction × vertical direction = 4 × 4 around the integer precision minimum value table element tm are used. An example is shown below.

  In this case, when the value of m is an odd number, such as the integer precision minimum value table element tm and its neighboring 8 table elements (3 × 3) and its neighboring 24 table elements (5 × 5) The integer precision minimum value table element tm is always the center of a plurality of table elements in the rectangular area to be used, and therefore the table range to be used is simply determined.

  On the other hand, when m is an even number, such as 15 neighboring table elements (4 × 4), the integer precision minimum value table element tm is the center position of a plurality of table elements in the rectangular area to be used. Since it is not possible, some ingenuity is necessary.

  That is, as seen from the integer precision minimum value table element tm, the SAD values of the left and right adjacent table elements are compared in the horizontal direction, and the table element adjacent to the adjacent table element in the direction in the direction of the smaller value is obtained. This is adopted as the fourth column of the neighborhood table element. Similarly, the SAD values of the upper and lower adjacent table elements are compared in the vertical direction, and the table element adjacent to the adjacent table element in that direction in the direction of the smaller value is adopted as the fourth row of the neighboring table element. .

  In the example of FIG. 16, since the SAD values of the left and right adjacent table elements in the horizontal direction of the integer precision minimum value table element tm are “177” and “173”, the SAD value of the right adjacent value “173” is small. The column to the right of the table element is adopted as the fourth column. Further, since the SAD values of the upper and lower adjacent table elements in the vertical direction of the integer precision minimum value table element tm are “168” and “182”, the SAD value is smaller than the table element of the adjacent value “168”. The upper adjacent row is adopted as the fourth row.

  In the case of the example of FIG. 16, the quadric surface 201 is generated by the least square method using the integer precision minimum value table element tm and the SAD values of 15 neighboring table elements. Then, the minimum value 202 of the generated quadratic surface 201 of the SAD value is detected, and the position (vx / n, vy / n) (position 203 in FIG. 16) corresponding to the SAD value taking the minimum value is detected. The detected position 203 is detected as a decimal precision minimum value table element position (decimal precision minimum value table address).

  Then, the vector (minimum value vector) 204 corresponding to the detected decimal precision table element position 203 is multiplied by n as shown in FIG. 11 to obtain the motion vector 205 with the original magnitude accuracy.

  Here, the calculation method of the position 203 corresponding to the minimum value 202 of the quadratic surface 201 of the SAD value in the case of this example is as follows. That is, as shown in FIG. 17, consider the (x, y) coordinates where the position of the integer precision minimum value table element tm is the origin (0, 0).

  In the case of this example, 4 as shown in FIGS. 17A, 17B, 17C, and 17D, depending on the position of the integer precision minimum value table element tm in a rectangular area consisting of 16 table elements. It is necessary to consider the arrangement of table elements on the street.

  In this case, the positions of the surrounding 15 table elements are four positions in the x-axis direction, that is, x = −, as can be seen from FIGS. 17 (A), (B), (C), and (D). 1, x = 0, x = 1, x = 2 or x = -2, and four positions in the Y-axis direction, that is, y = -1, y = 0, y = 1, y = 2 or y = 15 positions represented by the repetition of -2.

Then, the SAD value of each table element in the table of FIG. Thus, for example, SAD value of the integer precision minimum value table element tm (position (0, 0)) is expressed as _{S 00,} also, SAD value table element values of the position (1, 1) is expressed as _{S 11} The

  Then, the position (dx, dy) of decimal precision in the (x, y) coordinates with the origin (0, 0) as the center position in the rectangular area composed of the integer precision minimum value table element tm and the surrounding 16 table elements is It can be obtained by (Equation C) and (Equation D) shown in FIG.

  Here, in (Equation C) and (Equation D) in FIG. 18, Kx and Ky are the centers in the rectangular area consisting of the integer precision minimum value table element tm and the surrounding 16 table elements, as shown in FIG. When considering the (Kx, Ky) coordinates where the position is the origin (0, 0), the four table element arrangements shown in FIGS. 17A, 17B, 17C, and 17D are used. It becomes a corresponding value.

That is, in the case corresponding to FIG.
When x = -2, Kx = -1.5
When x = −1, Kx = −0.5
When x = 0, Kx = 0.5
When x = 1, Kx = 1.5
It becomes. Also,
When y = -2, Ky = -1.5
When y = −1, Ky = −0.5
When y = 0, Ky = 0.5
When y = 1, Ky = 1.5
It becomes.

In the case corresponding to FIG. 17B,
When x = -2, Kx = -1.5
When x = −1, Kx = −0.5
When x = 0, Kx = 0.5
When x = 1, Kx = 1.5
It becomes. Also,
When y = −1, Ky = −1.5
When y = 0, Ky = −0.5
When y = 1, Ky = 0.5
When y = 2, Ky = 1.5
It becomes.

In the case corresponding to FIG. 17C,
When x = −1, Kx = −1.5
When x = 0, Kx = −0.5
When x = 1, Kx = 0.5
When x = 2, Kx = 1.5
It becomes. Also,
When y = -2, Ky = -1.5
When y = −1, Ky = −0.5
When y = 0, Ky = 0.5
When y = 1, Ky = 1.5
It becomes.

In the case corresponding to FIG. 17D,
When x = −1, Kx = −1.5
When x = 0, Kx = −0.5
When x = 1, Kx = 0.5
When x = 2, Kx = 1.5
It becomes. Also,
When y = −1, Ky = −1.5
When y = 0, Ky = −0.5
When y = 1, Ky = 0.5
When y = 2, Ky = 1.5
It becomes.

Further, Δx and Δy in (Expression C) and (Expression D) in FIG. 18 are the table element arrangements in FIGS. 17A, 17B, 17C, and 17D with respect to the (Kx, Ky) coordinates. This represents the amount of deviation from the (x, y) coordinates, and as can be seen from FIG.
In the case corresponding to FIG. 17A, Δx = −0.5, Δy = −0.5,
In the case corresponding to FIG. 17B, Δx = −0.5, Δy = 0.5,
In the case corresponding to FIG. 17C, Δx = 0.5, Δy = −0.5,
In the case corresponding to FIG. 17D, Δx = 0.5, Δy = 0.5,
It becomes.

  Since the decimal precision position (dx, dy) thus obtained is a position having the position of the integer precision minimum value table element tm as the origin (0, 0), the decimal precision position (dx, dy) and the integer The position 203 from the center position of the search range to be obtained can be detected from the position of the minimum accuracy table element tm.

[Second Example of Interpolation Process for Detecting More Accurate Motion Vector]
A second example of the interpolation process for detecting a more accurate motion vector is a horizontal third order using SAD values of a plurality of horizontal table elements including the integer precision minimum value table element in the reduced SAD table. A curve is generated, and a vertical cubic curve is generated by using the SAD values of a plurality of vertical table elements including the integer precision minimum value table element, and the position (where the minimum value of each cubic curve becomes a minimum value ( vx, vy) is detected, and the detected position is used as the minimum value address with decimal precision.

  FIG. 20 is a diagram for explaining the second example. Similarly to the first example described above, the integer precision minimum value table element tm and a plurality of integer precision table elements centered on this integer precision minimum value table element, 4 × 4 = 16 in the example of FIG. The table elements are obtained (refer to the portion with a fill in FIG. 20A).

  Next, as in the first example, as shown in FIG. 20B, the position of the target frame is set to the reference position within the reference reduced vector range corresponding to the reduced SAD table in the reference frame search range. Assuming (0,0) the axis vx / n and the axis vy / n of the horizontal and vertical shift amounts (corresponding to the reference reduction vector), the axis vx / n and the axis vy / n are perpendicular to each other. As an axis, an SAD value axis is considered, and a coordinate space including these three axes is assumed.

  Next, among the 16 table elements around the integer precision minimum value table element tm, the horizontal SAD values of the four horizontal table elements including the integer precision minimum value table element tm are used to A cubic curve 206 of directions is generated. The horizontal position of the decimal precision minimum value table element position is detected as the horizontal position vx / n corresponding to the minimum value of the horizontal cubic curve 206.

  Next, among the 16 table elements around the integer precision minimum value table element tm, the SAD values of the four vertical table elements including the integer precision minimum value table element tm are used to A cubic curve 207 in the direction is generated. The vertical position of the decimal precision minimum value table element position is detected as the vertical position vy / n corresponding to the minimum value of the cubic curve 207 in the vertical direction.

  The decimal precision minimum value table element position (decimal precision minimum value table address) 208 is detected from the horizontal position and the vertical position of the decimal precision minimum value table element position obtained as described above. Then, the vector (minimum value vector) 209 corresponding to the detected decimal precision table element position 208 is multiplied by n as shown in FIG. 11 to obtain an original magnitude precision motion vector.

  That is, in the second example, four table elements in the horizontal direction and the vertical direction are determined by the method described in the first example, and as shown in FIG. 20B, the horizontal direction and the vertical direction are determined. This is a method for uniquely determining a cubic curve.

Here, the calculation method of the position 208 corresponding to the minimum value 202 of the cubic curves 206 and 209 of the SAD value is as follows. That is, in a cubic curve in either the horizontal direction or the vertical direction, four SAD values in the vicinity of the minimum value are set in the order along either the horizontal direction or the vertical direction, S ₀ , When S ₁ , S ₂ , and S ₃ are used, an equation for calculating the fractional component u that takes the minimum value depends on which of the three sections Ra, Rb, and Rc shown in FIG. Different.

Here, the section Ra is a section between the position where the SAD value S ₀ is located and the position where the SAD value S ₁ is located, and Rb is a section between the position where the SAD value S ₁ is located and the position where the SAD value S ₂ is located, Rc is the section between the position where the position and the SAD value _{S 3} as a SAD value _{S 2.}

  When the minimum value of decimal precision is in the interval Ra shown in FIG. 21, the decimal component u of deviation up to the position where the minimum value is obtained with respect to the position of the minimum value of integer precision is calculated by (Equation E) in FIG. Is done.

  Similarly, when the minimum value of decimal precision is in the section Rb shown in FIG. 21, the decimal component of the deviation up to the position where the minimum value is obtained with respect to the position of the minimum value of integer precision is obtained by (Equation F) in FIG. u is calculated.

  Further, when the minimum value of decimal precision is in the section Rc shown in FIG. 21, the decimal component u of deviation up to the position where the minimum value is obtained with respect to the position of the minimum value of integer precision is calculated by (Equation G) of FIG. Is done.

  Then, determination as to which of the three sections Ra, Rb, and Rc shown in FIG. 21 has the minimum decimal precision value is performed as follows.

That is, FIG. 23 is a diagram for explaining the determination. As shown in FIGS. 23A, 23B, and 23C, first, the minimum value Smin of the integer precision SAD value and the second smallest integer precision SAD value Sn2 are detected, and the decimal precision minimum is detected. The value is detected as existing in a section between the position of the minimum value Smin of the detected integer precision SAD value and the position of the second smallest integer precision SAD value Sn2. Next, the minimum value Smin of the integer precision SAD value and the second smallest integer precision SAD value Sn2 are the positions of any of the SAD values S ₀ , S ₁ , S ₂ , S ₃ shown in FIG. Whether or not the detected section is the section Ra, Rb, or Rc is determined depending on whether or not it is.

  As shown in FIG. 23D, when the minimum value Smin of the integer precision SAD value is at or at the position of the SAD value and is positioned at the end of the four table element values, the minimum position is estimated. In this embodiment, it is assumed that it cannot be performed, so that it is treated as an error and the minimum value position is not calculated. However, even in the case of FIG. 23D, the minimum value position may be calculated.

As described above, according to this embodiment, it is possible to detect a motion vector on the original image scale using a reduced SAD table having a small size scaled down to 1 / n ² . In this case, FIG. 24 shows that a vector detection result almost the same as the conventional one can be obtained despite the use of a reduced SAD table with a small size scaled down to 1 / n ² .

  The horizontal axis in FIG. 24 is the reduction factor n in the one-dimensional direction in either the horizontal direction or the vertical direction, and the vertical axis indicates an error (vector error) about the detected motion vector. The numerical value of the vector error in FIG. 24 is represented by the number of pixels.

  In FIG. 24, a curve 301 is an average value of vector errors with respect to the reduction magnification. A curve 302 indicates a triple value (3σ (99.7%)) of the vector error variance σ with respect to the reduction ratio. A curve 303 represents an approximate curve of the curve 302.

  FIG. 24 shows the vector error with respect to the reduction ratio n in the one-dimensional direction. However, since the SAD table is two-dimensional, the table size (number of table elements) is reduced by the ratio of the square shown in FIG. On the other hand, since the vector error increases only to a linear level, the usefulness of the method according to this embodiment can be understood.

  Further, even at a reduction ratio of n = 64 (reduction ratio 1/64) times, the vector error is small, and no failure that causes a completely different motion vector to be detected and output is observed. It can be said that the size of the table can be reduced.

  In addition, as described above, in the image stabilization of a moving image, real-time characteristics and reduction of system delay are strongly demanded. On the other hand, in terms of accuracy, a certain amount of vector is used except when a completely different broken motion vector is detected. Tolerant of detection error. Therefore, it can be said that this embodiment can greatly reduce the size of the SAD table without failing, and is highly useful.

  In this embodiment, the reference frame 102 is divided into a plurality of regions, and a motion vector (motion vector for each block) 205 is detected in each divided region. As described above, since there is a high possibility that a moving subject is included in the frame as described above, for example, 16 motion vectors 205 are detected in one frame of the reference frame 102 as shown in FIG. This is because one global motion vector, that is, a camera shake vector of the frame is determined for one frame by statistically processing the transition from the motion vector 205 in the frame.

  In this case, as shown in FIG. 25, search ranges SR1, SR2,..., SR16 centering on the respective reference positions PO1 to PO16 of the 16 motion vectors 205 to be detected are defined, and in each search range, the target Assume block projected image blocks IB1, IB2,..., IB16.

  Then, a reference block having the same size as the projected image blocks IB1, IB2,..., IB16 is set, and the set reference block is moved within each search range SR1, SR2,. In the same manner as described above, a reduced SAD table is generated, and the motion vector 205 in each search range SR1, SR2,. Therefore, in this embodiment, the SAD table TBLi is a configuration of a reduced SAD table.

  In this embodiment, the 16 reduced SAD tables obtained for the target blocks in the 16 search ranges are arranged so as to overlap as shown in FIG. 1, and the reference block positions corresponding to each other in the search range are arranged. That is, the SAD values at the same coordinate position are added together in the reduced SAD table to obtain the combined SAD value. Then, a combined reduced SAD table for a plurality of reference block positions within one search range is generated as an SAD table including the combined SAD values. Therefore, in this embodiment, the total SAD table SUM_TBL has the configuration of the total reduction SAD table.

  In this embodiment, the reduced SAD table TBLi, the block-by-block motion vector 205 obtained by performing the above-described approximate interpolation processing on the reduced SAD table TBLi, and the summed SAD table SUM_TBL are used. Reliability determination processing and re-summation SAD table RSUM_TBL generation as shown in FIG. 5, and curve approximation interpolation processing using the minimum SAD value of the re-summation SAD table RSUM_TBL generated and a plurality of SAD values in the vicinity thereof To calculate a global motion vector.

  The image processing method using the reduced SAD table according to the embodiment described above has the following two methods compared to the method of detecting a motion vector with a size obtained by reducing and converting the image described in Patent Document 3 described as the conventional method. In that respect, it has greatly different merits.

  First, unlike the conventional method described in Patent Document 3, the method according to this embodiment does not require any process for reducing and converting an image. This is because, in the method according to this embodiment, when the SAD value calculated for the reference block is substituted and added to the SAD table (reduced SAD table), address conversion corresponding to the reduction magnification is simultaneously performed.

  Thereby, in the method according to this embodiment, the logic for reducing and converting the image as in the conventional method described in Patent Document 3, and the time and bandwidth wasted for storing the reduced image in the memory are reduced. There is an advantage that it is not necessary to secure an area for pasting an image on a memory.

  As another important problem of the conventional method described in Patent Document 3, as described above, aliasing (folding distortion) when reducing and converting an image, and a low-pass filter for removing low illumination noise There is a problem of existence. That is, when the image is reduced, resampling must be performed after passing through an appropriate low-pass filter, otherwise unnecessary aliasing occurs and the accuracy of the motion vector using the reduced image is significantly impaired. It is.

  It is theoretically proved that the characteristic of an ideal low-pass filter at the time of reduction conversion is a function similar to a sinc function. The sinc function itself is an infinite tap FIR (Finite Impulse Response) filter having a cutoff frequency f / 2 expressed in the form of sin (xπ) / (xπ), which is ideal when the reduction ratio is 1 / n. A low-pass filter having a low cutoff frequency f / (2n) is expressed as sin (xπ / n) / (xπ / n). However, this may also be a form of the sinc function.

  The upper side of FIGS. 26 to 28 shows the shape of the sinc function (ideal characteristics of the low-pass filter) when the reduction magnification is 1/2, 1/4, and 1/8. From FIG. 26 to FIG. 28, it can be seen that the larger the reduction ratio is, the larger the function is in the tap axis direction. That is, it can be said that the number of taps of the FIR filter must be increased even when the infinite tap sinc function is approximated only by the main coefficients.

  In general, it is known that the number of taps in a filter that realizes a cut-off frequency in a lower band becomes dominant with respect to its performance rather than the filter shape.

  Therefore, in the case of the motion vector calculation method using the reduced image of the conventional method described in Patent Document 3, the larger the image reduction magnification, the greater the SAD table reduction effect, but the image generation The low-pass filter as the preprocessing filter has a contradiction that the cost increases as the reduction ratio increases.

  In general, when implementing an FIR filter with a high-order tap, the cost of the arithmetic logic increases in proportion to the square of the number of taps, but this is a problem, but the larger problem is the number of line memories for realizing the vertical filter. Is an increase. In recent digital still cameras, so-called strip processing is performed in order to reduce the size of the line memory as the number of pixels increases. For example, even if the size per line is reduced, the number of line memories increases. Doing so significantly increases the total cost converted in the physical layout area.

  As described above, it can be seen that the image reduction approach of the conventional method described in Patent Document 3 has a significant wall especially in realizing the vertical low-pass filter. In contrast, the method of this embodiment solves this problem in a completely different manner.

  The image of the low-pass filter in the method according to this embodiment is shown below FIGS. In the method according to this embodiment, the image of the low-pass filter in the generation calculation process of the reduced SAD table is illustrated without the image reduction process.

  As shown in the lower side of FIGS. 26 to 28, the characteristic of the low-pass filter is a simple filter characteristic obtained by linearly approximating the main coefficient portion of the sinc function, but the number of taps is linked to the reduction ratio. It can be seen that it has increased. This is suitable for the fact that the performance of the low-pass filter becomes dominant in the number of taps as the cut-off frequency becomes lower. That is, the processing itself for performing the dispersion addition of the SAD values in this embodiment, such as the processing for performing the linear weighted dispersion addition of the embodiment, realizes a high-performance low-pass filter linked with the magnification with a simple circuit. It is equivalent.

  There are other advantages associated with this low-pass filter. In the conventional method described in Patent Document 3, an image is reduced by applying a low-pass filter and then re-sampling, but a considerable amount of image information is lost at this point. That is, in the calculation of the low-pass filter, the word length of the luminance value of the image information is greatly rounded and stored in the memory, and the lower bits of most pixel information do not affect the reduced image.

  On the other hand, in the method according to this embodiment, the SAD value is calculated by using all bit information of the luminance values of all the pixels uniformly and equally, and the dispersion added value is obtained and added to the reduced SAD table. As long as the word length of each table element value of the reduced SAD table is increased, the calculation can be performed without including any rounding error until the final SAD value is output. Since the area of the reduced SAD table is smaller than that of the frame memory, the extension of the word length of the reduced SAD table is not a big problem. As a result, the reduced SAD table and motion vector detection can be realized with high accuracy.

[Embodiment of Image Processing Device According to the Invention]
Next, as an embodiment of an image processing apparatus using the image processing method according to the present invention, a case of an imaging apparatus will be described as an example with reference to the drawings. FIG. 29 shows a block diagram of an example of an imaging apparatus as an embodiment of the image processing apparatus of the present invention.

  As shown in FIG. 29, in the imaging apparatus of this embodiment, a CPU (Central Processing Unit) 1 is connected to the system bus 2, and the imaging signal processing system 10, the user operation input unit 3, and the like are connected to the system bus 2. The image memory unit 4 and the recording / reproducing device unit 5 are connected to each other. In this specification, the CPU 1 includes a ROM (Read Only Memory) that stores programs for performing various software processes, a work area RAM (Random Access Memory), and the like.

  In response to an imaging recording start operation through the user operation input unit 3, the imaging apparatus in FIG. 29 performs recording processing of captured image data as described later. In addition, upon receiving a reproduction start operation of the captured / recorded image via the user operation input unit 3, the imaging apparatus in FIG. 29 performs a reproduction process of the captured image data recorded on the recording medium of the recording / reproducing apparatus unit 5.

  As shown in FIG. 29, incident light from a subject through a camera optical system (not shown) including an imaging lens 10 </ b> L is applied to the imaging element 11 and imaged. In this example, the imaging device 11 is configured by a CCD (Charge Coupled Device) imager. Note that the image sensor 11 may be a CMOS (Complementary Metal Oxide Semiconductor) imager.

  In the image pickup apparatus of this example, when an image recording start operation is performed, the image pickup device 11 samples the red (R), green (G), and blue by sampling with the timing signal from the timing signal generating unit 12. An analog image pickup signal that is a RAW signal in a Bayer array composed of the three primary colors (B) is output. The output analog imaging signal is supplied to the preprocessing unit 13, subjected to preprocessing such as defect correction and γ correction, and supplied to the data conversion unit 14.

  The data conversion unit 14 converts the analog imaging signal input thereto into a digital imaging signal (YC data) composed of a luminance signal component Y and a color difference signal component Cb / Cr, and converts the digital imaging signal into a system. The image data is supplied to the image memory unit 4 via the bus.

  In the example of FIG. 29, the image memory unit 4 includes two frame memories 41 and 42, and the digital image pickup signal from the data conversion unit 14 is first stored in the frame memory 41. When one frame elapses, the digital imaging signal stored in the frame memory 41 is transferred to the frame memory 42, and the digital imaging signal of a new frame from the data conversion unit 14 is written in the frame memory 41. It is. Accordingly, the frame memory 42 stores a frame image one frame before the frame image stored in the frame memory 41.

  Then, the hand movement vector detecting unit 15 accesses the two frame memories 41 and 42 via the system bus 2, reads out the stored data, and has 16 SADs per frame as described above. Processing such as table generation processing, block-by-block motion vector detection processing, total SAD table generation processing, re-total SAD table generation processing, and global motion vector detection processing is executed. In this case, the frame image stored in the frame memory 42 is an original frame image, and the frame image stored in the frame memory 41 is a reference frame image. Actually, the frame memories 41 and 42 are rotated as a double buffer.

  The camera shake vector detection unit 15 transmits a camera shake vector (global motion vector) as a detection result to the resolution conversion unit 16 at the subsequent stage as a control signal.

  The resolution conversion unit 16 performs a process of converting the image data of the delay frame stored in the frame memory 42 to a necessary resolution and image size in accordance with the camera shake vector received from the camera shake vector detection unit 15. By cutting out according to the motion vector from the frame memory 42, the image after conversion becomes an image from which camera shake is removed.

  The image data from which the camera shake is removed from the resolution converting unit 16 is converted into an NTSC standard color video signal by an NTSC (National Television System Committee) encoder 18 and supplied to a monitor display 6 constituting an electronic viewfinder. The image at the time of shooting is displayed on the monitor screen.

  In parallel with this monitor display, the image data from which the camera shake from the resolution conversion unit 16 has been removed is subjected to coding processing such as recording modulation by the codec unit 17 and then supplied to the recording / reproducing device unit 5 to be a DVD (Digital Versatile). Disc) and the like on a recording medium such as an optical disk or a hard disk.

  The captured image data recorded on the recording medium of the recording / reproducing apparatus unit 5 is read in response to a reproduction start operation through the user operation input unit 3, supplied to the codec unit 17, and reproduced and decoded. The reproduced and decoded image data is supplied to the monitor display 6 through the NTSC encoder 18, and the reproduced image is displayed on the display screen. Although not shown in FIG. 29, the output video signal from the NTSC encoder 18 can be derived to the outside through a video output terminal.

  The above-described camera shake vector detection unit 15 can be configured by hardware, or can be configured by using a DSP (Digital Signal Processor). Furthermore, software processing can be performed by the CPU 1. Also, a combination of hardware or DSP processing and software processing by the CPU 1 can be used.

[Processing Operation in Camera Shake Vector Detection Unit 15]
An example of the flow of processing operations in the camera shake vector detection unit 15 will be described below with reference to the flowcharts of FIGS.

  30 and 31 show processing for one reference frame, and the processing routine of FIGS. 30 and 31 is executed in each reference frame. In this case, if the process of step S31 is set for the first reference frame, it can be omitted in the subsequent reference frames.

  First, as shown in FIG. 25, for the 16 search ranges for 16 target blocks, the center position of each target block is set to the center of each search range, and the search range offset is set to zero. In addition, each search range is set to the maximum range assumed in this embodiment (step S31 in FIG. 30).

  Next, the above-described reduction SAD table and block-by-block motion vector calculation processing is executed for each of the 16 target blocks in the set search range (step S32). The detailed processing routine of step S32 will be described later.

  When the generation of the reduced SAD table for the 16 target blocks is completed, the SAD value of the reference block position corresponding to each of the search ranges in the 16 reduced SAD tables according to (Equation 3) shown in FIG. Is added to generate a combined reduced SAD table for a plurality of reference block positions within one search range having the same size as the reduced SAD table (step S33).

  Next, in the generated combined reduced SAD table, the minimum SAD value is detected, and the above-described approximate curved surface interpolation process is performed using the detected minimum SAD value and a plurality of SAD values in the vicinity thereof to calculate the combined motion vector. (Step S34).

  Next, the conditions shown in FIG. 4 are determined from the SAD values of the 16 reduced SAD tables and the motion vectors for each block with reference to the combined motion vector calculated in step S34, and the 16 targets are determined. The labeling as described above for the SAD table for each of the blocks and the calculation process of the total score sum_score for the reference frame are performed, and the calculated labeling result and the total score sum_score are held (step S35).

  Next, a majority vote is taken for the 16 motion vectors per block calculated in step S32 (step S36), and based on the motion vector per block at the top of the majority vote, the SAD values of the 16 reduced SAD tables and each block The condition corresponding to FIG. 4 as described above is determined from the motion vector, and the labeling as described above for the SAD table for each of the 16 target blocks and the calculation of the total score many_score for the reference frame are performed. Processing is performed, and the calculated labeling result and the total score many_score are held (step S37).

  Then, the combined motion vector calculated in step S34 is compared with the motion vector of the majority vote detected as a result of the majority process in step S36, and both motion vectors are within one neighbor as coordinate positions on the reduced SAD table. It is determined whether or not the coordinate position is different in only one position in the diagonal direction (step S38).

  If it is determined in step S38 that the difference between the combined motion vector and the motion vector of the majority vote is not within one neighbor, it is determined that the global motion vector of the reference frame is not reliable, and the global motion of the reference frame is determined. For example, the global motion vector is set to zero so that the vector is not a camera shake vector (step S39). Then, this processing routine ends.

  In step S39, instead of zero, the global motion vector of the frame may be set to the same value as the global motion vector calculated in the reference frame of the previous frame.

  If it is determined in step S38 that the difference between the combined motion vector and the motion vector of the majority vote is within one neighbor, the total score sum_score obtained in step S35 is equal to or greater than a predetermined threshold value θth1. And it is discriminate | determined whether the total score many_score calculated | required by step S37 is more than predetermined threshold value (theta) th2 (step S41 of FIG. 31).

  In step S41, when one or both of the total score sum_score and the total score many_score do not satisfy the condition of the threshold θth1 and the threshold θth2 or more, the process proceeds to step S39, and the global motion vector of the reference frame has reliability. For example, the global motion vector is set to zero so that the global motion vector of the reference frame is not a camera shake vector. Then, this processing routine ends.

  In step S41, when both the total score sum_score and the total score many_score satisfy the condition that the threshold θth1 and the threshold θth2 are equal to or larger than “TOP” and “TOP” in the SAD table for the target block labeled in step S35. The total SAD value is recalculated using only the SAD value of the SAD table labeled “NEXT_TOP”, and the total reduced SAD table is recalculated (step S42).

  Then, approximate surface interpolation processing is performed using the coordinate position of the minimum SAD value and the plurality of SAD values in the vicinity of the coordinate position in the recombination reduced SAD table obtained by re-creation (step S43). In this example, the approximate curved surface interpolation process using the table elements of 3 × 3 rectangular areas described with reference to FIG. 13 is executed.

  Then, the motion vector detected as a result of the approximate curved surface interpolation process is secured as a global motion vector, and is passed to the resolution conversion unit 16 as a camera shake vector (step S44). Thus, the processing operation in the camera shake vector detection unit 15 for one reference frame is completed.

  In the flowcharts of FIGS. 30 and 31, the camera shake vector detection unit 15 may perform the processing from step S31 to step S34, and the subsequent processing may be performed by software. In this case, the calculated camera shake vector is supplied from the CPU 1 to the resolution conversion unit 16. Alternatively, the CPU 1 cuts out (reads out) the image data of the delay frame stored in the frame memory 42 of the image memory unit 4 according to the calculated camera shake vector, and supplies it to the resolution conversion unit 16. Also good.

  In addition, as described above, when detecting a camera shake vector (global motion vector), a processing method for securing a global motion vector as described above is further added to a conventionally proposed motion vector in the time axis direction. Further reliability and accuracy may be improved by combining methods for predicting global motion vectors from frequencies. However, in the case of moving images, it is necessary to be careful because processing time is required to be extremely short.

  In the above-described example, the recombination reduction SAD table uses only the SAD values of the reduction SAD table of the blocks assigned with the labels “TOP” and “NEXT_TOP” among the blocks labeled in step S35. However, only the SAD value of the reduced SAD table of the blocks labeled “TOP” and “NEXT_TOP” among the blocks labeled in step S37 may be used. Of the blocks labeled in both step S35 and step S37, a recombined reduced SAD table is generated using the SAD values of the reduced SAD table of the blocks labeled “TOP” and “NEXT_TOP”. You may do it.

  In the above example, the sums sum_score and many_score of the scores corresponding to the labels given to the motion vectors for each block are used as one of the global motion vector evaluation determinations for the reference frame for which the motion vector is calculated. However, instead of this total score, whether the label given to the motion vector for each block is “TOP” or “NEXT_TOP” is greater than or equal to a predetermined threshold value is used as an evaluation judgment material. If the number of “TOP” and “NEXT_TOP” is equal to or greater than a predetermined threshold, it may be determined that the evaluation is high.

[Example of processing routine of step S32]
Next, an example of a reduced SAD table generation process and a block-by-block motion vector calculation process routine for each target block in step S32 of FIG. 30 will be described.

<First example>
32 and 33 show a first example of the reduced SAD table and the block-by-block motion vector calculation processing routine for each target block in step S32.

  First, a reference vector (vx, vy) corresponding to one reference block position in the search range SR as shown in FIG. 25 is specified (step S101). As described above, (vx, vy) is a position indicated by the designated reference vector when the position on the frame of the target block (which is the center position of the search range) is the reference position (0, 0). Vx is a horizontal deviation component from the reference position due to the designated reference vector, and vy is a vertical deviation component from the reference position due to the designated reference vector. Similarly to the above-described conventional example, the shift amounts vx and vy are values in units of pixels.

Here, when the center position of the search range is the reference position (0, 0) and the search range is ± Rx in the horizontal direction and ± Ry in the vertical direction,
-Rx≤vx≤ + Rx, -Ry≤vy≤ + Ry
It is supposed to be.

Next, the coordinates (x, y) of one pixel in the target block Io are designated (step S102). Next, the pixel value Io (x, y) of one designated coordinate (x, y) in the target block Io and the pixel value Ii (x + vx, y + vy) of the corresponding pixel position in the reference block Ii The absolute difference value α is calculated as shown in (Equation 1) described above (step S103).
Then, the calculated absolute difference value α is added to the SAD value so far of the address (table element) indicated by the reference vector (vx, vy) of the reference block Ii, and the SAD value that is the addition is added to the address (Step S104). That is, when the SAD value corresponding to the reference vector (vx, vy) is expressed as SAD (vx, vy), this is expressed by the above-described (Equation 2), that is,
SAD (vx, vy) = Σα = Σ | Io (x, y) −Ii (x + vx, y + vy) |
... (Formula 2)
And written to the address indicated by the reference vector (vx, vy).

  Next, it is determined whether or not the above-described operations of Step S102 to Step S104 have been performed on all the coordinates (x, y) in the target block Io (Step S105), and all the coordinates in the target block Io are determined. For the pixel of (x, y), when it is determined that the calculation has not been completed yet, the process returns to step S102, and the pixel position of the next coordinate (x, y) in the target block Io is designated. The processes after step S102 are repeated.

  The processing from step S101 to step S105 is exactly the same as step S1 to step S5 in the flowchart shown in FIG.

  In this embodiment, when it is determined in step S105 that the above calculation has been performed for all the pixels of the coordinates (x, y) in the target block Io, the reduction ratio is set to 1 / n, and the reference vector (vx, vy) is set. ) Is reduced to 1 / n, and a reference reduced vector (vx / n, vy / n) is calculated (step S106).

  Next, a plurality of reference vectors in the vicinity of the reference reduced vector (vx / n, vy / n), in this example, four neighboring reference vectors are detected as described above (step S107). Then, as described above, the values to be distributed and added as the table elements corresponding to the detected four neighboring reference vectors, as described above, based on the positional relationship indicated by the reference reduced vector and the neighboring reference vector, step S104. Is obtained as a linear weighted variance value from the SAD value obtained in step S108. Then, the obtained four linear weighted variance values are added to the SAD table element value corresponding to each of the neighborhood reference vectors (step S109).

  When step S109 is completed, it is determined that the calculation of the SAD value for the reference block of interest has been completed, and the above-described processing is performed for all reference blocks in the search range, that is, all reference vectors (vx, vy). It is determined whether or not the arithmetic processing from step S101 to step S109 has been completed (step S111 in FIG. 33).

  If it is determined in step S111 that there is a reference vector (vx, vy) for which the above arithmetic processing has not yet been completed, the process returns to step S101, and the next reference vector (vx, vy) for which the above arithmetic processing has not been completed. ) Is set, and the processes after step S101 are repeated.

  If it is determined in step S111 that the reference vector (vx, vy) for which the above arithmetic processing has not been completed is no longer in the search range, it is determined that the reduced SAD table is completed, and the minimum value is determined in the completed reduced SAD table. The SAD value is detected (step S112).

  Next, the SAD value (minimum value) of the table element address (mx, my) having the minimum value and a plurality of neighboring SAD values, in this example, the SAD values of 15 neighboring table elements as described above. Is used to generate a quadric surface (step S113), and a minimum value vector (px, py) indicating the position of decimal precision corresponding to the minimum SAD value of the quadric surface is calculated (step S114). The minimum value vector (px, py) corresponds to the minimum table element address with decimal precision.

  Then, the motion vector (px × n, py × n) is calculated to be obtained by multiplying the calculated minimum value vector (px, py) indicating the position of decimal precision by n (step S115).

  This completes the motion vector detection processing by block matching in this embodiment for one target block. As shown in FIG. 25, when detecting a reduced SAD table and motion vectors for a plurality of target frames set in one frame, in this example, 16 target blocks, the search range is changed every time the target block is changed. Then, the reduction ratio 1 / n is reset, and the processes shown in FIGS. 32 and 33 are repeated for each divided region.

  Needless to say, as a method of calculating the minimum value vector (px, py) indicating the position of decimal precision, the above-described method using the cubic curves in the horizontal direction and the vertical direction may be used.

  In the flowcharts of FIGS. 32 and 33, the processing from step S101 to step S111 may be performed by the hand movement vector detection unit 15, and the subsequent processing may be performed by the CPU 1 by software.

<Second example>
In the first example described above, after obtaining the SAD value for one reference block (reference vector), the variance addition value for a plurality of reference vectors in the vicinity of the reference reduced vector is obtained from the SAD value, and the variance is obtained. Addition processing was performed.

  On the other hand, in the second example, when a difference between each pixel in the reference block and a pixel in the target block is detected, the difference value is used to calculate a plurality of reference vectors in the vicinity of the reference reduced vector. A variance addition value (difference value, not SAD value) is obtained, and the obtained difference value is subjected to variance addition processing. According to the second example, when the difference calculation for all the pixels in one reference block is completed, a reduced SAD table is generated.

  34 and 35 show a flowchart of the motion vector detection process according to the second example.

  The processing from step S121 to step S123 in FIG. 34 is exactly the same as the processing from step S101 to step S103 in FIG. 32, and therefore detailed description thereof is omitted here.

  In this second example, when the difference value α between the reference block and the target block for the pixel at the coordinate (x, y) is calculated in step S123, the reference is made with a reduction ratio of 1 / n. A reference reduced vector (vx / n, vy / n) obtained by reducing the vector (vx, vy) to 1 / n is calculated (step S124).

  Next, a plurality of reference vectors in the vicinity of the reference reduced vector (vx / n, vy / n), in this example, four neighboring reference vectors are detected as described above (step S125). Then, the difference value to be distributed and added as a table element corresponding to each of the detected four neighboring reference vectors, as described above, the reference reduced vector and the neighboring reference vector are obtained from the difference value α obtained in step S123. Based on the positional relationship shown, a linear weighted variance value (difference value) is obtained (step S126).

  Then, the obtained four linear weighted variance values are added to the table element values corresponding to the neighborhood reference vectors (step S127).

  When step S127 is completed, it is determined whether or not the calculations of steps S122 to S127 have been performed for all the coordinates (x, y) in the target block Io (step S128). When it is determined that the calculation has not been completed for all the pixels of the coordinates (x, y), the process returns to step S122, and the pixel position of the next coordinates (x, y) in the target block Io is determined. The process after step S122 is repeated.

  When it is determined in step S128 that the above calculation has been performed for all the pixels of the coordinates (x, y) in the target block Io, it is determined that the calculation of the SAD value for the reference block being noticed is completed, It is determined whether or not the arithmetic processing from step S121 to step S128 has been completed for all reference blocks in the search range, that is, all reference vectors (vx, vy) (step S131 in FIG. 35).

  If it is determined in step S131 that there is a reference vector (vx, vy) for which the above arithmetic processing has not yet been completed, the process returns to step S121, and the next reference vector (vx, vy) for which the above arithmetic processing has not been completed. ) Is set, and the processes after step S121 are repeated.

  If it is determined in step S121 that the reference vector (vx, vy) for which the above arithmetic processing has not been completed is no longer in the search range, it is determined that the reduced SAD table has been completed. In the completed reduced SAD table, the minimum value The SAD value is detected (step S132).

  Next, the SAD value (minimum value) of the table element address (mx, my) having the minimum value and a plurality of neighboring SAD values, in this example, the SAD values of 15 neighboring table elements as described above. Is used to generate a quadric surface (step S133), and a minimum value vector (px, py) indicating the position of decimal precision corresponding to the minimum SAD value of the quadric surface is calculated (step S134). The minimum value vector (px, py) corresponds to the minimum table element address with decimal precision.

  Then, the motion vector (px × n, py × n) is calculated to be obtained by multiplying the calculated minimum value vector (px, py) indicating the decimal precision position by n (step S135).

  This completes the motion vector detection process by block matching in the second example for one target block. As shown in FIG. 25, when detecting a reduced SAD table and motion vectors for a plurality of target frames set in one frame, in this example, 16 target blocks, the search range is changed every time the target block is changed. Then, the reduction ratio 1 / n is reset, and the processing shown in FIGS. 34 and 35 is repeated for each divided region.

  In the second example as well, as a method of calculating the minimum value vector (px, py) indicating the position of decimal precision, the method using the above-described cubic curves in the horizontal direction and the vertical direction may be used. Needless to say.

  In the flowcharts of FIGS. 34 and 35, the processing from step S121 to step S131 may be performed by the camera shake vector detection unit 15, and the subsequent processing may be performed by the CPU 1 by software.

<Third example>
As shown in FIG. 24, when the motion vector detection method according to this embodiment is used, even when the reduction ratio of the reference vector is 1/64, there is a failure to output a completely different motion vector. Therefore, it is possible to reduce the SAD table to substantially 1/404096.

  That is, a reduced SAD table reduced to 1/4096 is prepared, and the first motion vector is detected at a reduction ratio of 1/64. Next, the search range may be narrowed around the motion vector detected at the first time, and the second detection may be performed at a reduction magnification of 1/8, for example, which is smaller than the first time. That is, if the reduction ratio is changed between the first and second times and the second reduction ratio is set so as to be within the first vector error range, the motion vector can be detected with considerably high accuracy. .

  The motion vector detection process in the case of the third example will be described with reference to the flowcharts of FIGS.

  The third example shown in FIGS. 36 to 39 uses the first example described above as the basic motion detection process. Therefore, the processing steps from step S141 to step S149 in FIG. 36 and the processing steps from step S151 to step S155 in FIG. 37 are the processing steps from step S101 to step S109 in FIG. 32 and from step S111 to step S115 in FIG. It is exactly the same as the processing step.

  In the third example, when the motion vector is calculated in step S155 of FIG. 37, the process is not ended there, but the motion vector calculated in step S155 is used as the first motion vector, and the next step S156 is performed. , The search range is narrowed down within the same reference frame based on the motion vector calculated in the first time, and the reduction ratio of the reference vector is reduced to a reduction ratio 1 / nb (less than the first reduction ratio 1 / na). Here, na> nb).

  That is, as shown in FIG. 40, when the motion vector BLK_Vi for the target block TB is calculated in the search range SR_1 set in the first process, the reference frame and the original frame are calculated from the calculated motion vector BLK_Vi. A correlated block range can be roughly detected. Therefore, as the search range SR_2 for the second processing, as shown in the lower side of FIG. 40, a narrowed search range centered on the correlated block range can be set. In this case, as shown in FIG. 40, a position shift (search range offset) between the center position POi_1 of the first search range SR_1 and the center position POi_2 of the second search range SR_2 is detected at the first time. Corresponds to the motion vector BLK_Vi.

  Further, in this embodiment, it can be expected that the second motion vector can be calculated with a smaller error by reducing the reduction ratio with respect to the reference vector than the first time.

  In this way, when the narrowed search range is set in step S156 and a new reduction ratio is set, the second motion vector detection process is performed in the same manner as in the first time, in steps S157 to S158 and the steps in FIG. The process is executed in steps S161 to S168 and steps S171 to S174 in FIG. The processing in these steps is exactly the same as the processing steps in steps S101 to S109 in FIG. 32 and the processing steps in steps S111 to S115 in FIG.

  Thus, finally, in step S174, the target motion vector for each block is obtained as the second motion vector.

  The above example is a case where the above-described first example is used as the motion vector detection method for each block and is repeated in two stages. The search range is further narrowed down, and the reduction ratio is set as necessary. Of course, it is possible to repeat two or more stages while changing.

  Further, it goes without saying that the second example described above can be used instead of the first example described above as a method for detecting the motion vector for each block. Further, as a method for calculating the minimum value vector (px, py) indicating the position of decimal precision, the method using the above-described horizontal and vertical cubic curves may be used as in the above example. is there.

  36 to 39, the processing from step S141 to step S168 may be performed by the camera shake vector detection unit 15, and the subsequent processing may be performed by the CPU 1 by software.

[Second Embodiment of Image Processing Apparatus]
In the camera shake vector detection unit 15 in the imaging apparatus as the first embodiment of the image processing apparatus described above, as shown in FIG. 29, the image memory unit 4 includes two images, that is, an image of the original frame, It was assumed that both of the reference frame images were stored in the frame memory. For this reason, the motion vector detection timing is delayed by one frame.

  On the other hand, in the second embodiment, the sagging image data from the image sensor 11 is used as a reference frame so that the SAD value can be calculated in real time for the raster scan stream data. ing.

  FIG. 41 shows a block diagram of a configuration example of the imaging apparatus in the case of the second embodiment. As can be seen from FIG. 41, the constituent blocks of the imaging signal processing system 10 and other constituent blocks are the same as those in the first embodiment shown in FIG. 29. In this second embodiment, however, the image The memory unit 4 is composed of one frame memory 43. In practice, when a frame memory that cannot be written and read simultaneously is used, the frame memory 43 switches two frame memories alternately for writing and reading every frame as is well known. It is what is used.

  In the second embodiment, the original frame is stored in the frame memory 43, and the reference frame is input from the data converter 14 as a stream. In the first embodiment, the camera shake vector detection unit 15 uses the two pieces of image data stored in the two frame memories 41 and 42 to perform processing for obtaining the SAD value for the reference block. On the other hand, in the second embodiment, as shown in FIG. 41, the stream image data from the data conversion unit 14 is used as the image data of the reference frame, and the image data stored in the frame memory 43 is changed. The SAD value for the reference block is obtained as the image data of the original frame.

  Then, the resolution conversion unit 16 outputs image data free from camera shake by cutting out image data from the frame memory 43 based on the motion vector detected by the camera shake vector detection unit 15. Other configurations and operations are the same as those in the first embodiment.

  As described above, in the second embodiment, the stream image data from the data converter 14 is used as the image data of the reference frame. For this reason, for a certain input pixel, a plurality of reference blocks having this pixel as an element simultaneously exist on the reference frame. FIG. 42 is a diagram for explaining this.

  That is, the input pixel Din in the search range 105 on the reference frame 102 is, for example, a pixel located on the left side of the reference block 1061 to which the reference vector 1071 corresponds, and is located on the upper right side of the reference block 1062 to which the reference vector 1072 corresponds. It can be seen from FIG.

  Therefore, if the input pixel Din belongs to the reference block 1061, it is necessary to read out the pixel D1 of the target block 103 and calculate the difference. If the input pixel Din belongs to the reference block 1062, it is necessary to read out the pixel D2 of the target block 103 and calculate the difference.

  For simplicity, only two reference blocks are shown in FIG. 42 and FIG. 43 to be described later. In practice, however, there are a large number of reference blocks in which the input pixel Din is a pixel in the reference block.

  The SAD calculation in the case of the second embodiment is the absolute difference between the luminance value Y of the input pixel Din and the luminance value Y of the pixel in the target block corresponding to the position of the input pixel Din in each reference block. A value is calculated, and the calculated absolute difference value is added to the SAD table according to the reference vector corresponding to each reference block.

  For example, when the input pixel Din belongs to the reference block 1061, the absolute difference value between the pixel D1 of the target block 103 and the input pixel Din corresponds to the reference vector 1071 of the SAD table 108 as shown in FIG. It is added to the SAD value of the SAD table element 1092 and written. Further, when the input pixel Din belongs to the reference block 1062, the absolute difference value between the pixel D2 of the target block 103 and the input pixel Din corresponds to the reference vector 1072 of the SAD table 108 as shown in FIG. It is added to the SAD value of the SAD table element 1092 and written.

  Accordingly, when the input pixels of all the regions within the search range are input and the processing is completed, the SAD table is completed.

  The description of FIG. 43 is a case where real-time SAD calculation processing is applied to the conventional method. In the second embodiment, in FIG. 43, each calculated absolute difference value is added and written as the SAD value of the SAD table element 1091 or 1092 to which the reference vector 1071 or 1072 of the SAD table 108 corresponds. Instead, as in the first embodiment described above, a reference reduced vector obtained by reducing the reference vectors 1071 and 1072 with the reduction ratio 1 / n is calculated, and the calculation is performed on a plurality of reference vectors in the vicinity of the reference reduced vector. From the absolute difference values thus obtained, variance addition values for variance addition are obtained, and the obtained variance addition values are added to SAD values corresponding to the plurality of neighboring reference vectors.

  The process for detecting an accurate motion vector after the SAD table (reduced SAD table) is completed is the same as that described in the first embodiment in the second embodiment. A technique using a quadric surface or a cubic curve in the horizontal direction and the vertical direction can be used.

  FIG. 44 and FIG. 45 are flowcharts of operations of the reduced SAD table generation process and the block-by-block motion vector detection process for each target block in step S32 of FIG. 30 in the camera shake vector detection unit 15 in the case of the second embodiment. Shown in

  First, the hand movement vector detection unit 15 receives pixel data Din (x, y) at an arbitrary position (x, y) in the frame (reference frame) of the input image (step S181). Next, a reference vector (vx, vy) corresponding to one of a plurality of reference blocks including the position (x, y) of the pixel is set (step S182).

Next, the pixel value Ii (x, y) of the reference block Ii of the set reference vector (vx, vy) and the corresponding pixel value Io (x-vx, y-vy) in the target block Io Is calculated (step S183). That is, the difference absolute value α is
α = | Io (x−vx, y−vy) −Ii (x, y) | (Expression 4)
Is calculated as

  Next, a reference reduction vector (vx / n, vy / n) is calculated by reducing the reference vector (vx, vy) to 1 / n with a reduction ratio of 1 / n (step S184).

  Next, a plurality of reference vectors in the vicinity of the reference reduced vector (vx / n, vy / n), in this example, four neighboring reference vectors are detected as described above (step S185). Then, the values (difference absolute values) to be distributed and added as table elements corresponding to each of the detected four neighboring reference vectors are in the positional relationship indicated by the reference reduced vector and the neighboring reference vector as described above. Based on the difference absolute value α obtained in step S183, a linear weighted variance value is obtained (step S186). Then, the obtained four linear weighted variance values are added to the SAD table element values corresponding to each of the neighborhood reference vectors (step S187).

  Next, it is determined whether or not the calculations in steps S182 to S187 have been performed for all the reference blocks including the input pixel Din (x, y) (step S188), and the input pixel Din (x, y) is included. When it is determined that there is another reference block, the process returns to step S182, another reference block (vx, vy) including the input pixel Din is set, and the processing from step S182 to step S187 is repeated.

  When it is determined in step S188 that the calculations in steps S182 to S187 have been performed for all the reference blocks including the input pixel Din (x, y), the above-described processing is performed for all the input pixels Din in the search range. It is determined whether or not the processing of the calculation step has been completed (step S191 in FIG. 45). If it is determined that the processing has not ended, the process returns to step S181, and the next input pixel Din within the search range is fetched. Repeat the process.

  If it is determined in step S191 that the processing of the above-described calculation step has been completed for all input pixels Din within the search range, the reduced SAD table is completed, and the minimum value is obtained in the completed reduced SAD table. The detected SAD value is detected (step S192).

  Next, the SAD value (minimum value) of the table element address (mx, my) having the minimum value and a plurality of neighboring SAD values, in this example, the SAD values of 15 neighboring table elements as described above. Is used to generate a quadric surface (step S193), and a minimum value vector (px, py) indicating the position of decimal precision corresponding to the minimum SAD value of the quadric surface is calculated (step S194). The minimum value vector (px, py) corresponds to the minimum table element address with decimal precision.

  Then, the motion vector (px × n, py × n) to be obtained is calculated by multiplying the calculated minimum value vector (px, py) indicating the decimal position by n (step S195).

  In this example as well, as described above, the method using the above-described cubic curve in the horizontal direction and the vertical direction may be used as a method for calculating the minimum value vector (px, py) indicating the position with decimal precision. This is the same as the example.

  In the same manner as the third example of the first embodiment described above, the second embodiment also has two or more steps while narrowing the search range and changing the reduction ratio as necessary. Of course, the motion vector detection process using the reduced SAD table may be repeated.

  The merit of the second embodiment is that the frame memory can be reduced by one frame compared to the first embodiment, and the time for storing the input image in the frame memory can be shortened. Needless to say, the effect of reducing the memory has been increasing in recent years. In particular, when handling moving images, the system delay is reduced as it is. Therefore, eliminating as much as possible the uncomfortable feeling between the actual subject and the panel display image caused by the system delay has a high appeal effect to the user.

[Third Embodiment]
In the first and second embodiments described above, an image stabilization system for moving images is assumed, and the discussion up to this point has proceeded. However, the block matching method according to the present invention is based on still image stabilization. It can be easily deployed in a correction system. The third embodiment is a case where the present invention is applied to the still image stabilization system. Note that the third embodiment is not limited to still images, and is essentially applicable to moving images. In the case of moving images, there is an upper limit on the number of frames to be added (the number of frames) due to real-time characteristics, but the application to a system that generates a moving image with a high noise reduction effect by using this method for each frame is exactly the same. It can be realized by means.

  In the third embodiment, as in the second embodiment, the input image frame is used as a reference frame, and the input image frame and an image frame obtained by delaying the input image frame in the frame memory by one frame are used. To detect motion vectors. The camera shake correction for the still image in the third embodiment is performed by superimposing a plurality of images taken continuously, for example, 3 fps images while performing camera shake correction.

  As described above, in the third embodiment, camera shake correction of a captured still image is performed while performing camera shake correction on a plurality of continuously shot images, so that accuracy close to pixel accuracy (one pixel accuracy) is achieved. Desired. That is, in the third embodiment, it is necessary to detect the rotation component between frames simultaneously with the horizontal and vertical translation components between the frames as the hand movement motion vector.

  FIG. 46 is a block diagram illustrating a configuration example of the imaging apparatus in the case of the third embodiment. In the example of FIG. 46, in the second embodiment shown in FIG. 41, a rotation / translation addition unit 19 is provided between the camera shake vector detection unit 15 and the resolution conversion unit 16, and the image memory unit 4 is also provided. In addition to the frame memory 43, another frame memory 44 is provided, and this frame memory 44 is used for detecting the rotation component of the motion vector and for superimposing the frame images. Others are the same as the structure of FIG.

  As in the case of the moving image described in the second embodiment, the camera shake vector detection unit 15 sets the input pixel data from the data conversion unit 14 as the pixel data of the reference frame and stores it in the frame memory 43. Using the data as original frame data, a reduced SAD table generation process, a block-by-block motion vector detection process, a combined SAD table generation process, and a global motion vector (camera shake vector) detection process are performed. In the third embodiment, the hand movement vector detection unit 15 detects the rotation angle of the reference frame with respect to the original frame in addition to the global motion vector (the hand movement parallel translation component).

  In the case of this example, the camera shake vector detection unit 15 always obtains a relative camera shake vector with respect to the image one frame before, so that it is relative to the first reference image (see the image frame 120 in FIG. 47). In order to calculate camera shake, the camera shake components from the first sheet to that are integrated.

  Then, the rotation / translation adder 19 rotates the image frame stored in the frame memory 43 at the same time as the cutout after one frame delay according to the detected translational component and the rotation angle of the camera shake. The image is added or averaged to the frame memory 44 image. By repeating this process, an image frame 120 of a still image having a higher S / N and a higher resolution without camera shake is generated in the frame memory 44 (see FIG. 47).

  Then, the resolution conversion unit 16 cuts out a predetermined resolution and a predetermined image size according to the control instruction from the CPU 1 from the frame image in the frame memory 44, and as described above, the resolution conversion unit 16 stores the captured image data as the recorded captured image data in the codec unit 17. At the same time, it is supplied to the NTSC encoder 18 as monitor image data.

  Also in the third embodiment, the hand movement vector detecting unit 15 uses the reduced SAD table and the combined SAD table in two or more stages while narrowing the search range and changing the reduction ratio as necessary. The motion vector detection process can be repeated. In the camera shake vector detection and camera shake correction processing for the still image according to the third embodiment, there are few real-time restrictions, the number of pixels is large, and high-precision motion vectors need to be detected. Hierarchical motion vector detection processing is very effective.

  When performing such multi-step hierarchical motion vector detection processing, the search range in the subsequent motion vector detection is set as the search range offset for the global motion vector calculated in the previous round, A narrow search range is set.

[Fourth Embodiment]
In the example of the third embodiment described above, the motion vector detection method in the second embodiment is used. However, the motion vector detection method in the first embodiment may be used.

  FIG. 48 shows a configuration example of the fourth embodiment of the image processing apparatus when the motion vector detection method according to the first embodiment is used in the still image stabilization system similar to the third embodiment. It is.

  In the fourth embodiment, in the first embodiment shown in FIG. 29, a rotation / translation addition unit 19 is provided between the camera shake vector detection unit 15 and the resolution conversion unit 16, and an image memory unit. 4, a frame memory 44 is provided in addition to the frame memory 41 and the frame memory 42 shown in FIG. 29 of the first embodiment. Then, using the motion vector and the global motion vector, the image data of the first frame, which is a reference for rotating and translating a plurality of frames and superimposing them, is shown in FIG. Also write to.

  For the second and subsequent image frames, after being stored in the frame memory 42, the image data stored in the frame memory 41 is always used to detect the camera shake vector relative to the image one frame before. Is executed by the hand movement vector detection unit 15. At this time, in order to calculate the relative camera shake with the first reference image, the camera shake vector up to that point is integrated. In addition, the camera shake vector detection unit 15 detects a rotation angle of the second and subsequent image frames relative to the first reference image frame.

  The camera shake vector detection unit 15 supplies the CPU 1 with information on a relative camera shake vector and a rotation angle for each of the second and subsequent image frames with respect to the detected first image frame.

  The second and subsequent images stored in the frame memory 42 are read from the frame memory 42 under the control of the CPU 1 so as to cancel the relative camera shake component with the calculated reference image of the first frame. And is supplied to the rotation / translation adder 19. Then, the rotation / translation addition unit 19 rotates each of the second and subsequent image frames in accordance with the relative rotation angle with respect to the first reference image frame by the control signal from the CPU 1, and the frame memory 44. Are added or averaged to the image frames read out from. The added or averaged image frame is written back to the frame memory 44.

  The image frame data in the frame memory 44 is cut out at a predetermined resolution and a predetermined image size according to the control instruction of the CPU 1 and supplied to the resolution conversion unit 16. Then, as described above, the resolution conversion unit 16 supplies the captured image data to the codec unit 17 and the monitor image data to the NTSC encoder 18.

[Fifth Embodiment]
In the third embodiment and the fourth embodiment described above, the camera shake vector and the rotation angle are always obtained by comparing the input image with the image one frame before, but actually, FIG. As shown in FIG. 6, since the subsequent frames are added with the first frame as a reference, the error is smaller when the motion vector detection is also based on the first frame. The fifth embodiment considers this point.

  FIG. 49 is a block diagram showing a configuration example of the imaging apparatus according to the fifth embodiment. The configuration example of FIG. 49 is an example applied to the case of the third embodiment.

  That is, in the example of FIG. 49, the image memory unit 4 is provided with a frame memory 45 in addition to the frame memory 43 and the frame memory 44 in the third embodiment of FIG. The image data from the data converter 14 is written in the frame memory 43 and the frame memory 45.

  In the fourth embodiment, the frame memory 45 is used for storing the first target frame (original frame and reference image frame), and the reference vector of the input image for this image is always used. The structure of the system to calculate is shown. Also in this configuration, the addition image result is stored in the frame memory 44.

  Also in this example, the image data of the first frame serving as a reference is written in the frame memory 44 as indicated by a broken line in FIG.

  The second and subsequent image frames are written to the frame memory 43 and supplied to the camera shake vector detection unit 15. The camera shake vector detection unit 15 detects a relative camera shake vector between each of the second and subsequent image frames from the data conversion unit 14 and the first image data read from the frame memory 45. And the rotation angle is detected.

  The camera shake vector detection unit 15 supplies the CPU 1 with information on a relative camera shake vector and a rotation angle for each of the second and subsequent image frames with respect to the detected first image frame.

  The second and subsequent images stored in the frame memory 43 are read from the frame memory 42 under the control of the CPU 1 so as to cancel the relative camera shake component with the calculated reference image of the first frame. And is supplied to the rotation / translation adder 19. Then, the rotation / translation addition unit 19 rotates each of the second and subsequent image frames in accordance with the relative rotation angle with respect to the first reference image frame by the control signal from the CPU 1, and the frame memory 44. Are added or averaged to the image frames read out from. The added or averaged image frame is written back to the frame memory 44.

  The image frame data in the frame memory 44 is cut out at a predetermined resolution and a predetermined image size according to the control instruction of the CPU 1 and supplied to the resolution conversion unit 16. Then, as described above, the resolution conversion unit 16 supplies the captured image data to the codec unit 17 and the monitor image data to the NTSC encoder 18.

  Also in the fourth embodiment, by applying the concept of the fifth embodiment described above, the relative camera shake vector and the rotation angle with respect to the first reference image frame are always set to the second and subsequent frames. 48, the frame memory 42 is fixed exclusively for the first image in FIG. 48, or the representative point of the first image is stored in the local memory (SRAM). What is necessary is just to perform a block matching using it and to use a SAD value.

  In the fourth and fifth embodiments described above, a system that can perform infinite addition or infinite average addition using the first frame of the input image as a reference image has been shown. However, there is plenty of memory capacity. Alternatively, if temporary evacuation to the recording / reproducing apparatus unit 5 is permitted, all the images to be added may be stored in advance and added to the tournament type or average added. good.

[Sixth Embodiment]
A higher effect can be obtained by combining the sensorless camera shake correction according to the first to fifth embodiments described above and the optical camera shake correction which is the existing technology.

  As explained at the beginning, optical camera shake correction using a gyro sensor is good at rough correction, and rotation correction is difficult, whereas sensorless camera shake correction using block matching includes rotation correction. This is because although the accuracy is high, if the search range is widened, the cost of the SAD table increases rapidly, or even if the method of this embodiment is used, processing time is required in the case of a multi-stage motion detection process.

  Therefore, by roughly correcting with optical image stabilization, narrowing the search range for motion vector detection for sensor level image stabilization, detecting the motion vector in the search range, and applying sensorless image stabilization, A low-cost, high-precision, high-speed image stabilization system can be realized.

  By the way, in recent years, as a technique for preventing not only camera shake but also moving subject shake, high sensitivity shooting is now in the market. I collect ears.

  The problem in this case is how much ISO sensitivity can be accommodated while suppressing S / N. Normally, when sensitivity is improved, noise becomes conspicuous at the same time, so digital camera companies can reduce noise using various methods and maintain a certain level of S / N. Say it as performance.

  One of the purposes of still image stabilization, which is one of the subjects of the present invention, is noise reduction, and when adding a plurality of images, a moving subject portion is detected and the addition is not performed. It is possible to realize apparent high-sensitivity noise reduction corresponding to moving subject blurring by separately searching only the part and performing tracking addition.

  When random noise is targeted, if N sheets (N is an integer of 2 or more) are added, the noise component is statistically reduced to the ratio of the square root of N. That is, in a digital camera having an ability value equivalent to ISO 3200, if 16 frames corresponding to a moving subject are added, the ISO sensitivity of the set can be obtained as ISO 12800 which is 4 times.

[Effect of the embodiment]
As described above, in the above-described first and second embodiments, a combined SAD is proposed as a new measure for determining the reliability of a motion vector, and a video stabilization algorithm using this is shown. . This makes it possible to avoid large and incorrect corrections for conventional subjects, such as blinking of the entire frame and wavefront of the water surface, as well as correction that is more accurate than before even when subjects are not good at it. Therefore, it is possible to realize an image stabilization system for moving images that achieves both robustness and high accuracy.

  In addition, the sensorless camera shake correction method using the block matching method of the third to fifth embodiments described above is cost, accuracy, processing time, and the like for the sensorless still image camera shake correction technology that has been proposed so far. Robustness is superior in both cases.

  All of the still image stabilization currently on the market is a system that uses a combination of gyro sensor and optical correction such as lens shift, but the error is large and the image quality is not satisfactory. On the other hand, the method according to the present invention realizes low-cost and high-precision camera shake correction that eliminates a sensor and a mechanism portion.

[Other variations]
In the description of the above-described embodiment, the reduction ratio with respect to the reference vector is the same in the horizontal direction and the vertical direction. However, as described above, the reduction ratio may be different in the horizontal direction and the vertical direction. It is.

  In the above-described embodiment, the SAD value is obtained for all the pixels in the reference block and the target block. For example, the SAD value is obtained by using only every k pixels (k is a natural number). Also good.

  In a motion vector detection system for real-time processing, SAD calculation is often performed in which only representative points in a target block are searched in a reference block for the purpose of reducing calculation cost and processing time.

  That is, as shown in FIG. 50, the target block 103 is divided into a plurality of pixels, for example, horizontal × vertical = n × m (n and m are integers of 1 or more), and a plurality of pixels which are the division units are divided. One of the pixels is set as a representative point TP. In the SAD calculation, for the target block 103, only a plurality of representative points TP determined in this way are used.

  On the other hand, in the reference block 106, all the pixels are subjected to SAD value calculation. For one representative point TP in the target block 103, in one reference block 106, the division point in which the representative point TP is set is used. All pixels included in a certain pixel area AR composed of n × m pixels are used.

  Between the target block 103 and one reference block 106, the pixel value of each representative point TP of the target block 102 and each area AR corresponding to each representative point TP of the reference block 106 are included. The sum of the differences from the respective pixel values of the plurality of n × m pixels is obtained, and the sum of the differences for the obtained representative points TP is summed for all the representative points TP of the target block 103. Will be. This is one element value of the SAD table.

  Then, the difference calculation using the representative point TP similar to the above is performed for all the reference blocks in the search range for the target block 103, and the SAD table is generated. However, in the case of this example, the plurality of reference blocks set in the search range are shifted by each of a plurality of pixels composed of n × m, which is the division unit, or by an integer multiple of the plurality of pixels. Is done.

  When the representative point is used for the target block as described above, the memory access for the target block when calculating the SAD value is one representative point for each of the plurality of pixels in the area AR of the reference block. One TP is sufficient, and the number of memory accesses can be greatly reduced.

  When only the representative point TP is used, only the pixel data of the representative point TP among all the pixels in the block needs to be stored as the target block data. Therefore, the target block of the original frame (target frame) is stored. The capacity of the frame memory for storing the data can be reduced.

  In addition to the frame memory, a small representative point memory (SRAM) is locally provided, and the data of the target block of the original frame (target frame) is held in the local memory, whereby the image memory 4 ( DRAM) may be reduced.

  The above description regarding the processing in the case where the representative point is used for the target block is for the method described with reference to FIGS. 51 to 53, but the second embodiment described with reference to FIGS. 41 to 45. Needless to say, this method can also be applied to the above method.

  In the method according to the second embodiment, when only the representative point TP is used for the target block, an area including the input pixel in the entire search range for each pixel (input pixel) of the input reference frame All reference blocks having AR (the positions of the pixels are not the same in the area AR) are detected, and the representative points of the target block corresponding to the area AR in each of the detected reference blocks are determined.

  Then, the pixel values of a plurality of representative points obtained as a result of the determination are respectively read from the memory storing the image data of the original frame (target frame), and the difference between the pixel value of the representative point and the input pixel is calculated. Each calculation is performed, and the calculation result is accumulated in the coordinate position of the corresponding reference block (reference vector) in the SAD table.

  In this case, since the memory access only reads out the representative point of the target block, the number of memory accesses can be greatly reduced.

  Needless to say, the processing using the representative points can also be applied to the case where the above-described reduced SAD table is used.

  In the above-described embodiment, the pixel difference value and the SAD value are calculated using only the pixel luminance value Y. However, not only the luminance value Y but also the color difference component Cb is used for motion vector detection. / Cr may be used. Further, the motion vector detection process may be performed on the RAW data before being converted into the luminance value Y and the color difference component Cb / Cr by the data conversion unit 14.

  As described above, the camera shake vector detection unit 15 is not limited to a configuration using hardware processing, and may be realized by software.

It is a figure for demonstrating the outline | summary of embodiment of the image processing method by this invention. It is a figure used in order to demonstrate the outline | summary of embodiment of the image processing method by this invention. It is a figure for demonstrating the outline | summary of embodiment of the image processing method by this invention. It is a figure used in order to demonstrate the outline | summary of embodiment of the image processing method by this invention. It is a figure which shows the flowchart used in order to demonstrate the outline | summary of embodiment of the image processing method by this invention. It is a figure for demonstrating the example of the process for detecting a motion vector for every block in embodiment of the image processing method by this invention. It is a figure for demonstrating the example of the process for detecting a motion vector for every block in embodiment of the image processing method by this invention. It is a figure for demonstrating the example of the process for detecting a motion vector for every block in embodiment of the image processing method by this invention. It is a figure for demonstrating the example of the process for detecting a motion vector for every block in embodiment of the image processing method by this invention. It is a figure for demonstrating the example of the process for detecting a motion vector for every block in embodiment of the image processing method by this invention. It is a figure for demonstrating the example of the process for detecting a motion vector for every block in embodiment of the image processing method by this invention. It is a figure for demonstrating the example of the process for detecting a motion vector for every block in embodiment of the image processing method by this invention. It is a figure for demonstrating the example of the process for detecting a motion vector for every block in embodiment of the image processing method by this invention. It is a figure for demonstrating the example of the process for detecting a motion vector for every block in embodiment of the image processing method by this invention. It is a figure for demonstrating the example of the process for detecting a motion vector for every block in embodiment of the image processing method by this invention. It is a figure for demonstrating the example of the process for detecting a motion vector for every block in embodiment of the image processing method by this invention. It is a figure for demonstrating the example of the process for detecting a motion vector for every block in embodiment of the image processing method by this invention. It is a figure for demonstrating the example of the process for detecting a motion vector for every block in embodiment of the image processing method by this invention. It is a figure for demonstrating the example of the process for detecting a motion vector for every block in embodiment of the image processing method by this invention. It is a figure for demonstrating the example of the process for detecting a motion vector for every block in embodiment of the image processing method by this invention. It is a figure for demonstrating the example of the process for detecting a motion vector for every block in embodiment of the image processing method by this invention. It is a figure for demonstrating the example of the process for detecting a motion vector for every block in embodiment of the image processing method by this invention. It is a figure for demonstrating the example of the process for detecting a motion vector for every block in embodiment of the image processing method by this invention. It is a figure for demonstrating the processing performance of the example of a process for detecting a motion vector for every block in embodiment of the image processing method by this invention. It is a figure for demonstrating the outline | summary of embodiment of the image processing method by this invention. It is a figure used in order to demonstrate the characteristic of embodiment of the image processing method by this invention compared with the conventional method. It is a figure used in order to demonstrate the characteristic of embodiment of the image processing method by this invention compared with the conventional method. It is a figure used in order to demonstrate the characteristic of embodiment of the image processing method by this invention compared with the conventional method. 1 is a block diagram illustrating a configuration example of a first embodiment of an image processing device according to the present invention. It is a figure which shows a part of flowchart for demonstrating the example of the hand movement vector detection process of 1st Embodiment of the image processing apparatus by this invention. It is a figure which shows a part of flowchart for demonstrating the example of the hand movement vector detection process of 1st Embodiment of the image processing apparatus by this invention. It is a figure which shows a part of flowchart for demonstrating the 1st example of the motion vector detection process for every block in 1st Embodiment of the image processing apparatus by this invention. It is a figure which shows a part of flowchart for demonstrating the 1st example of the motion vector detection process for every block in 1st Embodiment of the image processing apparatus by this invention. It is a figure which shows a part of flowchart for demonstrating the 2nd example of the motion vector detection process for every block in 1st Embodiment of the image processing apparatus by this invention. It is a figure which shows a part of flowchart for demonstrating the 2nd example of the motion vector detection process for every block in 1st Embodiment of the image processing apparatus by this invention. It is a figure which shows a part of flowchart for demonstrating the 3rd example of the motion vector detection process for every block in 1st Embodiment of the image processing apparatus by this invention. It is a figure which shows a part of flowchart for demonstrating the 3rd example of the motion vector detection process for every block in 1st Embodiment of the image processing apparatus by this invention. It is a figure which shows a part of flowchart for demonstrating the 3rd example of the motion vector detection process for every block in 1st Embodiment of the image processing apparatus by this invention. It is a figure which shows a part of flowchart for demonstrating the 3rd example of the motion vector detection process for every block in 1st Embodiment of the image processing apparatus by this invention. It is a figure used in order to demonstrate the 3rd example of the motion vector detection processing for every block in 1st Embodiment of the image processing apparatus by this invention. It is a block diagram which shows the structural example of 2nd Embodiment of the image processing apparatus by this invention. It is a figure for demonstrating the motion vector detection process for every block in 2nd Embodiment of the image processing apparatus by this invention. It is a figure for demonstrating the motion vector detection process for every block in 2nd Embodiment of the image processing apparatus by this invention. It is a figure which shows a part of flowchart for demonstrating the example of the motion vector detection process for every block in 2nd Embodiment of the image processing apparatus by this invention. It is a figure which shows a part of flowchart for demonstrating the example of the motion vector detection process for every block in 2nd Embodiment of the image processing apparatus by this invention. It is a block diagram which shows the structural example of 3rd Embodiment of the image processing apparatus by this invention. It is a figure for demonstrating the camera-shake correction process in 3rd Embodiment of the image processing apparatus by this invention. It is a block diagram which shows the structural example of 4th Embodiment of the image processing apparatus by this invention. It is a block diagram which shows the structural example of 5th Embodiment of the image processing apparatus by this invention. It is a figure used in order to explain other examples of the image processing method by this invention. It is a figure for demonstrating the process which detects a motion vector by block matching. It is a figure for demonstrating the process which detects a motion vector by block matching. It is a figure for demonstrating the process which detects a motion vector by block matching. It is a figure for demonstrating the process which detects a motion vector by block matching.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 101 ... Original frame, 102 ... Reference frame, 103 ... Target block, 105 ... Search range, 106 ... Reference block, 107 ... Reference vector, 15 ... Shake motion vector detection part, 41-45 ... Frame memory, TBLi ... SAD table, SUM_TBL: Total SAD table, TBLs ... Reduced SAD table, TBLo ... Conventional SAD table, RV ... Reference vector, CV ... Reference reduced vector, NV1 to NV4 ... Neighborhood reference vector, BLK_Vi ... Block-by-block motion vector, SUM_V ... Total motion vector

Claims (22)

An image processing apparatus that calculates a motion vector in units of one screen between a reference screen that is an attention screen and an original screen that is a screen before the reference screen for images that are sequentially input in units of screens. There,
In the original screen, a target block having a predetermined size composed of a plurality of pixels is set at a plurality of predetermined positions, and a reference block having the same size as the target block is set on the reference screen. A plurality of search ranges set in accordance with the position of the target block are set, and in each of the search ranges, each pixel in the reference block is set for each of the set reference blocks. A difference absolute value sum calculating means for obtaining a difference absolute value sum that is a sum of absolute values of differences between a value and a pixel value of each pixel at a corresponding position in the target block;
For each of the plurality of search ranges, the difference absolute value sum calculated by the difference absolute value sum calculation means is added to the sum of the reference block positions corresponding to each of the plurality of search ranges to add up the difference absolute value sum. And a sum difference absolute value sum table generating means for generating a sum difference absolute value sum table for storing the sum of sum difference absolute value sums for a plurality of reference block positions within one search range;
A motion vector detecting means for detecting a motion vector of the reference screen relative to the original screen from the summed absolute difference sum table generated by the summed absolute difference sum table generating means;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
A difference absolute value sum table generating means for generating a plurality of difference absolute value sum tables for storing the difference absolute value sums for a plurality of reference block positions in the search range in correspondence with the plurality of search ranges;
The minimum value of the difference absolute value sum in each of the plurality of difference absolute value sum tables generated by the difference absolute value sum table generation means, and the sum difference absolute value calculated by the sum difference absolute value sum calculation means Means for detecting the difference absolute value sum table for the target block considered to be a moving subject part from the minimum value of the sum;
The motion vector detection means recalculates the sum of absolute differences, excluding the sum of absolute differences of the difference absolute value sum table for the target block considered to be the moving subject part, and adds the sum of differences An image processing apparatus, wherein an absolute value sum table is recalculated, and the motion vector is detected from the recalculated summed difference absolute value sum table. An image processing apparatus that calculates a motion vector in units of one screen between a reference screen that is an attention screen and an original screen that is a screen before the reference screen for images that are sequentially input in units of screens. There,
In the original screen, a target block having a predetermined size composed of a plurality of pixels is set at a plurality of predetermined positions, and a reference block having the same size as the target block is set on the reference screen. A plurality of search ranges set in accordance with the position of the target block are set, and in each of the search ranges, each pixel in the reference block is set for each of the set reference blocks. A difference absolute value sum calculating means for obtaining a difference absolute value sum that is a sum of absolute values of differences between a value and a pixel value of each pixel at a corresponding position in the target block;
A reference reduction including a reference vector including a directional component is used as a positional deviation amount of each reference block on the reference screen relative to the target block on the screen, and the reference vector is reduced at a predetermined reduction ratio. A reference reduced vector obtaining means for obtaining a vector;
For each target block in the plurality of search ranges, the reference vector having a size corresponding to the reference reduction vector is used as the positional shift amount, and a plurality of the plurality of the reductions corresponding to the predetermined reduction ratio are performed. A reduced difference absolute value sum table generating means for generating a reduced difference absolute value sum table for storing the difference absolute value sum for each of the reference blocks;
For the plurality of reduced difference absolute value sum tables generated by the reduced difference absolute value sum table generating means, the difference absolute value sums of the reference block positions corresponding to each of the plurality of search ranges are added together to add up the difference. A summed reduced difference absolute value sum table generating means for obtaining an absolute value sum and generating a summed reduced difference absolute value sum table;
A motion vector calculating unit that calculates a motion vector of the reference screen with respect to the original screen from the combined reduced difference absolute value sum table generated by the combined reduced difference absolute value sum table generating unit;
With
The reduced difference absolute value sum table generating means includes:
Neighborhood reference vector detection means for detecting a plurality of the reference vectors that are neighborhood values of the reference reduction vector acquired by the reference reduction vector acquisition means;
Corresponding to each of the plurality of neighboring reference vectors detected by the neighboring reference vector detecting means from the sum of the absolute values of the differences for each of the reference blocks calculated by the difference absolute value sum calculating means. A variance difference absolute value sum calculating means for calculating each of the difference absolute value sums;
The difference absolute value sum corresponding to each of the plurality of neighboring reference vectors calculated by the variance difference absolute value sum calculating means is used as the difference absolute value corresponding to each of the plurality of neighboring reference vectors so far. Distributed addition means for adding to the sum of values;
An image processing apparatus comprising: An image processing apparatus that calculates a motion vector in units of one screen between a reference screen that is an attention screen and an original screen that is a screen before the reference screen for images that are sequentially input in units of screens. There,
In the original screen, a target block having a predetermined size composed of a plurality of pixels is set at a plurality of predetermined positions, and a reference block having the same size as the target block is set on the reference screen. A plurality of search ranges set in accordance with the position of the target block are set, and in each of the search ranges, each pixel in the reference block is set for each of the set reference blocks. Difference calculating means for obtaining a difference between the value and the pixel value of each pixel at the corresponding position in the target block;
A reference reduction including a reference vector including a directional component is used as a positional deviation amount of each reference block on the reference screen relative to the target block on the screen, and the reference vector is reduced at a predetermined reduction ratio. A reference reduced vector obtaining means for obtaining a vector;
For each target block in the plurality of search ranges, the reference vector having a size corresponding to the reference reduction vector is used as the positional shift amount, and a plurality of the plurality of the reductions corresponding to the predetermined reduction ratio are performed. A reduced difference absolute value sum table generating means for generating a reduced difference absolute value sum table for storing the reduced difference absolute value sum for each of the reference blocks;
For the plurality of reduced difference absolute value sum tables generated by the reduced difference absolute value sum table generating means, the difference absolute value sums of the reference block positions corresponding to each of the plurality of search ranges are added together to add up the difference. A summed reduced difference absolute value sum table generating means for obtaining an absolute value sum and generating a summed reduced difference absolute value sum table;
A motion vector calculating unit that calculates a motion vector of the reference screen with respect to the original screen from the combined reduced difference absolute value sum table generated by the combined reduced difference absolute value sum table generating unit;
With
The reduced difference absolute value sum table generating means includes:
Neighborhood reference vector detection means for detecting a plurality of the reference vectors that are neighborhood values of the reference reduction vector acquired by the reference reduction vector acquisition means;
Each difference value corresponding to each of the plurality of neighboring reference vectors detected by the neighboring reference vector detecting means is calculated from the difference value for each of the reference blocks calculated by the difference calculating means. Distributed addition value calculating means;
The difference value corresponding to each of the plurality of neighboring reference vectors calculated by the variance addition value calculating means is added to the sum of absolute differences corresponding to the plurality of neighboring reference vectors so far. Adding means;
An image processing apparatus comprising: The image processing apparatus according to claim 3 or 4,
The minimum value of the difference absolute value sum of the reduced difference absolute value sum table for each target block of the plurality of search ranges, and the combined reduced difference generated by the combined reduced difference absolute value sum table generating means Means for detecting the reduced difference absolute value sum table for the target block considered to be a moving subject part from the minimum value of the sum of absolute differences sum of the absolute value sum table;
The motion vector detection means recalculates the sum of the sum of absolute differences, excluding the sum of the sum of absolute differences of the reduced difference absolute value sum table for the target block considered to be the moving subject part, and sums up the sum An image processing apparatus characterized by recalculating a reduced difference absolute value sum table and detecting the motion vector from the recalculated summed reduced difference absolute value sum table. The image processing apparatus according to claim 3 or 4,
Based on the motion vector calculated by the motion vector calculation means, the search range with respect to the reference screen is made narrower and the predetermined reduction rate is made smaller, and again, the claim 3 or the claim. An image processing apparatus comprising: means for repeating the motion vector calculation process using each of the four means. The image processing apparatus according to claim 3.
In the variance difference absolute value sum calculation means, the sum of the absolute values of the differences corresponding to the plurality of neighboring reference vectors according to the distance between each reference reduced vector and the plurality of neighboring reference vectors. An image processing apparatus characterized by calculating each of the above. The image processing apparatus according to claim 4.
The variance addition value calculating means calculates each of the difference values corresponding to the plurality of neighboring reference vectors according to a distance between each of the reference reduced vectors and the plurality of neighboring reference vectors. An image processing method characterized by the above. The image processing apparatus according to claim 3 or 4,
The image processing apparatus characterized in that the predetermined reduction ratio can be set independently of a horizontal reduction ratio and a vertical reduction ratio of the imaging screen. The image processing apparatus according to claim 1 or 2,
The motion vector calculation means includes
Means for generating an approximate higher-order curved surface from the minimum value of the total sum of absolute differences in the total sum of absolute differences sum table and the sum of the sum of absolute differences near the minimum value;
Minimum value position detecting means for detecting a minimum value position of the approximate higher-order curved surface;
Means for calculating a vector corresponding to a positional deviation from a position corresponding to the target block corresponding to the minimum value position detected in the minimum value position detecting step, and calculating the motion vector from the calculated vector;
An image processing apparatus comprising: In the image processing apparatus according to any one of claims 3 to 5,
The motion vector calculation means includes
Means for generating an approximate higher-order curved surface from the minimum value of the total sum of absolute differences in the total reduced difference absolute value sum table and the sum of the sum of absolute differences near the minimum value;
Minimum value position detecting means for detecting a minimum value position of the approximate higher-order curved surface;
Means for calculating a vector corresponding to a positional deviation from a position corresponding to the target block corresponding to the minimum value position detected in the minimum value position detecting step, and calculating the motion vector from the calculated vector;
An image processing apparatus comprising: The image processing apparatus according to claim 10 or 11,
The approximate high-order curved surface is a quadratic curved surface obtained by a least square method using the minimum value and the sum of absolute differences in the vicinity of the minimum value. The image processing apparatus according to claim 12.
The total sum of absolute differences near the minimum value is an eight total sum of absolute differences around the minimum value centered on the minimum value. The image processing apparatus according to claim 12.
The total sum of absolute differences near the minimum value is 15 total sums of absolute differences around the minimum value. An image processing method for calculating a motion vector in units of one screen between a reference screen that is a screen of interest and an original screen that is a screen before the reference screen for images that are sequentially input in screen units. There,
In the original screen, a target block having a predetermined size composed of a plurality of pixels is set at a plurality of predetermined positions, and a reference block having the same size as the target block is set on the reference screen. A plurality of search ranges set in accordance with the position of the target block are set, and in each of the search ranges, each pixel in the reference block is set for each of the set reference blocks. A difference absolute value sum calculation step for obtaining a difference absolute value sum that is a sum of absolute values of differences between a value and a pixel value of each pixel at a corresponding position in the target block;
For each of the plurality of search ranges, the difference absolute value sum calculated in the difference absolute value sum calculation step is summed up at the reference block position corresponding to each of the plurality of search ranges, and is summed difference absolute value sum And a sum difference absolute value sum table generation step for generating a sum difference absolute value sum table for storing the sum of sum difference absolute value sums for a plurality of reference block positions within one search range;
A motion vector detecting step of detecting a motion vector of the reference screen relative to the original screen from the summed absolute difference sum table generated by the summed absolute difference sum table generating means;
An image processing method comprising: The image processing method according to claim 15, wherein
A difference absolute value sum table generating step of generating a plurality of difference absolute value sum tables for storing the difference absolute value sums for a plurality of reference block positions in the search range corresponding to the plurality of search ranges;
The minimum value of the difference absolute value sum in each of the plurality of difference absolute value sum tables generated in the difference absolute value sum table generation step, and the sum difference absolute value calculated in the sum difference absolute value sum calculation step Providing a step of detecting the difference absolute value sum table for the target block considered to be a moving subject part from the minimum value of the sum;
In the motion vector detection step, the difference absolute value sum of the difference absolute value sum table for the target block that is considered to be the moving subject part is excluded, and the sum difference absolute value sum is recalculated, and a sum difference is obtained. An image processing method, wherein an absolute value sum table is recalculated, and the motion vector is detected from the recalculated summed difference absolute value sum table. An image processing method for calculating a motion vector in units of one screen between a reference screen that is a screen of interest and an original screen that is a screen before the reference screen for images that are sequentially input in screen units. There,
In the original screen, a target block having a predetermined size composed of a plurality of pixels is set at a plurality of predetermined positions, and a reference block having the same size as the target block is set on the reference screen. A plurality of search ranges set in accordance with the position of the target block are set, and in each of the search ranges, each pixel in the reference block is set for each of the set reference blocks. A difference absolute value sum calculation step for obtaining a difference absolute value sum that is a sum of absolute values of differences between a value and a pixel value of each pixel at a corresponding position in the target block;
A reference reduction including a reference vector including a directional component is used as a positional deviation amount of each reference block on the reference screen relative to the target block on the screen, and the reference vector is reduced at a predetermined reduction ratio. A reference reduced vector obtaining step for obtaining a vector;
For each target block in the plurality of search ranges, the reference vector having a size corresponding to the reference reduction vector is used as the positional shift amount, and a plurality of the plurality of the reductions corresponding to the predetermined reduction ratio are performed. A reduced difference absolute value sum table generating step for generating a reduced difference absolute value sum table for storing the difference absolute value sum for each of the reference blocks;
For the plurality of reduced difference absolute value sum tables generated in the reduced difference absolute value sum table generating step, the difference absolute value sums of the reference block positions corresponding to each of the plurality of search ranges are added together to add up the difference. Obtaining an absolute value sum and generating a combined reduced difference absolute value sum table, and a combined reduced difference absolute value sum table generating step;
A motion vector calculating step of calculating a motion vector of the reference screen relative to the original screen from the combined reduced difference absolute value sum table generated in the combined reduced difference absolute value sum table generating step;
With
The reduced difference absolute value sum table generating step includes:
A neighborhood reference vector detection step of detecting a plurality of the reference vectors that are neighborhood values of the reference reduction vector acquired in the reference reduction vector acquisition step;
Corresponding to each of the plurality of neighboring reference vectors detected in the neighborhood reference vector detection step from the sum of absolute values of the differences for each of the reference blocks calculated in the difference absolute value sum calculation step. A variance difference absolute value sum calculation step of calculating each of the difference absolute value sums;
The difference absolute value sum corresponding to each of the plurality of neighboring reference vectors calculated in the variance difference absolute value sum calculating step is used as the difference absolute value corresponding to each of the plurality of neighboring reference vectors so far. A variance addition step of adding to the sum of values;
An image processing method comprising: An image processing method for calculating a motion vector in units of one screen between a reference screen that is a screen of interest and an original screen that is a screen before the reference screen for images that are sequentially input in screen units. There,
In the original screen, a target block having a predetermined size composed of a plurality of pixels is set at a plurality of predetermined positions, and a reference block having the same size as the target block is set on the reference screen. A plurality of search ranges set in accordance with the position of the target block are set, and in each of the search ranges, each pixel in the reference block is set for each of the set reference blocks. A difference calculating step for obtaining a difference between the value and a pixel value of each pixel at a corresponding position in the target block;
A reference reduction including a reference vector including a directional component is used as a positional deviation amount of each reference block on the reference screen relative to the target block on the screen, and the reference vector is reduced at a predetermined reduction ratio. A reference reduced vector obtaining step for obtaining a vector;
For each target block in the plurality of search ranges, the reference vector having a size corresponding to the reference reduction vector is used as the positional shift amount, and a plurality of the plurality of the reductions corresponding to the predetermined reduction ratio are performed. A reduced difference absolute value sum table generating step for generating a reduced difference absolute value sum table for storing the difference absolute value sum for each of the reference blocks;
For the plurality of reduced difference absolute value sum tables generated in the reduced difference absolute value sum table generating step, the difference absolute value sums of the reference block positions corresponding to each of the plurality of search ranges are added together to add up the difference. Obtaining an absolute value sum and generating a combined reduced difference absolute value sum table, and a combined reduced difference absolute value sum table generating step;
A motion vector calculating step of calculating a motion vector of the reference screen relative to the original screen from the combined reduced difference absolute value sum table generated in the combined reduced difference absolute value sum table generating step;
With
The reduced difference absolute value sum table generating step includes:
A neighborhood reference vector detection step of detecting a plurality of the reference vectors that are neighborhood values of the reference reduced vector acquired by the reference reduced vector acquisition means;
Each difference value corresponding to each of the plurality of neighboring reference vectors detected by the neighboring reference vector detecting means is calculated from the difference value for each of the reference blocks calculated by the difference calculating means. Dispersion added value calculation step;
The difference value corresponding to each of the plurality of neighboring reference vectors calculated in the variance addition value calculating step is added to the sum of absolute differences corresponding to the plurality of neighboring reference vectors so far. Adding step;
An image processing method comprising: The image processing apparatus according to claim 17 or 18,
The minimum value of the difference absolute value sum of the reduced difference absolute value sum table for each target block of the plurality of search ranges, and the combined reduced difference generated by the combined reduced difference absolute value sum table generating means Providing a step of detecting the reduced difference absolute value sum table for the target block that is considered to be a moving subject part from the minimum value of the sum of absolute differences of the absolute value sum table;
In the motion vector detection step, excluding the difference absolute value sum of the reduced difference absolute value sum table for the target block that is regarded as the moving subject portion, the sum of absolute differences is recalculated and summed. An image processing method, wherein a reduced difference absolute value sum table is recalculated, and the motion vector is detected from the recalculated summed reduced difference absolute value sum table. The image processing method according to claim 15 or 16,
The motion vector calculation step includes:
Generating an approximate higher-order curved surface from the minimum value of the total sum of absolute differences in the total sum of absolute differences and the sum of absolute differences near the minimum value;
A minimum value position detecting step for detecting a minimum value position of the approximate higher-order curved surface;
Calculating a vector corresponding to a positional deviation from a position corresponding to the target block corresponding to the minimum value position detected in the minimum value position detecting step, and calculating the motion vector from the calculated vector;
An image processing method comprising: In the image processing method in any one of Claims 17-19,
The motion vector calculation step includes:
Generating an approximate higher-order curved surface from the minimum value of the total sum of absolute differences in the total reduced difference absolute value sum table and the sum of the sum of absolute differences near the minimum value;
A minimum value position detecting step for detecting a minimum value position of the approximate higher-order curved surface;
Calculating a vector corresponding to a positional deviation from a position corresponding to the target block corresponding to the minimum value position detected in the minimum value position detecting step, and calculating the motion vector from the calculated vector;
An image processing method comprising: The image processing method according to claim 20 or 21,
The approximate high-order curved surface is a quadratic curved surface obtained by a least square method using the minimum value and the sum of absolute sums of differences near the minimum value.

Images