JP4052348B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program that reduce the effects of camera shake during imaging and camera shake caused by movement of the imaging system itself. The present invention can be applied to, for example, an imaging device, a digital still camera, a digital movie camera, and the like mounted on an information terminal device such as a mobile phone device or a PDA (Personal Digital Assistant).

  2. Description of the Related Art Conventionally, when a subject is photographed with a silver film camera or a digital still camera by hand, the camera shakes due to camera shake or the like, and a phenomenon in which a captured image is deteriorated by flowing in the camera shake direction is known.

  In order to correct such image degradation due to blurring, a method for reducing the blur by driving the optical system to perform imaging and a method for reducing the blur by performing image processing using the calculated blur amount is proposed. Has been.

  For example, Patent Document 1 describes a camera shake correction device that corrects camera shake by image processing. The camera shake correction apparatus described in Patent Literature 1 first obtains a motion vector for each block constituting image data. Next, if a predetermined number or more of the motion vectors in each block have the same direction and size, it is determined that the entire screen has moved, and this motion vector is used as the motion vector for the entire screen. Then, the cutout position of the image data is changed using the motion vector of the entire screen, and the image is output as the shake correction image data.

JP-A-9-261530 (paragraphs 0034-0036, FIG. 1)

  However, the conventional camera shake correction apparatus does not consider the motion vector detection accuracy of each block. Therefore, when obtaining the motion vector of the entire screen, a motion vector with low reliability may be used, and as a result, the motion vector of the entire screen may not be obtained correctly.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus and an image processing method capable of performing shake correction processing with high accuracy.

The image processing apparatus according to the present invention provides, for each area constituting the input image data, image data of a reference area in the previous image data temporally before the input image data and an image of the area of the input image data. A first motion vector output unit that outputs a motion vector obtained based on the data and a reliability index indicating the reliability of each motion vector; and the reliability index of each region as a motion vector of the corresponding region. performs weighting by multiplying a second motion vector output unit that outputs the average of the weighted motion vector of each of the areas as the motion vector of the input image data, based on the motion vector of the input image data, wherein An image data processing unit that performs image processing on the input image data.

  According to the present invention, it is possible to perform a shake correction process with high accuracy.

It is a figure which shows the structure of the image processing apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the state from which the motion vector of each block was detected. It is a figure which shows SAD when a motion vector is detected. It is a figure which shows the motion vector of each block in Embodiment 1 of this invention, the minimum value of SAD, and a reliability parameter | index. It is a figure explaining the image processing in Embodiment 1 of this invention.

Explanation of symbols

    1 image data processing unit, 2 frame memory, 3 first motion vector output unit, 4 second motion vector output unit, 5 blur determination unit, 8 image data, 9 blocks, 10 motion vector of each block, 11 upper left Block, 12 bottom right block, 100 image processing apparatus, 101 image sensor, 102 display element.

Embodiment 1 FIG.
Hereinafter, the present invention will be described in detail with reference to the drawings showing embodiments thereof. The image processing apparatus 100 according to the first embodiment detects a motion vector and a reliability index indicating the reliability of the motion vector for each region (hereinafter referred to as “block”) constituting the image data, Based on the detected motion vector and the reliability index, a motion vector of the entire image data is obtained.

<Configuration of image processing apparatus>
FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus 100 according to the first embodiment. In FIG. 1, solid arrows indicate image data, and broken arrows indicate data other than image data.

  First, image data output from the image sensor 101 such as a video camera (hereinafter referred to as “input image data”) is input to the image data processing unit 1, the frame memory 2, and the first motion vector output unit 3. The The image data processing unit 1 performs image processing on the input image data and outputs it to the display element 102 or the like as blur correction image data. Although FIG. 1 shows an example in which the shake correction image data is output to the display element, it may be recorded on a recording medium such as a hard disk or a DVD, or may be transmitted to another terminal by communication means.

  The frame memory 2 stores input image data output from the image sensor 101. The first motion vector output unit 3 includes input image data from the image sensor 101 and image data stored in the frame memory 2 that is temporally prior to the input image data (hereinafter, “previous image data”). Is entered).

  The first motion vector output unit 3 detects a motion vector for each block obtained by dividing the image data, and outputs the detected motion vector of each block to the second motion vector output unit 4. The first motion vector output unit 3 also detects a reliability index indicating the reliability of each motion vector, and outputs the detected reliability index to the second motion vector output unit 4.

  The second motion vector output unit 4 detects a motion vector indicating the motion of the entire input image data from the motion vector and reliability index of each block output from the first motion vector output unit 3, and detects the detected motion The vector is output to the blur determination unit 5.

  The shake determination unit 5 analyzes the motion vector output from the second motion vector output unit 4 and determines whether the motion of the input image data indicated by the motion vector is due to a shake such as a camera shake. When it is determined that the input motion vector is due to shaking, the camera shake determination unit 5 outputs the reference position change information to the image data processing unit 1.

  Based on the reference position change information output from the shake determination hand unit 5, the image data processing unit 1 receives the reference position of the image data output to the display element 102 as shake correction image data, and is input from the image sensor 101. Change from the reference position of the image data.

<First motion vector output unit>
Next, processing of each part constituting the image processing apparatus 100 will be described in detail. First, the motion vector and reliability index detection processing in the first motion vector output unit 3 will be described with reference to FIGS. 2, 3, and 4. FIG. 2 is a diagram showing motion vectors in each block of image data divided into a predetermined number (12 in the figure). The first motion vector output unit 3 detects the motion vector 10 for each block of the input image data. In FIG. 2, for example, the size of the image data 8 is 48 pixels × 64 pixels, and the size of the block 9 is 16 pixels × 16 pixels.

  Here, the detection processing of the motion vector 10 of each block in the first motion vector output unit 3 will be described. First, the image data of each block constituting the previous image data is acquired from the previous image data stored in the frame memory 2. Here, the block in the previous image data is referred to as a “reference block”. Further, image data of a block at the same position as the reference block is acquired from the input image data. This is called an “input block”. Then, the difference absolute value summation (SAD) of the image data is calculated while moving the position of the reference block with respect to the input block vertically and horizontally. An amount represented by (x, y) of the position of the reference block with respect to the input block, SAD (x, y), which is SAD when moved, is calculated as follows.

  Note that the amount of movement of the reference block may be one pixel unit or a plurality of pixel units.

  FIG. 3 shows SAD. FIG. 3 shows that SAD has a value of sadmin, which is the minimum value, at (xmin, ymin). Here, a small SAD value indicates that the image data of the reference block and the image data of the input block are very close. Further, the fact that SAD takes the minimum value at (xmin, ymin) means that the image data when the reference block is shifted by (xmin, ymin) and the image data of the input block are most approximated. Show. Therefore, in the case shown in FIG. 3, the first motion vector output unit 3 outputs (xmin, ymin) as the motion vector of the current input block.

  The first motion vector output unit 3 performs the process of Equation 1 in all the blocks, detects the motion vectors of all the blocks, and outputs them to the second motion vector output unit 4.

  Next, the motion vector reliability index detection processing of each block in the first motion vector output unit 3 will be described.

  FIG. 4A shows the state of motion vector detection in the first embodiment. The image data 8, the block 9, and the motion vector 10 for each block are the same as those described above with reference to FIG.

  Here, the number shown in each block is a value indicated as “sadmin” in FIG. 3 and is a “minimum value of SAD” obtained when a motion vector is obtained. For example, in the upper left block 11 of FIG. 4A, the value “10” is shown as the “minimum value of SAD”, and in the lower right block 12, “50” is shown as the “minimum value of SAD”. And the value is shown.

  This indicates that the value of the image data of the block 11, which is an input block, is close to the value of the image data of the reference block. That is, it can be considered that the image contents of the block 11 and the reference block are approximate.

  On the other hand, it shows that the value of the image data of the block 12 which is the input block and the value of the image data of the reference block are not relatively approximate. That is, it can be considered that the image contents of the block 12 and the reference block are not approximated as compared with the case of the block 11.

  That is, the motion vector 10 of the block 11 is more reliable than the motion vector 10 of the block 12. Here, for example, a value obtained by subtracting “minimum value of SAD” from 100 is used as a reliability index indicating the reliability of the motion vector, and the reliability index of the motion vector 10 of each block shown in FIG. As shown in FIG.

  As shown in FIG. 4B, the motion vector reliability index of the block 11 is “90”, while the motion vector reliability index of the block 12 is “50”. As described above, the first motion vector output unit 3 detects the motion vector 10 and the reliability index of each block constituting the image data, and outputs the detected motion vector 10 to the second motion vector output unit 4.

<Second motion vector output unit>
Next, a motion vector detection process in the first motion vector output unit 4 will be described. The second motion vector output unit 4 detects the motion vector of the entire image data based on the motion vector for each block and the reliability index output from the first motion vector output unit 3.

  For example, the second motion vector output unit 4 sets the motion vector 10 having the highest reliability index as the motion vector of the entire image data. In the case shown in FIG. 4B, since the reliability index of the motion vector of the upper left block and the lower left block is the highest value, and the direction and the size are the same, these motion vectors 10 are imaged. The motion vector of the entire data 8 is output to the blur determination unit 5.

<Blur determination unit>
The shake determination unit 5 analyzes the motion vector output from the second motion vector output unit 4 and determines whether or not the motion vector is caused by a shake such as a camera shake. For example, if the motion vector output from the second motion vector output unit 4 is directed in the same direction for a certain period of time, it is determined that this motion has occurred because the user intentionally moved the camera instead of shaking. May be. Further, when the motion vector output from the second motion vector output unit 4 is observed for a certain period of time, and the frequency of the motion of the screen is slower than a certain value, this motion is not a camera shake but the user It may be determined that the error occurred because the camera was moved intentionally.

  When it is determined that the input motion vector is due to camera shake, the camera shake determination unit 5 outputs the reference position change information to the image data processing unit 1. When it is determined that the input motion vector is due to something other than camera shake, the camera shake correction determination unit 5 outputs information that the reference position is not changed to the image data processing unit 1. Note that the information that the reference position is not changed may not be output.

<Image data processing unit>
FIG. 6 is a diagram for explaining image processing in the image data processing unit 1. The image data processing unit 1 changes the reference position of the input image data 13 output from the image sensor 101 in the direction of the motion vector 15 based on the reference position change information. As a result, the image data output to the display element 102 changes from the image data 16 before image processing to the shake-corrected image data 14 after image processing.

  That is, the image data processing unit 1 uses the motion vector of the entire image data from the image data 16 centered on the reference position of the input image data 13 output from the image sensor 101 as the blur-corrected image data output to the display element 102. The shake correction image data 14 moved in the direction 15 is output.

  Thus, according to the image processing apparatus 100 according to the first embodiment, when detecting the motion vector of the entire image data from the motion vector of each block, the reliability index of each motion vector is referred to. The motion vector of the entire image data can be detected well. Therefore, the blur correction process can be performed with high accuracy.

  In the above description, the method of using the “minimum value of SAD” as the motion vector reliability index has been described. However, the present invention is not limited to this, and any method may be used as long as it indicates the reliability of the motion vector. .

  For example, since the motion vector of the block 11 in FIG. 4A shows the same direction as the motion vectors of the surrounding blocks, it is considered that the reliability is high. On the other hand, since the motion vector of the block 12 is completely different from the motion vectors of the surrounding blocks, it seems that the reliability is low. Thus, the closeness with the motion vectors of the surrounding blocks may be output as a reliability index.

  In the above description, the method for obtaining the motion vector of the entire image data from the motion vector having the highest reliability index has been described. However, the present invention is not limited to this. For example, the motion vectors may be arranged in the descending order of the reliability index, and a predetermined number of motion vectors in the higher order may be used. A motion vector having a reliability index of a predetermined value or more may be used.

  Alternatively, a method may be used in which a motion vector is weighted using a motion vector reliability index, and then the weighted motion vector is averaged. For example, only the block 11 and the block 12 in FIG. It is assumed that the motion vector of the block 11 indicates the direction (2, 3), and the motion vector of the block 12 indicates the direction (−1, −2). In this case, since the reliability index of the motion vector of the block 11 is “90” and the reliability index of the motion vector of the block 12 is “50”, (90 × (2, 3) + 50 × (−1, −2). )) / (90 + 50) = (130, 170) / 140 = (0.929, 1.214), and the motion vector of the entire image data is about (1, 1).

  As another method, a motion vector may be weighted by setting the motion vector reliability index as “motion vector appearance frequency”. For example, in FIG. 4B, it is assumed that the motion vector appearance frequency of the block 11 is “90” and the motion vector appearance frequency of the block 12 is “50”. The appearance frequency (reliability index) of other blocks is also confirmed, and the sum of the appearance frequencies is taken for each same vector. The vector having the highest sum of appearance frequencies is set as the motion vector of the entire image data.

  Furthermore, only blocks whose reliability index is larger than a certain value are extracted, and only for these, “weighted motion vector averaging” or “vector with the highest sum of motion vector appearance frequencies” May be performed.

  Further, although the image pickup device 101 and the display device 102 are included in the image processing apparatus 100 as a configuration, they may be configured outside the image processing device. Further, the frame memory 2 may be configured outside the image processing apparatus.

  The image data used for the calculation of SAD may be pixel data such as a luminance signal, a color difference signal, and an RGB signal for each pixel, or a combination thereof.

  The pixel data of all the pixels in the block is not limited to the pixel data, and pixel data of one or more representative pixels may be used.

  Further, the motion vector output from the second motion vector output unit 4 may be directly output to the image data processing unit 1 without going through the blur determination unit 5.

Furthermore, the image data stored in the frame memory and the input image data output from the image sensor may be in units of frames or VOPs. Also, a frame memory capable of storing a plurality of frames or VOPs is prepared, and not only the image data whose “reference block” is temporally before the input image data but also the image data temporally after the input image data. It may be in. Further, the motion vector may be obtained by referring to both image data.

Claims (12)

For each region constituting the input image data, the image data of the reference region in the previous image data temporally before the input image data, and the motion vector obtained based on the image data of the region of the input image data, A first motion vector output unit that outputs a reliability index indicating the reliability of each motion vector;
It performs weighting by multiplying the confidence index of each of the areas on a motion vector of the corresponding region, a second motion vector output for outputting the average of the weighted motion vector of each of the areas as the motion vector of the input image data And
Based on the motion vector of the input image data, the image processing apparatus you characterized in that said input image data and an image data processing unit for image processing. For each region constituting the input image data, the image data of the reference region in the previous image data temporally before the input image data, and the motion vector obtained based on the image data of the region of the input image data, A first motion vector output unit that outputs a reliability index indicating the reliability of each motion vector;
Using the reliability index as the appearance frequency of the motion vector, the appearance frequency is integrated for each motion vector in all the areas constituting the input image data, and the motion vector having the highest appearance frequency is used as the motion vector of the input image data. A second motion vector output unit for outputting;
An image data processing unit that performs image processing on the input image data based on a motion vector of the input image data;
An image processing apparatus comprising: The second motion vector output unit performs weighting by extracting only a region having a reliability index greater than a certain value and multiplying the reliability index for each region by the motion vector of the corresponding region. The image processing apparatus according to claim 1 , wherein an average of weighted motion vectors for each region is output as a motion vector of the input image data . The second motion vector output unit extracts only a region having a reliability index greater than a certain value, and uses the reliability index as the frequency of appearance of the motion vector to configure all of the input image data The image processing apparatus according to claim 2 , wherein the appearance frequency is integrated for each motion vector in the region, and the motion vector having the highest appearance frequency is output as the motion vector of the input image data . The reliability index is
According to claim 1, wherein 4 of said pixel data of the pixels constituting the reference area is the sum of the absolute difference between the pixel data of the pixels constituting the region corresponding to the motion vector Image processing apparatus. A blur determination unit that determines the occurrence of blur in the input image data based on a motion vector of the input image data;
When the blur determination unit determines that blur has occurred in the input image data, the blur determination unit outputs reference position change information based on a motion vector of the input image data,
The image data processing unit performs image processing on the input image data based on the reference position change information.
The image processing apparatus according to claim 1, wherein: For each region constituting the input image data, the image data of the reference region in the previous image data temporally before the input image data, and the motion vector obtained based on the image data of the region of the input image data, A first motion vector output step for outputting a reliability index indicating reliability for each motion vector;
A second motion vector output that performs weighting by multiplying a reliability index for each region by a motion vector of the corresponding region, and outputs an average of the weighted motion vectors for each region as a motion vector of the input image data Steps,
Based on the motion vector of the input image data, and images data processing step of the image processing the input image data
An image processing method comprising: For each region constituting the input image data, the image data of the reference region in the previous image data temporally before the input image data, and the motion vector obtained based on the image data of the region of the input image data, A first motion vector output step for outputting a reliability index indicating reliability for each motion vector;
Using the reliability index as the appearance frequency of the motion vector, the appearance frequency is integrated for each motion vector in all the regions constituting the input image data, and the motion vector having the highest appearance frequency is used as the motion vector of the input image data. A second motion vector output step to output;
An image data processing step for performing image processing on the input image data based on a motion vector of the input image data;
An image processing method comprising: The second motion vector output step performs weighting by extracting only a region having a reliability index larger than a certain value and multiplying the reliability index for each region by the motion vector of the corresponding region. The image processing method according to claim 7 , wherein an average of weighted motion vectors for each region is output as a motion vector of the input image data . In the second motion vector output step, all regions constituting the input image data are extracted by extracting only a region having a reliability index greater than a certain value and using the reliability index as the appearance frequency of the motion vector. The image processing method according to claim 8 , wherein the appearance frequency is integrated for each motion vector in the region and the motion vector having the highest appearance frequency is output as the motion vector of the input image data . The reliability index is
The sum of absolute value differences between pixel data of pixels constituting the reference area and pixel data of pixels constituting the area corresponding to the motion vector
The image processing method according to claim 7, wherein: A blur determination step of determining occurrence of blur in the input image data based on a motion vector of the input image data;
In the blur determination step, when it is determined that blur has occurred in the input image data, reference position change information based on a motion vector of the input image data is output,
The image data processing step performs image processing on the input image data based on the reference position change information.
The image processing method according to claim 7, wherein:

Images