WO2007026452A1

WO2007026452A1 - Image processing apparatus, image processing method, and image processing program

Info

Publication number: WO2007026452A1
Application number: PCT/JP2006/310771
Authority: WO
Inventors: Shuji Sotoda; Yoshiko Hatano
Original assignee: Mitsubishi Electric Corporation
Priority date: 2005-08-29
Filing date: 2006-05-30
Publication date: 2007-03-08
Also published as: JPWO2007026452A1; JP4052348B2

Abstract

[PROBLEMS] To obtain a highly accurate shake-corrected image. [MEANS FOR SOLVING PROBLEMS] An image processing apparatus comprises a first motion vector output unit, a second motion vector output unit, and an image data processing unit. The first motion vector output unit outputs motion vectors and reliability indicators indicating reliabilities of the motion vectors on a per-region basis constituting input image data. The second motion vector output unit outputs the motion vectors of the input image data according to the motion vectors in each of the regions and to the reliability indicators of the motion vectors in each of the regions. The image data processing unit performs image processing on the input image data according to the motion vectors of the input image data.

Description

Specification

Image processing apparatus, image processing method, and image processing program

Technical field

The present invention relates to an image processing device, an image processing method, and an image processing program that reduce the effects of camera shake during imaging and camera shake caused by movement of the imaging system itself. The present invention is applicable to, for example, an imaging device, a digital still camera, a digital movie camera, and the like mounted on an information terminal device such as a mobile phone device or a PDA (Personal Digital Assistant).

Background art

[0002] Conventionally, when shooting a subject with a silver salt film camera or digital still camera by hand, the camera shakes due to camera shake, etc., and the captured image is subject to deterioration that flows in the direction of camera shake. .

[0003] In order to correct such image degradation due to camera shake, imaging is performed by driving the optical system, and a method of reducing camera shake or image processing using the calculated amount of camera shake is performed to reduce camera shake. A method has been proposed.

[0004] For example, Patent Document 1 describes a camera shake correction device that corrects camera shake by image processing. The camera shake correction apparatus described in Patent Document 1 first obtains a motion vector for each block constituting image data. Next, if a predetermined number or more of the motion vectors in each block have the same direction and size, it is determined that the entire screen has moved, and this motion vector is used as the motion vector for the entire screen. Then, the cutout position of the image data is changed using the motion vector of the entire screen, and output as shake correction image data.

Patent Document 1: Japanese Patent Laid-Open No. 9-261530 (paragraphs 0034-0036, FIG. 1)

Disclosure of the invention

Problems to be solved by the invention

[0006] However, in the conventional camera shake correction device, the detection accuracy of the motion vector of each block is not taken into consideration. Therefore, when determining the motion vector for the entire screen, a motion vector with low reliability may be used, and as a result, the motion vector for the entire screen must be determined correctly. May not be possible.

The present invention has been made to solve the above-described problems, and an object of the present invention is to provide an image processing apparatus and an image processing method capable of performing highly accurate shake correction processing.

Means for solving the problem

[0008] An image processing apparatus according to the present invention provides:

A first motion vector output unit that outputs a motion vector and a reliability index indicating the reliability of each motion vector for each region constituting the input image data;

A second motion vector output unit that outputs a motion vector of the input image data based on the motion vector for each region and a reliability index of the motion vector for each region; and the motion of the input image data An image data processing unit for image processing the input image data based on the vector;

Is provided.

The invention's effect

According to the present invention, it is possible to perform a shake correction process with high accuracy.

Brief Description of Drawings

FIG. 1 is a diagram showing a configuration of an image processing device according to Embodiment 1 of the present invention.

FIG. 2 is a diagram showing a state in which a motion vector of each block is detected.

FIG. 3 is a diagram showing SAD when a motion vector is detected.

FIG. 4 is a diagram showing a motion vector, a minimum value of SAD, and a reliability index of each block according to Embodiment 1 of the present invention.

FIG. 5 is a diagram for explaining image processing in Embodiment 1 of the present invention.

Explanation of symbols

[0011] 1 image data processing unit, 2 frame memory, 3 first motion vector output unit,

4 Second motion vector output unit, 5 blur detection unit, 8 image data, 9 blocks, 10 motion vectors of each block, 11 upper left block, 12 lower right block, 100 image processing device, 101 image sensor , 102 Display element. BEST MODE FOR CARRYING OUT THE INVENTION

[0012] Embodiment 1.

Hereinafter, the present invention will be described in detail with reference to the drawings showing embodiments thereof. The image processing apparatus 100 according to Embodiment 1 detects a motion vector and a reliability index indicating the reliability of the motion vector for each region (hereinafter referred to as “block”) constituting the image data. Then, the motion vector of the entire image data is obtained based on the detected motion vector and the reliability index.

FIG. 1 is a block diagram showing the configuration of the image processing apparatus 100 according to the first embodiment. In Fig. 1, solid line arrows indicate image data, and broken line arrows indicate data other than image data.

First, image data output from the image sensor 101 such as a video camera (hereinafter referred to as “input image data”) is an image data processing unit 1, a frame memory 2, and a first motion vector output. Input to part 3. The image data processing unit 1 performs image processing on the input image data, and outputs it to the display element 102 or the like as blur correction image data. Note that FIG. 1 shows an example in which the blur-corrected image data is output to the display element. The image may be recorded on a recording medium such as a hard disk or a DVD, or may be transmitted to another terminal by communication means.

The frame memory 2 stores input image data output from the image sensor 101. The first motion vector output unit 3 includes input image data from the image sensor 101 and image data that is stored in the frame memory 2 and temporally prior to the input image data (hereinafter referred to as “previous image data”). ) Is entered.

[0016] The first motion vector output unit 3 detects a motion vector for each block obtained by dividing the image data, and outputs the detected motion vector of each block to the second motion vector output unit 4. The first motion vector output unit 3 also detects a reliability index indicating the reliability of each motion vector, and outputs the detected reliability index to the second motion vector output unit 4.

[0017] The second motion vector output unit 4 detects a motion vector indicating the motion of the entire input image data from the motion vector of each block output from the first motion vector output unit 3 and the reliability index. The detected motion vector is output to the shake determination unit 5. [0018] The blur determination unit 5 analyzes the motion vector output from the second motion vector output unit 4, and determines whether or not the force of the input image data indicated by the motion vector is due to a shake such as a camera shake. To do. If it is determined that the input motion vector is due to shaking, the camera shake determination unit 5 outputs the reference position change information to the image data processing unit 1.

Based on the reference position change information output from the shake determination hand unit 5, the image data processing unit 1 determines the reference position of the image data output to the display element 102 as shake correction image data from the image sensor 101. The reference position force of the input image data to be output is also changed.

[0020] <First motion vector output unit>

Next, processing of each unit constituting the image processing apparatus 100 will be described in detail. First, the motion beta and reliability index detection processing in the first motion vector output unit 3 will be described with reference to FIGS. FIG. 2 is a diagram showing motion vectors in each block of image data divided into a predetermined number (12 in the figure). The first motion vector output unit 3 detects the motion vector 10 for each block of the input image data. In FIG. 2, for example, the size of the image data 8 is 48 pixels by 64 pixels by 64 pixels, and the size of the block 9 is 16 pixels by 16 pixels by 16 pixels.

Here, the process of detecting the motion vector 10 of each block in the first motion vector output unit 3 will be described. First, image data of each block constituting the previous image data is acquired from the previous image data stored in the frame memory 2. Here, the block in the previous image data is called a “reference block”. Further, image data of a block at the same position as the reference block is acquired from the input image data. This is called an “input block”. Then, the sum of absolute differences (SAD) of the image data is calculated while moving the position of the reference block up, down, left, and right with respect to the input block. SAD (X, y), which is the SAD when the position of the reference block relative to the input block is expressed by (x, y) and moved, is calculated as follows.

[0022] [Equation 1] .D (x,

/) | input image (j): Input hook data

rejerence imagei u]) '. [0023] It should be noted that the amount of movement of the reference block may be one pixel unit or multiple pixel units! /.

[0024] Figure 3 shows SAD. Figure 3 shows that SAD has the minimum sadmin value at (xmin, ymin). Here, the value of SAD means that the image data of the reference block and the image data of the input block are very close. In addition, when SAD takes the minimum value at (xmin, ymin), the image data when the reference block is shifted by (xmin, ymin) and the image data of the input block are the most approximate. It is shown that. Therefore, in the case shown in FIG. 3, the first motion vector output unit 3 outputs ( _xmin , ymin) as the motion vector of the current input block.

The first motion vector output unit 3 performs the processing of Equation 1 in all the blocks, detects the motion vectors of all the blocks, and outputs them to the second motion vector output unit 4.

Next, the motion vector reliability index detection processing of each block in the first motion vector output unit 3 will be described.

FIG. 4 (a) shows how motion vectors are detected in the first embodiment. Image data 8, block 9, and motion vector 10 for each block are the same as those in FIG. 2 described above. is there.

Here, the number shown in each block is a value shown as “sadmin” in FIG. 3, which is the “minimum value of SAD” obtained when the motion vector is obtained. For example, in the upper left block 11 of Fig. 4 (a), the value "10" is shown as the "minimum value of SAD", and in the lower right block 12, "the minimum value of SAD" is "50". ”Is shown! /

[0029] This indicates that the value of the image data of the block 11, which is an input block, is close to the value of the image data of the reference block. That is, it can be considered that the image contents of block 11 and the reference block are approximate.

[0030] On the other hand, it is shown that the value of the image data of the block 12, which is the input block, and the value of the image data of the reference block are not relatively approximate. In other words, it is possible to consider that the image content of the block 12 and the reference block are approximate compared to the case of the block 11 and are not.

In other words, the motion vector 10 of the block 11 is more reliable than the motion vector 10 of the block 12. Here, for example, a value obtained by subtracting the “minimum value of SAD” from 100 is changed. Figure 4 (b) shows the reliability index of motion vector 10 of each block shown in Fig. 4 (a).

[0032] As shown in FIG. 4 (b), the reliability index of the motion vector in block 11 is “90”,

The reliability index of the motion vector of block 12 is “50”. As described above, the first motion vector output unit 3 detects the motion vector 10 and the reliability index of each block constituting the image data, and outputs the detected motion vector 10 to the second motion vector output unit 4.

Next, the motion vector detection process in the first motion vector output unit 4 will be described. The second motion vector output unit 4 detects the motion vector of the entire image data based on the motion vector and reliability index for each block output from the first motion vector output unit 3.

For example, the second motion vector output unit 4 sets the motion vector 10 having the highest reliability index as the motion vector of the entire image data. In the case shown in Fig. 4 (b), since the reliability index of the motion vector of the upper left block and the lower left block is the highest value and the direction and size are the same, these motion vectors 10 are The image data 8 is output to the motion detection unit 5 as the motion vector of the entire image data 8.

[0035] <Shake determination unit>

The shake determination unit 5 analyzes the motion vector output from the second motion vector output unit 4, and determines whether or not the motion vector is caused by a shake such as a camera shake. For example, if the motion vector output from the second motion vector output unit 4 points in the same direction for a certain period of time, this motion occurs because the user intentionally moved the camera instead of camera shake. You may judge. In addition, if the motion vector output from the second motion vector output unit 4 is observed for a certain period of time, and the screen motion frequency is slower than a certain value, this motion is not a camera shake. Even if it is determined that the error occurred because the user intended to power the camera.

When it is determined that the input motion vector is due to camera shake, the camera shake determination unit 5 outputs the reference position change information to the image data processing unit 1. When it is determined that the input motion vector is due to something other than camera shake, the image stabilization determination unit 5 determines the reference position as Information that no change is made is output to the image data processing unit 1. In addition, do not output the information that V does not change the reference position!

[0037] <Image data processing unit>

FIG. 6 is a diagram for explaining image processing in the image data processing unit 1. The image data processing unit 1 changes the reference position of the input image data 13 output from the image sensor 101 in the direction of the motion vector 15 based on the reference position change information. As a result, the image data output to the display element 102 changes from the image data 16 before the image processing to the shake-corrected image data 14 after the image processing.

That is, the image data processing unit 1 uses image data 16 from the image data 16 centered on the reference position of the input image data 13 output from the image sensor 101 as the blur-corrected image data output to the display element 102. Outputs image data 14 for image stabilization 14 that is moved in the direction of the entire motion vector 15.

Thus, according to the image processing apparatus 100 according to the first embodiment, the motion vector of each block also refers to the reliability index of each motion vector when detecting the motion vector of the entire image data. Therefore, the motion vector of the entire image data can be detected with high accuracy. Therefore, the blur correction process can be performed with high accuracy.

In the above description, the method of using the “minimum value of SAD” as the motion vector reliability index has been described. However, any method that shows the reliability of the motion vector is not limited to this. Even things! / ヽ.

[0041] For example, the motion vector of the block 11 in FIG. 4 (a) shows the same direction as the motion vector of the surrounding blocks, and is considered to be highly reliable. On the other hand, since the motion vector of block 12 is completely different from the motion vectors of the surrounding blocks, it seems that the reliability is low. Thus, the closeness with the motion vectors of the surrounding blocks may be output as a reliability index.

[0042] In the above description, the reliability index is the highest, and the motion vector force is the force described for the method for obtaining the motion vector of the entire image data. For example, the motion vectors may be arranged in the descending order of the reliability index, and a predetermined number of motion vectors in the higher order may be used. A motion vector having a reliability index of a predetermined value or more may be used. [0043] Alternatively, a method may be used in which a motion vector is weighted using a motion vector reliability index, and then the weighted motion vector is averaged. For example, only block 11 and block 12 in FIG. The motion vector of block 11 is (2, 3), and the motion vector of block 12 is (1-1, 1). In this case, since the reliability index of the motion vector of block 11 is “90” and the reliability index of the motion vector of block 12 is “50”, (90 X (2, 3) + 50 X (—1, 1 2)) Z (90 + 50) = (130, 170) / 140 = (0. 929, 1. 214), and the motion vector of the entire image data is about (1, 1).

[0044] As another method, a method of weighting motion vectors using the motion vector reliability index as "motion vector appearance frequency" may be used. For example, in FIG. 4B, the motion vector appearance frequency of block 11 is assumed to be “90”, and the motion vector appearance frequency of block 12 is assumed to be “50”. Also check the appearance frequency (reliability index) of other blocks and sum the appearance frequency for each same vector. The vector having the highest sum of appearance frequencies is set as the motion vector of the entire image data.

[0045] Furthermore, only blocks whose reliability index is larger than a certain value are extracted, and only for these blocks, "weighted motion vector averaging process" or "sum of motion vector appearance frequencies" is extracted. It ’s the highest, you can do “vector extraction”!

[0046] Further, as a configuration, a force including the image sensor 101 and the display element 102 in the image processing apparatus 100 may be configured to be outside the image processing apparatus. Furthermore, the frame memory 2 may be configured outside the image processing device!

[0047] Further, the image data used for calculating the SAD may be pixel data such as a luminance signal, a color difference signal, and an RGB signal for each pixel, or a combination thereof.

[0048] In addition, the pixel data of all the pixels in the block is not limited to the pixel data, and pixel data of a single pixel or a plurality of representative pixels may be used.

[0049] Alternatively, the configuration may be such that the motion vector output from the second motion vector output unit 4 that passes through the blur determination unit 5 is directly output to the image data processing unit 1.

[0050] Furthermore, the image data stored in the frame memory and the input image data output from the image sensor may be in units of frames or VOPs. Also, multiple frames or A frame memory capable of storing VOPs may be prepared, and the “reference block” may be present in the image data temporally after the input image data including only the image data temporally prior to the input image data. Further, the motion vector may be obtained by referring to both image data.

Claims

The scope of the claims

[1] A first motion vector output unit that outputs a motion vector and a reliability index indicating the reliability of each motion vector for each region constituting input image data;

An image processing apparatus comprising:

[2] The reliability index is

Image data of any reference area of the area in the previous image data that is temporally prior to the input image data, which is referred to when determining the motion vector corresponding to the reliability index;

Determined based on the image data of the region corresponding to the motion vector of the input image data

The image processing apparatus according to claim 1, wherein:

[3] The reliability index is

The sum of absolute value differences between pixel data of pixels constituting the reference area and pixel data of pixels constituting the area corresponding to the motion vector

The image processing apparatus according to claim 2.

[4] A shake determination unit that determines the occurrence of shake in the input image data based on a motion vector of the input image data,

The blur determination unit has a blur in the input image data! When it is determined that the input / output is performed, the reference position change information based on the motion vector of the input image data is output,

The image data processing unit performs image processing on the input image data based on the reference position change information!

The image processing apparatus according to claim 1, wherein:

[5] The image processing includes:

Based on the motion vector of the input image data, the image data after the image processing is processed. Changing the reference position, the reference position force of the input image data

The image processing apparatus according to claim 1, wherein:

[6] A first motion vector output step for outputting a motion vector and a reliability index indicating the reliability of each motion vector for each area constituting the input image data;

A second motion vector output step for outputting a motion vector of the input image data based on the motion vector for each region and a reliability index of the motion vector for each region; and a motion vector of the input image data And an image data processing step for performing image processing on the input image data.

An image processing method comprising:

[7] The reliability index is

The image processing method according to claim 6.

[8] The reliability index is:

The image processing method according to claim 6.

[9] A shake determination step of determining occurrence of shake in the input image data based on a motion vector of the input image data,

In the blur determination step, when it is determined that blur has occurred in the input image data, reference position change information based on a movement margin of the input image data is output,

In the image data processing step, image processing is performed on the input image data based on the reference position change information! The image processing method according to claim 6.

[10] The image processing includes:

Based on the motion vector of the input image data, the reference position of the input image data is changed to the reference position of the image data after the image processing.

The image processing method according to claim 6.

[11] A first motion vector output step for outputting a motion vector and a reliability index indicating the reliability of each motion vector for each region constituting the input image data;

An image processing program comprising: