JP2012169772A - Image processing device, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform blur correction image processing to a focus oscillation video with ease.SOLUTION: To such a video with oscillated focus as being once focused, then out of focus and focused again, a focused frame is specified as a reference frame image. A time period to a frame image being focused once, then blurred and focused again is set as a blur video time period by using the reference frame image as a reference. Blur characteristics of each frame are calculated based on the reference frame image in the blur video time period. Blur correction to each frame in the blur video time period is performed using the calculated blur characteristics to enable a video with modified blur to be outputted.

Description

  The present invention relates to an image processing apparatus and an image processing method, and more particularly to a technique suitable for easily correcting an image generated by a focus lens that is vibrated when a focus lens is operated with a camera.

  In digital still cameras and video cameras, when taking a picture by manually operating the focus lens, it is difficult to focus on the subject because the display area is small when viewing the subject displayed on the viewfinder or the like. For this reason, when focusing on the subject while operating the focus lens, the focus may be once out of focus and the focus may be corrected again. At this time, the image changes so that the subject is once out of focus and refocused from the image of the proper focus (hereinafter referred to as focus vibration or focus vibration image).

  As a conventional technique, there is a technique (for example, Patent Document 1) that calculates a blur correction function from a focused image and a blur image of the same subject and performs blur correction. This technology calculates a blur correction function (detailed focus function) from a focused image and a blur image of the same subject, and further gives a training data to the blur correction function to calculate a smart focus function for blur correction. It is.

JP 2009-20844 A

However, according to Patent Document 1, it is necessary to prepare a focused image and a blurred image of the same subject in advance in order to calculate a smart focus function for blur correction. In addition, when the correction target is an image, all the target frames must be specified, and there is a problem that it takes time and effort for the user.
SUMMARY OF THE INVENTION In view of the above-described problems, an object of the present invention is to make it possible to easily perform blur correction image processing on a focused vibration image.

  An image processing apparatus according to the present invention includes a display unit that selects an input video and displays the selected video on a display device, and a reference frame that designates a frame that is in focus from the video displayed on the display device as a reference frame image. A reference means, a blur period detecting means for detecting a blurred frame period in which the focus is oscillating from the reference frame image; and the reference frame for the blurred video period detected by the blur period detecting means. A blur characteristic calculation unit that calculates a blur characteristic of each frame in the blur period based on an image, and a blur correction process that performs blur correction of each frame based on the blur characteristic of each frame calculated by the blur characteristic calculation unit Means.

According to the present invention, it is possible to use an image in which a focus is vibrated that cannot be removed even if there is a focus support function for focusing, by using the material image such as editing by correcting the blur. There is an effect.
Further, the present invention can be applied not only to manual focus but also to a focus vibration image generated during auto focus. There are an active method for performing autofocus using ultrasonic waves or infrared rays, and a passive method for performing autofocus based on the contrast or phase difference of a captured image. There are cases where the image is such that the autofocus becomes unstable and the focus vibrates due to the influence of the positional relationship with the subject, brightness, etc. The present invention can also be applied to such a problem.

It is a block diagram which shows the structural example of the image processing apparatus of the 1st Embodiment of this invention. It is a figure which shows an example of the user interface which performs video editing. It is a flowchart which shows the whole process of 1st Embodiment. It is a flowchart which shows the focus area | region designation | designated process of 1st Embodiment. It is a schematic diagram explaining a mode that an area | region is designated when correcting a focus area | region. It is a flowchart which shows an example of the blur period detection process of 1st Embodiment. It is a flowchart which shows the similarity calculation process of 1st Embodiment. It is a figure which shows the relationship of the effective frequency area | region and linear approximation in a similarity calculation process. It is a flowchart which shows the blur characteristic calculation process in a blur period. It is a flowchart which shows the blurring correction process in a blurring period. It is a figure explaining a mode that a focus vibrates. It is a figure explaining the mode of the blur correction performed during the period when the focus is vibrating.

(First embodiment)
An image processing apparatus according to a first embodiment of the present invention will be described below with reference to the drawings.
The present invention performs blur correction on an image in which the focus is vibrating by an editing application. The overall configuration of an image processing apparatus system that operates an editing application will be described with reference to the configuration diagram of FIG.

  An image processing apparatus 101 that corrects a blurred image includes an external recording interface (I / F) 102 that inputs and outputs video, and a user interface (I / F) 103 that specifies user control. In addition, a network interface card (NIC) 104 for connecting to a network and a display interface (I / F) 105 for displaying an image on a display are provided. Furthermore, a CPU 106 that performs program calculation / control processing, a ROM 107 that stores programs to be executed and various data, and a RAM 108 that develops video data and execution programs are provided.

Hereinafter, an example of a user interface of an editing application used for blur correction will be described with reference to FIG.
The video editing window 201 represents the entire video editing application. A video selection screen 202 for selecting a video to be edited from a plurality of videos and a preview screen 203 for displaying a frame of the edited video are provided. In addition, a timeline editing screen 204 in which video, audio, and the like are arranged on the time axis, and an editing target frame setting unit 205 that selects an editing target frame from the timeline are provided.

  In the following, an algorithm of a part for performing blur correction on an image whose focus is oscillating in an editing application will be described using the flowchart of FIG. Here, an outline of each step will be described, and detailed processing will be described later except for S301.

When the process is started, the input video is selected and displayed on the display (display device) (S301).
Next, it waits for the user to select a reference frame that is in focus (S302).
Next, frequency analysis processing is performed on the selected reference frame image to extract a focused subject area (S303).
As a result of the frequency analysis, the extracted focus subject area is displayed on the screen. If it is different from the user's intention, a mechanism for designating a focus area is provided (S304).

A similarity is calculated from the frequency component of the reference frame selected in S302 and the frequency component of each frame, and a focus vibration period for detecting a frame period in which the focus is vibrated and blurred is calculated (S305).
Next, a blur PSF function is calculated for each frame in the focus vibration period (S306).
Next, blur correction processing is performed on each frame based on the PSF function (S307).
Next, the corrected video is recorded on the recording medium from the external recording interface 102 (S308).

Hereinafter, the reference frame designation method in S302 will be described with reference to FIG.
A video to be corrected for blurring is selected on the video selection screen 202, and a frame in focus is searched for in the editing target frame setting unit 205 for the video arranged on the timeline editing screen 204. In the search method, the frame displayed on the preview screen 203 is changed by the editing target frame setting unit 205 by operating a pointing device such as a mouse on the timeline editing screen 204. While checking the image displayed on the preview screen 203, the frame in focus is checked. At this time, the most focused frame is selected while moving the frame from the past to the future on the time axis. The selected frame is set as a reference frame.

Hereinafter, the extraction of the subject area of the reference frame in S303 will be described with reference to the flowchart of FIG.
A process of obtaining a secondary differential image by applying a secondary differential filter to the image of the specified frame on the entire reference frame image specified in S302 (S401).
Next, the binary differential image is binarized using a predetermined threshold value, and a binary image generation process for generating a binary image is performed (S402).

An isolated point having a predetermined area or less is further removed from the binarized image, and edge image generation processing for generating an edge image is performed (S403).
Next, a rectangular region circumscribing all the edges is obtained, and a focus region estimation process for estimating a focus region is performed (S404).

Hereinafter, the display of the subject area in S304 will be described with reference to FIG. The in-focus subject area 501 obtained in S303 is displayed on the preview screen 203 in FIG.
As in the focus estimation area 503 surrounded by a thick line in FIG. 5A, a process of explicitly surrounding the area with a line is performed so that the focus area can be easily understood.
Also, the focus area is explicitly displayed by adding a color to the enclosed area, such as the focus estimation amount area 504 in FIG.

  On the other hand, as shown in FIG. 5C, when the detected focus estimation area 503 is different from the user's intention and the area 506 of the subject 502 is an area desired by the user, the user can use a user interface such as a mouse. It has a function of designating a desired area. Examples of the designation method include a method in which a focus area desired to be set with the pointing cursor 507 is enclosed with a pen-like tool, a square is set, and a square area enclosed in advance is called and enclosed.

Hereinafter, the blur period detection process in S305 will be described with reference to the flowchart of FIG.
The focus area of the reference frame obtained in S304 is acquired (S601).
Next, the search target frame is changed to a frame before the reference frame and initialized (S602).
Next, the search target frame is moved to the past by one frame (S603).
Next, alignment processing with the reference frame is performed on the search target frame (S604). Regarding the alignment process, a process using the phase only correlation method may be used. When the two-dimensional Fourier transform of the reference frame is I ₁ and the two-dimensional Fourier transform of the search target frame is I ₂ , the phase-only correlation is calculated by inverse Fourier transform of the following (1).

  As a result, the shift amount is obtained from the correlation peak coordinates. The alignment process may be another process and does not limit the processing method.

Next, a focus area is acquired from the frame after alignment (S605).
Next, the similarity is calculated from the focus area of the search target frame and the reference frame (S606). Details of the calculation process will be described later.
Next, it is determined whether the similarity calculated in S606 is greater than a predetermined value (S607). If it is large, the similarity is high, and if it is below a predetermined value, the similarity is low. If the similarity is low, the process proceeds to S608, and the frame movement amount is determined. If it is higher, the process proceeds to S610.

  Next, the frame movement amount is determined. If the focus vibration period (blurring period) to be searched is greater than or equal to a predetermined value, this is not the detection of the focus vibration section that is assumed this time, but the entire section is blurred, or the section is intentionally blurred for a long time Will go. Also, when reaching the leading edge or the ending edge of the video frame, it is determined that the threshold is equal to or greater than the threshold (S608). If it is less than or equal to the threshold, the process proceeds to S603, and if it is greater than the threshold, the similarity determination threshold is reset (S609).

  Thereafter, the process proceeds to S602, where the search target frame is initialized. In addition, when resetting, not only the resetting process is automatically performed, but also a warning is displayed on the display screen indicating that the resetting process is being performed, and the user is made aware that the resetting process is being performed. May be.

On the other hand, the threshold value for similarity determination is not reset, and there is no video section that is in focus vibration from the reference frame, so the reference frame may be reselected. Also in this case, a warning that “the focus vibration section cannot be found” is issued on the display screen, a display prompting the user to reselect the reference frame is displayed, and the process proceeds to S302 in the flowchart of FIG. 3 to perform reprocessing.
In step S610, the focus vibration period is calculated from a frame having a high degree of similarity to the reference frame, and the calculated number of frames in the focus vibration period and the frame in which the focus vibration period ends are stored.

Hereinafter, the calculation of the similarity in S606 will be described with reference to the flowchart of FIG.
The window function is multiplied by the focus area of the reference frame and the corresponding area in the search target frame (S701). The window function uses a Hamming window function, but is not limited to this. Fourier transform is performed on the image multiplied by the window function to obtain a frequency component (S702).

The frequency component is converted into one-dimensional data (S703). The conversion method integrates frequency components for each circumferential frequency. The frequency component in the circumferential direction may be integrated, or the vertical component and the horizontal component may be integrated.
An effective frequency region to be used for similarity calculation is calculated from the focus region of the reference frame (S704). The focus area of the reference frame is Fourier-transformed to obtain an absolute value, and an area where the frequency amplitude component becomes a predetermined threshold is set as an effective frequency area.

By dividing the one-dimensional data of the frequency component of the search target frame by the one-dimensional data of the frequency component of the reference frame, blur frequency information is acquired (S705).
The blur frequency information and the effective frequency region are stored as PSF calculation information (S706). Here, PSF is a point spread function, which is used as a method of expressing blur characteristics as data.
Using the blur frequency information in the effective frequency region, a predetermined section is approximated to a straight line, and the slope of the approximated straight line is set as similarity information (S707). The predetermined section is determined from the relative position in the effective frequency region.

  FIG. 8A shows the one-dimensional data of the frequency components of the image and the effective frequency region. FIG. 8B is a schematic diagram showing the relationship between the blur frequency information and the straight line approximation section.

Hereinafter, the blur function calculation process (blur characteristic calculation process) in S306 will be described with reference to the flowchart of FIG. In the blur characteristic calculation process, the number of frames in the focus vibration period is set as the upper limit, and the process is repeated up to the upper limit.
First, the search target frame is initialized (S901).
Next, PSF calculation information of the target frame is acquired (S902).
Next, a Gaussian function approximating the blur frequency information is obtained in the effective frequency region (S903).

Next, inverse Fourier transform is applied to the approximated Gaussian function, and the calculated coefficient is set as PSF (S904).
Next, the calculated PSF is stored, and the target frame is updated to the next frame (S905).
Next, it is determined whether the updated frame exceeds the number of frames in the correction period (S906). When it does not exceed, the process proceeds to S902, and when it exceeds, the blur characteristic calculation process ends.

  Next, the blur correction process in S307 will be described. In the present embodiment, blur correction processing is performed based on the MAP estimation method. (Equation 2) is an evaluation function to be minimized in the MAP estimation method.

  Here, x is a corrected image, and y is a one-dimensional representation of a frame image. The matrix P is a blur model, and represents the PSF convolution in matrix representation. Here, if the image size is horizontal W pixels and vertical H pixels, x and y are column vectors of H × W rows, and P is a square matrix of (H × W) × (H × W).

The first term on the right side of (Formula 2) is the square error between the image obtained by blurring the corrected image x with the matrix P and the frame image y, and corresponds to the conditioning probability. σ is a standard deviation of the noise amount of the frame image. The second term on the right side is an evaluation value of the smoothness of the image and corresponds to the prior probability of non-edge.
∇ ² x _{m, n} corresponds to the second derivative of the image and can be calculated by the following (Equation 3). Α is a coefficient.

Hereinafter, the blur correction process for the blur correction period using the MAP estimation method described above will be described with reference to the flowchart of FIG. In this embodiment, the steepest descent method is used for minimizing the evaluation function.
First, a target frame to be subjected to blur correction is set to a frame that is one past of the reference frame and is initialized (S1001).

Next, the PSF corresponding to the target frame is acquired, and the matrix P is updated (S1002).
Next, the corrected image x is initialized with the target frame image (S1003).
Next, the update amount of the image is calculated by the following (formula 4) and (formula 5) (S1004).

次に、補正画像の各画素に更新量の各画素を足す（Ｓ１００５）。t回の繰り返し処理が行われた補正画像をx^(t)とすると、x^(t+1)とは下記の（６式）より計算される。

Next, the update amount of each pixel is added to each pixel of the corrected image (S1005). If a corrected image that has been subjected to t iterations is x ^(t) , x ^{(t + 1)} is calculated from (Equation 6) below.

Here, ε is the width of the update. If the value is large, the convergence approaches quickly, but if it is too large, the error becomes large or diverges.
Next, it is determined whether the number of processing times is equal to or greater than a predetermined value (S1006). If the determination is true, the process proceeds to S1007, and if it is false, the process proceeds to S1004 to continue the correction image correction process.

If the number of processing times is equal to or greater than the predetermined value, the frame subjected to blur correction is stored (S1007). Next, the correction target frame is moved to the next frame (S1008).
Next, it is determined whether or not the moved blur correction target frame exceeds the processing frame for the blur correction period (S1009). If the number of processing frames has not been exceeded, the process moves to S1002 to perform repeated processing. On the other hand, if the number of processing frames is exceeded, the blur correction process is terminated.

  The blur correction process of the present embodiment is an example of a process using the MAP estimation method, but is not limited to this, and other methods such as an inverse filter or a Wiener filter based on PSF may be used.

  Further, although the blur characteristic calculation process and the blur correction process are described separately for each processing block, the process may be sequentially performed for each frame. Further, in the present embodiment, an embodiment such as an application that operates on a computer has been described. However, the present invention may be implemented even when mounted in a photographing device such as a digital still camera or a video camera.

  Further, in the present embodiment, for the designation of the focus vibration period, a reference frame as a starting point is set and the end point is automatically obtained. However, the user may designate a frame as the end point. In the present embodiment, an external recording medium is used for the video for correcting the focus vibration period. However, either or both of the input video and the output video may be via a network.

  As described above, according to the present invention, when focusing on a subject at the time of shooting as shown in FIG. 11, a frame in focus is designated for an image whose focus has vibrated. Thus, the blurred video period is automatically detected. Then, a blur characteristic for each frame is calculated for the blur period in which the focus is oscillating, and blur correction is performed using the characteristic as shown in FIG. 12, and a corrected video can be output.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (computer program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various computer-readable storage media. Then, the computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program.

DESCRIPTION OF SYMBOLS 101 Image processing apparatus 102 External recording interface 103 User interface 104 Network interface card 105 Display interface 106 CPU, 107 ROM, 108 RAM

Claims (7)

Display means for selecting and displaying the input video on a display device;
Reference frame designating means for designating a frame in focus from the video displayed on the display device as a reference frame image;
A blur period detecting means for detecting a frame period in which the focus is vibrating from the reference frame image as a blur period;
A blur characteristic calculation unit that calculates a blur characteristic of each frame in the blur period based on the reference frame image with respect to a blurred frame period detected by the blur period detection unit;
An image processing apparatus comprising: a blur correction processing unit that performs blur correction of each frame based on the blur characteristic of each frame calculated by the blur characteristic calculation unit. A focus area estimation means for estimating a focus area of the reference frame;
The image processing apparatus according to claim 1, wherein the focus area estimation unit explicitly displays a focus area of the reference frame and displays a focus estimation area to a user.   The image processing apparatus according to claim 2, wherein the focus area estimation unit includes a changing unit that changes the estimation area by a user.   The image processing apparatus according to claim 1, further comprising a storage unit that specifies a reference frame image and stores a frame in which the focus vibration period ends in a storage medium. A display process of selecting the input video and displaying it on the display device;
A reference frame designating step for designating a frame in focus from the video displayed on the display device as a reference frame image;
From the reference frame image, a blur period detection step of detecting a frame period in which the focus is vibrating as a blur period;
A blur characteristic calculating step of calculating a blur characteristic of each frame in the blur period based on the reference frame image with respect to the blurred frame period detected in the blur period detecting step;
An image processing method comprising: a blur correction process step of performing blur correction of each frame based on the blur characteristic of each frame calculated in the blur characteristic calculation step. A display process of selecting the input video and displaying it on the display device;
A reference frame designating step for designating a frame in focus from the video displayed on the display device as a reference frame image;
From the reference frame image, a blur period detection step of detecting a frame period in which the focus is vibrating as a blur period;
A blur characteristic calculating step of calculating a blur characteristic of each frame in the blur period based on the reference frame image with respect to the blurred frame period detected in the blur period detecting step;
A program causing a computer to execute a blur correction processing step of performing blur correction of each frame based on the blur characteristic of each frame calculated in the blur characteristic calculation step.   A computer-readable storage medium storing the program according to claim 6.

Images