JP2009111921A - Image processing device and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing device and an image processing method capable of extracting a foreground with a simple technique. <P>SOLUTION: The image processing device comprises: an estimating part 11 for estimating a state of defocus of each image data out of a plurality of image data about an identical object; a dividing part 12 for dividing the background area and the foreground area of each of the image data based on the estimated result by the estimating part 11; and an extracting part 16 for extracting the foreground of each of the image data in response to the divided result 31 of the dividing part 12. Each of the image data is the image focused on the foreground compared with the background. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method, for example, a method for extracting a foreground included in an image.

  Conventionally, techniques for extracting a foreground region in an image have been studied. For example, the present technology is used for the purpose of, for example, extracting an actor only by removing a background from an image taken by a camera, and combining the extracted actor image with another background.

  The standard foreground extraction method used in the movie industry is to extract the foreground part on the assumption that the background is, for example, a single blue color and that the color of the foreground object does not include blue (for example, patents). References 1 and 2). Also proposed is a method of eliminating the assumption that the foreground object color does not include a specific color by using two images taken in front of two different colors (eg blue and green) respectively. (For example, refer to Patent Document 3). However, with these methods, the shooting location is limited to an environment where background control is easy, such as a studio, and it is difficult to improve the extraction accuracy.

  In this regard, there is a technique that greatly simplifies the problem by considering it as a binary discrimination problem in which each pixel of the captured image is either the foreground or the background. However, with this method, when shooting a complex silhouette object such as hair or fur, or an object with a translucent part, the foreground color is partially extracted or the foreground color including the original background color is extracted. There is a problem to do. In the technique disclosed in Patent Document 4, this problem is reduced by filtering the extracted foreground, but it is not an essential solution.

  Furthermore, many methods for extracting the foreground part by providing the user with rough information on the foreground part and the background part through some user interface have been proposed (see, for example, Patent Documents 5 to 7). However, with these methods, there is a problem that the user has to become familiar with the interface and it takes time and effort to input information. In general, the input work is often trial and error.

Therefore, in order to automatically extract the foreground without human intervention in a state where the background is unknown, a method of increasing the number of images taken of the same object has been proposed (for example, Patent Documents 3, 8, 9, and Non-patent documents 1 to 3). However, these methods require very precise equipment such as preparing a large number of cameras and calibrating the cameras.
US Pat. No. 6,525,741 US Patent 3,595,987 US Pat. No. 6,301,382 US Patent 7,024,054 US Pat. No. 6,721,446 US Pat. No. 6,288,703 US Pat. No. 6,134,346 US Pat. No. 6,903,738 US 7,206,000 specification McGuire, Matusik, Pfister, Hughes, Durand, “Defocus video matting”, ACM Trans. Graphics 24 (3), pages 567-576, 2005 McGuire, Matusik, Yerazunis, “Practical, real-time studio matting using dual imagers”, Eurographics Symposium on Rendering, page 235-244, 2006 Joshi, Matusik, Avidan, “Natural video matting using camera arrays”, ACM Trans. Graphics 25 (3), page 779-786, 2006

  The present invention provides an image processing apparatus and an image processing method capable of extracting a foreground by a simple method.

  An image processing apparatus according to an aspect of the present invention includes: an estimation unit that estimates a blur state in each of a plurality of image data related to the same target; and a background in each of the image data based on an estimation result in the estimation unit. A dividing unit that divides the region to be the foreground and the region to be the foreground, and an extraction unit that extracts the foreground in each of the image data according to the division result in the dividing unit, each of the image data, The image is focused on the foreground compared to the background.

  An image processing method according to an aspect of the present invention includes a step of receiving a plurality of first image data and a state in which a plurality of processors are blurred for a plurality of regions included in each of the received plurality of first image data. Estimating each of the first image data into a background area and a foreground area based on the blur state; and Extracting the foreground for first image data; and the processor synthesizes the foreground with second image data different from the first image data, wherein each of the first image data includes the first image data The image is focused on the foreground compared to the background.

  According to the present invention, it is possible to provide an image processing apparatus and an image processing method capable of extracting a foreground by a simple method.

  Embodiments of the present invention will be described below with reference to the drawings. In the description, common parts are denoted by common reference symbols throughout the drawings.

[First Embodiment]
An image processing apparatus and an image processing method according to the first embodiment of the present invention will be described with reference to FIG. FIG. 1 is a block diagram of an image processing system according to the present embodiment.

  As illustrated, the image processing system 1 includes a plurality of cameras 2 and an image processing device 3. Each of the cameras 2 captures the target object and outputs the obtained image data to the image processing device 3. The camera 2 captures the same target object from different angles. At that time, the foreground as the target object is focused, and the background is photographed with the focus shifted. The target object is arranged such that the depth of the target object from the camera 2 is sufficiently closer than the depth of the background. In addition, the camera 2 shoots so that the shape of the foreground taken by each camera is substantially the same. In other words, there is no way to shoot such that one camera 2 shoots the target object from the front, another camera 2 shoots from behind, and another camera 2 shoots from the side. That is, all the cameras 2 photograph the target object from the same direction (or substantially the same direction). Therefore, the variation between viewpoints in each camera 2 is substantially perpendicular to the depth direction of the scene. Then, since there is a difference in depth between the foreground and the background, a plurality of pieces of image data in which the foreground object is moved with respect to the background can be obtained when shooting from different viewpoints. Also, the background only needs to be more blurred than the foreground, and the degree is not a problem.

  The image processing device 3 includes a foreground extraction unit 10 and an image synthesis unit 20. The foreground extraction unit 10 extracts a foreground in a captured image using a plurality of image data given from the camera 2. The image composition unit 20 composes the foreground extracted by the foreground extraction unit 10 with another background image to generate composite image data.

<Configuration of Image Processing Device 3>
Next, details of the image processing apparatus 3 will be described with reference to FIG. FIG. 2 is a block diagram illustrating a configuration of the image processing apparatus 3.

  First, the configuration of the foreground extraction unit 10 will be described. As illustrated, the foreground extraction unit 10 includes a blur estimation unit 11, a region division unit 12, a background corresponding point calculation unit 13, a foreground corresponding point calculation unit 14, a foreground / background color estimation unit 15, and an α value estimation unit 16. Yes.

  The blur estimation unit 11 receives a plurality of image data captured by the camera 2 as an input image. Then, the degree of blur in each input image is estimated. By this estimation, information indicating how much of which area is blurred in each input image can be obtained. This information is hereinafter referred to as a blur amount 30.

  The area dividing unit 12 creates a trimap 31 corresponding to each input image based on the blur amount 30 obtained by the blur estimation unit 11. A trimap is an image obtained by dividing an input image into three regions: a clearly foreground region, a clearly background region, and a foreground or background unknown region. The region dividing unit 12 estimates the three regions from the degree of blur in the input image.

  The background corresponding point calculation unit 13 aligns the background between a plurality of input images. That is, it is determined which point (pixel) in the background in one input image corresponds to which point (pixel) in the background in the other input image, and the amount of movement is obtained as the background warp function 32.

  The foreground corresponding point calculation unit 14 performs the same processing as the background corresponding point calculation unit 13 for the foreground. That is, the foreground is aligned between a plurality of input images. In other words, the foreground warp function 33 is obtained by grasping which point (pixel) in the foreground in one input image corresponds to which point (pixel) in the foreground in the other input image.

  The foreground / background color estimation unit 15 is based on the background warp function 32 obtained by the background corresponding point calculation unit 13, the foreground warp function 33 obtained by the foreground corresponding point calculation unit 14, an input image, and an α value (described later). The background color 34 and the foreground color 35 in each input image are estimated.

  The α value estimation unit 16 estimates the α value for each input image to obtain a mask image 36. The α value indicates a mixing ratio of the foreground color and the background color at a certain coordinate, and is a scalar value that takes a value in the range [0, 1]. For example, an area where α is “0” means that only the background is visible and there is no foreground. Conversely, in the region where α is “1”, only the foreground is visible, which means that the background is completely blocked by the foreground. An image obtained by obtaining this α value for the entire input image is a mask image. It is sufficient that the α value at each point of the input image is obtained, and it is not always necessary to create an image itself called a mask image. The α value estimating unit 16 first obtains an initial value of the α value based on the trimap 31 obtained by the region dividing unit 12. Thereafter, the α value is estimated using the foreground warp function 33 obtained by the foreground corresponding point calculation unit 14 and the background color 34 and foreground color 35 obtained by the foreground / background color estimation unit 15.

  Next, the configuration of the image composition unit 20 will be described. As illustrated, the image composition unit 20 includes a memory 21 and a composition unit 22.

  The memory 21 holds a new background 40. The new background 40 is image data serving as a new background that is different from the input image.

  The synthesizing unit 22 synthesizes the foreground and the new background 40 in the input image using the mask image 36 (α value) obtained by the α value estimating unit 16, the foreground color 35, and the new background 40. As a result, a composite image 41 in which the background in the input image is replaced with the new background 40 is obtained.

<Operation of Image Processing Device 3>
<< About the concept of foreground extraction >>
First, the general concept of foreground extraction in the image processing apparatus 3 according to the present embodiment will be described below.

In the foreground extraction or matting, the input image I is a linear blending of the foreground color F (the foreground color 35 described in FIG. 2) and the background color B (the background color 34 described in FIG. 2). ) Is assumed to be the result generated by

Here, x is a two-dimensional vector representing coordinates (pixel positions) on the image. Therefore, a notation such as I (x) represents a value at the point x of the input image I. If grayscale, I (x) is a scalar value, and if it is an RGB image, I (x) is a three-dimensional vector. . The same applies to the foreground color F and the background color B. Therefore, α is a function of x, and its value is expressed as α (x). As described above, α (x) indicates the mixing ratio of the foreground color F (x) and the background color B (x) at the coordinate x. Therefore, a pixel with α (x) of 0 can see only the background, and a pixel with α (x) of 1 can see only the foreground. When α (x) takes an intermediate value (0 <α (x) <1), it means that the foreground blocks a part of the background at the target pixel. There are two causes for this partial occlusion. One is a case where a boundary line between the foreground and the background (the outline of the foreground object) passes through the pixel, and the foreground color and the background color contribute to the same pixel. The other is a case where the background color can be seen through because the foreground object is translucent. α represents the two effects in an integrated manner, and the two effects are not distinguished in foreground extraction. This α (x) is called a foreground mask or matte.

The purpose of foreground extraction is to obtain a mat α (x), foreground color F (x), and background color B (x) from a given input image I (x). If this is achieved, an image I ′ (x) obtained by synthesizing the foreground part with the new background B ′ (x) is obtained by the following equation.

In the foreground extraction problem, the equation (1) alone has too few constraints on the number of values to be obtained, and there are an infinite number of solutions. Particularly in the case of an RGB image, the number of constraint equations for each pixel x is three (because it is a three-dimensional vector), whereas the unknowns are seven of α, F, and B (F , B are three-dimensional vectors). Therefore, in this embodiment, the solution space is significantly narrowed by using a plurality of images and image blur.

<< Flow of image processing method >>
Next, the flow of the image processing method based on the above concept will be described with reference to FIG. FIG. 3 is a flowchart showing a process for obtaining the composite image 41 from the input image in the image processing apparatus 3 according to the present embodiment.

  As shown in the drawing, first, the image processing device 3 receives a plurality of image data obtained by blurring the background and photographing the target object from different angles as an input image (step S10).

  Then, the blur estimation unit 11 estimates blur for a plurality of input images. Thereby, a blur amount 30 indicating how much the periphery of each pixel of each input image is blurred is obtained (step S11).

  Next, the region dividing unit 12 performs region division using a blur amount 30 for a plurality of input images to obtain a trimap (step S12). That is, the region dividing unit 12 classifies each pixel in each input image into one of three regions: “a region that is definitely a background”, “a region that is definitely a foreground”, and “an unknown region”.

  Next, the α value estimating unit 16 estimates an initial value of the α value from the trimap 31 obtained in step S12 (step S13). Thereby, the mask image 36 is obtained.

  Next, the background corresponding point calculation unit 13 and the foreground corresponding point calculation unit 14 calculate the corresponding positional relationship between the plurality of input images using the mask image and the input image (step S14). That is, when two images are selected from the input image to create a pair, corresponding points between the foreground region and the background region are calculated for each pair. In other words, it is calculated which pixel in the region that is the background reliably in one input image corresponds to which pixel in the region that is the background in the other input image. The same applies to the foreground, and it is calculated which pixel in the region that is surely the foreground in one input image corresponds to the pixel in the region that is definitely the foreground in the other input image. It should be noted that there is a case where there is no “area that is definitely the foreground” at the stage where the process of step S13 is performed. In the present embodiment, such a case will be described in detail below, and a method for obtaining “a region that is surely a foreground” in step S13 will be described in a third embodiment. Since the amount of movement of the object differs between the foreground and the background from the shooting conditions of the input image, two of the foreground corresponding point and the background corresponding point are calculated. From this calculation result, a foreground warp function 33 and a background warp function 32 indicating how much the pixels in one input image have moved in the other input image are obtained (step S15).

  Next, the foreground color estimation unit 15 estimates the foreground color 35 and the background color 34 in each input image based on the currently obtained foreground warp function 33, background warp function 32, and mask image (step) S16).

  Next, the α value estimation unit 16 estimates the α value in each input image based on the foreground and background colors obtained in step S16 and the foreground warp function obtained in step S15, and creates a mask image 36. Obtain (step S17). That is, the α value obtained so far is updated.

  Subsequently, the α value estimation unit 16 determines whether or not the α value has converged (step S18). That is, it is determined whether or not the change amount of the α value is sufficiently small before and after the update of the α value in step S17. If the amount of change is large, it is determined that the α value has not yet converged (NO in step S19), and the process returns to step S14. Conversely, if the amount of change is sufficiently small, it is determined that the α value has converged (step S19, YES), and the process proceeds to step S20.

  In step S20, the synthesizing unit 22 synthesizes the foreground of the input image and the new background 40 using the obtained foreground color and the mask image. Thereby, the composite image 41 is obtained.

<< Details of each step >>
Next, details of each step in FIG. 3 will be described. Hereinafter, each of M (M is a natural number of 2 or more) input images will be referred to as an input image Ii (x) (i is one of 0 to (M-1)).

(Step S11: blur estimation)
First, the blur estimation in step S11 will be described. Since the input image Ii (x) was taken with the foreground focused and the background blurred, it is considered that only the background portion Bi (x) is blurred. The blur due to the camera lens can be modeled by convolution with a blur function h (x; r) similar to the shape of the opening of the camera. r represents the size of the blur function and corresponds to a blur amount of 30. Therefore, when the background color not including blur is Ci (x), Equation (1) can be rewritten as the following equation.

Here, “*” represents convolution. Equation (3) is an equation that assumes that the amount of blur is constant over the entire area of the background.However, if the amount of blur changes but is smooth, it can be locally expressed by Equation (3). Can be modeled. In the present embodiment, since the background is sufficiently far away, the blur amount change may be considered to be smooth.

  It should be noted that non-uniform blur estimation needs to be performed because there is a blurred region and a non-blurred region in the image and the amount of blur may not be uniform. However, blur cannot be determined by only one pixel. Therefore, a small window (usually a square) is set around each pixel x, and estimation is performed assuming a uniform blur for the window. This is shown in FIG.

  FIG. 4 is a schematic diagram of an input image, and is a diagram showing a window set in blur estimation. As shown in the drawing, a window WND (WND1, WND2, and WND3 are shown in FIG. 4) having a predetermined area is set around the pixel of interest (in FIG. 4, x = x1, x2, and x3 are shown). . As a blur estimation method, for example, a method using a local Fourier transform can be used.

  As shown in Equation (3), in order to be able to assume a uniform blur h (x; r) in the window, all the windows must be α = 1 (all background) (FIG. 4). In window WND1). The blur estimation method extracts the influence of the blur function h (x; r) on the frequency component of the image in the window. Therefore, when α changes like the window WND3 and the influence of the foreground is included, the effect of the blur function is concealed. Accordingly, the area of the window WND3 is estimated to have no blur as in the area of the window WND2 that is the foreground part.

(Step S12: area division)
Next, the area division in step S12 will be described.

  The region dividing unit 12 classifies the region estimated to be blurred by blur estimation as “a region that is definitely a background”. As described in the blur estimation, it is determined that there is no blur at the point where the window overlaps the region of α> 0. Therefore, the “region that is definitely the background” is a conservative estimate that is separated from the foreground object by the size of the window. This is shown in FIG.

  FIG. 5 is a schematic diagram of the input image, showing the result of the area division, and the hatched area is the area that reliably becomes the background. As shown in the figure, the edge of the region that is surely the background is separated from the contour portion of the foreground object by approximately the size of the window.

  The area dividing unit 12 sets an area other than “certainly background” as an “unknown area”. Accordingly, in FIG. 5, all of the areas not hatched are “unknown areas”. In this way, a trimap as shown in FIG. 5 is obtained. Note that a “trimap” is an image that includes three areas, but since there is no “area that is definitely the foreground” at this point, “area that is definitely the background” and “area that is unknown” There are only two areas. The “region that is surely the foreground” appears by the iterative calculation in the processing after step S14.

  A specific example of step S12 will be described with reference to FIGS. FIG. 6 is a schematic diagram of the input images I1 (i = 1) and I2 (i = 2), and FIG. 7 is a schematic diagram showing a trimap of the input images I1 and I2.

  As shown in FIG. 6, it is assumed that, for example, pictures of a house are taken from two positions against a mountain background. At this time, the foreground house is in focus, and the background is mountains, clouds, and sky. Then, the trimap obtained from FIG. 6 is as shown in FIG. That is, an area expanded from the outline of the house by the window size used at the time of blur estimation is set as “unknown area”, and the other areas are set as “reliably background (hatched area)”. FIG. 7 schematically shows a state where the “reliable background” area is set on the entire surface except for the periphery of the house. However, the blur cannot be estimated due to the absence of a pattern (texture) in the background. There may be areas that remain as “unknown”.

(Step S13: Estimation of α value initial value)
Next, estimation of the initial value of α value in step S13 will be described.

  The α value estimation unit 16 generates an initial value of the α value using the trimap obtained in step S12. The α value estimation unit 16 sets α = 0 for “a region that is surely a background” and α = 0.5 for “an unknown region”. The resulting mask image is shown in FIG. FIG. 8 is a schematic diagram of a mask image obtained from the trimap shown in FIG. As shown in the figure, the area expanded by the window size used at the time of blur estimation from the outline of the house is set to α = 0.5, and the other areas are set to α = 0.0 “reliably background (area with hatching). "

(Steps S14 and S15: corresponding point calculation and warp function calculation)
Next, the corresponding point calculation in step S14 will be described. Hereinafter, a pair of input images will be referred to as Ii and Ij, respectively. As described above, both are functions of x.

In step S14, the coordinates x on the input image Ii correspond to the coordinates x on the input image Ii for each of the foreground colors Fi and Fj and the background colors Bi and Bj included in the pairs Ii and Ij of each input image. Vij (x) and Wij (x) are calculated. This calculation is performed by the foreground corresponding point calculation unit 14 for the foreground and the background corresponding point calculation unit 13 for the background. Vij (x) can be considered as a warp function (foreground warp function 33) when transforming the foreground Fi to match the foreground Fj. Wij (x) can be considered as a warp function (background warp function 32) that deforms the background Bi to match the background Bj. Therefore, the following relationship is established.

Warp functions Vji and Wji from Ij to Ii can be defined similarly.

  This will be described with reference to a specific example of FIG. FIG. 9 is a schematic diagram of the input images I1 (i = 1) and I2 (j = 2), and illustrates how the correspondence between the foreground and the background is obtained. As shown in the figure, for example, it is calculated which point of the input image I2 corresponds to the point x1 with the house which is the foreground in the input image I1. At this time, the pixel x1 corresponds to the pixel V12 (x1) calculated by the foreground warp function V12. The same applies to the background. As shown in the figure, for example, it is calculated which point of the input image I2 corresponds to a point x0 having a mountain as a background in the input image I1. At this time, the pixel x0 corresponds to the pixel W12 (x0) calculated by the background warp function W12.

First, how to obtain the background warp function Wij will be described. For this, an optical flow method is used to determine Wij that minimizes the following equation Eo.

The first term of equation (6) means to satisfy equation (5) as much as possible. However, since there are many candidates for corresponding points only with information that the colors are similar, the second term imposes a constraint that Wij is smooth. Here, N (x) represents a set of pixels in the vicinity of the pixel x. For example, it is assumed that there are 4 neighboring vertical, horizontal, and 8 neighboring areas including diagonally. Decreasing the second term means that the value of Wij (x) is small compared to the value Wij (y) in the vicinity y of x, that is, smooth. λo is a parameter that adjusts the strength of the smoothness constraint. Since the background is far enough, the warp function is considered very smooth, so λo may be large. As a result, the solution can be obtained more stably.

In equation (6), it is assumed that the backgrounds Bi and Bj are known. However, only the input images Ii and Ij are actually given. Therefore, Equation (6) is modified as follows so that it can be applied to the input image pairs Ii and Ij, and weighting is performed according to the ratio of the background in the input image.

The newly introduced S (α) is set to 1 when α = 0, ie, the background. On the other hand, when α is larger than a certain threshold value, the ratio of the foreground is large, so that it cannot be used for the calculation of the corresponding points of the background. That is, a weighting function.

  An example of the weighting function S (α) is shown in FIG. As shown in the figure, the function S (α) takes a maximum value of 1 when α = 0, and decreases as α increases. Thus, equation (5) is considered only when the background ratio is high at both the point x on the input image Ii and the corresponding point Wij (x) on Ij. However, since the desired variable Wij appears as S (αj (Wij (x))) in S (α), it may be difficult to minimize Equation (7) at a time. In such a case, S (αj (Wij (x))) may be fixed by an estimate of the current Wij (x). As a result, like the equation (6), it can be solved by the conventional optical flow method. When the solution is obtained, S (αj (Wij (x))) is updated and repeated to converge.

  Next, how to obtain the foreground warp function Vij (x) will be described. When the “foreground” region is obtained at the time of the region division in step S12 (as described above, this case is a situation that occurs in the third embodiment. In the case where a region having a large α value is obtained by performing the processing up to step S19 once or a plurality of times, the same method as the background warp function can be obtained. However, instead of S (α), the weighting function uses a function T (α) having a larger weight for the foreground. An example of the weighting function T (α) is shown in FIG. As shown in the figure, the function T (α) increases as α increases, and takes a maximum value of 1 when α = 1.

  In the case where the “region for surely foreground” is not obtained at the time of step S12, the following is performed. As described with reference to FIG. 5, the “unknown region” surrounds the foreground object with a gap of approximately the estimated blur window size. Therefore, if the amount of movement from the “unknown area” in Ii to the “unknown area” in Ij is obtained, a rough estimate of the foreground warp function can be obtained.

  Further, a high-pass filter that removes low-frequency components may be applied to the input images Ii and Ij to reduce the influence of the background. For this purpose, for example, a weighting function T ′ (α) shown in FIG. 12 may be used. As shown in the figure, the function T ′ (α) is a function that does not set the weight to 0 even when the value of α is smaller than the function T (α).

  Note that a high-pass filter can be applied even when a region having a large α value is obtained. However, in this case, since the low-frequency component of the foreground is ignored, the accuracy of the corresponding point calculation may be reduced. Therefore, the following two-stage process is performed. First, Vij is obtained using the weighting function T (α) with the input images Ii and Ij as they are. Next, Vij (x) in a region with a high foreground ratio such that T (α (x))> 0 is fixed. For the remaining regions, Vij is updated using the weighting function T '(α) for Ii and Ij subjected to the high-pass filter.

(Step S16: Foreground / Background Color Estimation)
Next, foreground / background color estimation in step S16 will be described. This process is performed by the foreground / background color estimation unit 15.

The foreground / background color estimation unit 15 estimates the foreground color Fi and the background color Bi of each image from the foreground and background corresponding point information (warp functions) Vij and Wij and the estimated α value αi (x) of each image. . Specifically, Fi, Bi (i = 0, 1,... (M−1)) that minimizes the following expression Ec is obtained.

Equation (8) represents that the cost function Ec to be minimized is the sum of the cost functions Ec, i for the i-th image and is optimized for all images. The first term in equation (9) means that the matting equation (1) is satisfied for the i-th image. This term is weighted by the parameter λm. The second and third terms require that Fi and Bi be smooth. The overall smoothness is adjusted by parameters λf and λb, and the local smoothness is adjusted by Uf, i and Ub, i. For example, the following formula is used to adjust local smoothness.

Expression (10) is an expression based on the fact that it is considered that the change in both the foreground and the background color is small in the region where the change of α is large, that is, in the transition part from the foreground to the background.

  The fourth and fifth terms of Equation (9) require that the foreground and background are similar in color between corresponding points, respectively. For the i-th image of interest, since all the images j except the i-th have corresponding points, they are summed up. The influence of these terms is determined by the parameters κf and κb.

  In equation (9), the variables (unknown numbers) to be calculated are Fi (x) and Bi (x) in the “unknown” region of the trimap. In the “certain foreground” area, Fi (x) = Ii (x) is fixed, and Bi (x) is undefined. In the “definite background” area, Bi (x) = Ii (x) is fixed, and Fi (x) is undefined. If the corresponding points Vij (x) or Wij (x) are coordinates outside the image in the fourth and fifth terms, and if you refer to an undefined foreground / background color, include that term in the formula Make it not exist.

  At the stage where the current α value is the initial value obtained in step S13, the estimation of the α value is not yet reliable. In this case, λm is decreased to reduce the effect of the expression including α. For the local weighting of the smooth constraint, an expression where α does not appear, such as Uf, i (x, y) = Ub, i (x, y) = 1, is used. More simply, without using Equation (8), the foreground color interpolates the color of the other area from the color of the `` definitely foreground '' area, and the background color of the other area from the color of the `` definitely background '' area Colors may be interpolated. If there is no “certain foreground” area, the foreground color is initialized with the input image itself.

  Since equation (8) is quadratic with respect to the unknowns Fi (x) and Bi (x), it can be solved by the least-squares method. In order to further narrow the solution space, it is also possible to minimize Equation (8) by imposing constraints of 0 ≦ Fi (x) and Bi (x) ≦ 1. In that case, quadratic programming may be used.

  To further narrow the solution space, the possible values of Fi (x) and Bi (x) pairs at each pixel x are K discrete colors (K is a natural number of 1 or more) sampled from around x. Equation (8) can be minimized by assuming that only the pair {Fi, k (x), Bi, k (x)} (k = 0, 1,... (K-1)). The target area for sampling the foreground color for the pixel x of the i-th image is the “definitely foreground” area around the coordinate x of the i-th image and the coordinates Vij (x of the j (≠ i) -th image. ) Is a "certainly foreground" area. The target area for sampling the background color is a “reliably background” area around the coordinate x of the i-th image and a “reliably background” area around the coordinate Wij (x) of the j (≠ i) -th image. It is. This is shown in FIG. FIG. 13 is a schematic diagram of a trimap of the input images I1 and I2.

  As shown in the figure, the target area for sampling the foreground color for the pixel x1 of the input image I1 is a “certainly foreground” area around the coordinate x1 of the input image I1, and a foreground corresponding point Vij (x ) Is a "certainly foreground" area. The target area for sampling the background color of x1 is a “reliably background” area around the coordinate x1 of the input image I1 and a “reliably background” area around the coordinate Wij (x) of the input image I2.

  When an appropriate number of foreground / background color samples are obtained, a pair is formed by combining the foreground color and the background color, and the top K pairs having the highest importance are taken. Then, assuming that the range of possible values of {Fi (x), Bi (x)} is the K pairs, Equation (8) is minimized. That is, each pixel x can take only K states. For the optimization of such conditions, a probability propagation method may be used.

Note that the importance of a pair may be as follows, or a combination thereof.
・ Distance Gd with sampling point
・ Fitness Gb to linear mixed model
The importance based on the distance to the sampling point is such that the closer the foreground sample point xf and the background sample point xb are to the point of interest x, the more important those samples are, and can be expressed by the following equation.

Where Df = | x − xf | (xf is a point on image i)
Df = | Vij (x) −xf | (xf is a point on the image j)
Db = | x−xb | (xb is a point on the image i)
Db = | Wij (x) −xb | (xb is a point on the image j)
The importance based on the degree of conformity to the linear mixed model is that the α value estimated from the foreground sample color F, the background sample color B, and the color Ii (x) of the pixel of interest (referred to as α *) is a matting equation. Those samples are important enough to satisfy (1), and can be expressed by the following equation.

However, α * = (Ii (x) −B) · (F−B) / | F−B | ² . “·” Represents an inner product of vectors.

(Step S17: α value estimation)
Next, the α value estimation process in step S17 will be described. This process is performed in the α value estimation unit 16.

The α value estimation unit 16 estimates the mat αi of each image from the foreground corresponding point information (warp function) Vij and the foreground color estimates Fi and Bi of each image. Specifically, αi (i = 0, 1,... (M−1)) that minimizes the following expression Ea is obtained.

Equation (13) represents that the cost function Ea to be minimized is the sum of the cost functions Ea, i for the i-th image and is optimized for all images. The first term in equation (14) means that the matting equation (1) is satisfied for the i-th image. The second term requires that αi be smooth. The overall smoothness is adjusted by the parameter λa, and the local smoothness is adjusted by Ua, i. For example, the following formula is used to adjust local smoothness.

Expression (13) is an expression based on the fact that the α value is considered to be small in a region where the change of the input image is small.

Also, using the fact that the background is blurred, even if the change in the input image is small between adjacent pixels, if the change is different from the surroundings, equation (15) was modified to reduce the weight The following formula is also conceivable.

Where sigmoid (t) = 1 / (1 + exp {−σt}) is a sigmoid function, z = y − x, ω is an offset parameter, and σ is a scale parameter. Since the image changes smoothly in the blurred background, the image difference Ii (x) − Ii (y) between the adjacent pixels x and y is more distant from the pixels xz and y + z. It is expected that the ratio Ii (xz) −Ii (y + z) of the difference between the two is greater than a certain value ω (about 2). Where this is not satisfied, there is a possibility of a boundary between the foreground and the background, and the sigmoid function reduces the smoothness constraint.

  The third term in equation (14) requires that the α value be close to the foreground corresponding points. For the i-th image of interest, since all the images j except the i-th have corresponding points, they are summed up. The influence of these terms is determined by the parameter κa.

  The fourth term of Equation (14) is effective when it is known from some information that the pixel x is likely to be the foreground. By setting the weight Rf, i (x) to be large, the estimated value of α can be biased in a direction approaching 1. The overall influence of the fourth term is adjusted by the parameter γf. Conversely, the fifth term is to bias the estimated value of α toward 0 by increasing the weight Rb, i (x) when it is known that the pixel x is likely to be the background. Can do. The overall influence of the fifth term is adjusted by the parameter γb.

For example, if it is known in advance that the color of the foreground object is brighter than the background color, Rf, i (x) is increased when the luminance of the pixel x is high, and Rb, i when the luminance is low. Increase (x). By this. The estimation accuracy may be improved. In addition, if a plurality of input images are similar in a certain range surrounding the foreground corresponding point, it is considered that there is a high possibility that the input image is the foreground. The same applies to the background. Rf, i (x) and Rb, i (x) are given by the following equations.

However, L is a set of points included in a circle or square of an appropriate size centered on the origin.

  In equation (14), the variable (unknown number) to be obtained is αi (x) in the “unknown” region of the trimap. In the “definitely foreground” region, αi (x) = 1 is fixed, and in the “definitely background” region, αi (x) = 0 is fixed. When the corresponding point Vij (x) is a coordinate outside the image in the third term, the term is not included in the equation.

  Since Equation (13) is quadratic with respect to αi (x), which is an unknown, it can be solved by a least-squares method. In order to further narrow the solution space, the constraint of 0 ≦ αi (x) ≦ 1 can be imposed to minimize Equation (13). In that case, quadratic programming may be used.

  It can be considered that the estimation of the α value minimizes the expression (Ec + Ea) obtained by integrating the expressions (8) and (13) together with the foreground / background color estimation. If you try to optimize Fi, Bi, and αi at the same time, you will not be able to use the least-squares method or quadratic programming because it is not a quadratic equation, but you should optimize it using gradient descent, etc. Can do.

(Steps S18 and S19: Determination of convergence of α value)
Next, α value convergence determination in step S18 will be described. This process is performed in the α value estimation unit 16.

The α value estimation unit 16 determines that the convergence has been achieved if the update amount of the α value by the α value estimation is sufficiently small. When the α value before update is αi (x) (i = 0, 1,... (M−1)) and after update is α ⁺ i (x), the following equation is satisfied for a certain threshold θ. Suppose that it has converged.

Equation (19) means that the change in the value of α is small before and after the update at all points of all images.

If the α value has not converged, the process returns to the corresponding point calculation in step S14. At this time, the trimap is updated as follows. A point where the updated α value is estimated to be sufficiently close to 1, that is, x satisfying α ⁺ i (x)> (1-ε) for a small ε is newly included in the “certainly foreground” region. . A point where the updated α value is estimated to be sufficiently close to 0, that is, x satisfying α ⁺ i (x) <ε is newly included in the “certainly background” region.

  As described above, α values for a plurality of input images are obtained, and a mask image 36 is obtained for each of them. FIG. 14 shows mask images for the input images I1 and I2 shown in FIG. As shown in the drawing, the foreground house portion has α = 1.0, and the other regions have α = 0.0. FIG. 14 shows a case where the outline of the house is clear. Therefore, there is no region where 0 <α <1, but this is only for the sake of simplicity.

(Step S20: Composition with new background)
Next, the synthesis process in step S20 will be described.

  The combining process in step S20 is performed in the combining unit 22. The synthesis unit 22 reads the new background 40 from the memory 21 and obtains a new background color B ′ (x). Then, by substituting the foreground color F (x), the mat α (x) and the new background color B ′ (x) obtained in Expression (22) into Expression (2), the composite image I ′ (x ) This is shown in FIG.

  FIG. 15 is a schematic diagram showing how the new background 40 and the foreground of the input image I1 are combined. As shown in the figure, for example, the new background 40 is assumed to be an image of the earth viewed from the moon. By combining the foreground of the input image I1 with this, a composite image in which the house in the input image I1 is on the moon as shown in the figure is obtained. Of course, the foreground in the input image I2 may be used.

<Effect>
As described above, the image processing apparatus and the image processing method according to the first embodiment of the present invention can achieve the following effect (1).

(1) A foreground can be extracted from image data by a simple technique.
In the method according to the present embodiment, a plurality of pieces of image data are used for foreground extraction. As this image data, image data photographed by focusing on the same target object and shifting the focus of the background of the target object is used. Then, the initial value of the α value is estimated by estimating the degree of blur for each image data.

  That is, the foreground is extracted by estimating the difference in blur between the target object that is the foreground and the background. Therefore, calibration is not necessary for the camera used for photographing, and high-precision foreground extraction can be performed in a very simple photographing environment. Further, it is particularly effective when extracting a complex object with a silhouette such as hair or fur or an object with a translucent part. An example in such a case will be described with reference to FIGS. 16A, 16B, 17A, 17B, and 18. FIG.

  FIGS. 16A and 16B are photographs taken of the same foreground from the left side and the right side. 17A is a final mask image corresponding to FIG. 16A, and FIG. 17B is a foreground color image extracted by the mask image of FIG. 17A. FIG. 18 is a composite image obtained by combining the foreground color of FIG. 17B with a new background using the mask image of FIG. Assume that the foreground object is, for example, a sheep (stuffed animal) as shown in FIGS. Then, the outline of the foreground becomes a region where the foreground and the background are mixed by the sheep's hair. According to the conventional method, it is very difficult to extract an object having such a complicated contour. However, in the method according to the present embodiment, blur is extracted and the α value is estimated, so that the foreground can be extracted with high accuracy as shown in FIGS.

  As described above, according to the first embodiment of the present invention, even a complex object can be extracted with high accuracy from a plurality of images taken by an uncalibrated camera. In this case, manual information input as described in the background art is also unnecessary. Therefore, an object can be extracted by a very simple method.

[Second Embodiment]
Next, an image processing apparatus and an image processing method according to the second embodiment of the present invention will be described. This embodiment obtains a foreground viewed from an intermediate viewpoint by performing interpolation on the foreground obtained for a plurality of input images in the first embodiment. Hereinafter, only differences from the first embodiment will be described.

<Configuration of Image Processing Device 3>
FIG. 19 is a block diagram of the image processing apparatus 3 according to the present embodiment. As shown in the figure, the image processing apparatus 3 according to this embodiment further includes a foreground interpolation unit 23 in the configuration of FIG. 2 described in the first embodiment.

  The foreground interpolation unit 23 uses the foreground warp function 33, the foreground color 35, and the mask image 36 to obtain a foreground color interpolation image 42 and an α value interpolation image 43. The foreground color interpolation image 42 and the α value interpolation image 43 are obtained when the foreground object (hereinafter sometimes referred to as an intermediate foreground) is viewed from an intermediate position of a plurality of viewpoints with respect to a plurality of foreground objects. Foreground color and α value.

  The synthesizing unit 22 does not directly use the mask image 36 and the foreground color 35, but uses the foreground color interpolation image 42 and the α value interpolation image 43 obtained by the foreground interpolation unit 23 to generate an intermediate foreground image. Synthesis with the new background 40 is performed.

<Operation of Image Processing Device 3>
Next, the flow of the image processing method based on the above concept will be described with reference to FIG. FIG. 20 is a flowchart showing processing for obtaining the composite image 41 from the input image in the image processing apparatus 3 according to the present embodiment.

  As shown in the drawing, after the steps S10 to S18 described in the first embodiment, if the α value converges (step S19, YES), the process of step S21 is performed. In step S <b> 21, the foreground interpolation unit 23 calculates the foreground color interpolation image 42 and the α value interpolation image 43. Thereafter, the synthesis unit 22 performs synthesis of the intermediate foreground and the new background 40 (step S22).

(Step S21: Foreground interpolation)
Next, details of the foreground interpolation processing in step S21 will be described. Foreground interpolation is performed in the foreground interpolation unit 23.

The foreground interpolation unit 23 interpolates a plurality of foreground colors Fi (x) and mats αi (x) obtained in steps S10 to S18 to generate intermediate foreground information F (x) and α (x). If the contribution rate of the i-th image is βi (Σi {βi} = 1), the foreground interpolation unit 23 first interpolates the warp function 33 of each image with the weight of this contribution rate.

However, warping to itself is the identity transformation Vii (x) = x. Next, using this interpolated warp function Vi (x), Fi (x) and αi (x) are warped. In this process, an inverse function Vi ⁻¹ (x) of Vi (x) is used as follows.

Finally, the foreground information obtained by interpolating the warped foreground information according to the contribution ratio is obtained.

In the image processing system 1 shown in FIG. 1, a plurality of input images correspond to a plurality of viewpoints. Then, if the three-dimensional coordinate at the viewpoint i of the input image Fi is Xi, the foreground information shown in the equation (22) corresponds to the foreground viewed from the viewpoint X below.

Thus, the interpolated foreground color F (x) and the interpolated α value α (x) are obtained.

<Effect>
As described above, the image processing apparatus and the image processing method according to the second embodiment of the present invention have the following effect (2) in addition to the effect (1) described in the first embodiment. can get.

(2) A new foreground viewed from an arbitrary viewpoint can be obtained.
In the configuration according to the present embodiment, the image processing apparatus 3 includes a foreground interpolation unit 23. Then, the foreground interpolation unit 23, based on the plurality of foregrounds obtained by the foreground / background color estimation unit 15 for a plurality of input images, the foreground foreground (foreground color interpolation image 42) viewed from an arbitrary viewpoint, and corresponding to it The calculated α value (α value interpolated image 43) is calculated. Therefore, a foreground viewed from an arbitrary viewpoint can be obtained. That is, a new foreground that is not present in the input image can be extracted.

  This situation is shown in FIG. FIG. 21 is a schematic diagram showing how a new foreground is calculated using the foreground extracted from two input images. As shown in the figure, the two input images are the same as those in FIG. 6 described above. The input image I1 is an image of the house that is the foreground, taken from the left side of the drawing, and the input image I2 is taken from the right side. It is what. From these two house images, the foreground interpolation unit 23 can obtain, for example, a house image viewed from the front.

[Third Embodiment]
Next, an image processing apparatus and an image processing method according to the third embodiment of the present invention will be described. In this embodiment, in the first and second embodiments, when a trimap is obtained, information regarding a region that is surely a foreground is given by the user. Hereinafter, only differences from the first and second embodiments will be described.

  In the configuration according to the present embodiment, the region dividing unit 12 holds the upper limit distance d1 from the background region of the mixed region. This information is, for example, information given in advance by the user. The upper limit distance d1 will be described with reference to FIG. FIG. 22 is a trimap relating to an input image obtained by photographing a foreground object.

  As shown in the figure, depending on the input image, there is a case where a region that is at least a certain distance away from the end of the region that is reliably the background is surely the foreground. In other words, the region from the end of the region that is surely the background to the certain distance satisfies 0 <α <1. This is the mixing region. In some cases, the user knows that there is an area that is surely the foreground. In such a case, the user gives the fixed distance d1 to the image processing apparatus 3, and the area dividing unit 12 holds the fixed distance d1.

  Then, the area dividing unit 12 reliably obtains the area as the background by the method described in the first embodiment, and then reliably identifies the area inside the “unknown area” by the distance d1 from the area. It becomes an area to become. Then, an area between a surely background area and a surely foreground area is set as an unknown area. Then, in the process of step S13 in FIG. 13, the region dividing unit 12 sets α = 0 for the “certainly background region”, sets α = 1 for the “certainly foreground region”, and sets the “unknown region”. Let α = 0.5.

  In addition, if this method is used, “a region that is surely a foreground” has already been obtained in the first step of steps S14 and S15 that are repeated a plurality of times. Therefore, as described in the first embodiment, the foreground warp function can be obtained in the same manner as the background warp function. Further, the function T (α) can be used as the weighting function.

<Effect>
In the method according to this embodiment, the following effect (3) is obtained in addition to the effects (1) and (2) described in the first and second embodiments.

(3) The processing speed related to foreground extraction can be improved.
In the present embodiment, after blur estimation, the region dividing unit 12 can reliably grasp the region that is the foreground. Accordingly, at the initial value stage of the α value, the ratio of the “unknown area” in the entire image is very small. Therefore, it is possible to reduce the load on the foreground / background color estimation unit 15 and the α value estimation unit 16 and improve the foreground extraction speed.

[Fourth Embodiment]
Next, an image processing apparatus and an image processing method according to the fourth embodiment of the present invention will be described. The present embodiment relates to details of the blur estimation method in the first to third embodiments. Hereinafter, only differences from the first to third embodiments will be described.

  In this embodiment, the blur estimation unit 11 performs different blur removal for each input image, performs region division according to the image content of the input image, and performs local blur estimation for each region. Then, the images from which the proper blur removal has been performed for each region are integrated. FIG. 23 is a block diagram of the blur estimation unit 11 according to the present embodiment.

  As illustrated, the blur estimation unit 11 includes a PSF model storage device 103, a deconvolution unit 104, a blur removal image storage device 105, a local blur parameter estimation unit 106, a local estimation blur parameter storage device 107, an image integration unit 108, and a restoration. An image storage device 109 is provided. Note that the blur parameter in the present embodiment corresponds to the blur amount 30 described in the above embodiment.

  The PSF model storage device 103 stores a PSF (blur function) 103a obtained by modeling a blur state, and a plurality of blur parameters 3b applied to the PSF 3a. The PSF 103a corresponds to the blur function h (x; r) described in the first embodiment.

  The deconvolution unit 104 generates, from the input image, the deblurred image data 151 to 15n from which the blur represented by the PSF 103a to which the blur parameter 103b corresponding to the step is applied is removed from the input image. The deblurred image data 151 to 15n corresponding to the stage is stored in the deblurred image storage device 105.

  The local blur parameter estimation unit 106 reads the input image, and the region division unit 161 of the local blur parameter estimation unit 106 divides the input image into a plurality of regions based on the content of the image such as color change. .

  The local blur parameter estimation unit 106 reads the blur-removed image data 151 to 15n corresponding to each stage, and the local blur estimation unit 162 of the local blur parameter estimation unit 106 performs the blur-removed image data 151 to 150 for each region. From 15n, data satisfying a predetermined appropriate condition is selected in the area, and the local estimation blur parameter 107a for specifying the selected data for each area is estimated.

  Then, the local blur parameter estimation unit 106 stores the local estimated blur parameter 107 a estimated for each region in the local estimated blur parameter storage device 107. This local estimation blur parameter corresponds to the blur amount 30 in each window described in the above embodiment.

  The image integration unit 108 reads the blur-removed image data 151 to 15n at each stage and the local estimated blur parameter 107a of each region. Then, for each area, the corresponding area data of the blur-removed image data 107a corresponding to the local estimated blur parameter 107a estimated for the area is extracted, and the restored area data 109a is combined with the extracted corresponding area data of each area. And the restored image data 109a is stored in the output image storage device 109.

Details of the configuration of the blur estimation unit 11 will be described below.
The deconvolution unit 104 includes a parameter changing unit 141 and a deconvolution processing unit 142. Further, the deconvolution processing unit 142 includes a differentiation unit 143, a differential domain deconvolution processing unit 144, and an integration unit 145. The deconvolution unit 104 performs deconvolution on the input image on the assumption that the blur is uniform. The PSF 103a is parameterized with respect to the magnitude of blur. The parameter changing unit 141 of the deconvolution unit 104 changes the parameter r of the PSF 103a according to the blur parameter 103b in a stepwise manner, for example, r = 0.5, 1.0, 1.5, 2.0,. To change.

  The deconvolution unit 104 performs deblurred image data indicating the result of deconvolution using the PSF 103a to which the blur parameter of r = 0.5 is applied, and reverses using the PSF 103a to which the blur parameter of r = 1.0 is applied. Deblurred image data 151 to 15n corresponding to each blur parameter is generated, such as deblurred image data indicating the result of the convolution.

  Various functions are used as the PSF 103a parameterized with respect to the magnitude of blur. The Gaussian distribution with the blur parameter as the standard deviation r can be a model of PSF in many fields such as MRI (Magnetic Resonance Imaging) and CT (Computed Tomography).

Equation (24) is a Gaussian distribution with the blur parameter as the standard deviation r. In the following, the pixel coordinate x (two-dimensional vector) described in the above embodiment is expressed as an (x, y) coordinate.

Equation (24) is generalized and expressed as Equation (25) in the frequency domain.

  Here, F [h] is a Fourier transform of PSF f and is called an optical transfer function (OTF).

  ξ and η are spatial frequencies. When b = 1, the expression (25) corresponds to the expression (24). When b = 5/6, the PSF is caused by atmospheric turbulence, and b = 1/2 is the PSF / OTF model caused by X-ray scattering.

Also, in the case of defocusing by a lens such as a camera, PSF can be expressed as in equation (26) using the blur radius as a blur parameter.

Also, an arbitrary PSF can be defined by using r as a scaling parameter for scaling a certain basic PSF q (x, y).

  The basic PSF is obtained by calibration, for example.

In the deconvolution process in the deconvolution processing unit 142, a degradation model of image data as shown in Expression (28) is used.

  Here, g, f, and n are the input image, the original clear image data, and noise, respectively, and the pixel values that have been discretized are arranged in the order of the dictionary and expressed as vectors.

H is a matrix representing a two-dimensional convolution by PSF h (x, y; r). Deconvolution corresponds to estimating the original image data f given the input image g and the blur matrix H, and is known to be a defect setting problem. When simply solving as a least squares problem that minimizes | g−Hf | ² , noise is magnified in the resulting image data even if an accurate blur matrix H is used.

In order to solve this noise expansion, prior knowledge about the original image data f is required. As prior knowledge, it is used that the distribution of the luminance gradient of natural image data is generally as shown by the solid line in FIG. The luminance gradient distribution of the natural image data has narrower vertices and wider bases than the Gaussian distribution (broken line) that is generally used. The distribution of the luminance gradient of the natural image data can be modeled as a generalized Laplacian distribution, for example, as shown in Expression (29).

The distribution of this equation (29) is called differential coefficient prior distribution. Since this prior knowledge is applied to the luminance gradient, that is, the differential coefficient, Equation (29) is differentiated to transform the image degradation model as Equation (30) and Equation (31).

  Here, the subscripts x and y represent differentiations in the respective directions. nx and ny represent noise in the differential region and are generally different from noise n in the original image region.

  gx and gy can be obtained by differentiating the deteriorated image data g.

From the expressions (30) and (31), an estimated value f ^ x and an estimated value f ^ y of the x derivative of the original image data are obtained, respectively. The estimated value f ^ of the original image data is obtained by integrating the estimated value f ^ x of the x derivative and the estimated value f ^ y of the y derivative. The integration can be done by solving the Poisson equation of Equation (32).

  The deconvolution processing unit 42 executes the processing shown in FIG.

  That is, the differentiation unit 143 of the deconvolution processing unit 42 executes a process 110a for differentiating the input image in the x direction and a process 110b for differentiating the input image in the y direction.

  Next, the differential domain deconvolution processing unit 144 of the deconvolution processing unit 42 executes processing 110c for performing deconvolution on the x-direction differentiation of the input image and processing 110d for performing deconvolution on the y-direction differentiation of the input image. . The processes 110c and 110d are executed step by step based on the stepwise blur parameters 103b and PSF 103a obtained by the parameter changing unit 141 and the differential coefficient prior distribution 111a stored in the storage device 111. The differential domain deconvolution processing unit 144 prompts the image data obtained as a result of the deconvolution processing to follow the differential coefficient prior distribution 111a (statistic distribution of the differential coefficient of the image) (so as to follow the differential coefficient prior distribution 111a as much as possible). Impose some constraints).

  Then, the integration unit 142 of the deconvolution processing unit 142 integrates the result of deconvolution to the x-direction derivative of the input image and the result of deconvolution to the y-direction derivative of the input image, and the blur parameter 103b A process 110e for obtaining the blur-removed image data 151 to 15n at each stage is executed.

  The deconvolution process represents deconvolution of the degradation model equations (30) and (31) of the image in the differentiated state.

  Since Equation (30) and Equation (31) can be handled in the same manner, the deconvolution of Equation (30) will be described below, and the description of the deconvolution of Equation (31) will be omitted.

Assuming that the noise n′x is Gaussian noise with variance σ ² , the conditional probability of the x differential fx of the original image data given the x differential gx of the deteriorated image data 2a is given by equation (33). .

  Here, p (.) Is the probability density function of the variable serving as an argument, and fx, i is the i-th pixel value of the x differential fx of the original image data. Further, it is assumed that the differential coefficient prior distribution can be applied to each pixel.

Considering a MAP estimation (Maximum A Posteriori estimate) with fx that maximizes Expression (33) as an estimated value, the position of the maximum value is invariant even when taking a logarithm (ln), and therefore Expression (34) is obtained.

  Equation (34) is reduced to a linear equation if the differential prior distribution p (fx, i) is a Gaussian distribution that is generally used. However, the Gaussian distribution has an effect of smoothing every part of the image data, and may not be suitable for removing blur and restoring clear image data. Therefore, in the present embodiment, the distribution indicated by the solid line in FIG. 24 is used.

  By using the solid line in FIG. 24, the equation (34) becomes a nonlinear optimization problem. In the present embodiment, the solid line in FIG. 24 is transformed into an iterative linear equation by the EM (Expectation Maximization) method to speed up the processing.

Below, the case where EM method is applied to differential image data is demonstrated. First, the differential coefficient prior distribution p (fx, i) is expressed as a superposition of Gaussians having an average of 0 and different variances. That is, in equation (35), p (fx, i | zi) is a Gaussian with an average 0 variance zi, and p (zi) is a probability distribution with variance zi.

When {EM} is applied assuming that {zi} is missing (unobservable), Equation (34) becomes an iterative estimate update equation such as Equation (36). The estimated value of fx is assumed to be f ⁽ⁿ⁾ x.

E ⁽ⁿ⁾ i [. ] Represents an expected value based on the probability distribution p (zi | f ⁽ⁿ⁾ x, i) of the variance zi for the i-th differential coefficient. The second term of equation (36) is expressed by equation (37) because p (fx, i | zi) is Gaussian.

However,

It is. For example, when the prior distribution model of the differential coefficient shown in Expression (29) is used, Expression (38) is specifically expressed as Expression (39).

Thus, Equation (36) becomes a quadratic equation with respect to fx, and can be maximized by solving the linear equation of Equation (40) with respect to fx.

Here, D ⁽ⁿ⁾ is a diagonal matrix having E ⁽ⁿ⁾ i [1 / zi] shown in Equation (37) as a diagonal component. H ^T represents the transpose of the blur matrix H.

An example of the deconvolution processes 10c and 10d for the differential image data of the input image in FIG. 25 is shown in FIG. That is, in the n-th iteration, the differential domain deconvolution processing unit 144 performs a matrix D ⁽ⁿ⁾ having a diagonal component calculated by the equation (38) based on the current estimated deblurred differential image data f ⁽ⁿ⁾ x. ⁾ is calculated (the coefficient of 40) (sigma ² D ⁽ⁿ⁾ linear equations ^{+ H} T H) and ^H T gx calculating the (step S1).

  Next, the differential domain deconvolution processing unit 144 solves the linear equation (40) (step S2).

Similarly, steps S1 and S2 are executed for new estimated blur-removed differential image data f ^{(n + 1)} x.

The differential region deconvolution processing unit 144 repeats the above process starting from the initial estimated blur removal differential image data f ⁽⁰⁾ x until the change in the estimated value becomes smaller than a predetermined value, and the final estimated value is removed by blurring. It outputs as differential image data (step S3).

For solving the linear equation, for example, a conjugate gradient method or a second-order stationary iterative method can be used. The main process during the solution is the calculation of the product of the blur matrix H and its transpose H ^T and the vector. The calculation of the product of the blur matrix H and its transpose H ^T and the vector can be performed at high speed using a fast Fourier transform.

As the initial estimated differential image data f ⁽⁰⁾ x, the differential data gx of the input image is most simply used.

The noise variance σ ² can be estimated from the differential data gx of the deteriorated image data. Further, σ ² may be regarded as a weighting coefficient of the prior distribution term in Expression (17), and an arbitrary value may be set. As the value of σ ² is larger, the noise of the restored image data is reduced. However, if the value is too large, the restored image data becomes unclear. A method of increasing the initial value of σ ² and decreasing it with repeated deconvolution calculations is effective in achieving both noise reduction and sharpness.

  As for the deconvolution processing by the differential domain deconvolution processing unit 144 as described above, the case where each pixel is a single-element grayscale image has been described. When deconvolution is performed on a color image, it can be applied by repeating the same process for the number of elements.

  In the image processing apparatus 1, while the processing by the deconvolution unit 104 is performed, the local blur parameter estimation unit 106 obtains r (x, y) which is the local estimation blur parameter 107 a at each coordinate (x, y).

  FIG. 27 shows an example of processing of the local blur parameter estimation unit 7a. In step T1, the local blur parameter estimation unit 106 divides the input image into a plurality of regions according to the image contents. This area is a window.

  In step T <b> 2, the local blur parameter estimation unit 106 obtains a value (divergence) that serves as an index of a good blur removal result for each region based on the plurality of blur removal image data 151 to 15 n. Here, the degree of divergence refers to the appearance of noise or a periodic pattern, as will be described later, and represents the degree of deviation from a general natural image as a result of blur removal. That is, in the present embodiment, for each region, a determination is made as to whether blur removal satisfies an appropriate condition depending on the state of divergence.

  In step T3, the local blur parameter estimation unit 106 estimates (selects) a blur parameter having a good blur removal result for each region based on the divergence.

  In step T4, the local blur parameter estimation unit 106 performs post-blurring of blur parameters for regions where blur parameters are not estimated or smoothing of blur parameters between regions based on the blur parameters estimated for each region. Execute the process.

  In the processing of the local blur parameter estimation unit 6, the input image may be divided into regions by collecting pixels having similar colors. FIG. 28 shows an example of the result of area division according to the contents of image data. The method of dividing the input image into regions is arbitrary, and for example, simple rectangular division may be used. By performing the division according to the content of the image, it is possible to improve the estimation accuracy of the blur parameter. The shape of the region is not limited. Where the area is too small, the number of pixels is small and a sufficient sample cannot be obtained, so it may be merged with the surrounding area, and the area may be excluded from the estimation target. An area that is too large may be subdivided appropriately because the locality of estimation becomes low. The reliability of the estimation may be improved by excluding an elongated shape region, a region where the change in pixel value is minute, a region where the pixel value is saturated, and the like.

Assuming that the divided area is Aj and the area of the entire input image is I, Expressions (41) and (42) hold.

  In each region Aj, blur estimation is performed assuming that the blur parameter is constant in that region. In the present embodiment, the blur parameter corresponding to the blur-removed image data that provides the best restoration result in the region is estimated from the plurality of blur-removed image data 151 to 15n. For example, there are a plurality of deblurred image data 151 to 15n that are deconvolved with the blur parameter r = 0.5, 1.0, 1.5, 2.0,..., And the true blur parameter in the attention area is r. When 1.6 = 1.6, it is considered that the deblurred image data obtained by deconvolution with r = 1.5 for the attention area among the plurality of deblurred image data 151 to 15n gives the best restoration result. Therefore, if the quality of the restoration result can be determined, the blur parameter in the attention area can be estimated as r = 1.5 in this example. Since the estimated value of the blur parameter is one of the stepwise blur parameters prepared in advance, it is necessary to prepare a blur parameter group according to the required accuracy.

The divergence of the image data is considered as a criterion for the goodness of the restoration result. Note that divergence means that a pattern having periodicity and messy noise that do not exist in the original image data appear in the blur-removed image data. The degree of divergence represents a pattern with periodicity that is not present in the original image data or the level of random noise. If the deblurred image data is f ^, the following expressions (43) to (46) It can be evaluated with such an index.

  Here, f ^ xx and f ^ yy are second-order derivatives with respect to x and y, respectively, of the deblurred image data. δ (.) is a function that returns 1 when the argument is true, and 0 when the argument is false, and [θmin, θmax] is a range of pixel values.

  When the true blur parameter is rt and the deconvolution is performed with the blur parameter r, the degree of divergence is relatively small in the deblurred image data when r ≦ rt. However, when r> rt, the divergence increases rapidly as r increases. The reason for this will be described below.

  As the blur parameter r is larger, the high-frequency component is attenuated in the image data blurred by the PSF at that time.

  For example, a graph in the frequency domain of the PSF in Expression (24) is obtained when b = 1 in Expression (25). FIG. 29 is an example in which the graph when b = 1 in this equation (25) is shown in one dimension for simplicity. As shown in the figure, it can be seen that as the blur parameter r increases, the attenuation of the high-frequency component increases. Deconvolution corresponds to a process of amplifying the attenuated frequency component. Therefore, the smaller the blur parameter, the larger the smaller frequency component is amplified.

  Therefore, when r <rt, since the degree of amplification by deconvolution is weaker than the degree of attenuation by the true blur parameter rt, the high-frequency component of the blur-removed image data is still smaller than the original image data.

  On the other hand, when r> rt, the degree of amplification by deconvolution is stronger than the degree of attenuation by the true blur parameter rt, so that the deblurred image data has a higher frequency component than the original image data. Diverge.

  When r = rt, ideally the filter is 1 everywhere and the original image data is restored. Actually, because of the influence of noise, even if r ≦ rt, the noise is amplified together and a certain degree of divergence occurs. However, when the noise is sufficiently smaller than the content of the image, when r> rt Small enough compared to the method of divergence.

In order to simplify the explanation, it is considered that the original image data f is blurred by a Gaussian with a blur parameter rt and deconvolved with a Gaussian with a blur parameter r, and a simple inverse filter ignoring noise. Is used to obtain the relationship of equation (47).

If exp {− (rt ² −r ² ) (ξ ² + η ² )} is considered as a filter in the frequency domain with respect to the original image data f in the equation (47), the graph of this filter also ignores η and is one-dimensional. In this case, FIG. 30 shows the relationship between r and rt. It can be seen that the high-frequency component of the blur-removed image data when r> rt is excessively amplified with respect to the original image data. This discussion is made for the case where b = 1 in the equation (25), but the same applies when the value of b is not 1.

A case where the defocused PSF of Expression (26) is used will be described below. Assuming a one-dimensional case for simplicity, Equation (48) is obtained.

The frequency conversion of h (x) is given by equation (49).

  This equation (49) is a graph shown in FIG. 31 and not only attenuates high frequency components but also periodically sets the frequency to 0 at ξ = mπ / r (m = ± 1, ± 2,...). It becomes a filter to do. For this reason, unlike the PSFs of the equations (24) and (25), excessive amplification occurs due to deconvolution even in the low frequency part.

Similarly to the above, it is assumed that the original image data f is deconvolved with the blur of the blur parameter r by defocusing the image data blurred by the blur of the blur parameter rt. In this case, since the denominator is 0 in a simple inverse filter, using a pseudo inverse filter to which a minute value ε is added, the relationship of Expression (50) is obtained.

The value at the point ξ = mπ / r where the value of equation (49) becomes 0 in equation (50) is given by equation (51).

  Since the frequency is divided by the minute value ε, the corresponding frequency component is greatly expanded. Since image data usually contains many low frequency components, the component of ξ = ± π / r where m = ± 1 is particularly divergent.

  When a graph is drawn using the value of the equation (51) when m = 1 as a function of the blur parameter r, it is as shown in FIG. It can be understood that the corresponding frequency component of the deblurred image data when r> rt is excessively amplified with respect to the original image data.

  From the above, the PSFs of the equations (24) and (25) and the PSF of the equation (26) differ in the manner of image data divergence when deconvolution is performed, but both are larger than the true blur parameter rt. The degree of divergence increases when deconvolution is performed with a blur parameter. A general PSF is in a similar state in many cases. Therefore, as shown in FIG. 27, first, in each region Aj, a divergence degree vj (r) is calculated for a plurality of deblurred image data deconvolved with the blur parameter r.

  In the present embodiment, the deblurred image data 151 to 15n obtained by the deconvolution unit 104 is used, but is separately generated by a simple deconvolution method using a pseudo-inverse filter or the like. It is good. The blur-removed image data necessary for blur parameter estimation need not have good image quality, and the blur-removed image data determined to be appropriate may diverge to some extent.

  Next, in each region, the blur-removed image data immediately before the image starts to diverge greatly is selected as the best restoration result. That is, r immediately before vj (r) starts to increase rapidly is taken as the estimated value of the blur parameter in this region.

  FIG. 33 shows an example of the relationship between the divergence degree and the blur parameter. For example, vj (0), that is, a threshold value based on the divergence of the input image itself is considered, and the maximum r that gives vj (r) that does not exceed this threshold value is selected. Good. A method of setting the threshold value by normalizing the divergence by the area of the region is also effective. Alternatively, for example, the maximum point of the second order derivative with respect to r of vj (r) may be selected.

  By smoothing vj (r) and estimating the parameters while suppressing the influence of noise, the determination accuracy may be improved while suppressing the influence of noise. Further, in the vicinity of r = r0, a straight line is fitted to the value of vj (r) when r <r0 and the value when r> r0, and r0 is set so that the angle of the two straight lines becomes maximum. You may choose.

  As a result of the above processing, the blur parameter rj is obtained for each region Aj. Therefore, the local estimated blur parameter r (x, y) can be obtained by connecting the blur parameter rj for each region Aj.

As post-processing of FIG. 27 described above, additional processing such as filling of estimated values and smoothing between regions is performed. First, when the region was divided, the region that was not estimated due to the region being too small, or the region that could not be reliably estimated because the change in image value within the region was very small and the change in divergence was small. Fill estimates based on surrounding area. For example, the Laplacian equation of Equation (52) is solved for the region Aj where there is no estimated value.

  Then, if the blur parameter greatly differs at the boundary of the region, there is a possibility that discontinuity appears in the final restored image data, and thus the local estimated blur parameter r (x, y) is smoothed.

Alternatively, the equation (53) may be solved for r (x, y) in the entire image region I, and hole filling and smoothing may be performed simultaneously.

  However, for (x, y) ∈ Aj, if there is an estimated value of the blur parameter in the region Aj, λ (x, y) = λc, e (x, y) = rj, and λ if there is no estimated value. (X, y) = 0 and e (x, y) = 0. The constant λc is a positive number indicating how much the blur parameter r (x, y) is adapted to the estimated value e (x, y) in each region. When λc is small, the effect of smoothing by Laplacian is relatively large. Become.

The image integration unit 108 integrates the plurality of blur-removed image data 151 to 15n generated by the deconvolution unit 104 based on r (x, y) of the blur parameter 7a estimated by the local blur parameter estimation unit 106. . Since each coordinate (x, y) is blurred by the blur parameter r (x, y), the blur closest to r (x, y) among the blur parameters changed in stages. The parameters r1 and r2 are selected (r1 ≦ r (x, y) ≦ r2), and the defocused image data f ^ 1 and f ^ 2 deconvolved with the parameters r1 and r2 are used to express the integrated image data f ^ Interpolate as shown in 54).

  By applying the above method to each input image, a blur amount 30 (blur parameter) for each input image can be obtained.

<Effect>
In the method according to this embodiment, in addition to the effects (1) to (3) described in the first to third embodiments, the following effect (4) can be obtained.

(4) The foreground extraction accuracy can be improved (part 1).
With the blur estimation method according to the present embodiment, the blur estimation accuracy can be improved. As a result, the foreground extraction accuracy can be improved. Regarding this effect, the blur estimation accuracy will be described below.

  In the method according to the present embodiment, blur estimation is performed locally. Therefore, the size of the blur need not be obtained in advance. Therefore, the size of the blur does not have to be uniform in the input image, and even when the blur of the input image changes relatively greatly, the blur can be removed or the blur state can be converted.

  In the present embodiment, the shape of region division for local blur estimation is not limited. Therefore, blur estimation along the image content of the image data can be performed, and the locality and accuracy of the estimation can be improved.

  Furthermore, in this embodiment, the input image is subjected to blur removal at a plurality of stages using PSF, and appropriate data is integrated for each region based on the plurality of blur removal image data 151 to 15n, and the restored image data 109a. Create Thereby, non-uniform blur removal can be performed by a process of performing multiple deconvolutions of image data by uniform PSF, and an efficient deconvolution method that hardly increases noise can be used. .

  In the present embodiment, the PSF only needs to be parameterized with respect to the magnitude of blur, and various types of PSF models can be used.

  In the present embodiment, the deconvolution process is performed on the luminance gradient obtained by differentiating the input image, and a prior distribution with respect to the luminance gradient of the input image is applied. Thereby, high-quality blur-removed image data can be obtained at high speed.

  In the present embodiment, the deconvolution unit 104 applies deconvolution to the differential image data of the input image in the differential domain deconvolution processing unit 144, and removes blur according to the differential coefficient prior distribution in the deconvolution. To generate first deconvolution data. Further, the deconvolution unit 104 may generate the second deconvolution data by removing blur that does not depend on the differential coefficient prior distribution by using a simple pseudo inverse filter for the input image. In this case, the integration unit 145 integrates the first deconvolution data to generate first deblurred image data. Since the second deconvolution data is not differentiated from the original, it becomes the second deblurred image data as it is without integration. Then, the local blur parameter estimation unit 106 generates a local estimation blur parameter 107a based on the second blur-removed image data, and the image integration unit 108 applies to each region based on the local estimation blur parameter 107a. The first blur-removed image data is integrated to generate restored image data 109a.

[Fifth Embodiment]
Next, an image processing apparatus and an image processing method according to the fifth embodiment of the present invention will be described. In the fourth embodiment, the restored image data 109a obtained by the blur estimation unit 11 is used for foreground and background alignment (corresponding point calculation) in the fourth embodiment. Hereinafter, only differences from the above embodiment will be described.

  FIG. 34 is a block diagram of the foreground extraction unit 10 according to the present embodiment. As illustrated, the background corresponding point calculation unit 13 and the foreground corresponding point calculation unit 14 perform alignment using an input image 37 having no blur instead of the input image. The input image 37 without blur is the restored image data 109a obtained by the image integration unit 108 of the blur estimation unit. Other configurations and operations are the same as those in the first to fourth embodiments, and a description thereof will be omitted.

<Effect>
In the image processing apparatus and the image processing method according to the present embodiment, in addition to the effects (1) to (4) described in the first to fourth embodiments, the following effect (5) can be obtained. .

(5) The foreground extraction accuracy can be improved (part 2).
In the configuration according to the present embodiment, the background and foreground are aligned using the input image 37 without blur. According to this, the accuracy of alignment can be improved, and as a result, the foreground extraction accuracy can be improved.

[Sixth Embodiment]
Next, an image processing apparatus and an image processing method according to the sixth embodiment of the present invention will be described. In this embodiment, blur estimation is performed by a method different from that of the fourth embodiment, and the method disclosed in Japanese Patent Application Laid-Open No. 2007-201533 is used.

  FIG. 35 is a block diagram of the blur estimation unit 11 according to the present embodiment. As shown in the figure, the blur estimation unit 11 includes an edge detection unit 200, a storage device 210, and an estimation unit 220.

  The edge detection unit 200 reads the input image I (x, y), executes edge detection calculation of the input image, and coordinates (ai, bi) of pixels (edge points) determined to be edges and the coordinates (ai). , Bi) store the edge direction or normal vector ni in the storage device 210. Here, i is an edge point number. An existing method can be applied to the edge detection method.

  The estimation unit 220 reads the input image and the edge position and edge direction stored in the storage device 210, estimates blur based on the value of input image data around the edge point, and blurs each edge point. Outputs the blur parameter that is the estimation result. This blur parameter is the blur amount 30 described in the above embodiment.

  Next, the blur estimation method in the estimation unit 220 will be described. The blur of the input image is modeled by the blur function h (x, y; r) described above. r is a blur parameter, that is, a blur amount of 30.

Consider a case where step edges as shown in the following equations (55) and (56) are blurred by the blur function h. In equation (55), g0 is the luminance on one side of the edge, and in equation (56), g1 is the luminance on the other side of the edge.

  In the following, the case where the step edge is along the y-axis will be described for the sake of simplicity. However, in other general cases, the edge point of interest is the origin and the y-axis is in the direction of the edge. By taking the x axis in the direction perpendicular to the edge, a similar method can be used.

The luminance distribution when the step edge is blurred is expressed by the following equation (57).

Here, “*” is a two-dimensional convolution. For example, when blur is modeled by a Gaussian of the following formula (24) having one parameter of blur radius r, the following formula (58) is obtained.

  Here, erf (x) represents an error function. Considering an edge along the y-axis as described above, f is a function independent of y as shown in equation (58).

  FIG. 36 shows an example of a cross-section g (x, 0) at a step edge y = 0, FIG. 37 shows an example of a cross-section h (x, 0; r) at a blur function y = 0, and FIG. 38 shows a blurred edge. An example of a cross section f (x, 0; g0, g1, r) at y = 0 is shown.

The blur is estimated by obtaining g0, g1, and r such that equation (57) is close to the luminance distribution I (x + ai, y + bi) of the input image data around the edge point (ai, bi). In this blur estimation, a least square fitting is performed. The least square fitting may be performed in one dimension in the direction of the normal line ni of the edge. In the case of the edge along the y axis, f = 0 (x, 0; g0, g1, r) and I (x + ai, bi) are fitted in the x direction with y = 0.
As described above, the estimation unit calculates the blur parameter r at each edge point.

<Effect>
In the method according to the present embodiment, the following effect (6) can be obtained in addition to the effects (1) to (3) described in the first to third embodiments.

(6) The foreground extraction accuracy can be improved (part 3).
In the method according to the present embodiment, the estimation unit 105 fits an input image and a step edge blurred by a blur function, and estimates blur based on this. Thereby, the problem of blur estimation can be simplified to the problem of fitting between a function and data. Therefore, it is possible to reduce the load on the blur estimation unit 11 and improve the foreground extraction accuracy.

[Seventh Embodiment]
Next, an image processing apparatus and an image processing method according to the seventh embodiment of the present invention will be described. The present embodiment uses a moving image as an input image in the first to sixth embodiments. The other points are the same as those in the first to sixth embodiments, and thus the description thereof is omitted.

  FIG. 39 is a block diagram of the image processing system 1 according to the present embodiment. As shown in the figure, the configuration of FIG. 1 described in the first embodiment is that the camera 2 is replaced with a video camera 4. Then, the video camera 4 outputs the captured moving image data to the image processing device 3.

  As in the first embodiment, the video camera 4 focuses on the target object and shoots with the background shifted. The video camera 4 need not be calibrated. Further, if the condition that the background is sufficiently far from the foreground is maintained, the foreground object may move or the background may change with respect to the foreground. That is, the first embodiment differs from the first embodiment in that the first embodiment obtains a plurality of image data from a plurality of viewpoints at the same time, whereas the present embodiment has a plurality of image data from the same viewpoint at a plurality of times. Is to get.

  The moving image data is a set of a plurality of still images (referred to as frames) along the time axis. This is shown in FIG. FIG. 40 is a schematic diagram of a moving image, and shows how a plurality of frames are obtained with time.

  When the image processing apparatus 3 receives the moving image data from the video camera 4, for example, an internal moving image receiving unit (not shown) determines which frame is used as the input image. For example, when the first two frames of moving image data are used as input images, as shown in FIG. 40, the respective frames are input images F1 (x) and F2 (x) to the foreground extraction unit 10. Output.

Note that the frame that can be selected as the input image needs to satisfy the following conditions as described above. This is the same as in the first to sixth embodiments.
(A) The foreground object is an image sequence that moves relative to the background while maintaining substantially the same shape.
(B) The foreground object is in focus and the background is blurred.
Condition (A) is satisfied if the scene is stationary, and the video camera 4 is photographed while moving in a plane substantially perpendicular to the depth direction of the scene. This is equivalent to taking the camera 2 as a single camera 2 in the first embodiment while moving. When the foreground object moves, the condition (A) is satisfied even if the video camera 4 is fixed. However, since the foreground must maintain substantially the same shape, for example, when the foreground object moves 180 degrees backward, the moving image data is maintained in substantially the same shape. It is necessary to divide it up to such an extent that it can be regarded as being.

In the present embodiment, a plurality of images correspond to a plurality of times. Therefore, if the time of the image i is ti, the foreground information represented by the equation (22) described in the second embodiment corresponds to the foreground observed at the time represented by the following equation (59).

  As described above, the video camera 4 may be used instead of the camera 2 and a plurality of frames included in the moving image may be used as the input image. Further, in the case where the foreground is extracted from the moving image, even if the third embodiment in which the foreground information is given by the user is applied, as described in the third embodiment, the information to be given by the user is the distance d1. Only. Therefore, even when handling moving images, the foreground can be extracted with high accuracy by a simple method.

[Eighth Embodiment]
Next explained is an image processing device and image processing method according to the eighth embodiment of the invention. In the present embodiment, the image processing method described in the first to seventh embodiments is realized by a processor system including a plurality of processors.

  FIG. 41 is a block diagram of a processor system according to the present embodiment. As illustrated, the processor system 300 includes a first processor 310, a plurality of second processors 320, a first decoder 330, a second decoder 340, a DMA (direct memory access) controller 350, a memory 360, a main memory 370, an interface 380, And a memory controller 390.

  The first processor 310 manages main control in the system 300. The second processor 320 performs calculations according to the control of the first processor 320. Each of the second processors 320 includes a control unit 321, a calculation unit 322, and a memory 323.

  The first decoder 330 decodes moving picture expert group (MPEG) moving image data input from the outside. The second decoder 340 receives the H.264 input from the outside. H.264 format moving picture data is decoded. The DMA controller 350 manages data transfer between memories included in the system 300.

  The memory 360 is a non-volatile semiconductor memory such as a flash memory, and includes a control program 361, a blur estimation program 362, a trimap creation program 363, an α value estimation program 364, a registration program 365, a color estimation program 366, and a composite image creation program. 367 and the new background 40 are held. The programs 361 to 367 will be described later.

  The main memory 370 is a semiconductor memory such as a DRAM (Dynamic Random Access Memory), for example, and serves as a work area for the first processor 310. The interface 380 manages connection between the system 300 and the outside. Thus, for example, user instructions or the like are provided to the system 300 via the interface 380. The memory controller 390 manages data exchange between the system 300 and the outside. Therefore, for example, data given from the camera 2 or the video camera 4 is written in the memory 360 under the control of the memory controller 390.

  Next, the foreground extraction / image composition method by the system 300 will be described with reference to the flowchart of FIG. The basic foreground extraction method and image composition method are the same as those in the first to seventh embodiments. Hereinafter, how each processing step described in the first to seventh embodiments is executed in the system 300 will be described.

  First, the memory controller 390 receives an input image from the outside, and stores the input image in the memory 360 (step S30). In the first to sixth embodiments, the input image is a still image provided from the camera 2, and in the seventh embodiment, the input image is a moving image provided from the video camera 4. Input images are already in MPEG format or H.264 format. It may be encoded in the H.264 format. In this case, the first decoder 330 or the second decoder 340 decodes the input image.

  Next, the first processor 310 reads the input image and the control program 361 from the memory 360 to the main memory 370, and executes the control program 361 (step S31). The control program 361 is a program relating to the main processing flow for performing foreground extraction and image synthesis. By executing the control program 361, foreground extraction and image composition processing is started.

  According to the control program 361, the first processor 310 first starts blur estimation processing (step S32). In the blur estimation process, the first processor 310 sets, for example, the window WND described with reference to FIG. Subsequently, the first processor 310 instructs the plurality of second processors 320 to estimate blur for each window WND (step S33).

  Then, the control unit 321 of each second processor 320 reads the blur estimation program 362 from the memory 360 into the memory 323. Then, the control unit 321 executes the blur estimation program 362 to cause the calculation unit 322 to perform calculation. As a result, the second processor 320 obtains the blur amount 30 regarding the window WND that it is in charge of (step S34). This process is the process of step S11 described in the first embodiment, and the specific method thereof is the contents described in the fourth and sixth embodiments. The blur estimation program 361 is a program for executing the above method.

  Next, the first processor 310 reads the trimap creation program 363 into the main memory 370. Then, the trimap 31 is obtained by executing the trimap creation program 363 based on the blur amount 30 obtained by each second processor 320 in step S34 (step S35). The trimap creation program 363 is a program for executing step S12 described in the first embodiment. Of course, not only the first processor 310 but also the second processor 320 may take part of the processing.

  Subsequently, the first processor 310 reads the α value estimation program 364 into the main memory 370. Then, by executing the α value estimation program 364 using the trimap 31 obtained in step S35, an initial value of α value is obtained (step S36). The α value estimation program 364 is a program for executing steps S13, S17, and S18 described in the first embodiment. Of course, not only the first processor 310 but also the second processor 320 may take part of the processing.

  Next, according to the control program 361, the first processor 310 starts corresponding point calculation and warp function calculation processing (step S37). In this process, the first processor 310 allocates the pixel x to be dealt with to the plurality of second processors 320 as described with reference to FIG. 9, for example (step S38).

  Then, the control unit 321 of each second processor 320 reads the alignment program 365 from the memory 360 to the memory 323. Then, the control unit 321 executes the alignment program 365 and causes the calculation unit 322 to perform calculation. As a result, the second processor 320 obtains the warp functions 32 and 33 relating to the pixel x that it is in charge of (step S39). This process is the process of steps S14 and S15 described in the first embodiment, and the alignment program 365 is a program for executing this process. For example, in FIG. 9, if a certain second processor 320 takes charge of the pixel x0, the second processor 320 calculates which pixel in the input image I2 (x) corresponds to the pixel x0 in the input image I1 (x). , A warp function W12 (x0) is obtained.

  Subsequently, the first processor 310 starts the foreground / background color estimation process according to the control program 361 (step S40). In this processing, the first processor 310 assigns each pixel for which the foreground color and the background color are to be calculated to the second processor 320 (step S41).

  Then, the control unit 321 of each second processor 320 reads the color estimation program 366 from the memory 360 into the memory 323. Then, the control unit 321 executes the color estimation program 366 and causes the calculation unit 322 to perform calculation. As a result, the second processor 320 obtains the foreground color 35 (Fi) and the background color 34 (Bi) related to the pixel x that it is in charge of (step S42). This process is the process of step S16 described in the first embodiment, and the color estimation program 366 is a program for executing this process.

  Next, the first processor 310 reads the α value estimation program 364 into the main memory 370. Then, the α value estimation program 364 is executed using the foreground color 35 and the background color 34 obtained in step S42, thereby updating the α value and determining whether the α value has converged (step S43). This process corresponds to the processes in steps S17 to S19 described in the first embodiment. Of course, not only the first processor 310 but also the second processor 320 may perform a part of this processing.

  When determining that the α value has converged, the first processor 310 reads the composite image creation program 367 and the new background 40 from the memory 360 into the main memory 370. Then, using the foreground color 35 obtained in step S42 and the α value obtained in step S43, the foreground of the input image and the new background 40 are synthesized (step S44). This process corresponds to the process of step S20 described in the first embodiment. Of course, not only the first processor 310 but also the second processor 320 may perform a part of this processing.

  As described above, the image processing method described in the first to sixth embodiments may be realized by the processor system 300 including the plurality of processors 310. In this case, the first processor 310 and the second processor 320 function as the foreground extraction unit 10 and the image composition unit 20 described with reference to FIGS.

  As described above, the image processing apparatus and the image processing method according to the first to eighth embodiments of the present invention use a plurality of images that are photographed with a focus on the foreground and a blurred background. The degree of blur in each image is estimated. The α value is estimated according to the degree of blur. Therefore, highly accurate foreground extraction is possible without requiring a special shooting environment.

  In addition, corresponding point calculation of a plurality of input images is performed between the areas that are the background and the areas that are the foreground. And about an unknown area | region, it estimates from the value of the area | region used as a background, and the area | region used as a foreground. That is, matching is performed for areas where the correspondence can be reliably obtained, and thereafter, estimation is performed for areas where the correspondence is unknown. As a result, it is possible to grasp which pixel in the unknown area of one input image corresponds to which pixel of the other input image. Further, the α value is estimated based on the result.

  Therefore, the correspondence can be taken with higher accuracy than conventional stereo matching and the like, and as a result, the foreground extraction accuracy is improved.

  In the first embodiment, the image processing system 1 including a plurality of cameras 2 has been described as an example. However, only one camera 2 may be provided. That is, when the target object is stationary, images from a plurality of viewpoints can be obtained by moving the photographer. When the target object is moving, images from a plurality of viewpoints can be obtained without the photographer moving.

  Further, it is only necessary that at least a part of the foreground is the same among a plurality of input images. However, in order to increase the accuracy of the corresponding point calculation, it is preferable that the foreground is substantially the same between the input images. However, even if the foreground does not match in many parts, the above embodiment can be applied. is there.

  Further, the system 1 itself does not have to have the camera 2 or the video camera 4. In other words, for example, image data serving as an input image may be given to the image processing apparatus 3 via a network or the like.

  In the first embodiment, depending on the input image, the mask image obtained in step S13 may be used as the final mask image. In this case, steps S14 to S20 are omitted. Further, if the system omits steps S14 to S20, it is sufficient that the foreground extraction unit 10 includes only the blur estimation unit 11, the region division unit 12, and the α value estimation unit 16.

  Note that the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the scope of the invention in the implementation stage. Furthermore, the above embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some constituent requirements are deleted from all the constituent requirements shown in the embodiment, the problem described in the column of the problem to be solved by the invention can be solved, and the effect described in the column of the effect of the invention Can be extracted as an invention.

1 is a block diagram of an image processing system according to a first embodiment of the present invention. 1 is a block diagram of an image processing apparatus according to a first embodiment of the present invention. 1 is a flowchart of an image processing method according to the first embodiment of the present invention. It is a schematic diagram of an input image, and is a diagram showing a state of blur estimation in the image processing method according to the first embodiment of the present invention. It is a schematic diagram of an input image, and is a diagram showing a state of blur estimation in the image processing method according to the first embodiment of the present invention. The schematic diagram of an input image. The schematic diagram of the trimap obtained by the image processing method which concerns on 1st Embodiment of this invention. The schematic diagram of the mask image obtained by the image processing method concerning a 1st embodiment of this invention. It is a schematic diagram of an input image, and is a diagram showing a state of alignment in the image processing method according to the first embodiment of the present invention. The graph of the weighting function used with the image processing method which concerns on 1st Embodiment of this invention. The graph of the weighting function used with the image processing method which concerns on 1st Embodiment of this invention. The graph of the weighting function used with the image processing method which concerns on 1st Embodiment of this invention. It is a schematic diagram of a mask image obtained by the image processing method according to the first embodiment of the present invention, and shows a state of color estimation. The schematic diagram of the mask image obtained by the image processing method concerning a 1st embodiment of this invention. FIG. 3 is a schematic diagram showing a state of image composition in the image processing method according to the first embodiment of the present invention. It is a photograph replacing the drawing, and is an input image used in the image processing method according to the first embodiment of the present invention, (a) a photograph taken from the left side, and (b) a photograph taken from the right side. FIG. 4A is a photograph replacing a drawing, and FIG. 5A is a mask image obtained by the image processing method according to the first embodiment of the present invention, and FIG. It is a photograph replaced with drawings, Comprising: The composite image obtained by the image processing method which concerns on 1st Embodiment of this invention. The block diagram of the image processing apparatus which concerns on 2nd Embodiment of this invention. The flowchart of the image processing method which concerns on 2nd Embodiment of this invention. The schematic diagram which shows the mode of foreground interpolation in the image processing method which concerns on 2nd Embodiment of this invention. The schematic diagram of the trimap obtained by the image processing method concerning the 3rd Embodiment of this invention. The block diagram of the blur estimation part which concerns on 4th Embodiment of this invention. The graph which shows an example of distribution of the differential coefficient of natural image data, and the Gaussian distribution for a comparison object. The block diagram which shows an example of the process of the deconvolution part which concerns on 4th Embodiment of this invention. The flowchart which shows an example of the deconvolution process performed by the deconvolution part which concerns on 4th Embodiment of this invention. The flowchart which shows an example of the process of the local blur parameter estimation part which concerns on 4th Embodiment of this invention. The figure which shows an example of the result of the area | region division in 4th Embodiment of this invention. The graph which shows the filter response value used with the image processing method which concerns on 4th Embodiment of this invention. The graph which shows the filter response value used with the image processing method which concerns on 4th Embodiment of this invention. The graph which shows the filter response value used with the image processing method which concerns on 4th Embodiment of this invention. The graph which shows the decompression | restoration image frequency component obtained with the image processing method which concerns on 4th Embodiment of this invention. The graph which shows the divergence degree obtained with the image processing method which concerns on 4th Embodiment of this invention. The block diagram of the foreground extraction part which concerns on 5th Embodiment of this invention. The block diagram of the blur estimation part which concerns on 6th Embodiment of this invention. The graph which shows the example of the cross section g (x, 0) in y = 0 of a step edge. The graph which shows the example of the cross section h (x, 0; r) in y = 0 of the blurring function in 6th Embodiment of this invention. The graph which shows the example of the cross section f (x, 0; g0, g1, r) in y = 0 of the blurred edge. The block diagram of the image processing system which concerns on 7th Embodiment of this invention. The conceptual diagram which shows the structure of a moving image. The block diagram of the processor system which concerns on 8th Embodiment of this invention. The flowchart of the image processing method by the processor system concerning 8th Embodiment of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Image processing system, 2 ... Camera, 3 ... Image processing apparatus, 4 ... Video camera, 10 ... Foreground extraction part, 11 ... Blur estimation part, 12 ... Area division part, 13 ... Background corresponding point calculation part, 14 ... Foreground Corresponding point calculation unit, 15 ... foreground / background color estimation unit, 16 ... α value estimation unit, 20 ... image synthesis unit, 21, 360 ... memory, 22 ... synthesis unit, 23 ... foreground interpolation unit, 30 ... blur amount, 31 ... Tri-map, 32 ... Background warp function, 33 ... Foreground warp function, 34 ... Background color, 35 ... Foreground color, 36 ... Mask image, 40 ... New background, 41 ... Composite image, 42 ... Foreground color interpolation image, 43 ... α Value interpolation image, 103 ... PSF model storage device, 104 ... Deconvolution unit, 105 ... Deblurred image storage device, 106 ... Local blur parameter estimation unit, 107 ... Local estimation blur parameter storage device, 108 ... Image integration unit, 109 ... Restore Image storage device 141 ... parameter changing unit 142 ... deconvolution processing unit 143 ... differentiation unit 144 ... differential region deconvolution unit 145 ... integration unit 161 ... region division unit 162 ... local blur estimation unit 200 Edge detection unit 210 ... Storage device 220 ... Estimation unit 300 ... Processor system 310 ... First processor 320 ... Second processor 330 ... First decoder 340 ... Second decoder 350 ... DMA controller 361 ... Control program, 362 ... blur estimation program, 363 ... trimap creation program, 364 ... α value estimation program, 365 ... registration program, 366 ... color estimation program, 367 ... composite image creation program, 370 ... main memory, 380 ... interface 390 ... Memory controller

Claims (5)

An estimation unit for estimating a blur state in each of a plurality of image data related to the same object;
Based on the estimation result in the estimation unit, a dividing unit that divides a background region and a foreground region in each of the image data;
An extraction unit that extracts a foreground in each of the image data according to a division result in the division unit, and each of the image data is an image focused on the foreground compared to the background. An image processing apparatus. An estimation unit for estimating a blur state in each of a plurality of image data related to the same object;
Based on the estimation result in the estimation unit, in each of the image data, a dividing unit that divides the first region as the background and the other second region,
An alignment unit for grasping a positional relationship between pixels corresponding to each other between the image data;
An extraction unit that extracts a foreground in each of the image data according to a division result in the division unit and the positional relationship grasped in the alignment unit, and each of the image data is included in the background Compared to the foreground, the image is
The dividing unit regards an area estimated to be blurred in the estimation result as the background,
The image processing apparatus, wherein the alignment unit performs alignment between the first regions and the second regions. The image processing apparatus according to claim 1, wherein the extraction unit calculates a ratio of the foreground color and a ratio of the background color for each pixel included in the image data. . The estimation unit generates blur-removed image data from which the blur is removed for each of the image data,
The image processing apparatus according to claim 2, wherein the alignment unit grasps the positional relationship between the deblurred image data. Receiving a plurality of first image data;
A plurality of processors estimating a blur state for each of a plurality of areas included in each of the plurality of received first image data;
The processor dividing each of the first image data into a background area and a foreground area based on the blur state;
The processor extracting the foreground for each of the first image data;
The processor comprising combining the foreground with second image data different from the first image data, each of the first image data being an image focused on the foreground compared to the background. An image processing method characterized by that.

Images