JP2011227909A - Image separation device, image separation method, and image separation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To appropriately separate a foreground and a background in an image, while reducing a work load and work time of a user.SOLUTION: An image separation device is a device for separating an image into a foreground and a background. The image separation device comprises data input means for inputting the image and storing the image in storage means; separation means for defining F as a foreground image shade value and B as a background image shade value when image shade values (or luminance) of the foreground and the background are defined as I, and defines α as a decimal number concerning α blending, minimizing a target function under a prescribed restriction condition, acquiring unknown α, F, and B by pixel unit, determining whether a pixel is the pixel of the foreground or background through the use of the image shade values of F and B acquired by pixel unit, and separating the foreground and the background; and display means for displaying the separated background and foreground respectively.

Description

  The present invention relates to a technique for separating a foreground and a background included in an image.

When extracting the foreground (object of interest, object) and the background constituting the image from one image, for example, using the image editing tool such as Non-Patent Document 1, for example, Separating and extracting images is performed.
“Photoshop”, [online], [Search September 12, 2008], Internet <URL: http://web.kyoto-inet.or.jp/people/hikeda/justnet/smu/ind ex. html>

When using the image editing tool as described above, the user manually designates / sets the outline of the foreground to be extracted. Therefore, when the outline of the extraction target is fine or unclear, such as smoke or hair, it is difficult to properly separate the foreground and the background, and the user's work load is heavy, and the user's skilled skills and experience Is required.

In addition, a foreground and a background are separated and extracted from a moving image, and a new moving image is created by combining the separated and extracted foreground with a background of another scene. A moving image is composed of a plurality of image frames, and the contour to be extracted is manually specified for each image frame. Therefore, in the case of a moving image, separation and extraction work is performed over several days or more.

The present invention has been made in view of the above circumstances, and an object of the present invention is to appropriately separate the foreground and background of an image while reducing the work load and work time of the user.

The present invention is an image separation device for separating an image into a foreground and a background,
Data input means for inputting an image and storing it in a storage means;
If the foreground and background image intensity values (or luminance) are I, F is the foreground image intensity value, B is the background image intensity value, α is the fractional value for α blending,

To generate for each pixel unit constituting one image,
From equation (1)

    Generate

As a constraint,
Objective function

_{Formula (9)}

  And the objective function is

_{Formula (10)}

e is the unknowns α, F, and B. Partial differentiation is performed for each variable to obtain the unknowns α, F, and B in units of pixels, and the F and B image grayscale values that are acquired in units of pixels are used. Separating means for determining whether the pixel is a foreground pixel or a background pixel and separating the foreground and the background;
Display means for displaying the separated background and foreground, respectively.

Further, the present invention is an image separation method for separating an image into a foreground and a background performed by an image separation device,
A data input step of inputting an image and storing it in a storage means;
If the foreground and background image intensity values (or luminance) are I, F is the foreground image intensity value, B is the background image intensity value, α is the fractional value for α blending,

To generate for each pixel unit constituting one image,
From equation (1)

    Generate

As a constraint,
Objective function

_{Formula (9)}

  And the objective function is

_{Formula (10)}

e is the unknowns α, F, and B. Partial differentiation is performed for each variable to obtain the unknowns α, F, and B in units of pixels, and the F and B image grayscale values that are acquired in units of pixels are used. A separation step of determining whether the pixel is a foreground pixel or a background pixel and separating the foreground and the background;
Displaying the separated background and foreground, respectively.

The present invention is also a spatio-temporal image separation program that causes a computer to execute the image separation method.

According to the present invention, it is possible to appropriately separate the foreground and the background of an image while reducing the user's workload and working time.

  Hereinafter, embodiments of the present invention will be described.

FIG. 1 is a schematic configuration diagram of an image separation apparatus according to an embodiment of the present invention. In the image separation apparatus according to the present embodiment, an image gray value (luminance) I is synthesized by combining a foreground F and a background B at a certain ratio α.
In an image model that is assumed to be αF + (1−α) B, an objective function for estimating F, B, and α is generated, and the objective function is analyzed to separate the image into the foreground and the background. Extract. Note that the foreground is an object of interest.

The illustrated image separation apparatus includes a data input unit 100, an instruction reception unit 110, and an image separation unit 1.
20, a display unit 130, a data storage unit 140, and an instruction displacement unit 150.

The data input unit 100 inputs one image (still image) or space-time image (moving image),
The data is stored in the data storage unit 140 serving as storage means. The instruction receiving unit 110 receives a user instruction input to a part of the foreground of the image and a user instruction input to a part of the background. The image separation unit 120 generates an objective function using the foreground and background image grayscale values specified by the user as constraints, and analyzes the objective function using the least squares method, thereby converting the image into the foreground and background. To separate. The display unit 130 displays the separated background and foreground, respectively. When the moving image is input, the instruction displacement unit 150 detects the foreground and background movement between the preceding and following image frames, and the user instruction input to the first image frame is determined according to the detected movement. It is displaced and reflected after the second image frame.

The image separation apparatus described above uses, for example, a general-purpose computer system including a CPU, a memory, an external storage device, an input device, an output device, and a bus that connects these devices. Can do. In this computer system, each function of the image separation device is realized by the CPU executing a program for the image separation device loaded on the memory. Note that a memory or an external storage device is used for the data storage unit 140 of the image separation device. Note that the image separation device may include a communication control device for connecting to other devices as necessary.

Also, the program for the image separation device is a hard disk, a flexible disk, a CD
-It can be stored in a computer-readable recording medium such as ROM, MO, or distributed via a network.

  Next, a method for separating an image into a foreground and a background will be described.

FIG. 2 is a flowchart of image separation processing. First, the data input unit 100 inputs a single image (still image) or a spatio-temporal image via a camera or a network and stores it in the data storage unit 140 (S11). A spatiotemporal image is moving image data composed of a plurality of image frames (video frames) continuous in time series.

The instruction receiving unit 110 reads out the image stored in the data storage unit 140 and displays it on the output device. When the image stored in the data storage unit 140 is a spatiotemporal image, the instruction receiving unit 110 displays the first (first) image frame on the output device.

The user interactively designates (defines) the foreground and background areas on the image displayed on the output device by using an input device such as a mouse or a keyboard. Then, the instruction receiving unit 110
Accepts a user instruction (S12). It is possible to use general image editing roots for the instruction receiving unit 110.

FIG. 3 is a diagram illustrating an example of a user instruction input to an image and a result of separation and extraction into a foreground and a background.

Using a mouse or the like, the user inputs an arbitrary line (curve or straight line) 202 to a part of the foreground area 201 of the displayed image 200, and an arbitrary line 20 to a part of the background area.
Enter 3. Then, by processing described later, the image 200 is converted into an image 210 with only a background,
The image is separated into the foreground-only image 220.

Then, the image separation unit 120 separates the image stored in the data storage unit 140 into the foreground and the background by a process described later (S13). In other words, the image separation unit 120 determines whether each pixel constituting the image is a foreground pixel or a background pixel. Display unit 130
Displays the separated foreground (pixels determined to be the foreground) and the background (pixels determined to be the background) on the display device (S14). In addition, the display unit 130 displays the separated foreground and background.
The data is stored in the data storage unit 140 in association with the original image before separation.

Then, the instruction displacement unit 150 determines whether or not the image input and stored in S11 includes the next image frame (that is, whether or not it is a moving image) (S15). Instruction displacement part 15
0 determines whether the image is a still image or a time-series moving image from the file number / frame number of the input image, the attribute of the image, and the like. If the input image is a single still image (S15: NO), the process ends.

On the other hand, when the input image is a moving image and the next image frame exists (S15: Y
ES), the instruction displacement unit 150 reflects (copies) the user instruction (see FIG. 3) input in S12 in the second image frame (S16).

Here, the foreground and background may change in time and space. Therefore, the instruction displacement unit 150 uses two image frames, that is, an image frame that has undergone separation processing (here, the first image frame) and an image frame at the next time (here, the second image frame). Then, the motion of the image is acquired by the motion estimation method. As the motion estimation method, an optical flow, a cross-correlation method, or the like can be used. The instruction displacement unit 150 displaces (moves, deforms) the position and shape of the line defined by the user as the foreground and background based on the acquired image movement, and sets the line in the second image frame. To do.

Then, returning to S13, the second image frame is separated into the foreground and background, and the separated foreground and background are displayed and stored in the data storage unit 140 (S14). Then, it is determined whether or not the next image frame exists, and the subsequent processing is repeated for all the image frames.

  FIG. 4 is a diagram schematically showing processing for separating a moving image into a foreground and a background.

FIG. 4 shows a first image frame 400 in which user instructions 401 and 402 specifying the foreground and background are input. FIG. 5 also shows a moving image 410 composed of a plurality of image frames, and a foreground 420 and a background 430 separated for each image frame.
Here, as can be seen from the separated foreground 420, the foreground is deformed and moved over time.

Here, the instruction displacement unit 150 detects the displacement of the foreground between the first image frame 411 and the second image frame 412 by a motion estimation method such as an optical flow or a cross-correlation method. Then, based on the detected displacement, the instruction displacement unit 150 deforms and moves the user instructions 401 and 402 input to the first image frame 400 as indicated by 413 to move the second image. Set (reflect) in the frame. For the third image frame, the motion between the second image frame and the third image frame is detected, and the user instruction set for the second image frame is displaced based on the motion. To set. Thus, the user simply inputs the foreground and background instructions to the first image frame, and the user instructions are automatically reflected in all the image frames.

Next, the process of the image separation unit 120 in S13, that is, a method for separating and extracting the foreground and the background from one image will be described. This method is also called a matting method.

First, it is assumed that two objects, foreground and background, are combined at a certain ratio as one image model. This is expressed as a mathematical expression when the gray value (or luminance) of a pixel is I,
It can be expressed by the following formula (1).

F is the foreground image intensity value, and B is the background image intensity value. α is a decimal for α blending and is in the range of 0.0 to 1.0. Expression (1) is generated for each pixel unit constituting one image.

In the present embodiment, when one image is given, an object is to estimate F, B, and decimal number α for each pixel and separate the foreground and the background. There are the following four methods for this estimation.
The first and second methods use the foreground and background information specified by the user in S12, while the third and fourth methods do not use the foreground and background information specified by the user in S12. When the third and fourth methods are used, the processes of S12 and S16 can be omitted. It is assumed that a method to be used is determined in advance according to an image to be separated.
[First method]
First, consider an image model that assumes that α can be approximated by a linear function of the image gray value I of a pixel. In the case of this image model, α can be expressed as the following equation (2). Further, it is assumed that the foreground and the background are relatively smooth images (the spatial differentiation is small) with little texture.

w is a small area (window area) in the image, and is a set of neighboring pixels. The size of the window is “3 pixels × 3 pixels” to “5 pixels × 5 pixels”. i indicates the position of the pixel in the window region.

In order to estimate the unknowns α, a, and b in Equation (2), it can be reduced to the objective function minimization problem in Equation (3) shown below.

ε is an adjustment coefficient, and a predetermined value is set according to the smoothness of the foreground and background images. Further, the term of εa _j ² is a regularization term, and has a role to alleviate the numerical instability problem. j indicates the j-th window region.

However, in this state, in order to uniquely determine the objective functions of a _j and b _j , conditions are insufficient (under-constraint), so prior knowledge is required. (3) is α, a, b
For N pixels, the unknown is 3N.

Therefore, as a constraint condition, for the window region j input by the user as an arbitrary curve by the mouse operation, the image gray value of the pixel at the position designated as the foreground is set to F, and the image gray value of the pixel at the position designated as the background is set. Set (reflect) the value to B.

Then, by analyzing the expression (3) using the least square method, α, a, b are used for each pixel.
To get. Then, the obtained α, a, and b are substituted into the corresponding part of the equation (2), and the equation (2)
F and B are obtained for each pixel by solving the simultaneous equations of the above portion.

Then, using the F and B image grayscale values acquired in units of pixels, it is determined whether the pixel is a foreground pixel or a background pixel. For example, a pixel in which at least one of the F image gradation value and the B image gradation value is smaller than a predetermined threshold (for example, 10) is determined as a background pixel, and the other pixels are determined as foreground pixels. .

Then, by extracting each pixel determined to be the foreground, the foreground image 22 as shown in FIG.
0 can be extracted separately from the background. Also. By extracting only each image determined to be the background, only the background image 210 with the foreground part masked (displayed in black) as shown in FIG. 3 can be separated and extracted. In addition to separating and extracting the foreground and background, it is possible to estimate α in the region where the foreground and background overlap.
[Second method]
In the case of an image with a smooth foreground and background, the foreground and the background can be appropriately separated and extracted by the above-described equation (3) of the first method. However, when the object (foreground) is rich in texture, the change in luminance is large, and the least square method of Equation (3) may cause an estimation error.

Therefore, in the second method, the following formula (4) is applied instead of formula (3). Formula (4)
Is an expression of a nonlinear least square method using a robust function ρ used in robust statistics.

The robust function can be expressed by the following equation (5), where λ is an adjustment parameter and is in the range of 0.001 to 0.1.

The necessary condition for minimizing the equation (4) is the following equation (6). By analyzing the equation (4) using a nonlinear least square method such as a conjugate gradient method, α, a, b can be obtained.

Then, the obtained α, a, and b are substituted into the corresponding portion of the equation (2), and the simultaneous equations of the remaining portion of the equation (2) are solved to obtain F and B for each pixel. As a result, the F and B image grayscale values acquired in units of pixels as in the first method are used to determine whether the pixel is a foreground pixel or a background pixel, and the foreground and background. Isolate.

FIG. 5 shows a result of separating an image 300 having a texture-rich foreground by the first method and the second method. FIG. 5A shows the result of separation by the first method, and FIG. 5B shows the result of separation by the second method.

In the first method of FIG. 5A, an estimation error is caused, and the image 310 separated and extracted as the foreground includes a part of the background. That is, the foreground and background are not properly separated. On the other hand, in the second method shown in FIG. 5B, the foreground 330 and the background 340 are appropriately separated by using a robust function. That is, in the second method using the robust function, the foreground 330 and the background 340 are obtained from one image 330 rich in texture.
Can be appropriately separated and extracted.
[Third method]
The third method separates images using only the above-described equation (1) without using the foreground and background information designated by the user in S12.

  First, the following formulas (7) and (8) are generated from the formula (1).

That is, from the viewpoint of the minimization problem, Expression (7) can be easily derived from Expression (1). This becomes a minimization problem of the function V ₁ . However, this is not enough (under-
constraint), a constraint condition is required. Thus, when focusing on a certain pixel point p, it may be possible to impose a smoothness condition. Therefore, Equation (8) is generated with q as a 4-point connected neighborhood point of p, and Equation (8) is minimized.

Then, equations (7) and (8) can be put together to obtain the objective function of the following equation (9).

_{Formula (9)}

Here, e and f are coefficients. The necessary condition for minimizing the equation (4) is the following equation (10
).

_{Formula (10)}

Here, e is the unknowns α, F, and B, and partial differentiation is performed for each variable. Thereby, the unknowns α, F, and B can be acquired in units of pixels. As a result, the F and B image grayscale values acquired in units of pixels as in the first method are used to determine whether the pixel is a foreground pixel or a background pixel, and the foreground and background. Isolate.
[Fourth method]
In the fourth method, a robust function is applied to the third method. That is, the third method formula (
The following formula (11) is applied instead of 7) and formula (8). Expression (11) is an expression of a nonlinear least square method using a robust function ρ used in robust statistics.

  The objective function of the following formula (12) can be obtained from the formula (11).

Here, c and d are coefficients. By solving this equation (12) with the framework of the nonlinear least square method, the unknowns α, F, and B can be acquired in units of pixels. As a result, the F and B image grayscale values acquired in units of pixels as in the first method are used to determine whether the pixel is a foreground pixel or a background pixel, and the foreground and background. Isolate.

In the present embodiment described above, an objective function is generated using the foreground and background image grayscale values at a location designated by the user as a constraint condition, and the objective function is analyzed using the least square method, whereby the user's workload In addition, the foreground and the background of the image can be appropriately separated while reducing the work time. In addition, in the present embodiment, an object is to estimate F, B, and α in an image model in which an image gray value I of an image is synthesized with a foreground F and a background B at a certain ratio α. By generating the objective function and analyzing the objective function, the image can be separated into the foreground and the background even when the outline of the foreground (target) is composed of fine hairs or thin lines. .

Further, in the present embodiment, an object function (equation (4)) that the similarity of estimated values in neighboring pixels is high is generated as a constraint condition of the object function, and is analyzed as a minimum value problem. The background can be properly separated.

Further, in the present embodiment, by using a robust function, the foreground and the background can be appropriately separated and extracted from an image rich in texture.

In the present embodiment, the foreground and the background can be separated for each image frame even in the case of a moving image composed of a plurality of image frames. Thereby, compared with the case where the user separates manually, the workload and the time required for the separation work are greatly reduced.

In the present embodiment, the foreground and background motion between the image frames is detected, and in response to the motion, the user's instruction input to the first image frame is displaced (moved / deformed), Sequentially applied (reflected) to the second and subsequent image frames. As a result, all the image frames are automatically separated into the foreground and the background with only one instruction from the user. Further, even when the foreground and the background are deformed and moved, the foreground and the background can be appropriately separated.

In the case of a known physical phenomenon in which the foreground (target) motion is oscillating like a spring,
By correcting the motion estimation result using an elastic method, more accurate motion estimation is performed.
By this motion estimation, the foreground and the background can be properly separated even when the foreground is in amplitude motion.

The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the gist.

1 is a block diagram of an image separation device according to an embodiment of the present invention. 3 is a flowchart of a method for separating an image into a foreground and a background. It is a specific example which shows the instruction | indication of the user input into the image, and the result separated and extracted by the foreground and the background. It is explanatory drawing in order to demonstrate the process which isolate | separates an image into a foreground and a background. It is a figure which shows the result of having isolate | separated the image which has foreground rich in a texture with the 1st method and the 2nd method.

DESCRIPTION OF SYMBOLS 100 Data input part 110 Instruction reception part 120 Image separation part 130 Display part 140 Data storage part 150 Instruction displacement part

Claims (7)

An image separation device for separating an image into a foreground and a background,
Data input means for inputting an image and storing it in a storage means;
If the foreground and background image intensity values (or luminance) are I, F is the foreground image intensity value, B is the background image intensity value, α is the fractional value for α blending,


To generate for each pixel unit constituting one image,
From equation (1)


Generate

As a constraint,
Objective function
_{Formula (9)}

And the objective function is
_{Formula (10)}

e is the unknowns α, F, and B. Partial differentiation is performed for each variable to obtain the unknowns α, F, and B in units of pixels, and the F and B image grayscale values that are acquired in units of pixels are used. Separating means for determining whether the pixel is a foreground pixel or a background pixel and separating the foreground and the background;
An image separating apparatus comprising: display means for displaying each of the separated background and foreground. The image separation device according to claim 1,
The image separation apparatus characterized in that the separation means generates an objective function via a robust function and analyzes the objective function using a nonlinear least square method to separate the image into a foreground and a background. The image separation device according to claim 1 or 2,
The image input by the data input means is a spatio-temporal image composed of a plurality of image frames,
The foreground and background motion between the preceding and following image frames is detected, and the F and B image grayscale values acquired for each pixel with respect to the first image frame of the spatiotemporal image are used. An image separating apparatus, further comprising: a displacement unit configured to cause a determination as to whether the pixel is a pixel or a background pixel to be reflected according to the detected motion and to reflect it after the second image frame. An image separation method for separating an image into a foreground and a background performed by an image separation device,
A data input step of inputting an image and storing it in a storage means;
If the foreground and background image intensity values (or luminance) are I, F is the foreground image intensity value, B is the background image intensity value, α is the fractional value for α blending,


To generate for each pixel unit constituting one image,
From equation (1)

Generate

As a constraint,
Objective function
_{Formula (9)}

And the objective function is
_{Formula (10)}

e is the unknowns α, F, and B. Partial differentiation is performed for each variable to obtain the unknowns α, F, and B in units of pixels, and the F and B image grayscale values that are acquired in units of pixels are used. A separation step of determining whether the pixel is a foreground pixel or a background pixel and separating the foreground and the background;
And a display step of displaying the separated background and foreground, respectively. The image separation method according to claim 4,
The separation step generates an objective function via a robust function and analyzes the objective function using a nonlinear least square method to separate the image into a foreground and a background. The image separation method according to claim 4 or 5, wherein
The image input in the data input step is a spatio-temporal image composed of a plurality of image frames,
Detecting foreground and background motion between preceding and following image frames, and determining whether the pixel is a foreground pixel or a background pixel with respect to the first image frame of the spatiotemporal image An image separation method, further comprising: a displacement step of displacing in accordance with the performed movement and reflecting it after the second image frame. An image separation program for causing a computer to execute the image separation method according to any one of claims 4 to 6.