JP2010224592A - Object extraction device and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an object extraction device, a method and a program for accurately extracting an object area in an image while reducing a burden on a user. <P>SOLUTION: The object extraction device includes: an object specification line setting means for setting an object specification line composed of a closed curve in an image according to operation thereof; a first area setting means for setting an inside area of the object specification line as a first area; a second area setting means for setting, as a second area, an area obtained by subtracting the first area from an expanded area including the first area; a dominant parameter extracting means for extracting a dominant parameter in the first and second areas of an image; a feature amount extracting means for extracting an object feature amount based on the dominant parameter; and an area extracting means for extracting an object area in an image based on the object feature amount. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an object extraction apparatus and method for extracting an object region from an image.

  As a method for extracting a part of an object (object) such as a person or animal specified by a user in an image, commercially available image editing software for a personal computer uses a mouse operation to draw an outline of the object in the displayed image. A method of extracting a designated area as an object by tracing and specifying is generally used.

  Also, a method for extracting an object based on color and visual texture is known (see Patent Document 1). In this extraction method, the user designates an object and a part of the background in the image, performs learning based on the color information at the designated position, and obtains the feature amount of the object.

  Furthermore, there is a method in which the start point of the contour of the object is designated by the user, and each pixel included in a region within a predetermined range from the pixel designated as the start point is used as a contour candidate point, and a continuous contour of the object is automatically extracted. (See Patent Document 2).

  Further, there is a method in which the user sets a rough initial contour along the contour of the object to be extracted (see Non-Patent Document 1). In this method, a rectangle (small area) with a side length of several pixels and a slightly enlarged area (medium area) around the initial outline are considered. Since there is a high tendency for similarity (fractal) to be seen in the image in the image, the object contour is extracted based on this.

Japanese Patent Laid-Open No. 2005-129065 JP-A-5-324830

"Self-Affine Mapping System and Its Application to Object Contour Extraction", IEEE Transaction on Image Processing, vol.9, No.11, November, 2000

  However, in the conventional object region extraction technique described above, if the user does not carefully specify the object outline, a part of the object may be missing or a background pixel may be included in the object. It becomes unsightly. As described above, in order to increase the accuracy of object extraction, there is a problem that the burden on the user increases.

  Therefore, the problems to be solved by the present invention include the above-mentioned drawbacks as an example, and provide an object extraction apparatus, method, and program for accurately extracting an object region in an image while reducing the burden on the user. This is an object of the present invention.

  The object extraction device of the present invention according to claim 1 is an object extraction device for extracting an object region in an image, and an object specification line setting means for setting an object specification line comprising a closed curve in the image according to an operation. And a first area setting means for setting an inner area of the object designation line as a first area, and a second area that is obtained by subtracting the first area from an enlarged area including the first area. 2-region setting means, dominant parameter extracting means for extracting dominant parameters in the first area and the second area of the image, and feature quantity extracting means for extracting object feature quantities based on the dominant parameters; Area extracting means for extracting the object area in the image based on the object feature amount. It is characterized in.

  The object extracting method of the present invention according to claim 9 is an object extracting method for extracting an object region in an image, and an object specifying line setting step for setting an object specifying line consisting of a closed curve in the image according to an operation. And a first area setting step for setting the inner area of the object designation line as a first area, and an area obtained by subtracting the first area from an enlarged area including the first area as a second area. A two-region setting step, a step of extracting a dominant parameter in the first region and the second region of the image, a feature amount extracting step of extracting an object feature amount based on the dominant parameter, and the object feature A region extracting step of extracting the object region in the image based on a quantity; It is characterized in that.

  The program of the present invention according to claim 10 is a program for extracting an object area in an image, and sets an object designation line that is a closed curve in the image in response to an operation on a computer. A first area setting step for setting an internal area of the object designation line as a first area, and an area obtained by subtracting the first area from an enlarged area including the first area as a second area A second region setting step, a step of extracting a dominant parameter in the first region and the second region of the image, a feature amount extracting step of extracting an object feature amount based on the dominant parameter, and the object A region extracting step of extracting the object region in the image based on a feature amount; It is characterized in that to the row.

It is a block diagram which shows the 1st Example of this invention. It is a flowchart which shows the object extraction operation | movement of the apparatus of FIG. It is a figure which shows the target image of object area | region extraction. It is a figure which shows the object designation | designated line in a target image. It is a figure which shows the initial object area | region and initial background area | region in a target image. It is a figure which shows the extraction range in a target image. It is a figure which shows the object extracted from the target image. It is a flowchart which shows the object extraction operation | movement of a moving image as 2nd Example of this invention. It is a figure which shows the image of the 1st-3rd frame of a moving image, an object area | region, and the next frame extraction range. It is a figure which shows deletion of the parts in an initial object area | region.

  In the object extraction device of the present invention according to claim 1, the object extraction method of the present invention according to claim 9 and the program of the present invention according to claim 10, an object designation line consisting of a closed curve is provided in an image according to an operation. Set, the internal area of the object designation line is set as the first area, the area obtained by subtracting the first area from the enlarged area including the first area is set as the second area, and the first area and the first area of the image The dominant parameter in the two regions is extracted, the object feature amount is extracted based on the dominant parameter, and the object region in the image is extracted based on the object feature amount. Therefore, when extracting an object area from an image, the user only needs to perform an operation for specifying an object designation line in the image, so that the burden on the user can be reduced. In addition, since the object region is determined by extracting the object feature amount based on the dominant parameter in the first region in the inner region of the object designation line and the second region in the outer region, more accurate object extraction is realized. be able to.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 shows an object extracting apparatus as a first embodiment of the present invention. The object extraction apparatus includes an image input unit 101, an image storage unit 102, an image display unit 103, an operation unit 104, a control unit 105, and an object extraction unit 110. The image input unit 101, the image storage unit 102, the image display unit 103, the operation unit 104, and the object extraction unit 110 are connected to the control unit 105.

  The image input unit 101 is a part that inputs image data and supplies it to the control unit 105, and is, for example, a tuner that receives and decodes broadcast waves, an Internet connection unit, or a photographing device such as a camera. The image storage unit 102 stores the input image data via the control unit 105, and writing and reading of the image data is controlled by the control unit 105. Note that the image data used here is not particularly limited in image size, color format, and compression format such as JPEG and GIF.

  The image display unit 103 displays an image indicated by the image data supplied from the control unit 105. The operation unit 104 includes input devices such as a keyboard, a mouse, and a touch pen operated by the user, and supplies commands to the control unit 105 in accordance with user operations.

  The object extraction unit 110 is a part that extracts an object area from the image displayed on the image display unit 103.

  The control unit 105 controls the image input unit 101, the image storage unit 102, the image display unit 103, and the object extraction unit 110 according to the operation of the operation unit 104.

  As shown in FIG. 1, the object extraction unit 110 includes an initial object / background region determination unit 106, a dominant color estimation unit 107, an object feature amount extraction unit 108, and an object region determination unit 109, as shown in FIG.

  The initial object / background area determination unit 106 determines an initial object area (first area) surrounded by an object specification line specified by the user's operation of the operation unit 104 for the image displayed on the image display unit 103. To do. The initial object / background area determination unit 106 determines an initial background area (second area) based on the initial object area. The object designation line, the initial object area, and the initial background area will be described later.

  The dominant color estimating unit 107 extracts the dominant color (dominant parameter) of the initial object region and the initial background region determined by the initial object / background region determining unit 106.

  The object feature amount extraction unit 108 extracts an object feature amount based on the dominant color extracted by the dominant color estimation unit 107.

  The object region determination unit 109 determines a final object region based on the object feature amount extracted by the object feature amount extraction unit 108.

  The control unit 105 controls the initial object / background region determination unit 106, the dominant color estimation unit 107, the object feature amount extraction unit 108, and the object region determination unit 109 and displays them on the image display unit 103. Existing image data can be supplied.

  Next, an object extracting operation of the object extracting apparatus having such a configuration will be described.

  In the object extraction operation, as shown in FIG. 2, first, the control unit 105 causes the image display unit 103 to display an image (step S1). In step S <b> 1, the control unit 105 reads out image data from the image storage unit 102 and supplies the image data to the image display unit 103 in accordance with the operation of the operation unit 104 by the user. The control unit 105 may supply the image data input to the image input unit 101 to the image display unit 103 in accordance with a user operation on the operation unit 104. The image display unit 103 displays an image according to the supplied image data. In the following description, it is assumed that the displayed image is a target image for object extraction, and the target image is an image as shown in FIG.

  After executing step S <b> 1, the user designates an object area to be extracted by operating the operation unit 104 while viewing the target image displayed on the image display unit 103. As a specification method, a method of drawing a line roughly surrounding an object is used. This line is called an “object designation line”. It is desirable that the object to be extracted is completely included in the object designation line, but there is no problem even if the object slightly protrudes. If the object designation line is not a closed curve, the line start point and end point are connected to form a closed curve. The control unit 105 accepts the object designation line designated by the operation of the operation unit 104 (step S2). For example, an object designation line is formed on the target image in FIG. 3 as shown in FIG.

  After execution of step S2, the initial object / background area determination unit 106 determines an initial object area based on the object designation line supplied as data from the control unit 105 (step S3). Specifically, the initial object area is an internal area surrounded by an object designation line.

  Next, the initial object / background area determination unit 106 determines an initial background area (step S4). An area obtained by subtracting the initial object area from the area obtained by enlarging the initial object area is determined as the initial background area. That is, the initial background area is an area surrounding the object area. The amount of enlargement of the initial object area is not particularly limited, but it is preferable that the area of the initial object area and the area of the initial background area are substantially equal. For example, as shown in FIG. 5, the initial background area is determined for the initial object area.

  When the initial object area and the initial background area are determined by the initial object / background area determination unit 106, the dominant color estimation unit 107 extracts the dominant colors of the initial object area and the initial background area from the image data (step S5).

  The dominant color of the initial object area refers to the color of a pixel that appears frequently in the initial object area in the target image but does not appear much in the initial background area. The dominant color of the initial background area is the opposite. As the simplest dominant color extraction method, color histograms of the initial object region and the initial background region are generated from the target image, and the dominant color may be obtained based on the number of bins for each color. For example, when the target image is an image of RGB 8-bit color image data, the number of bins (number of pixels, where [R, G, B] = [[0,0,0] to, [255,255,255]]) , RGB, one pixel) is obtained, and the dominant color is obtained according to the number of bins of each of 16,777,216 colors.

  When this dominant color is extracted, it is possible not to simply compare the number of bins, but to use only colors having a greatly different ratio as the dominant color. For example, if the larger number of bins is Cb and the smaller number of bins is Cs, only the color satisfying Cb> Cs * Th can be set as the dominant color on the Cb side. Here, Th is a real value equal to or greater than 1.0. For example, by setting Th to be 1.2, a color having a slight difference in the number of bins can be removed and a more appropriate dominant color can be obtained. When the area of the initial object area and the area of the initial background area are different from each other, the bin numbers are divided by the respective areas, normalized, and then compared.

  The dominant color extraction method is not limited to the above. For example, when the difference in the number of bins is a small difference, processing such as not using the dominant color may be added. In addition to color, dominant parameters (that is, dominant luminance and dominant texture) may be extracted based on characteristics obtained from the target image such as luminance (mainly in the case of a monochrome image) and texture. The dominant parameter may be at least one of the color, brightness, and texture of the image.

  When the dominant color is extracted, the object feature amount extraction unit 108 extracts the object feature amount based on the extracted dominant color (step S6). The simplest object feature amount is the dominant color of the initial object area extracted in step S5. For example, when the target image is an image using RGB 8-bit color image data, the object feature amount is [R, G, B] = [[0,0,0], [128,0,128], [64, 32,192],. ] Is represented by an RGB data group.

  The object feature amount is not limited to the above, but by machine learning such as a support vector machine, according to the dominant color of each of the initial object region and the initial background region, the object and the background are classified into two classes (clustering), The object feature amount may be extracted based on the result.

  When the object feature amount is extracted, the object region determination unit 109 determines a final object region (step S7). Specifically, pixels satisfying the object feature amount extracted in step S6 are extracted from a region (hereinafter referred to as “extraction region”) that combines the initial object region and the initial background region, as shown in FIG. Is done. The extracted pixel group itself may be determined as the object region, but an object region that has been shaped by morphological processing such as closing may be used as the object region.

  After execution of step S7, although not shown in FIG. 2, the control unit 105 receives the object area, displays an image obtained by cutting the object area from the target image on the image display unit 104, and outputs the object area data itself. For example, an image of the object area is displayed on the image display unit 104 as shown in FIG. The output destination of the object area data or the image data of the object area portion is not limited to the image display unit 104, and may be output to the image storage means 102, and may be distributed via the Internet. Not limited.

  By performing the processes in steps S1 to S7 described above, it is possible to realize more accurate object extraction while reducing the operation burden on the user when extracting an object region from the target image. In particular, in the first embodiment, it is only necessary to designate the object so as to roughly surround the object, so that the burden on the user is reduced. Compared with the single point designation in Patent Document 2, the object designation can be performed more intuitively. In addition, since the object feature amount is determined based on the dominant colors of the initial object area and the initial background area, the processing load can be very light.

  In the determination of the initial background area in step S4, if the initial background area is too small, the object color is included in the initial background area when the object designation line by the user passes through the actual range of the object. The ratio increases, and the extraction accuracy of the object area deteriorates. On the other hand, if the initial background area becomes too large as in the entire screen, the object area extraction accuracy also deteriorates in the case where the object color is included at a location away from the object. Further, as the parameter extraction area increases, the processing load also increases. Even considering the balance between accuracy and processing load, it is more preferable that the areas of the initial object area and the initial background area are substantially equal.

  In the first embodiment described above, the object region is extracted assuming that the target image is a still image. However, the object region may be extracted for each frame using the target image as each frame of the moving image. .

  Next, an object extraction operation for extracting an object area from a moving image will be described as a second embodiment of the present invention.

  In the object extraction apparatus of FIG. 1, it is assumed that at least the image input unit 101, the image storage unit 102, the image display unit 103, and the control unit 105 have a function for handling moving images. Image data indicating a moving image is input from the image input unit 101, and the image data is supplied to the image storage unit 102 or the image display unit 103 via the control unit 105. The image storage unit 102 stores image data indicating the supplied moving image, and the image display unit 103 displays the moving image in accordance with the image data indicating the supplied moving image.

  In the object extraction operation, as shown in FIG. 8, first, the control unit 105 causes the image display unit 103 to display the first frame of the moving image (step S11), and the object designation line designated by the operation of the operation unit 104. Is accepted (step S12). After execution of step S12, the initial object / background region determination unit 106 determines an initial object region based on the object designation line supplied as data from the control unit 105 (step S13), and then determines an initial background region (step S13). Step S14). After execution of step S14, the dominant color estimation unit 107 extracts the dominant color of the initial object region and the initial background region from the image data (step S15). When the dominant color is extracted, the object feature amount extraction unit 108 extracts the dominant color. An object feature amount is extracted based on the dominant color (step S16). Then, the object area determination unit 109 determines the final object area of the first frame (step S17). The operations in steps S11 to S17 are the same as those in steps S1 to S7 shown in FIG. 2 except that the target image is the first frame of the moving image.

  At the end of the execution of step S17, the object area of the first frame has been extracted. Based on this, the object areas of the second and subsequent frames are extracted.

  First, the control unit 105 makes the frame number k equal to 2 (step S18), and expands the object area of the (k−1) th frame (step S19). That is, an extended area centered on the object area of the previous frame (first frame when k = 2) is set. The size of the extension region is sufficient if it can cover the range in which the object moves between adjacent frames ((k-1) th frame and kth frame).

  Next, the object area determination unit 109 extracts an object area for the k-th frame (current frame) according to the object feature amount extracted in step S16 from the extension range set in step S19 (step S20). . The process of step S20 is completely equivalent to the process of extracting the object area by regarding the object extension area as the “extraction range” (see FIG. 6) in step S7. Therefore, this object extension area is called a “next frame extraction range”.

  After execution of step S20, the control unit 105 determines whether or not the extraction of the object area for all the frames of the moving image has been completed (step S21). If there is a next frame, the frame number k is incremented by 1 (step S22), and steps S19 and S20 are similarly executed to extract the object region. When the extraction of object areas for all frames is completed, the object extraction operation is terminated.

  For example, when the moving image changes from the first frame to the third frame as shown in FIG. 9A, the object extraction result for them changes as shown in FIG. 9B. First, an object designation line is drawn so as to surround an object designated by the user, that is, a spherical object, from the image of the first frame. An initial object area and an initial background area are set from the object designation line, and an object feature amount is obtained by executing steps S11 to S16. Here, the pixel group region that satisfies the object feature amount is set as the object region in the first frame. An object extension area expanded around this object area is set as the next frame extraction range. Next, in the second frame image, an object region is extracted within the next frame extraction range, and the same processing as that for the second frame is performed for the subsequent third frame, fourth frame,. Are extracted.

  As described above, in the object extraction operation for extracting the object region from the moving image, the method in which the user designates the object designation line frame by frame as in the case of the still image increases the burden on the user. An extraction operation is performed to deal with it. The general moving image has a feature that the range in which an object moves between adjacent frames is narrow, and the color change of the background and the object is small between adjacent frames. Therefore, the object extraction operation of the second embodiment described above So these features are used. Therefore, if the user performs an object designation line designation operation for the first frame only, it is not necessary to perform an object designation line designation operation for the second and subsequent frames. Therefore, when extracting an object region from a still image, the burden on the user is reduced. It is possible to reliably extract an object from a moving image while reducing the burden of the above.

  When extracting an object region based on the object feature amount (object dominant color) in step S7 or S17, the extraction accuracy can be further improved by comparing with the initial object region. For example, as shown in FIG. 10A, an object designation line is designated for the target image (same as in FIG. 4), and as shown in FIG. Assume that the object region is extracted in the first embodiment. This object area is classified for each pixel connected component, and labels are assigned from a to d to become parts a to d. The pixel connected component is a collective part that can be regarded as one part in the object region. Here, for each part, what percentage of the part area is included in the initial object area is calculated, and if it is smaller than the threshold, the part is deleted from the object area. For example, when the threshold value is set as 50%, the part b is smaller than 50% in the example of FIG. 10B, so the part b is deleted from the object area as shown in FIG. 10C. Become. At this time, the object feature amount (object dominant color) may be relearned based on the remaining parts a, c, and d. The accuracy of the object area extraction can be further improved by calculating the ratio of the parts of the object area extracted in this way to be included in the initial object area and deleting the parts having a small value.

  Of course, the control unit 105 and the object extraction unit 110 in the apparatus of FIG. 1 may be configured by a computer, and the computer may execute the above-described object extraction operation according to a program stored in the memory.

  The present invention can be applied to application software such as image editing software for personal computers, and devices such as DVD recorders, cameras, and videophones.

DESCRIPTION OF SYMBOLS 101 Image input means 102 Image storage means 103 Image display part 104 Operation part 105 Control part 106 Initial object / background area determination part 107 Dominant color estimation part 108 Object feature-value extraction part 109 Object area determination part 110 Object extraction part

Claims (10)

An object extraction device for extracting an object region in an image,
Object designation line setting means for setting an object designation line consisting of a closed curve in the image according to an operation;
First area setting means for setting an inner area of the object designation line as a first area;
A second area setting means for setting, as a second area, an area obtained by subtracting the first area from an enlarged area including the first area;
Dominant parameter extracting means for extracting dominant parameters in the first region and the second region of the image;
Feature quantity extraction means for extracting an object feature quantity based on the dominant parameter;
An object extraction apparatus comprising: an area extraction unit that extracts the object area in the image based on the object feature amount.   The object extraction apparatus according to claim 1, wherein an area of the first region is substantially equal to an area of the second region.   The object extracting apparatus according to claim 1, wherein the second area is a portion in the vicinity of the first area of the image.   The object extracting apparatus according to claim 1, wherein the dominant parameter is at least one of color, luminance, and texture of the image.   The object extracting apparatus according to claim 1, wherein the feature amount extraction unit uses the dominant parameter in the first region as the object feature amount.   The feature amount extraction unit extracts the object feature amount based on a learning result by a machine including a support vector machine based on the dominant parameter in the first region and the second region. Object extraction device.   The area extraction means has a means for determining whether or not to include the part in the object area based on the ratio of the part in the first area as a part for each set part of the extracted object area. The object extracting device according to claim 1, wherein The image is a moving image;
The object designation line setting means, the first area setting means, the second area setting means, the dominant parameter extraction means, the feature amount extraction means, and the area extraction means are for the first frame image of the moving image. Work,
The region extracting means extracts the object region in the image of the current frame based on the object feature amount in a region obtained by enlarging the object region of the image of the previous frame for the image of the second and subsequent frames of the moving image. The object extracting apparatus according to claim 1, wherein the object extracting apparatus extracts the object. An object extraction method for extracting an object region in an image,
An object designation line setting step for setting an object designation line consisting of a closed curve in the image according to an operation;
A first area setting step of setting an internal area of the object designation line as a first area;
A second area setting step of setting an area obtained by subtracting the first area from an enlarged area including the first area as a second area;
Extracting dominant parameters in the first region and the second region of the image;
A feature amount extracting step of extracting an object feature amount based on the dominant parameter;
A region extracting step of extracting the object region in the image on the basis of the object feature amount. A program for extracting an object area in an image.
An object designation line setting step for setting an object designation line consisting of a closed curve in the image according to an operation;
A first area setting step of setting an internal area of the object designation line as a first area;
A second area setting step of setting an area obtained by subtracting the first area from an enlarged area including the first area as a second area;
Extracting dominant parameters in the first region and the second region of the image;
A feature amount extracting step of extracting an object feature amount based on the dominant parameter;
A region extraction step of extracting the object region in the image based on the object feature amount.

Images