JP3411971B2 - Human image extraction device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image input device such as a digital camera or a scanner, or a human image extraction device for extracting a human image from a human image input from an image data file that has already been captured by a computer. is there.

[0002]

2. Description of the Related Art FIG. 33 shows, for example, Japanese Patent No. 3034224.
It is a figure for explaining the whole composition of the conventional composition information extraction device shown by the gazette No. 4 by a subject extraction circuit.
The subject area is extracted from the input photographic image.

Next, the operation will be described. In a subject extraction circuit of a conventional composition information extraction device, an image is first divided into regions by a region dividing method using a Gabor filter bank, and contrast parameters of each divided region (differences between regions and the difference between each region and the entire image). By measuring parameters for measuring the difference) and inputting them to a neural network constructed in advance, each region is separated into a subject and a background.

Here, the neural network is
This is obtained as a result of learning subject area data and background area data classified by humans as learning data with respect to areas cut out from various images.

In another conventional foreground image extraction method disclosed in Japanese Patent Laid-Open No. 2000-36032, a foreground, that is, a subject region is extracted by obtaining a difference from a background image previously photographed. There is.

[0006]

Since the subject extraction circuit of the conventional composition information extraction device performs subject extraction based on area division by the Gabor filter bank, real-time processing is difficult. Further, although the region division by the Gabor filter bank is suitable for dividing a texture region, there is a problem that the region cutting accuracy (the ratio of the region boundary on the edge) is not good.

Further, since the subject extracting circuit of the conventional composition information extracting device is for extracting the subject from an arbitrary image, the human image extracting device of the present invention is capable of extracting the subject image. However, there is a problem that good results are not always obtained.

Further, in another conventional foreground image extraction method, the system must always hold the background image, which causes a problem that the user's time and effort at the time of photographing increases. Further, in order to extract a subject from an existing photographic image, a background image before the subject is photographed is required, and there is a problem that the subject cannot be extracted if the background image is not present.

The present invention has been made in order to solve the above-mentioned problems, and for an image taken by a bust shot in front of a uniform or blurred background of one or more persons, In real time, with good edge accuracy,
Moreover, it is an object of the present invention to provide a human image extraction device that can always extract a subject with high extraction accuracy.

[0010]

A person image extracting device according to the present invention comprises an image inputting means for inputting an image and a person area candidate determining means for determining a candidate for a person area from the image input by the image inputting means. And a person image extracting means for selecting an area forming a person image from the person area candidates determined by the person area candidate determining means and extracting the person image from the image, and outputting the extraction result of the person image extracting means. And an extraction result output means for performing the extraction result output.

Further, the person area candidate determining means is an edge extracting means for extracting an edge of the input image, a color of the input image is subtracted, and an index map in which each pixel of the image is represented by an ID based on the ID of each color. It is provided with image reducing means for producing and area map producing means for producing an area map based on the index map produced by the image reducing means.

Further, the area map creating means obtains a histogram of indexes for each pixel of the index map created by the image color-reducing means, and removes noise by substituting the index of the pixel with the index that maximizes the histogram. The processing is performed to exclude the minute area.

The edge extraction means performs edge extraction processing on each of the R, G, B planes of the image input by the image input means, and determines the maximum value of the edge strength in each plane of each pixel as the pixel value. The edge image is obtained by performing the processing with the edge strength of.

Further, the person image extracting means uses, in the preprocessing of the background removal, a conditional expression for determining whether or not the area in contact with the left and right edges of the image is the background area. It introduces weights that change depending on the coordinates.

Further, the person image extraction means performs the background removal processing while referring to the edge map created by the edge extraction means in the background removal processing.

The person area candidate determining means is adapted to determine a person area candidate according to the template.

The person area candidate determining means subtracts the color of the input image and creates an index map in which each pixel of the image is represented by an ID based on the ID of each color.
An area map creating means for creating an area map based on the index map created by the image color-reducing means, a template presenting means for presenting a template for cutting out a human image, and a template selected by the user are matched with the input image. Template fitting means to correct by
A person pose template DB, which stores templates of various person poses according to the number of subjects, is provided.

Further, the extraction result correction means for easily correcting the extraction result of the human image by the human image extraction means is provided.

Further, the extraction result correction means presents the correction target area to the user in order from the boundary area between the human area and the background area of the human area extraction result, and the user can easily select the result. It can be modified to.

[0020]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings. Embodiment 1. The person image extracting apparatus according to the first embodiment is capable of automatically extracting a person image as a subject in real time from an image input apparatus such as a camera or a scanner, or a person photograph image input from an existing image file. To do.

The overall configuration of the person image extracting device in the first embodiment will be described with reference to FIG. 1 is a diagram showing a first embodiment, and is a diagram showing an overall configuration of a person image extracting device.
As shown in FIG. 1, the person image extracting device 1 includes an image input device such as a camera or a scanner, or an image input unit 2 for inputting image data from an image data file already loaded into a computer to the person image extracting device 1. , A person area candidate determining unit 3 for determining a person area candidate for the input full-color image, an image color reducing unit 4 for reducing the color of the full-color image in the person area candidate determining unit 3, and an area division of the color-reduced image Then, the area map creating means 5 for creating an area map, the edge extracting means 6 for extracting the edges of the input image and creating the corresponding edge map, and the candidate of the person area determined by the person area candidate determining means 3 From the image, a person image extracting means 7 for extracting the person image from the image, and an extraction result outputting means for outputting the extraction result. It is those with a door.

Next, the outline of the operation of the person image extracting device 1 according to the first embodiment will be described with reference to FIG. FIG. 2 is a diagram showing the first embodiment and is a flow chart for explaining the outline of the processing of the person image extracting device. As shown in FIG. 2, first, the image input unit 2 extracts an image from the person image extracting apparatus 1
(Step S1-2). The captured image is input to the edge extraction unit 6 and the image color reduction unit 4. The image color-reducing means 4 reduces a full-color image (an image capable of expressing 16.7 million colors) to a preset number of colors and stores a color map for storing color-reduced color information, and each pixel value of the image is a color map ID. Create an index map created by replacing (step S1-3).

Next, in the area map creating means 5, based on the index map created by the image color reducing means 4, adjacent pixels having the same ID are connected 4 or 8
The index map is divided into areas by connecting them and the area map is created by newly assigning an ID to each of the divided areas (step S1-4).

On the other hand, the edge extraction means 6 applies an edge extraction operator to the input image to extract the edge of the image, and creates an edge map in which the pixel value on the edge is 1 and the other pixel values are 0. Create (step S1-4). The person image extracting means 7 removes the background area based on the edge map created by the edge extracting means 6 and the area map created by the area map creating means 5, and cuts out only the person image (step S1-5). Finally, the person image extracted by the extraction result output means 8 is output.

Next, the image color reduction means 4 will be described in detail. In the field of computer graphics, in order to save a frame memory for displaying an image on a display, research has been conducted to reduce a color image with good reproducibility. A representative example of this research is the color quantization method proposed by Xiaolin et al.
Xiaolin Wu, “Color Quantization by Dynamic Progra
mming and PrincipalAnalysis, ”Transaction on Grap
hics, Vol. 11, No. 4, 348-372, 1992).

In this method, the color data used in the original image is subjected to the principal component analysis, and the colors are classified on the plane orthogonal to the first principal component. First, the color data of each pixel is mapped on the axis of the first principal component. Then, a boundary point for classifying the mapped pixels into two is obtained. At this time, the boundary point is the value of the first principal component that minimizes the difference between the primary color and the representative color of each classified category. If further division is required, that is, if the specified number of colors has not been reached, the one with a large difference from the primary color is selected from among the categories, and the selected category is classified into two by similar means. . This process is repeated until the number of classified categories and the designated number of colors match. By doing so, the image after color reduction is a good reproduction of the original image.

The image color-reducing means 4 of the human-image extracting apparatus 1 according to the first embodiment subtracts the color of each pixel using the above-described method, and creates a color map in which IDs are given to the representative colors after color reduction. Then, by referring to the color map created for each pixel of the image, an index map having an ID corresponding to the color is created.

Next, the area map creating means 5 will be described with reference to FIGS.
This will be described in detail with reference to FIG. 3 to 6 show the first embodiment.
FIG. 3 is a flow chart for explaining the operation of the area map creating means, FIG. 4 is a view for explaining noise removal processing performed on the index map created by the image color-reducing means, and FIG. FIG. 6 is a diagram showing a result of extracting a pixel having a specific ID (here, 1) from the generated index map, and FIG. 6 is a diagram showing a result of the region map creating means 5 newly assigning an ID to FIG. Is.

Next, the operation will be described. For each pixel of the index map created by the image color reduction unit 4,
Pixels satisfying the following conditions are regarded as noise and are removed (step S2-2).

As shown in FIG. 4, when the IDs of the pixel of interest 10 (pixels surrounded by circles) and the surrounding pixels 9 are referred to and there are more than half of the IDs and the IDs of the pixel of interest 10 are different. , The target pixel 10 is regarded as noise, and the target pixel 10
Replace the IDs of with more than half the IDs.

The above-described noise-removed index map is divided into areas, and a unique ID is assigned to each area. Here, the following processing is performed for all IDs assigned to the index map.

One of the IDs assigned to the index map is selected. For example, select ID = 1 here. next
When the pixel with ID = 1 is extracted from the index map,
For example, as shown in FIG. 5 (step S2-3, step S2-
Four).

The extracted pixels and adjacent pixels are
A region is obtained by connecting by 4 connection or 8 connection (step S2-5). In FIG. 5, two regions 11 and 12 (hatched portions) are obtained.

Different IDs are assigned to the areas obtained by the above processing (step S2-6). For example,
As a result of re-assigning IDs to the respective areas in FIG.
The area 11 is the area 13 in FIG. 6, and the area 12 in FIG. 5 is the area 14) in FIG. By performing the above processing for all the IDs, a region map with a unique ID assigned to each region is created.

Next, the edge extraction processing by the edge extraction means 6 will be described in detail with reference to FIGS. Figure 7
9 is a diagram showing the first embodiment, FIG. 7 is a flow chart showing the operation of the edge extracting means, and FIG. 8 is a horizontal Sobe.
Figure showing the kernel of the l operator, Figure 9 shows the Sob in the vertical direction
It is a figure which shows the kernel of an el operator.

Since the image input by the image input means 2 is a color image, the image is composed of three planes R, G and B. The edge extraction process is usually performed by converting a color image into a grayscale image and then applying a differential operator to each pixel to obtain the intensity of the gradient. However, for example, when the colors of adjacent pixels are different but the brightness is the same, the difference becomes 0, and the edge cannot be extracted. Therefore, in the human image extracting apparatus 1 of the present invention, the edge extraction operator is applied to each of the R, G, and B planes, and the maximum edge strength of each pixel for each plane is defined as the edge strength of that pixel. To do.

The edge extraction operator is, for example, Sobel.
There is an operator. The Sobel operator is a horizontal 15 and vertical differential operator 16 shown in FIGS.
(FIGS. 8 and 9 are operators for 3 × 3 neighborhoods), and the values in the grids of FIGS. 8 and 9 represent the coefficients of the local sum of products operation. That is, the pixel value of the pixel of interest (x, y) is changed to f (x,
y) and the value f _* (x, y) of the gradient obtained as a result of applying the Sobel operator of size 2 * hk-1 is expressed by the following formula.

[0038]

[Equation 1]

W (x, y) is the coefficient of the Sobel operator, * is the direction of the operator, and if * = x, the operator 1 in the horizontal direction
5 is used, and if * = y, the vertical operator 16 is used.

The edge strength is obtained from the following equation from the horizontal and vertical gradients obtained by the above equation 1.

[0041]

[Equation 2]

Fx and fy are horizontal and vertical gradients, respectively.

In the edge extraction means 6, the Sobel operators 15 and 16 are applied to the R, G and B planes to obtain the edge strength of each plane (step S3-2), and for all pixels,
The maximum value of the edge intensity of each plane of each pixel is obtained, and the edge image is obtained by performing the processing in which the obtained maximum value is the edge intensity of that pixel (step S3-3 to step S3-
Five).

The edge image thus obtained is subjected to threshold processing to create an edge map in which the value of a pixel having an edge strength equal to or higher than the threshold is 1, and the values of other pixels are 0 ( Step S3-6).

Next, the person image extracting means 7 will be described with reference to FIGS.
This will be described with reference to FIG. 10 to 16 are diagrams showing the first embodiment, FIG. 10 is a flowchart of a pre-processing of the background removal processing in the person image extracting means, and FIG. 11 is a flowchart of the background removal processing in the person image extracting means 7. 12 is a diagram showing an example of an edge map created by the edge extraction means, FIGS. 13 to 15 are diagrams showing a processing process of the person image extraction means, and FIG. 16 is a left and right edge of the image where the arm portion of the person is shown. It is a figure which shows the image at the time of touching.

Next, the operation will be described. Since it is a condition that a person be photographed by a bust shot, there is a high probability that the regions in contact with the left and right edges of the image are background regions. Therefore, of the areas on the image that are in contact with the left and right edges, one that satisfies the following conditions is set as the background area. Area bordering the top edge of the image:

[0047]

[Equation 3]

Uth is a threshold value, upix is the number of pixels in the outer peripheral pixels of the area in contact with the upper edge of the image, and wIDth is the width (number of pixels) of the image. The area touching the edge on either side of the image:

[0049]

[Equation 4]

[0050]

[Equation 5]

[0051]

[Equation 6]

[0052]

[Equation 7]

Lth and Rth are threshold values, lpix and rpix are the outer peripheral pixels of the area, and are the number of pixels in contact with either the left or right edge of the image, height is the height (number of pixels) of the image, and w is a weighting factor. p is a peripheral pixel of the area and is in contact with either the left or right edge of the image, v (p) is a function with the y coordinate of p (vertical direction of the image) as a variable, and y is the y coordinate of p.

Here, the reason for introducing the weighting factor w in the equations 4 and 5 is the bust shot condition as shown in FIG. 16, which is a condition for determining whether or not the area in contact with the left and right edges of the image is the background area. In some cases, a part of the body from the shoulder to the arm may be photographed in contact with the edge of the image. Therefore, as shown in Expression 7, in particular, for a pixel in contact with the lower part of the image, Since it may be a region, the pixel y
The influence on the values obtained by Expression 4 and Expression 5 is suppressed according to the coordinates.

Based on the above conditions, it is determined as a pre-process of the background removal process whether or not the area in contact with the left and right edges of the image is the background area.

First, areas on the image that are in contact with the left and right edges are obtained (step S4-2). This can be done by focusing on the pixels on the left and right edges of the image based on the area map created by the area map creating means 5 and determining the area ID of the pixels to determine the target area.

Next, it is judged whether or not there is an area satisfying the above condition for all the obtained areas (step S4).
-3 ~ step S4-11).

First, the number of pixels in contact with the upper edge of the image is obtained, and it is judged by Expression 3 whether or not it is the background area (steps S4-4 and S4-5). If the condition of Expression 3 is satisfied, , That area is set as the background area (step S4-11).

When it is determined in the above processing that the pixel is not the background area, the number of pixels in contact with the left edge of the image is calculated, and it is determined whether or not the pixel is the background area by the equation 4 (steps S4-6 and S4). -7), if the condition of Expression 4 is satisfied, the area is set as the background area (step S4-11).

When it is determined in the above processing that the pixel is not the background area, the number of pixels in contact with the right edge of the image is calculated, and it is determined by Expression 5 whether or not the pixel is the background area (steps S4-8 and S4). -9), if the condition of Expression 5 is satisfied, the area is set as the background area (step S4-11).

For example, when the area map 18 created by the area map creating means 5 is as shown in FIG. 13, the black-painted area 19 in FIG. Become.

Then, the background area is removed from the remaining area. The condition that the region is the background other than the condition shown in the pre-processing of the background removal process is that there is no edge on the outer periphery of the region and at the boundary with the background region. That is, the area satisfying the following conditions is set as the background area.

[0063]

[Equation 8]

Eth is a threshold value, outpix is the number of pixels in contact with the background area in the area in contact with the background area, and epix is the number of pixels on the edge among the pixels in contact with the background area.

Here, since the area boundary and the edge position are not always coincident with each other when the epix in the equation 8 is obtained, the edge map created by the edge extracting means 6 is referred to in the vicinity of the target pixel. If there is an edge, the pixel is regarded as a pixel on the edge. Here, the pixel of interest and neighboring pixels are referred to.

First, an area in contact with the background area is obtained, and it is determined whether or not all the obtained areas are background areas based on the above conditions (steps S5-2 to S5-7).

Of the outer peripheral pixels of the area which is in contact with the background area,
The number of pixels outpix in contact with the background area is calculated (step S5-
3). Next, with respect to all the obtained pixels, the edge map created by the edge extraction means 6 is referred to, and it is checked whether or not an edge exists at a position near the pixel of interest and each pixel.
Then, the number of pixels epix in which the edge exists is obtained (step S5-4).

Whether or not the area is the background area is determined by the expression 8 (step S5-5), and if the condition of the expression 8 is satisfied, the area is set as the background area (step S5-6).

When the background area is found when the processing of steps S5-2 to S5-7 is completed for all areas in contact with the background area, the process returns to step S5-2 to repeat the processing (step S5 -8).

For example, when the result of the pre-processing of the background removal processing is as shown in FIG. 14 and the edge map 17 extracted by the edge extracting means is as shown in FIG. 12, the above processing is performed to obtain the result 20 of FIG. can get.

The area finally left by the background removal processing performed by the person image extraction means 7 is set as a person area, and the extraction result output means 8 outputs the person area.

According to the above described first embodiment, the following effects can be obtained. (1) A person image can be automatically extracted from an image. (2) Since the image color reduction unit 4 and the region map creation unit 5 divide the regions based on the color information, when the boundary between the background and the person is clear, the clipping precision is higher than that of the texture-based region division. (3) The algorithm used in the image color reduction unit 4 and the area map creation unit 5 is simple and can be processed in real time. (4) In the area map creating means 5, noise removal processing is performed to exclude minute areas. As a result, the number of areas after area division is reduced, and processing efficiency can be improved. (5) The edge extraction means 6 performs edge extraction processing on each of the R, G, and B planes, and the maximum value of the edge strength of each plane of each pixel is represented by the edge strength of that pixel to obtain a gray level. When extracting edges from the scale image, it is possible to solve the problem that "adjacent pixels have different hues, but proper edge extraction is not possible when the luminance is the same". (6) In the background removal pre-processing of the person image extracting means 7, a conditional expression for determining whether or not the area in contact with the left and right edges of the image is the background area is determined according to the coordinates of the pixels in contact with the edge. By introducing the changing weight, it is possible to cut out as a person region even if a part of the body is in contact with the edge of the image, for example, from the shoulder to the arm of the person. (7) Since the background removal processing is performed while referring to the edge map created by the edge extraction means 6 in the background removal processing of the person image extraction means 7, not only the image taken in front of a uniform background but also the focus Is only a person, and if the image has a blurred background, it is possible to appropriately cut out the person image even with an arbitrary background.

Embodiment 2. In the above-described first embodiment, the person area candidate determining means 3 of the person image extracting apparatus 1 has the configuration shown in FIG. 1, but another configuration will be described here. FIG. 17 is a diagram showing the second embodiment, and is a diagram showing an overall configuration of the person image extracting device. As shown in FIG. 17, the person area candidate determination unit 3 of the person image extraction device 1 according to the second embodiment has a template presentation unit 22 that presents candidates for a shooting pose and a template selected by the user. It is composed of a template fitting unit 23 that fits to a person image, and a person pose template DB 24 that stores a plurality of assumed shooting poses of a person.

Further, unlike the human image extracting means 7 in the first embodiment, the human image extracting device 1 in the second embodiment uses the human image extracting means 25 for extracting the human image using the template information. .

Next, the outline of the operation will be described. Image input means
When an image is input to the person image extracting device 1 by 2, the image color reduction means 4 and the area map creation means 5 create an area map in which the areas are divided and the divided areas are given IDs, as in the first embodiment. . Next, the template presenting unit 22 acquires a human pose template for cutting out a human image from the image from the human pose template DB 24 and presents it to the user. The user selects an optimum one for cutting out a person image from the image from the templates presented by the template fitting means 23, and fits the template according to the image. When the fitting of the template is completed, the human image extracting unit 25 cuts out the human image from the image along the template, and the extraction result output unit 8 outputs the result.

Details of the operation of the template presenting means 22 will be described below with reference to FIGS. 18 to 21. 18 to 21 are views showing the second embodiment, FIG. 18 is a flow chart for explaining the operation of the template presenting means, FIG. 19 is a view showing template data stored in the person pose template DB, and FIG. Is a human pose template
FIG. 21 is a diagram showing a dialog for searching a template according to the number of subjects from the templates stored in the DB, and FIG. 21 is a diagram showing a template display window for displaying the searched template.

The template presenting means 22 first presents the subject number input dialog 27 to the user in order to present a template according to the number of photographed subjects (step S6-2). Here, when the user inputs the number of subjects and presses the OK button, the template presenting means 22
Acquires template data corresponding to the number of subjects from the template data 26 stored in the person pose template DB 24 and displays the template in the template display window (step S6-3, step S6-5, template S6-6). When the user presses the cancel button,
The template presenting means 22 does nothing and ends the process (steps S6-4 and S6-9). At this time, the processing of the person image extracting device 1 also ends at the same time.

When the user selects one of the templates shown in the template display window 28, the template presenting means 22 stores the template ID (steps S6-7, S6-8), and the template is displayed. The ID is handed over to the fitting means 23.

Next, the operation of the template fitting means 23 will be described in detail with reference to FIGS. 22-
FIG. 25 is a diagram showing the second embodiment, FIG. 22 is a flow chart for explaining the operation of the template fitting means, and FIG. 23 is a template for fitting the template selected by the user from the template display window to the subject. FIG. 24 is a diagram showing a fitting dialog.
FIG. 25 is a flowchart for explaining the process of overlapping the template with the original image, and FIG. 25 is a diagram showing template data.

The template fitting means 23 acquires the corresponding template data from the person pose template DB 24 based on the template ID input from the template presenting means 22, and enlarges or reduces it according to the size of the original image (step S7- 2).

Next, an image in which the scaled-up and down template data overlaps the original image is created, and the image is displayed together with the template fitting dialog 29 shown in FIG. 23 (steps S7-3 and S7-4).

The user changes the template data so that the object and the template match by using the tool (here, the scroll bar) for enlarging / reducing and moving in the horizontal / vertical directions provided in the template fitting dialog 29 ( Steps S7-5 to S7-11). And
Finally, when the OK button is pressed, the changed template data is stored, the template data and the area map created by the area map creating means 5 are delivered to the person image extracting means 25, and the processing is terminated (steps S7-12, step S7-12). S7-1
3).

The process of overlapping the template data with the original image will be described in detail with reference to FIGS. 24 and 25. In the template data 30, as shown in FIG. 25, the pixel value at the boundary of the template is 1, the pixel value inside the boundary is 2, and the pixel value outside the boundary is 0.

Therefore, in the template fitting means 23, in step S7-3, the original image is first copied to the image display buffer of the template fitting dialog 29 (step S8-2), and all the pixels of the template data 30 are copied. On the other hand, a pixel having a pixel value of 1 is obtained (step S8-3, step S8-4), and the color of the pixel of the corresponding display buffer is changed to a preset color (step S8-5). , The template line drawing can be overlapped with the original image.

Next, the process of extracting a person image from an image based on the template data set by the user will be described with reference to FIG. FIG. 26 is a diagram showing the second embodiment and is a flow chart for explaining the process of the person image extracting means. The person image extracting means 25 sets the pixel value of the template data to 1 for all the pixels of the template data,
Alternatively, the ID of the pixel in the area map corresponding to the pixel that is 2.
(Area ID) is acquired (step S9-2, step S9-3),
It is stored (step S9-4). Then, the person image is extracted by extracting the regions corresponding to all the stored IDs (step S9-6).

According to the above described second embodiment, the following effects can be obtained. Although the human region candidate determining means 3 in the second embodiment is configured to extract the human image using the template, it is manual, but (1) the human image is extracted regardless of the background condition and the human photographing condition. Can be extracted. (2) Not only bust shots but also standing people can be extracted. In the first embodiment, the condition was that the uniform background or the background was blurred, and further, only the bust shot.

Third Embodiment In the person image extracting device 1 shown in the above-described first embodiment, the extraction result of the person image extracting means 7 is output by the extraction result outputting means 8. However, as shown in FIG. 27, the extraction result correcting means is provided, The extraction error may be easily corrected.

FIG. 27 is a diagram showing the third embodiment, and is a diagram showing the overall configuration of the person image extracting device. As shown in FIG. 27, the extraction result correction unit 31 detects that the region extracted by the human image extraction unit 7 is erroneously extracted from the extraction result of the human image extraction unit 7.
This area is for easily making corrections by pointing the area with a pointing device, a button, or the like.

Next, the operation of the human image extraction device 1 according to the third embodiment will be described with reference to FIGS. 28 to 32. FIG. 28
32 is a diagram showing the third embodiment, FIG. 28 is a diagram showing an extraction result correction window which is an example of a user interface of the extraction result correction means, and FIG. 29 is a portion for explaining the operation of the extraction result correction means. Flowchart diagram, Fig. 3
0 is a partial flow chart for explaining the operation of the extraction result correction means, FIG. 31 is a view showing an array for storing an area at the boundary between the background area and the person area, and FIG. 32 is for correcting an extraction result. It is a figure for demonstrating operation | movement of the extraction result correction means when doing.

A person image is input to the person image extracting device 1 through the image inputting means 2, the person area candidate determining means 3 determines a person area candidate, and the person image extracting means 7 extracts the person area. It is assumed that it is displayed in the extraction result correction window 32 of FIG. The extraction result correction means 31 sequentially sets the areas on the boundary between the background area and the person area as the correction target area, highlights the correction target area, and presents it to the user to prompt the correction.

In FIG. 28, the correction target area 33 is highlighted around the left shoulder of the person. If the highlighted correction target area 33 is an erroneous extraction area, the user selects the area with the input device,
The background area can be modified to a person area, or the person area can be modified to a background area.

First, all the IDs of the areas located at the boundary between the background area and the person area are obtained (step S10-2). this is,
Outer peripheral pixels of the person area determined by the person image extracting means 7 are excluded from the pixels which are in contact with the lower edge of the image, and the person area candidate determining means 3 including peripheral pixels of each pixel is also included.
By referring to the area map created in
You can ask for an ID.

Next, the obtained ID is stored in the array 34 (step S10-3).

Of the area IDs stored in the array 34, the ID stored at the beginning of the array is set as the correction target area (step S1
0-4), highlight the area (step S10-
Five).

If the user selects the extraction result correction window 3
When the next button is selected in step 2 (step S10-6),
It is determined whether or not the position where the current ID is stored is at the end of the array (step S10-7), and if it is not the end, the next stored ID is set as the correction target area (step S10-
8), the process is repeated from step S10-5. If the current storage position is at the end of the array (step S10-7),
The process is repeated from step S10-4.

Similarly, if the user selects the previous button in the extraction result correction window 32 (step S10-
11) It is determined whether or not the position where the current ID is stored is the head of the array (step S10-12), and if it is not the head, the previously stored ID is set as the correction target area (step S1).
0-13), and the process is repeated from step S10-5. If the current storage position is the beginning of the array (step S10-1
2) The area to be modified is set to the ID stored at the end of the array (step S10-14), and the process is repeated from step S10-5.

If the user selects the extraction result correction window 3
When the selection button is selected in step 2 (step S10-
9) It is determined whether the selected area is a person area (step S10-10). For example, it is assumed that the person area 5 in the array 34 and the person area (ID = 5) surrounded by a circle in the area map 36 of FIG. 32 are selected.

When the area (ID = 5) selected by the user is the person area, the area stored in the array 34 in the background areas adjacent to the area is deleted from the array (step S10-17). ). In the example of FIGS. 31 and 32, the area of ID = 6 is deleted from the array.

Next, the person area selected by the user (ID = 5)
An area not stored in the array 34 is obtained from the person areas adjacent to (step S10-18). In the example of the area map 36 of FIG. 32, ID = 15 is obtained.

The person area (ID = 5) selected by the user is changed to the background area (step S10-19), and the area ID is arrayed 3
Delete from 4 (step S10-20).

Next, the person area (I
Among the background areas adjacent to (D = 15), areas not stored in the array 34 are obtained (step S10-21). In the example of the area map 36 of FIG. 32, the area of ID = 5 is obtained.

All area IDs (ID = 15, ID =
5) is inserted into the position of the correction target area ID of the array 34 (step S10-22), and the processing is repeated from step S10-5. In the example of FIG. 31, the array 34 becomes the array 35.

Similarly, when the area selected by the user in step S10-10 is the background area, first, the areas stored in the array 34 among the adjacent person areas are obtained,
They are deleted from the array 34 (step S10-23).

Next, among the adjacent background areas, areas not stored in the array 34 are obtained (step S10-24), and the background area selected by the user is changed to the person area (step S1).
0-25), the ID of the area is deleted from the array 34 (step S10-26).

An area which is not included in the array 34 in the person area adjacent to the background area obtained in step S10-24 is obtained (step S10-27), and all the areas obtained above are arranged in the array 3
It is inserted into the position where the correction target area of No. 4 was stored (step S10-22), and the processing is repeated from step S10-5.

When the user selects the OK button in the extraction result correction window 32 (step S10-15), the extraction result output means 8 displays the extraction result.

The next button, the previous button, the select button, and the OK button of the extraction result correction window 32 may correspond to the keys of the keyboard on a PC. Also, if it is a mobile phone, it may correspond to a jog button or a confirm button.

[0108]

Since the present invention is constructed as described above, it has the following effects.

From the image input by the image input means,
It is possible to extract a human figure automatically and in real time.

Further, since the image color reduction means and the area map creation means divide the area based on the color information, when the boundary between the background and the person is clear, the cutout accuracy is higher than that of the area division based on the texture. Further, the algorithm used in the image color reduction means and the area map creation means is simple and can be processed in real time.

Further, in the area map creating means, noise removal processing is performed to exclude minute areas, and as a result, the number of areas after area division is reduced and processing efficiency can be improved.

In the edge extraction means, the edge extraction processing is performed on each of the R, G, and B planes, and the maximum value of the edge strength of each plane of each pixel is represented by the edge strength of that pixel. When extracting edges from a grayscale image, the problem that "adjacent pixels have different hues but proper edge extraction cannot be performed when the luminance is the same" can be solved.

Further, in the background removal pre-processing of the person image extracting means, the conditional expression for determining whether or not the area in contact with the left and right edges of the image is the background area is determined according to the coordinates of the pixels in contact with the edge. By introducing the weights that vary with each other, it is possible to cut out as a person region even if a part of the body is in contact with the edge of the image, for example, from the shoulder to the arm of the person.

In the background removal processing of the person image extracting means,
Since the background removal processing is performed while referring to the edge map created by the edge extraction means, it is not limited to an image taken in front of a uniform background, but an arbitrary image if the image is focused only on a person and the background is blurred. It is possible to appropriately cut out a human figure even in the background of.

Also, by cutting out the human image using the template, the human image can be extracted irrespective of the background though it is a manual operation.
Even standing people can be extracted.

The extraction result correction means can easily correct the human image extraction result.

Further, the extraction result correction means presents the user with the correction target area in order from the boundary area between the human area and the background area of the human area extraction result, and the user can easily obtain the result simply by selecting it. This is easier and more efficient than the method of manually tracing the boundary between the person and the background.

[Brief description of drawings]

FIG. 1 is a diagram showing a first embodiment and is a diagram showing an overall configuration of a person image extracting device.

FIG. 2 is a diagram showing the first embodiment and is a flowchart for explaining the outline of the processing of the person image extracting device.

FIG. 3 is a diagram showing the first embodiment and is a flow chart for explaining the operation of the area map creating means.

FIG. 4 is a diagram showing the first embodiment and is a diagram for explaining noise removal processing performed on an index map created by an image color reduction unit.

FIG. 5 shows the first embodiment, and is a diagram showing a result of extracting a pixel having a specific ID (here, 1) from the created index map.

FIG. 6 is a diagram showing the first embodiment and is a diagram showing a result in which the area map creating means newly gives an ID to FIG. 5;

FIG. 7 shows the first embodiment and is a flowchart showing the operation of the edge extracting means.

FIG. 8 is a diagram showing the first embodiment and is a Sobel in a horizontal direction.
It is a figure which shows the kernel of an operator.

FIG. 9 is a diagram showing the first embodiment and is a Sobel in a vertical direction.
It is a figure which shows the kernel of an operator.

FIG. 10 shows the first embodiment and is a flowchart of a pre-process of the background removal process in the person image extracting means.

FIG. 11 shows the first embodiment and is a flowchart of background removal processing in the person image extracting means.

FIG. 12 is a diagram showing the first embodiment and is a diagram showing an example of an edge map created by an edge extracting unit.

FIG. 13 is a diagram showing the first embodiment and is a diagram showing a processing process of a person image extracting means.

FIG. 14 is a diagram showing the first embodiment and is a diagram showing a processing process of a person image extracting means.

FIG. 15 is a diagram showing the first embodiment and is a diagram showing a processing process of the person image extracting means.

FIG. 16 is a diagram illustrating the first embodiment and is a diagram illustrating an image when the arm portion of a person touches the left and right edges of the image.

FIG. 17 is a diagram showing the second embodiment and is a diagram showing an overall configuration of a person image extracting device.

FIG. 18 is a diagram showing the second embodiment, and is a flowchart for explaining the operation of the template presenting means.

FIG. 19 is a diagram showing the second embodiment and is a diagram showing template data stored in a person pose template DB.

FIG. 20 is a diagram showing the second embodiment and is a diagram showing a dialog for searching a template corresponding to the number of subjects from the templates stored in the person pose template DB.

FIG. 21 is a diagram showing the second embodiment and is a diagram showing a template display window for displaying a searched template.

FIG. 22 is a view showing the second embodiment and is a flow chart for explaining the operation of the template fitting means.

FIG. 23 is a diagram showing the second embodiment and is a diagram showing a template fitting dialog for fitting the template selected by the user from the template display window to the subject.

FIG. 24 is a diagram showing the second embodiment and is a flowchart for explaining the process of overlapping the template with the original image.

FIG. 25 is a diagram showing the second embodiment and is a diagram showing template data.

FIG. 26 is a diagram showing the second embodiment and is a flowchart for explaining the process of the person image extracting means.

FIG. 27 is a diagram showing the third embodiment and is a diagram showing an overall configuration of a person image extracting device.

FIG. 28 is a diagram showing the third embodiment and is a diagram showing an extraction result correction window which is an example of a user interface of the extraction result correction means.

FIG. 29 shows the third embodiment and is a partial flowchart for explaining the operation of the extraction result correction means.

FIG. 30 shows the third embodiment and is a partial flow chart for explaining the operation of the extraction result correction means.

FIG. 31 is a diagram showing the third embodiment and is a diagram showing an array for storing an area at the boundary between the background area and the person area.

FIG. 32 shows the third embodiment and is a diagram for explaining the operation of the extraction result correction means when correcting a certain extraction result.

FIG. 33 is a diagram for explaining the overall configuration of a conventional composition information extraction device.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 person image extraction device, 2 image input means, 3 person area candidate determination means, 4 image color reduction means 4, 5 area map creation means 5, 6 edge extraction means, 7 person image extraction means,
8 Extraction Result Output Means, 9 Surrounding Pixels, 10 Target Pixels, 11 to 14 Regions, 15 Horizontal Differential Operators, 16 Vertical Differential Operators, 17 Edge Maps, 18 Region Maps, 19 Black Filled Regions, 22 Template presenting means, 23 Template fitting means, 24 Person pose template DB, 25
Person image extraction means, 26 template data, 27 subject number input dialog, 28 template display window, 29 template fitting dialog, 30
Template data, 31 Extraction result correction means, 32
Extraction result correction window, 33 correction target area, 34,
35 sequence, 36 region map.

─────────────────────────────────────────────────── ─── Continuation of the front page (58) Fields surveyed (Int.Cl. ⁷ , DB name) G06T 1/00 340 G06T 7/00 150 JISST file (JOIS)

Claims (8)

(57) [Claims] 1. An image inputting means for inputting an image, a person area candidate deciding means for deciding a person area candidate from the image inputted by the image input means, and a person area candidate decided by the person area candidate deciding means. A person image extracting unit that selects a region forming a human image from the image and extracts the human image from the image; and an extraction result output unit that outputs the extraction result of the human image extracting unit. The means determines whether or not the area in contact with the edge of the image is a background area from the person area candidates based on a predetermined conditional expression, and removes the area determined to be the background area from the person area candidates. Tomo and performing pre-processing of the background removal
In the pre-processing of the background removal, on the left and right edges of the image
Article for determining whether or not the contact area is the background area
Changes according to the coordinates of the pixels that touch the left and right edges
A human image extraction device characterized by introducing weights . 2. The person area candidate determination means, edge extraction means for extracting an edge of an input image, color reduction of the input image, and an index map in which each pixel of the image is represented by an ID based on the ID of each color. 2. The person image extracting device according to claim 1, further comprising: an image color-reducing means for creating the area map and an area map creating means for creating the area map based on the index map created by the image color-reducing means. 3. The area map creating means obtains an index histogram for each pixel of each pixel of the index map created by the image color reducing means, and replaces the pixel index with the index that maximizes the histogram. The person image extracting device according to claim 2, wherein a small area is excluded by performing noise removal processing. 4. The edge extraction means performs edge extraction processing on each of the R, G, B planes of the image input by the image input means, and determines the maximum value of the edge strength of each plane of each pixel. 3. The human image extracting apparatus according to claim 2, wherein the edge image is obtained by performing processing for setting the edge strength of the pixel. 5. The person image extracting means refers to the edge map created by the edge extracting means, determines an area in which no edge exists from the candidate person areas as a background area, and determines that the area is a background area. The person image extracting device according to claim 2, wherein background removal processing is performed to remove the determined area from the person area candidates. 6. The person image extracting device according to claim 1, wherein the person region candidate determining means determines a person region candidate according to a template. 7. The person area candidate determining means reduces the color of the input image, and creates an index map that represents each pixel of the image by an ID based on the ID of each color,
An area map creating means for creating an area map based on the index map created by the image color-reducing means, a template presenting means for presenting a template for cutting out a human image, and a template selected by the user are matched with the input image. Template fitting means to correct by
The person image extracting apparatus according to claim 6 , further comprising: a person pose template DB that stores templates of various person poses according to the number of subjects. 8. The person image extracting device further presents a correction target area to a user in order from a boundary area between a person area as a result of the person area extraction by the person image extracting means and the other background area, 2. The person image extracting apparatus according to claim 1, further comprising extraction result correction means for correcting the person region extraction result only by the user selecting it.