JP2004280148A - Method, apparatus and program for color model construction - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To construct a color model from a simple and natural action in which a user waves his or her hand at any place toward a camera. <P>SOLUTION: An action area detecting part 11A extracts a candidate for an arm area by generating a difference image between an frame image of an imaged motion image at each time and an frame image at a time t<SB>0</SB>. An arm area determining part 12A further generates difference images among generated difference images between frames, generates an image of which only the arm area is colored to make it a closed area to a colored pixel area and obtains an arm area image at each time. A flesh-colored area determining part 13A determines hand areas from each of arm area images. A flesh color model determining part 14A calculates an average color and a variance value of a color in the hand areas at each time to make them the flesh color. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a method and an apparatus for constructing a skin color model of a hand region of a user, which is suitable for an environment at the time of photographing, while photographing a user moving his / her hand with a video camera.
[0002]
[Prior art]
In the technology for realizing the above, in Patent Document 1, a hand color model (previously referred to as a skin color template in Patent Document) is used as an initial value and a hand shape template is determined based on a predetermined hand shape template. We propose a method of tracking the movement of the area and updating the skin color template of the tracked hand area sequentially.
[0003]
In Non-Patent Document 1, for example, it is assumed that only a hand is present within a certain distance range from a stereo camera, and a skin color model of the hand region is determined.
[0004]
On the other hand, in Non-Patent Document 2, a model posture in which the face and both hands are captured is set in advance so that the position of the face and hands can be easily estimated. At the time of photographing, a region where the face and hand are shown is determined from template matching based on the result of the user's posture, and a skin color model of the user is determined from the region color.
[0005]
[Patent Document 1]
JP-A-11-167455 [Non-Patent Document 1]
"Construction of Wearable 3D Desktop Environment Using Range Images" (Technology of IEICE, PRMU2001-222, pp.1-8, 2002-2)
[Non-patent document 2]
"Finger Feature Extraction and Sign Language Recognition Under Complex Background" (Image Recognition and Understanding Symposium Vol.II, pp.105-110, July 2002)
[0006]
[Problems to be solved by the invention]
The conventional method described above has the following problems.
[0007]
In the method of Patent Document 1, the user must first specify a skin color model (skin color template) or an accurate hand shape template as an initial value, and the shape of the moving hand must match the template. Must.
[0008]
The method of Non-Patent Document 1 requires a user to use an expensive stereo camera and place his hand only within a predetermined distance from the camera.
[0009]
The method of Non-Patent Document 2 forces the user to a predetermined posture.
[0010]
An object of the present invention is to provide a method and an apparatus for constructing a user's skin color model from a simple and natural motion in which the user shakes his hand at a free position toward the camera (FIG. 9A). is there.
[0011]
[Means for Solving the Problems]
In order to achieve the above object, a color model construction method of the present invention includes:
An image input stage for photographing the user;
A motion area detecting step of extracting a moving area in the captured image;
An arm area determining step of determining an arm area from the areas extracted in the moving area detecting step,
A skin color area determining step of determining an area representing a skin color from the arm areas determined in the arm area determining step;
And a skin color model determining step of determining a target skin color model from the color information of the skin color area determined in the skin color area determining step.
[0012]
Hereinafter, each stage will be described.
[0013]
First, in the moving area detecting step, an area in which a color change between frame images of the captured moving image in the image input step has been extracted as a moving area.
[0014]
In the next arm area determination step, first, only the human arm area is extracted from the motion area extracted in the previous step. By closing the result, a final arm region is created. If there are a plurality of closed regions, 1) the image is photographed with a certain size or more with respect to the image, 2) the ratio between the major axis and the minor axis (see FIG. 9 (2)) depends on individual differences. Utilizing the property of being substantially constant, a region having a specified area or more and a ratio of a minor axis to a major axis within a specified range from among a plurality of closed regions is defined as an arm region. FIG. 9B shows an image diagram of the determined arm region.
[0015]
In the next skin color region determination stage, the hand region is located at the tip of the arm region, and when a human shakes his hand, the movement amount is larger on the hand side and the smaller movement amount is on the body side. It is determined that, of the two end regions in the major axis direction, the one with the larger motion amount is the hand region. FIG. 9C shows an image of the determined hand region.
[0016]
In the final skin color model determination step, a skin color model is determined using the color information in the extracted hand region.
[0017]
In the present invention, a pointing system using a hand that creates a recognized color model as a pointer in order to extract a hand region from a natural and simple motion of waving a hand by a moving image processing and determine a skin color model Can be created. That is, it is possible to contribute to the construction of a pointing system that is highly convenient for the user.
[0018]
Here, the terms used in the present specification will be described.
1. This is the range of colors taken by the user's hand or face skin color area under the skin color model shooting environment. In general, each component value is represented by an average value and a variance value using an RGB or HSV color system, or represented by using a color histogram.
2. This is the minimum element constituting the pixel image. An image is formed by regularly arranging pixels in the x and y directions. The appearance of an image is determined by the color value of each pixel (generally, 256 levels of R, G, and G values are used using the RGB color system).
3. This is a still image cut out from a frame image photographed moving image for each unit time.
4. The difference image is an image generated according to the algorithm of FIG. 10 using two images as inputs. The algorithm first determines the value ^{L 2} for the input image pair I1, I2 (Step 31). If the L ² is in the predetermined A- threshold above, the pixel color difference B- than the pixel threshold, which maps pixel color of an image of time t _i, did not map Is white (steps 32 and 33), and this generated image is called a difference image (step 34). Here, between the two images L ² is the square sum of the differences of the color of each pixel, A- threshold in determining whether to generate a difference image from the input image pair, L ² And B-threshold is a threshold indicating the minimum value of the difference between the target pixel colors of the input image pair. If the value is equal to or less than this value, the pixels of interest are regarded as having the same color.
5. Arm region Here, the actual hand and arm regions of the user are collectively referred to as an arm region (the region shown in FIG. 9B).
6. The longer one of the long and short arm regions is called the long diameter, and the short one is called the short diameter.
7. An area that can be enclosed by connecting line segments around a closed area. When a closed region is created from a focused pixel, the closed region is created by connecting pixels on the outline.
8. The purpose is to reduce the line width of the thinned colored pixel region and extract the center line of one pixel in width.
9. The purpose is to recognize the result as a line segment while connecting adjacent colored pixels that have been thinned.
10. An optical flow that expresses the apparent moving distance of an object or small area in an image between frames using the difference between frames.
11. It represents the number of pixels having a certain range of values among the pixels in the histogram designated area. It can also be said that it represents the color distribution of the pixels in the area.
[0019]
BEST MODE FOR CARRYING OUT THE INVENTION
Next, embodiments of the present invention will be described with reference to the drawings.
[0020]
[First Embodiment]
FIG. 1 is a block diagram of a color model construction apparatus according to a first embodiment of the present invention, and FIG. 2 is a flowchart showing the flow of the processing.
[0021]
The color model construction apparatus according to the present embodiment includes an image input unit 10, a motion region detection unit 11A, an arm region determination unit 12A, a skin color region determination unit 13A, a skin color model determination unit 14A, an image memory 15, and an information storage unit 16. I have. The information storage unit 16 stores motion information, arm area information, color display area information, and the like obtained by each unit.
[0022]
Next, in the present embodiment, an operation of constructing a skin color model of a hand from a user's hand gesture will be described. Here, it is assumed that the time at which the user starts the hand gesture is t ₀ , and the time at which the user ends the motion is t _n .
[0023]
First, the image input unit 10 captures a user and stores the captured image in the image memory 15 (Step 20). The motion area detection unit 11A for each time of the frame image captured moving image stored in the image memory 15, by generating a difference image between the frame image at time t _0, to extract a candidate of the arm region (step 21A). The information on the extracted arm region is stored in the information storage unit 16. Type inter-frame difference image generation algorithm for time t _i of a frame image shows an image of the generated image in FIG. In the difference image thus created, the arm region at time t _i is ideally the color of the actual arm region, the arm region at time t ₀ is the actual background color, and the rest is white.
[0024]
In the next arm region determination unit 12A, first, a difference image is created for the inter-frame difference images generated by the motion region detection unit 11A, and the difference image is compared with the original inter-frame difference images. An image in which only the arm region is colored is generated (Step 22A). FIG. 4 shows an algorithm for generating a difference image from the inter-frame difference image at time t _{i and} the inter-frame difference image at time t _j to generate an arm region image at time t _j , and an image of an input / output image. Next, a closed region is generated for the colored pixel region. Among the generated closed regions, a region having an area equal to or larger than the threshold value and within a specified range of the ratio of the minor axis to the major axis is selected and stored in the information storage unit 16. This area becomes a temporary arm area at each time.
[0025]
The above processing is performed on each inter-frame image obtained by the motion area detection unit 11A, and an image having the largest extracted arm area area at each time is set as a true arm area image at that time.
[0026]
The skin color region determination unit 13A cuts out the region near both tips in the major axis direction into a rectangle of a designated size from the extracted region of each image (step 23A, FIG. 5). Along the distance of the barycentric coordinates. The obtained moving distance represents the approximate magnitude of the movement of the arm area, and the area having the larger value is regarded as the hand area.
[0027]
The last skin color model determination unit 14A calculates the average value and the variance of the color of the hand region at each time determined by the skin color region determination unit 13A, and uses them as a skin color model (step 24A).
[0028]
[Second embodiment]
FIG. 6 is a block diagram of a color model construction apparatus according to the second embodiment of the present invention, and FIG. 7 is a flowchart showing the flow of the processing.
[0029]
The color model construction apparatus according to the present embodiment includes an image input unit 10, a motion region detection unit 11B, an arm region determination unit 12B, a skin color region determination unit 13B, a skin color model determination unit 14B, an image memory 15, and an information storage unit 16. I have.
[0030]
Also in the present embodiment, the time at which the user starts the hand gesture is time t ₀ , and the time at which the user finishes the motion is time t _n .
[0031]
The motion area detection unit 11B in the present embodiment, by generating a difference image from the frame image at time t _i and the time t _{i + 1,} detects the there has been a change in color between adjacent frame area (step 21B). FIG. 8 shows an image of an algorithm and input / output images. In this case, an image representing a line segment surrounding the arm region, in which the body side of the arm region is not connected, is generated.
[0032]
The arm area determination unit 12B creates a closed area by connecting the unconnected end points on the body side after the thinning processing (step 22B). An image mapping the color of the closed region of the pixel time t _i of a frame image for, and images showing the arm region at time t _i.
[0033]
First, the skin color region determination unit 13B obtains an optical flow representing the amount of movement of each small region in the arm region image at each time, obtains a region having a large amount of movement from the obtained optical flows, and sets it as a hand region. Consider (step 23B).
[0034]
The last skin color model determining unit 14B extracts the color of the area near the barycenter coordinate of the hand area determined at each time as described above, obtains a histogram of each color component, and uses it as a skin color model (step 24B).
[0035]
In the above embodiments, all applicable ranges are not limited to these. For example, it is conceivable to apply the moving region detecting unit 11A and the arm region determining unit 12A in the first embodiment to the moving region detecting unit 11B and the arm region determining unit 12B in the second embodiment, respectively, and vice versa. Is also possible. Furthermore, the same can be said for the skin color area determination units 13A and 13B and the skin color model determination units 14A and 14B.
[0036]
In addition, the present invention records a program for realizing the function other than that realized by dedicated hardware on a computer-readable recording medium, and stores the program recorded on the recording medium in a computer system. It may be read and executed. The computer-readable recording medium refers to a recording medium such as a floppy disk, a magneto-optical disk, a CD-ROM, or a storage device such as a hard disk device built in a computer system. Further, the computer-readable recording medium is one that dynamically holds the program for a short time (transmission medium or transmission wave), such as a case where the program is transmitted via the Internet, and serves as a server in that case. It also includes those that hold programs for a certain period of time, such as volatile memory inside a computer system.
[0037]
【The invention's effect】
As described above, according to the present invention, it is possible to construct a skin color model only from a natural hand movement at a free position without forcing the user to pose or setting an initial value. The effect is obtained.
[Brief description of the drawings]
FIG.
It is a lineblock diagram of a color model construction device of a 1st embodiment of the present invention.
FIG. 2
5 is a flowchart illustrating a flow of processing of the color model construction device according to the first embodiment.
FIG. 3
In the first embodiment is an image diagram showing the inter-frame difference image generation algorithm for time t _i of the frame image time t ₀ of the frame image.
FIG. 4
FIG. 9 is an image diagram showing an arm area determination algorithm for generating an arm area image at time t _j from the frame difference image at time t _{i and} the frame difference image at time t _j in the first embodiment.
FIG. 5 is an image diagram for performing rectangular clipping of a region near both ends from an extracted arm region image in the first embodiment.
FIG. 6 is a configuration diagram of a color model construction device according to a second embodiment of the present invention.
FIG. 7 is a diagram showing a flow of processing of a color model construction device of a second embodiment.
[8] in the second embodiment is an image diagram showing the inter-frame difference image generation algorithm for time t _i at time t _{i + 1.}
FIG. 9 is a diagram showing an image of a hand region and an arm region and a hand region determined by the present invention.
FIG. 10 is a flowchart illustrating an algorithm for generating a difference image.
[Explanation of symbols]
Reference Signs List 10 Image input units 11A, 11B Motion region detection units 12A, 12B Arm region determination units 13A, 13B Skin color region determination units 14A, 14B Skin color model determination unit 15 Image memory 16 Information storage units 20, 21A to 24A, 21B to 24B, 31 ~ 34 steps

Claims (10)

A color model construction method of photographing a user and using the photographed moving image to construct a skin color model of a hand moved by the user,
An image input stage for photographing the user;
A motion area detecting step of extracting a moving area in the captured image;
An arm area determining step of determining an arm area from the areas extracted in the moving area detecting step;
A skin color region determining step of determining a region representing a skin color from the arm regions determined in the arm region determining step;
A method for constructing a color model, comprising: a skin color model determining step of determining a target skin color model from color information of a skin color region determined in the skin color region determining step. The method according to claim 1, wherein the moving area detecting step extracts an area where a color change between frame images of the captured moving image has occurred as a moving area. The arm region determination step is to extract only the human arm region from the extracted motion region, create a final arm region by closing the result, if there are multiple closed regions, In a plurality of closed regions, a region having a specified area or more and a ratio of a major axis, which is a longer diameter of the region, and a minor axis, which is a shorter diameter, within a specified range is determined as an arm region. Item 3. The method according to Item 1 or 2. 4. The skin color region determining step according to claim 1, wherein, of the end regions in the major axis direction determined in the arm region determining step, a region having a larger amount of motion is determined as a skin color region. 5. the method of. The skin color model determining step calculates an average value and a variance value of the color of the skin color region at each time determined in the skin color region determining step, and uses the calculated values as a skin color model. The method described in. The arm area determination step includes, after the thinning processing of the extracted motion area, creating a closed area by connecting the unconnected end points on the body side, and mapping the color of the frame image at each time for the pixels in the area. The method according to claim 1, wherein the determined image is determined as an image representing the arm region at the time. The skin color region determination step is to determine an optical flow representing the amount of movement for each small region in the arm region image at each time, to determine a region with a large amount of movement from the optical flow, consider the region as a hand region, The method of claim 6. 8. The method according to claim 7, wherein the skin color model determining step extracts a color of a region near the barycenter coordinate of the hand region at each time, obtains a histogram of each color component, and uses the histogram as a skin color model. A color model construction device that photographs a user and uses the photographed moving image to construct a skin color model of a hand moved by the user,
Image input means for photographing the user,
A motion region detecting means for extracting a moving region in the captured image,
Arm area determining means for determining an arm area from among the areas extracted by the motion area detecting means,
A skin color region determining unit that determines a region representing a skin color from the arm regions determined by the arm region determining unit;
A color model construction device having a skin color model determining unit for determining a target skin color model from color information of a skin color region determined by the skin color region determining unit. A color model construction program for causing a computer to execute the color model construction method according to any one of claims 1 to 8.

Images