JP2006244272A - Hand position tracking method, device and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce erroneous recognition of a similar object in a method for specifying the two-dimensional position of a hand from an image photographed by a camera. <P>SOLUTION: An image is inputted from an image picking up part 11, a limited area setting part 12A determines a tracking range, and a limited area image creating part 14 creates a camera image (limited area image) limited only to information within the limited range. A skin color tracking part 15 uses the limited area image to track the position of the hand according to skin color information. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an interface method and apparatus for operating a computer, home appliances, etc. according to the movement and position of a human hand.

  An interface for recognizing a bare hand and operating a computer or a household appliance according to the movement and position of the hand has been studied. The usage scene of the interface operated by the position and movement of the hand is assumed to be a normal room where there are multiple users at a distance from the screen and the background is not specified, and the user wears nothing on the body. It is desirable that it can be used without it. Furthermore, real-time performance that can follow the movement of the hand is required.

  It is actually difficult to recognize the hand position in such a situation.

  As an approach, there is a method of detecting the shape of the hand shape using shape restoration technology such as stereo vision, but the shape of the hand can be recognized with sufficient accuracy unless means such as direct close-up of the hand are used. It is difficult. When taking an image without performing close-up or the like, the rough shape (distance information from the camera) of the entire person is used. However, since it is difficult to recognize a hand only with a rough shape, erroneous recognition is likely to occur. In addition, in the case of a system that performs three-dimensional pointing, misrecognition of the pointing position also occurs because the measurement error of the fingertip position increases in proportion to the distance from the object being pointed. In addition, in the case of detecting and tracking the position of the hand mainly using the skin color information, there are cases where conditions other than the hand such as the face and surrounding objects are met, and thus erroneous recognition is unavoidable.

Here, the problems of typical prior art will be described. The technique described in Non-Patent Document 1 tracks an object in real time using an algorithm called Continuously Adaptive Mean Shift (CAMSSHIFT). May be misrecognized that there is the same color in the vicinity tracking background starting from the designated color detection position. In particular, the accuracy of color information is reduced when a shadow or the like enters an object due to lighting. The technique described in Non-Patent Document 2 uses distance (the one that is in the foreground), background difference (ignoring anything that is placed between the camera and the hand), and the condition of being near the face and skin color information. Face detection is difficult to identify the operator in complicated backgrounds such as when someone is looking over your shoulder and is strongly affected by face misrecognition. The technique described in Non-Patent Document 3 tracks each time-series image in the designated color region, and must use a designated color marker. If color information that has a high possibility of not being in the background is used and there is a designated color in the background, position recognition fails.
Gary R. Bradski, “Real Time Face and Object Tracking as a Component of a Perceptual User Interface” 1998 IEEE p. p. 214-219 Yasuaki Kaneji et al. “Identification Method of Cursor Position on Pointer Pointer” IEICE ITS Study Group 2002.1 Eiichi Hosoya et al. “Real World Interaction between Remote Locations Using Mirror Interface” FIT (Information Science and Technology Forum) 2003, p. p. 591-592

  An object of the present invention is to provide a hand position tracking method, apparatus, and program that reduce false recognition of similar objects.

According to the first aspect of the present invention, the hand position tracking method comprises:
A limited area determination step for determining a limited area in the image for limiting the search range of the hand using the captured image captured by the imaging means as an input;
A limited area image creating step of creating a limited area image which is image information obtained by deleting information outside the limited area from the captured image;
A skin color tracking step of tracking the region on the image that matches the skin color information that specifies the color of the skin color region with the limited region image as a search range.

  In this aspect, a range for tracking the position of the hand is determined, a camera image limited to information within the limited range (limited area image) is created, and the position of the hand is determined based on skin color information using the limited area image. To track. By limiting the range to be tracked, the information to be processed is reduced, the possibility of erroneous recognition is reduced, and the calculation cost is also reduced.

  According to the second aspect of the present invention, the limited area determining step generates distance information of each pixel of the image by a stereo vision technique from a plurality of images obtained from the first and second imaging means. And a limited area creating step of obtaining an area on the image that matches the target distance information for designating the search range from the distance information and outputting the area as a limited area.

  In this aspect, distance information is created by a stereo vision technique using two imaging means, and a limited region is calculated based on target distance information obtained from a target distance setting unit.

  According to the third aspect of the present invention, the limited region determination step may calculate the absolute value of the difference between the pixel values of every two adjacent images with respect to a plurality of consecutive image sequences obtained in time series from the imaging unit. A motion information creation stage for extracting points on the image whose sum is equal to or greater than a certain value, and a connected region set in which the area of the connected region defined by the adjacent relationship on the image of the extracted points is equal to or greater than a certain value is defined as a limited region Includes a limited area creation stage to output.

  In this aspect, an image of one imaging unit is input, motion information is created using a difference between images that are time-sequentially continuous, and a range where there is a motion is defined as a limited region.

According to a fourth aspect of the present invention, a hand position tracking method comprises:
An area on the image that matches the target distance information for specifying the search range from the distance information by creating distance information of each pixel of the image from a plurality of images obtained from the first and second imaging means by stereo vision technology A first limited region determination step for obtaining and outputting as a first limited region;
For a plurality of image sequences that are continuous in time series obtained from the first imaging means, extract points on the image in which the sum of absolute difference values of each pixel value for every two adjacent images is a certain value or more, A second limited region determination step of outputting a connected region set in which the area of the connected region defined by the adjacent relationship on the image of the extraction points is a certain value or more as a second limited region;
A limited area image creation step of creating a limited area image that is image information obtained by deleting information outside the limited area that is a product set of the first and second limited areas from the captured image;
A skin color tracking step of tracking the region on the image that matches the skin color information that specifies the color of the skin color region with the limited region image as a search range.

    In this aspect, the motion information and the distance information are used together to limit an area that is in motion and within a certain distance range as a search range.

  In the present invention, by limiting the range to be tracked, it is possible to reduce false recognition of similar color objects, which is a drawback of the color-based method, and to reduce calculation costs.

  Next, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
Referring to FIG. 1, the hand position tracking device according to the first embodiment of the present invention includes an imaging unit 11, a limited region setting unit 12 </ b> A, a limited region image creating unit 13, a skin color tracking unit 14, and a display unit 15. ing.

  In this embodiment, an image from the imaging unit 11 is input, a range to be tracked by the user is determined, a camera image (limited region image) limited to only information within the limited range is created, and the limited region image is used. The position of the hand is tracked by the skin color information. This embodiment shows a method for specifying a region manually (input from a file input or application user interface) instead of automatically determining a limited region based on movement and distance information. It can be used when only the target is recognized.

  In particular, this embodiment is intended for application to a mirror interface, and is one method for touching a menu icon (an object to which a command is input by touching it) on the screen. The reason for recognizing the menu is that it is meaningless if the application side only accepts commands from the menu even if a hand is found elsewhere. In addition, when the system is installed at a location other than the location of the menu icon, if there is a door in the background and it moves well and it is known that a person passes and there is no need to recognize the hand at that location, By designating a place as a recognition target, the opening and closing of the door and the background person can be ignored as noise, and in this case, the present embodiment can be used.

  Next, the function of each unit will be described with reference to the flowchart of FIG.

  The limited area determination unit 12A is for the user to designate in advance a specific area on the screen as a limited area (steps 101 and 102). In the case of a rectangular limited area, the limited area can be specified by specifying the coordinates of two points on the diagonal of the square or by specifying the coordinates of one point and the width and height of the square. In the case of a limited area other than a quadrangle, the limited area can be specified by specifying coordinates, size, and the like. When it is assumed that the hand position tracking device is used as the user interface, a range including a command designation area (icon or the like) on the screen that can be selected by the user can be set in advance as a limited area. Among the outputs of the limited region determination unit 12A, pixels within the limited range have a value of “1”, and other pixels have a value of “0”. In FIG. 1, three square areas are limited areas.

  The imaging unit 11 is a camera and inputs a human body image including a hand (step 103).

  The limited region image creation unit 13 inputs a camera image (reference image) from the imaging unit 11 (camera), and uses only the limited region created by the limited region determination unit 12A, and only the region limited within this limited region. A limited area image having the following information is created (step 104). In this limited area image, for the pixels in the limited area, the color pixel value at the position of the same pixel in the camera image (reference image) is held, and for the pixels outside the limited area, the same pixel in the camera image (reference image). The color pixel value at the position is “0”. In FIG. 1, an image of a part of the hand is in the right limited area among the three square limited areas.

  The skin color tracking unit 14 tracks the position of the hand based on the skin color in the limited region image created by the limited region image creating unit 13 (step 105) and displays it on the display unit 15 (step 106). The skin color tracking unit 14 can be realized by using an existing method such as CAMSHIFT.

  Instead of designating the limited area by the user, the limited area specifying unit 11A may read the limited area designation information from a file in which coordinates for designating the area or the like (limited area designation information) are stored.

  In this embodiment, tracking is limited only to an area necessary as a user interface, thereby reducing calculation cost and reducing misrecognition and ensuring command selection.

[Second Embodiment]
Referring to FIG. 3, the hand position tracking apparatus according to the second embodiment of the present invention includes imaging units 11 ₁ and 11 ₂ , a limited region determining unit 12B, a limited region image creating unit 13, a skin color tracking unit 14, and a display unit 15. It is composed of As shown in FIG. 4, the limited region determination unit 12 </ b> B includes a distance information creation unit 21, a target distance setting unit 22, and a limited region creation unit 23.

  The limited area image creation unit 13, the skin color tracking unit 14, and the display unit 15 are the same as those in the first embodiment.

  In the present embodiment, distance information is created by a stereo vision technique using two imaging units, and a limited region is calculated based on target distance information obtained from the target distance setting unit 22.

Commercially available cameras can be used as the imaging units 11 ₁ and 11 ₂ . For example, Sensor Technology Co., Ltd. STC-R640. When creating distance information using two commercially available cameras, a commercially available camera capable of non-interlaced synchronous control is used. A commercially available compound eye camera in which a plurality of special cameras are installed in advance can also be used. For example, Point Gray Research Inc. It is a camera that has been calibrated in advance for stereo vision, such as Digitallops.

  The target distance setting unit 22 sets a target distance range (distance and range from the camera) for recognizing a hand. The target distance range is a value that specifies where the target is from the distance information between the camera and the object obtained by stereo viewing. The target distance range is a distance 1 (front) and a distance 2 (back) from the camera. It consists of a range between two points (see FIG. 6). As a setting interface, there is a method of setting the distance by two numerical values (for example, between 1 m and 3 m). On the other hand, the “distance information” is a so-called depth map (depth information of the entire image viewed from the camera) for stereo viewing. The distance information of each pixel is changed to luminance, and the gray scale image is shown with the near part being bright (white) and the far part being dark (black). In other words, a picture with gradation is a distance image.

  The configuration of the target distance setting unit 22 is not limited to the above. For example, a method of presenting a pattern (object) at two points in front of the camera and measuring the distance of the pattern (object) from the camera with the stereo vision function to two points in the setting range, or from an arbitrarily shot image There is a method of creating a range image and determining the range by looking at the range image while changing the set range by trial and error.

  Another method of setting the target distance is to use the stereo vision function dynamically and continue in time series in the form of how many meters to how many meters from the distance based on the distance of the pixel closest to the imaging unit. The target distance is dynamically changed for each image. There is also a method of dynamically setting the distance range by setting a certain range in the distance range between the nearest pixel and the farthest pixel as a percentage.

  Note that the target distance Z is converted into a parallax value d using a conversion formula of d = f × B / Z using a camera focal length f and a camera baseline distance B used in the distance information creation unit 21 described later. (See FIG. 7), the processing amount of the limited area creating unit 23 can be reduced.

  The distance information creation unit 21 determines the position on the image where the same point is captured from two or more images based on the similarity of the images, and the principle of triangulation from the positional deviation (parallax) between the plurality of images. The stereo view for obtaining the distance from the camera on the reference image is performed (step 107 in FIG. 5).

Hereinafter, a case where two cameras (imaging units) are placed in parallel will be described as an example. The parallax between the two camera images taken by the camera 11 ₁ (right) and the camera 11 ₂ (left) placed in parallel in the horizontal direction at two fixed distances synchronized in time is captured from the camera. Distance information.

As a method for calculating the parallax, for example, there are the following methods. The camera 11 ₁ as a reference, the camera image captured by the reference camera 11 ₁ (reference image) in a camera image captured by the camera 11 _2, similar image subregion (block) between the particular size By calculating the degree, the parallax is calculated by performing block matching to obtain the corresponding pixel. When the parallax is d, the distance Z between the imaging means and the imaging object is Z = f × B / d. ). d = u−u ′, B is the distance between the camera 11 ₁ and the camera 11 _2, and f is the focal length of the camera. FIG. 7 shows an explanatory diagram. Here, the left image and the right image are images captured by the left camera and the right camera, respectively.

The distance image is generated by expressing the parallax between the right camera image and the left camera image captured at the same time by the pixel value of each pixel of the entire image. The parallax indicates that the position of the person is closer to the camera 11 ₁ as the value is larger, and the position of the person is farther from the camera 11 ₁ as the value is smaller.

  The configuration of the distance information creation unit 21 is not limited to the above. For example, when a commercially available stereo camera such as Digitallops is used, it is also possible to create distance information from the stereo camera and its attached program.

  FIG. 6 is a diagram for explaining the operation of the limited area creation unit 23. The limited area creating unit 23 inputs the distance image created by the distance information creating unit 21 and the range of the target distance set by the target distance setting unit 22, and represents the pixel value of each pixel in the distance image. Using the distance information thus obtained, only pixels having a distance (parallax) within the range set by the target distance setting unit 22 are extracted (step 108 in FIG. 5).

  For example, “1” is given to the pixels within the range set by the target distance setting unit 22 as being within a certain range of the hand, and other pixels are outside the certain range of the hand. A limited area limited to a certain distance range is created by a method such as giving a value of “0” as a thing.

  FIG. 8 shows that a limited area limited to the target distance is obtained from the camera image 1 (right) and the camera image 2 (left), and that a limited area image having image information only in the limited area is obtained.

[Third Embodiment]
Referring to FIG. 9, the hand position tracking apparatus according to the third embodiment of the present invention includes an imaging unit 11, a limited region determination unit 12 </ b> C, a limited region image creation unit 13, a skin color tracking unit 14, and a display unit 15. Yes. As shown in FIG. 10, the limited region determination unit 12 </ b> C includes a motion information creation unit 31 and a limited region creation unit 32.

This embodiment is as shown in FIG. Using the image (A _t ) of one imaging unit (camera) as an input, the limited region determination unit 12C creates motion information using the difference between images that are continuous in time series (step 109), and a range where there is motion. Is a limited area (step 110).

  The generation of the limited area image in the limited area image creation unit 13 after the calculation of the limited area and the skin color tracking of the skin color tracking unit 14 are the same as in the first embodiment.

  Hereinafter, the motion information creation unit 31 and the limited region creation unit 32 of the limited region determination unit 12C will be described in detail with reference to FIG.

The motion information creation unit 31 receives a plurality (k + 1) of captured images (A _tk , A _{t-k + 1} ,..., A _t ) that are continuous in time series and inputs two adjacent (continuous) in time series. An image composed of the sum of each pixel by calculating a difference absolute value (| A _{t−k + 1} −A _tk |,..., | A _t −A _t−1 |) of the captured image.

The calculated, the value for each pixel of S _t is set to 1 pixel exceeds a predetermined value, the pixel is not the case and 0, creates a bitmap B _t.

The configuration of the motion information creation unit 31 is not limited to the above. For example, as a method for determining the in motion pixel, in the above, and detects the motion area by the sum S _t of the difference absolute values of the image, a difference square sum

There is also a method of detecting by. Further, there is a method for detecting motion by creating a background difference that is a difference between a background image captured in advance and a camera input image. In addition, by calculating the optical flow, a moving pixel can be obtained.

Limited area creating unit 32, is created by the motion information generating unit 31 divides the pixels in B _t to the pixel set with adjacency (connected set), the number of pixels belonging to the region of each connected set A limited area is created by treating an area having a size equal to or smaller than a predetermined threshold as an error as the same area as a non-motion area (eg, setting the value to “0”). This means that “labeling” is performed to obtain the area of the object divided as one object, and the area having a very small area is ignored.

Figure 13 shows the manner in which the camera image A _t and the camera image A _t-1, to obtain a limited area is limited to a range of motion limited region image is obtained further having image information only limited areas with motion ing.

  According to the present embodiment, by determining the limited area using the motion, for example, even when an object having a color close to the skin color and the hue value exists in the same distance range from the camera as the hand, If the object is a still life or if the amount of movement is small, it can be ignored, and erroneous detection of the skin color object can be suppressed.

[Fourth Embodiment]
Referring to FIG. 14, the hand position tracking apparatus according to the fourth embodiment of the present invention includes imaging units 11 ₁ and 11 ₂ , limited region determination units 12B and 12C in the second and third embodiments, respectively, and an AND circuit. 16, a limited area image creation unit 13, a skin color tracking unit 14, and a display unit 15.

  In the present embodiment, in addition to the contents of the first embodiment, the limited area determination unit 13 shown in the first embodiment uses movement information and distance information together, and there is movement, so that a certain distance range is obtained. A certain area is limited as a search range (steps 102B, 102C, 111).

  That is, the logical product of the limited areas used in the second and third embodiments output from the limited area determining units 12B and 12C is output as the limited area.

Figure 16 is restricted area creates camera image 1 (right) by the distance from the camera image 2 (left), limited areas due to the movement is created from the camera image A _t and the camera image A _t-1, in both confined area This shows that the limited area image creating unit 13 can obtain a limited area image having image information only in an area where there is a motion and within a certain distance range.

  The program for realizing the function of the hand position tracking device described above is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed. May be. The computer-readable recording medium refers to a recording medium such as a flexible disk, a magneto-optical disk, and a CD-ROM, and a storage device such as a hard disk device built in a computer system. Furthermore, a computer-readable recording medium is a server that dynamically holds a program (transmission medium or transmission wave) for a short period of time, as in the case of transmitting a program via the Internet, and a server in that case. Some of them hold programs for a certain period of time, such as volatile memory inside computer systems.

It is a block diagram of the hand position tracking apparatus of the 1st Embodiment of this invention. It is a flowchart which shows the flow of a process of the hand position tracking apparatus of 1st Embodiment. It is a block diagram of the hand position tracking apparatus of the 2nd Embodiment of this invention. It is a block diagram of the limited area | region determination part in FIG. It is a flowchart which shows the flow of a process of the hand position tracking apparatus of 2nd Embodiment. It is explanatory drawing of limited area creation in 2nd Embodiment. It is explanatory drawing of the calculation method of parallax. It is explanatory drawing until limited area image creation in 2nd Embodiment. It is a block diagram of the hand position tracking apparatus of the 3rd Embodiment of this invention. It is a block diagram of the limited area | region determination part in FIG. It is a flowchart which shows the flow of a process of the hand position tracking apparatus of 3rd Embodiment. It is explanatory drawing of limited area creation in 3rd Embodiment. It is explanatory drawing until limited area image creation in 3rd Embodiment. It is a block diagram of the hand position tracking apparatus of the 4th Embodiment of this invention. It is a flowchart which shows the flow of a process of the hand position tracking apparatus of 4th Embodiment. It is explanatory drawing until limited area image creation in 4th Embodiment.

Explanation of symbols

11, 11 ₁ , 11 ₂ Imaging unit 12A, 12B, 12C Limited region determination unit 13 Limited region image creation unit 14 Skin color tracking unit 15 Display unit 16 AND circuit 21 Distance information creation unit 22 Target distance setting unit 23 Limited region creation unit 31 Motion information creation unit 32 Limited area creation unit 101-111 steps

Claims (9)

A method of tracking a hand position by tracking a skin color region on a captured moving image obtained by imaging an imaging target by an imaging means,
A limited area determination step of determining a limited area in the image for limiting the search range of the hand using the captured image captured by the imaging means as an input;
A limited area image creating step of creating a limited area image which is image information obtained by deleting information outside the limited area from the captured image;
A skin color tracking step of tracking a region on the image that matches the skin color information that specifies the color of the skin color region with the limited region image as a search range.   The limited region determining step includes a distance information creating step of creating distance information of each pixel of the image from a plurality of images obtained from the first and second imaging means by a stereo vision technique, and a search range from the distance information. The hand position tracking method according to claim 1, further comprising a limited area creating step of obtaining an area on the image that matches target distance information to be specified and outputting the area as the limited area.   The limited region determination step is performed on an image in which a sum of absolute differences of pixel values of two adjacent images is equal to or greater than a certain value with respect to a plurality of continuous image sequences obtained from the imaging unit. A motion information creation stage for extracting points, and a limited area creation stage for outputting a connected area set in which the area of the connected area defined by the adjacent relationship on the image of the extracted points is a certain value or more as the limited area, The hand position tracking method according to claim 1. A method of tracking a hand position by tracking a skin color region on a captured moving image obtained by imaging an imaging target by an imaging means,
The distance information of each pixel of the image is created from a plurality of images obtained from the first and second imaging means by a stereo vision technique, and on the image that matches the target distance information for designating the search range from the distance information A first limited area determining step for obtaining the area of the first limited area and outputting it as a first limited area;
For a plurality of image sequences that are continuous in time series obtained from the first imaging means, extract points on the image in which the sum of absolute difference values of each pixel value for every two adjacent images is a certain value or more, A second limited region determination step of outputting a connected region set in which the area of the connected region defined by the adjacent relationship on the image of the extraction points is a certain value or more as a second limited region;
A limited area image creation step of creating a limited area image that is image information obtained by deleting information outside the limited area that is a product set of the first and second limited areas from the captured image;
A skin color tracking step of tracking a region on the image that matches the skin color information that specifies the color of the skin color region with the limited region image as a search range. An apparatus for tracking a hand position by tracking a skin color region on a captured moving image obtained by imaging an imaging target by an imaging means,
Limited area determination means for determining a limited area in the image for limiting the search range of the hand by using the captured image captured by the imaging means as an input;
Limited area image creating means for creating a limited area image, which is image information obtained by deleting information outside the limited area from the captured image;
Skin color information setting means for holding skin color information to be searched;
A hand position tracking device comprising: a skin color tracking unit configured to track a region on an image that matches the skin color information that designates a color of a skin color region using the limited region image as a search range.   The limited area determining means is a distance information creating means for creating distance information of each pixel of the image from a plurality of images obtained from the first and second imaging means by a stereo vision technique, and for specifying a search range 6. The hand position according to claim 5, comprising target distance setting means for holding target distance information, and limited area creating means for obtaining an area on the image that matches the target distance information from the distance information and outputting the area as the limited area. Tracking device.   The limited region determining means is for an image whose sum of absolute differences of pixel values of every two adjacent images is equal to or greater than a certain value for a plurality of continuous image sequences obtained in time series from the imaging means. A motion information creating means for extracting points; and a limited area creating means for outputting, as a limited area, a connected area set in which the area of the connected area defined by the adjacent relationship on the image of the extracted points is a predetermined value or more. Item 6. The hand position tracking device according to Item 5. An apparatus for tracking a hand position by tracking a skin color region on a captured moving image obtained by imaging an imaging target by an imaging means,
Target distance creation means for creating distance information of each pixel of an image by a stereo vision technique from a plurality of images obtained from the first and second imaging means;
First limited area determination means including limited area creation means for obtaining an area on the image that matches target distance information for designating a search range from the distance information and outputting the area as a first limited area;
For a plurality of image sequences that are continuous in time series obtained from the first imaging means, extract points on the image in which the sum of absolute difference values of each pixel value for every two adjacent images is a certain value or more, A second limited area determining means including a limited area creating means for outputting a connected area set in which the area of the connected area defined by the adjacent relationship on the image of the extraction points is a certain value or more, as a second limited area;
Limited area image creating means for creating a limited area image that is image information obtained by deleting information outside the limited area that is a product set of the first and second limited areas from the captured image;
Skin color information setting means for holding skin color information to be searched;
A hand position tracking device comprising: a skin color tracking unit configured to track a region on an image that matches the skin color information that designates a color of a skin color region using the limited region image as a search range. A hand position tracking program for causing a computer to execute the hand position tracking method according to claim 1.

Images