JP2007233544A - Object position tracking method, device, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an object position tracking method for reducing the erroneous recognition of objects in similar colors by increasing a recognition rate. <P>SOLUTION: In a skin color probability distribution calculating part 14, the pickup image of each frame picked up by an imaging part 11 is inputted, and the probability distribution of skin colors within a designated rectangular region is calculated by using the calculation formula of probability that the skin colors exist or a lookup table showing relationship between pixel values and the probability. A rectangular region decision part 15 decides the rectangular region applied with width and height according to the total sum of the probability distribution values and the function whose value is the maximum of the probability distribution with the center of gravity of the probability distribution as a center. When a difference in size between the rectangular region decided in the previous loop and the rectangular region currently decided is within a threshold, or the rectangular region is decided to the same frame with designated loop frequency or more, or the size of the rectangular region exceeds the designated size, a skin color region decision part 16 decides that the rectangular region is the result of tracking the site in the current frame, and shifts to the calculation of the next frame. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an interface method for operating a computer, a household appliance, and the like by the movement and position of an object having a color and size specified in advance, such as a human hand.

  An interface for recognizing a bare hand and operating a computer or a household appliance according to the movement and position of the hand has been studied. The usage scene of the interface operated by the position and movement of the hand is assumed to be a normal room where there are multiple users at a distance from the screen and the background is not specified, and the user wears nothing on the body. It is desirable that it can be used without it. Furthermore, real-time performance that can follow the movement of the hand is required.

  It is actually difficult to recognize the hand position in such a situation.

  As an approach, there is a method of detecting the shape of the hand shape using shape restoration technology such as stereo vision, but the shape of the hand can be recognized with sufficient accuracy unless means such as direct close-up of the hand are used. It is difficult. When taking an image without performing close-up or the like, the rough shape (distance information from the camera) of the entire person is used. However, since it is difficult to recognize a hand only with a rough shape, erroneous recognition is likely to occur. In addition, in the case of a system that performs three-dimensional pointing, misrecognition of the pointing position also occurs because the measurement error of the fingertip position increases in proportion to the distance from the object being pointed. In addition, in the case of detecting / tracking the position of the hand mainly using skin color information, there are cases where the condition other than the hand such as the face and surrounding objects is met, and thus erroneous recognition cannot be avoided.

  Here, a typical prior art will be described.

  The technique described in Non-Patent Document 1 tracks an object in real time using an algorithm called Continuously Adaptive Mean Shift (CAMSHIFT). Then, by determining a rectangular area having a specified skin color distribution probability in the vicinity while changing the position and size as in the first aspect, the skin color area in the current frame is determined. However, the algorithm does not limit the size of the rectangular area that can be taken, and if there is only one skin color area in the image, it can be tracked relatively stably, but the hand is in front of the skin color object in the background. In the case of passing, there is a problem that a large rectangular area including both skin color areas is erroneously recognized as a skin color area immediately after passing.

  The technique described in Non-Patent Document 2 uses distance (the one that is in the foreground), background difference (ignoring anything that is placed between the camera and the hand), and the condition of being near the face and skin color information. Face detection is difficult to identify the operator in complicated backgrounds such as when someone is looking over your shoulder and is strongly affected by face misrecognition.

  The technique described in Non-Patent Document 3 is a method of tracking the position in the current frame based on the position of the previous frame image with respect to an area that takes the designated color, which is operated by a user having a designated color marker. . Here, color information that does not exist in the background is used, and position recognition fails if there is a designated color in the background.

The technique described in Non-Patent Document 4 adds a restriction based on distance and motion to the method of Non-Patent Document 1, and makes it possible to track a skin-colored object relatively stably even when there is a skin-colored area in the background. However, since the tracking target is only an area where there is movement, tracking fails when the hand is stopped, and there is a restriction that the hand must be placed within a specified distance range due to the distance limitation.
Gary R. Bradski, "Real Time Face and Object Tacking as a Component of a Perceptual User Interface" 1998 IEEE pp. 214-219 Yasuaki Kaneji et al. “Specifying the cursor position on the pointing pointer” (ITS Information Technology Society 2002) Eiichi Hosoya et al. “Real-world interaction between remote locations using mirror interface” FIT (Information Science and Technology Forum) 2003 pp. 591-592 Kitabata Miki et al. "A Study of Hand Tracking Method on Mirror Interface"

  An object of the present invention is to provide an object position tracking method, apparatus, and program that prevent erroneous recognition of similar color objects and increase the recognition rate.

The object tracking method of the present invention includes:
A method of tracking the position of an object by tracking an area of a predetermined color and size specified in advance on an imaged moving image obtained by imaging an imaging object by an imaging means,
A captured image input stage for inputting a captured image of each frame imaged by the imaging means;
Predetermined color for calculating the probability distribution of the predetermined color in a specified rectangular area using a formula for calculating the probability that the predetermined color exists or a look-up table representing the relationship between a pixel value and the probability A probability distribution calculation stage;
When the center of gravity of the probability distribution is at the center and the sum M ₀₀ of probability distribution values and the maximum value of the probability distribution are P _max , the width l _w and height l _h of one side of the rectangular area are M ₀₀ and P _max. A rectangular area determining step for determining a rectangular area as given by: l _w = f _w (M ₀₀ , P _max ), l _h = f _h (M ₀₀ , P _max ),
The size of the rectangular area determined in the rectangular area determination stage in the previous loop of the processing performed in the captured image input stage, the predetermined color probability distribution calculation stage, and the rectangular area determination stage and determined in the rectangular area determination stage in this loop When the difference in size with the rectangular area is within the threshold, or when the rectangular area has been redetermined more than the specified number of loops for the same frame, or the size of the determined rectangular area exceeds the specified size The rectangular area determined this time in the rectangular area determination stage is the tracking result of the object in the current frame, and the predetermined color area determination stage that returns to the captured image input stage and moves to calculation for the next frame;
Have

  The present invention inputs a moving image obtained by capturing an object having a color and size designated in advance, obtains a probability distribution in which a predetermined color exists from color information of a frame image, and has a rectangular area centered on the center of gravity of the probability distribution. When a predetermined end condition is satisfied, for example, when the size of the rectangular area exceeds the specified size, the rectangular area is set as an area where the object exists.

  When obtaining the above probability distribution, it is also possible to obtain an integrated probability distribution in consideration of three-dimensional distance information and further motion information, and obtain a rectangular area from this probability distribution.

  According to the first and sixth aspects of the present invention, it is possible to prevent a plurality of predetermined color areas from being erroneously recognized as one large predetermined color area by providing an upper limit value in the search rectangular area.

  According to the inventions of claims 2, 3, 7, and 8, as a probability distribution calculation method, not only a probability distribution based only on predetermined color information but also a probability distribution based on distance information is considered (claim 2), Considering the probability distribution based on the motion information (Claim 3) and using the probability distribution obtained by integrating these, the recognition rate can be increased. Here, the probability distribution based on the distance information dynamically gives a constraint on the distance at which the object will exist in the current frame based on the distance of the object searched in the previous frame. In the probability distribution based on the motion information, the tracking target is limited to only a predetermined color region having a motion on the input image.

  According to the fourth and ninth aspects of the present invention, if the rectangular area becomes larger than the specified value, it can be considered that the tracking of the object has failed. In this case, only the first probability distribution is considered. Thus, by performing a search specialized for searching only the predetermined color region, it becomes easy to re-search for an object that has failed to be tracked.

  According to the invention of claim 5, when the maximum value of the third probability distribution value is small, if the probability distribution is ignored, the tracking can be continued even if the object is stopped. . In addition, even if the calculation of distance information of the second probability distribution fails and changes significantly from the value of the previous frame, by ignoring it, the object can be tracked robustly with respect to the accuracy of the distance information calculation. It becomes possible to do.

  Next, embodiments of the present invention will be described with reference to the drawings. In the following embodiments, the object is limited to “hand” and the color is limited to “skin color”, but it goes without saying that the present invention is not limited to this.

[First Embodiment]
Referring to FIG. 1, the hand position tracking apparatus according to the first embodiment of the present invention includes an imaging unit 11, an image input unit 13, a skin color probability distribution calculation unit 14, a rectangular region determination unit 15, and a skin color region determination unit 16. And a data storage unit 17. As shown in FIG. 2, the skin color probability distribution calculation unit 14 includes a first probability distribution calculation unit 21, a distance information creation unit 22, a second probability distribution calculation unit 23, and an integrated probability distribution calculation unit 24.

  Next, the processing flow of the hand position tracking device of this embodiment will be described with reference to FIGS.

  First, a rectangular area designated in advance in the first frame and a rectangular area in the previous frame in the second and subsequent frames are given as initial areas (step 101).

  As shown in FIG. 4, the skin color probability distribution calculation unit 14 calculates the skin color probability value P (x, y) at all pixel positions (x, y) according to the following procedure (step 102). The obtained skin color probability values of all the pixels represent the skin color probability distribution.

1. In the first probability distribution calculation unit 21, the first probability distribution value P _c is determined from the relationship between a predetermined color and a probability value given by a value from 0 to 255 (255 is closest to the skin color). Calculate as (x, y) (step 201).

Here, the skin color probability distribution value P _c (x, y) is calculated as follows. First, as shown in FIG. 5, an image showing a hand is photographed by the imaging unit 11, and as shown in FIG. 6, a rectangular region in the palm of the photographed image is manually designated as a skin color region. The RGB color system is converted to the HSV color system for all pixels in the specified skin color area, and a histogram is created for all H values whose S and V values are within the threshold values. An image of the created histogram is shown in FIG. Thereafter, the obtained histogram is normalized so that the maximum value is 255. This is a graph representing the skin color probability values for all H satisfying the threshold values of S and V, and P _c (x, y) can be represented by the following equation.

Here, V _h (H) is a function representing the above-described graph value. By holding the value of V _h (H) directly in the data storage unit 17 as a lookup table, each value can be held as it is and can be cited as a value for H. Further, if an approximate expression such as a Gaussian function is used to express it as a function expression, it can be obtained from the expression.

  2. The distance information creation unit 22 creates distance information of a frame image using a stereo vision technique which is one of known techniques as a method for acquiring three-dimensional distance information (step 202). An image from the imaging unit 12 is also used for stereo viewing. At this time, the position on the image where the same point is photographed from two or more images is determined based on the similarity of the images, and on the reference image based on the principle of triangulation from the positional deviation (parallax) between multiple images. The distance from the imaging unit 11 at is obtained.

  Hereinafter, as shown in FIG. 8, a case where two imaging units (cameras) are arranged in parallel will be described as an example. The parallax between two camera images captured by the camera 11 (right) and the camera 12 (left), which are synchronized in time and separated by a certain distance B and parallel to the horizontal direction, is captured from the camera ( Distance information.

  As a parallax calculation method, for example, there is the following method (FIG. 8). Using the camera 11 as a reference, the similarity between image small regions (blocks) of a specific size is calculated between the camera image (reference image) captured by the camera 11 and the camera image (reference image) captured by the camera 12. By doing so, the parallax is calculated by performing block matching to find the corresponding pixel. When the parallax is d, the distance Z between the cameras 11 and 12 and the imaging object is Z = f × B / d, where f is the focal length of the cameras 11 and 12. Here, d = u−u ′, and B is the distance between the camera 11 and the camera 12. The distance information is generated by expressing the parallax d between the image of the camera 11 and the image of the camera 12 captured at the same time as the pixel value of each pixel of the entire image. This parallax indicates that the position of the person is closer to the camera 11 as the value is larger, and the position of the person is farther from the camera 11 as the value is smaller.

  The distance information creating means is not limited to the above, and for example, Point Gray Research Inc. The distance information can also be created by using a commercially available stereo vision camera such as Digitallops.

3. In the second probability distribution calculation unit 23, the distance within the specified range is 1 around the distance d from the camera 11 created by the distance information creation unit 22 (or searched in the previous frame) to the hand, Otherwise, the second probability distribution value P _d (x, y) is calculated as 0 (step 203, FIG. 9).

  3. The integrated probability distribution calculation unit 24 calculates the integrated probability distribution P (x, y) with a function using the following coefficients a and b (step 204).

The rectangular area determination unit 15 calculates the center of gravity position of the rectangular area using the integrated probability distribution P (x, y) obtained above, and calculates the size l _x and l _y of one side of the next rectangular area. (Step 103). The gravity center position (x _c , y _c ) of the rectangular area can be calculated by the following equation.

At this time, the width l _w and the height l _h of the rectangular area are set with P _max as the maximum probability value.

It expresses by. Here, α and β are coefficients determined by the aspect ratio of the size of the hand.

  The skin color area determination unit 16 determines the following three conditions, and if at least one of the conditions is satisfied, the process for the frame is terminated, and the rectangular area at that time is used as a hand tracking result (step 104). If none is satisfied, the rectangular area at that time is set as an initial value, and the recalculation process of the rectangular area is repeated for the current frame.

  1) When the difference in size between the rectangular area in the previous loop and the rectangular area calculated by the rectangular area determining unit 15 is within the threshold. If the difference between the rectangular areas between the loops is small, it can be determined that the rectangular area has converged, and thus the process ends.

  2) When rectangular area recalculation processing is performed for the same frame more than the specified number of loops. This is because the calculation could not be converged even if the calculation was performed more than the specified number of loops, so it can be determined that it is better to stop calculating further and calculate the next frame. .

  3) When the size of the rectangular area is larger than the specified size. If the rectangular area is clearly larger than the specified size, which is the size of the hand that was assumed, the frame image can be determined to have failed to track the hand, so the process is terminated. To do.

  In addition, the calculation method of the 2nd probability distribution in this embodiment is not restricted above. For example, it is possible to use a trapezoidal function or a Gaussian function centered on d.

  Further, the skin color probability value P (x, y) is not limited to the above formula. Any expression that emphasizes each probability distribution value independently may be used. For example, the following linear sum function may be used.

  In addition to the lookup table, the data storage unit 17 stores information such as rectangular area information, probability distribution information, and processing loop count information.

[Second Embodiment]
FIG. 10 is a block diagram of the flesh color probability distribution calculation unit of the hand position tracking apparatus according to the second embodiment of the present invention, and FIG. 11 is a flowchart showing the processing flow.

  In this embodiment, motion information is used as the third probability distribution in addition to the first and second probability distributions calculated in the first embodiment.

Here, the third probability distribution calculation unit 25 calculates the probability P _m (x, y) based on the motion information as follows (step 205).

P _m (x, y) = 1: When the color difference value from the previous frame is greater than or equal to the threshold value
0: When the color difference value with respect to the previous frame is less than the threshold value The method of calculating the third probability distribution in the present embodiment is not limited to the above. For example, it may be considered that motion between a plurality of adjacent frames (difference value between frames) is considered, or motion information between the latest frames is large and motion information between other frames is small.

  In this case, the probability value P (x, y) of the pixel (x, y) calculated by the rectangular area determination unit 15 is expressed by the following expression.

  In this case, as in the first embodiment, the skin color probability value P (x, y) is not limited to the above equation. It is also possible to use the following linear sum function.

[Third Embodiment]
FIG. 12 is a block diagram of a hand position tracking apparatus according to the third embodiment of the present invention, and FIG. 13 is a flowchart showing the processing flow.

  In the present embodiment, the coefficient determination unit 18 is further added to the first and second embodiments, and the coefficients a, b, and c for calculating the probability value P (x, y) vary according to the region determination result. (Step 105) is different.

Here, at the beginning of tracking and when the rectangular area is the maximum value,
a = 1, b = c = 0
In the state where only the skin color that is the first probability distribution is considered and the hand is normally tracked,
a = b = c = 1
As such, consider both distance and movement.
Furthermore, when the movement of the skin color area is small and the third probability distribution value is equal to or less than the threshold value,
a = b = 1, c = 0
As such, do not consider movement.

  In the above, the combination of the values of a, b, and c is not limited to this.

  For example, if the distance calculation by stereo vision is unstable and the time series change of the distance d is large, b = 0 may be considered, and c = 0.5 is regularly used to neglect the motion information. A small value can be considered.

  In the example in which a, b, and c are set to 0 or a constant other than 0 in this embodiment, when P is obtained in the form of a linear sum of Pc, Pd, and Pm, the values of a, b, and c are directly set. By substituting into the expression of P, the value of P can be calculated. When P is calculated by the product of Pc, Pd, and Pm as in Equation 2, if any of a, b, and c is 0, P = 0, so the calculation formula is as follows: It is also possible to do.

  Note that the object position tracking apparatus of the present invention records a program for realizing the function on a computer-readable recording medium, and causes the computer to read and execute the program recorded on the recording medium. It may be. The computer-readable recording medium refers to a recording medium such as a flexible disk, a magneto-optical disk, and a CD-ROM, and a storage device such as a hard disk device built in a computer system. Further, the computer-readable recording medium is a medium that dynamically holds the program for a short time (transmission medium or transmission wave) as in the case of transmitting the program via the Internet, and in the computer serving as a server in that case Such as a volatile memory that holds a program for a certain period of time.

1 is a block diagram of a hand position tracking device according to a first embodiment of the present invention. FIG. It is a block diagram of a skin color probability distribution calculation part. It is a flowchart which shows the flow of the whole process of the hand position tracking apparatus by 1st Embodiment. It is a flowchart which shows the flow of a process of the skin color probability distribution calculation part in 1st Embodiment. It is a figure which shows a mode that the image in which the hand is reflected is image | photographed with an imaging part. It is a figure which shows a skin color area | region. It is a histogram of all H values in which S and V values are within threshold values. It is an image figure of distance calculation by stereo vision. It is an image figure showing the calculation formula of the 2nd probability distribution in 1st Embodiment. It is a block diagram of the skin color probability distribution calculation part in the hand position tracking apparatus by the 2nd Embodiment of this invention. It is a flowchart which shows the flow of a process of the skin color probability distribution calculation part in 2nd Embodiment. It is a block diagram of the hand position tracking apparatus by the 3rd Embodiment of this invention. It is a flowchart which shows the flow of the whole process of the hand position tracking apparatus by 3rd Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 Image pick-up part 12 Image pick-up part 13 Image input part 14 Skin color probability distribution calculation part 15 Rectangular area determination part 16 Skin color area determination part 17 Data storage part 18 Coefficient determination part 21 1st probability distribution calculation part 22 Distance information creation part 23 2nd probability Distribution calculation unit 24 Integrated probability distribution calculation unit 25 Third probability distribution calculation unit 101-105, 201-205 steps

Claims (10)

A method of tracking the position of an object by tracking an area of a predetermined color and size specified in advance on an imaged moving image obtained by imaging an imaging object by an imaging means,
A captured image input stage for inputting a captured image of each frame imaged by the imaging means;
Predetermined color for calculating the probability distribution of the predetermined color in a specified rectangular area using a formula for calculating the probability that the predetermined color exists or a look-up table representing the relationship between a pixel value and the probability A probability distribution calculation stage;
When the center of gravity of the probability distribution is at the center and the sum M ₀₀ of probability distribution values and the maximum value of the probability distribution are P _max , the width l _w and height l _h of one side of the rectangular area are M ₀₀ and P _max. A rectangular area determining step for determining a rectangular area as given by: l _w = f _w (M ₀₀ , P _max ), l _h = f _h (M ₀₀ , P _max ),
The size of the rectangular area determined in the rectangular area determination stage in the previous loop of the processing performed in the captured image input stage, the predetermined color probability distribution calculation stage, and the rectangular area determination stage, and the rectangular area determination in the current loop. The difference in size from the rectangular area determined in the stage is within the threshold value, or the rectangular area has been redetermined more than the specified number of loops for the same frame, or the size of the determined rectangular area is the specified size A predetermined color region determination step in which the rectangular region determined this time in the rectangular region determination step is used as the tracking result of the object in the current frame, and the process returns to the captured image input step and proceeds to calculation for the next frame. When,
An object position tracking method comprising: The predetermined color probability distribution calculation step includes:
A first probability distribution calculating step in which a value calculated using the calculation formula or the lookup table is a first probability distribution;
A distance information creation step of creating distance information of each pixel of the captured image by a three-dimensional distance information acquisition technique;
A second probability distribution calculation stage for calculating a distance probability distribution for calculating a distance probability distribution in the current frame from the distance information in the previous frame as a second probability distribution;
An integrated probability distribution calculation stage for calculating a probability distribution with a function having the first probability distribution and the second probability distribution as variables;
The object position tracking method according to claim 1, comprising: The predetermined color probability distribution calculation step includes:
A first probability distribution calculating step in which a value calculated using the calculation formula or the lookup table is a first probability distribution;
A distance information creation step of creating distance information of each pixel of the captured image by a three-dimensional distance information acquisition technique;
A second probability distribution calculation stage for calculating a distance probability distribution for calculating a distance probability distribution in the current frame from the distance information in the previous frame as a second probability distribution;
Generated by extracting the points on the image where the sum of absolute differences of each pixel value for each image between two adjacent frames is greater than or equal to a certain value for an image sequence of a plurality of consecutive frames before the current frame A third probability distribution calculation stage for generating a motion probability distribution as a third probability distribution using the motion information to be
An integrated probability distribution calculating step of calculating a probability distribution with a function having the first, second and third probability distributions as variables;
The object position tracking method according to claim 1, comprising:   4. The object position tracking method according to claim 2, wherein, in the integrated probability distribution calculation step, when the rectangular area is larger than a specified size, only the first probability distribution is set as the probability distribution. 5.   4. The object according to claim 2, wherein the integrated probability distribution calculation step dynamically changes a function for calculating the probability distribution based on at least one of a size of the rectangular area and a calculation result of each probability distribution. Location tracking method. An apparatus for tracking the position of an object by tracking an area of a predetermined color and size specified in advance on an imaging moving image obtained by imaging an imaging object by an imaging means,
Captured image input means for inputting a captured image of each frame imaged by the imaging means;
Predetermined color for calculating the probability distribution of the predetermined color in a specified rectangular area using a formula for calculating the probability that the predetermined color exists or a look-up table representing the relationship between a pixel value and the probability A probability distribution calculation means;
When the center of gravity of the probability distribution is at the center and the sum M ₀₀ of probability distribution values and the maximum value of the probability distribution are P _max , the width l _w and height l _h of one side of the rectangular area are M ₀₀ and P _max. A rectangular area determining means for determining a rectangular area as given by the function l _w = f _w (M ₀₀ , P _max ), l _h = f _h (M ₀₀ , P _max ),
The size of the rectangular area determined by the rectangular area determining means in the previous loop of the processing sequentially performed by the captured image input means, the predetermined color probability distribution calculating means, and the rectangular area determining means, and the rectangular area in the current loop The difference in size from the rectangular area determined by the determining means is within the threshold value, or the rectangular area that has been redetermined or exceeds the specified number of loops for the same frame, or the size of the determined rectangular area is specified. When the size is exceeded, the rectangular area determined this time by the rectangular area determining means is set as the tracking result of the object in the current frame, and the process returns to the captured image input means, and the predetermined color area determination moves to the calculation for the next frame. Means,
An object position tracking device. The predetermined color probability distribution calculating means includes:
A first probability distribution calculating means for setting a value calculated using the calculation formula or the lookup table as a first probability distribution;
Distance information creating means for creating distance information of each pixel of the captured image by a three-dimensional distance information acquisition technique;
A second probability distribution calculating means for calculating a distance probability distribution for calculating a distance probability distribution in the current frame from the distance information in the previous frame as a second probability distribution;
An integrated probability distribution calculating means for calculating the probability distribution using a function having the first probability distribution and the second probability distribution as variables;
The object position tracking device according to claim 6, comprising: The predetermined color probability distribution calculating means includes:
A first probability distribution calculating means for setting a value calculated using the calculation formula or the lookup table as a first probability distribution;
Distance information creating means for creating distance information of each pixel of the captured image by a three-dimensional distance information acquisition technique;
A second probability distribution calculating means for calculating a distance probability distribution for calculating a distance probability distribution in the current frame from the distance information in the previous frame as a second probability distribution;
Generated by extracting the points on the image where the sum of absolute differences of each pixel value for each image between two adjacent frames is greater than or equal to a certain value for an image sequence of a plurality of consecutive frames before the current frame Third probability distribution calculation means for generating a motion probability distribution as the third probability distribution using the motion information
An integrated probability distribution calculating means for calculating a probability distribution with a function having the first, second and third probability distributions as variables;
The object position tracking device according to claim 6, comprising:     The object according to claim 7 or 8, wherein the integrated probability distribution calculation means dynamically changes a function for calculating the probability distribution based on at least one of the size of the rectangular area and the calculation result of each probability distribution. Location tracking device.   An object position tracking program for causing a computer to execute each means of the object position tracking device according to any one of claims 6 to 9.

Images