JP2003076488A - Device and method of determining indicating position - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enhance the determination accuracy of an indicating position or an indicating direction in indicating operations. SOLUTION: A situation that an information input person 10 is indicating an optional spot on the display surface of a display 12 is imaged from plural directions and it is determined based on plural images obtained by imaging that which arm of the operator is used for the indicating operation. When it is determined that his/her left arm is used, a reference point PO is set in the vicinity of his/her left shoulder, and when it is determined that his/her right arm is used, the reference point PO is set in the vicinity of his/her right shoulder. Thereby, three-dimensional coordinates of both the reference point PO and a characteristic point PX corresponding to a fingertip performing the indicating operation are determined. An extending direction of a virtual line 54 connecting the reference point PO and the characteristic point PX together is determined as the indicating direction of the information input person 10, and coordinates (plane coordinates) of an intersection S of a plane including the display surface of the display 12 and the virtual line 54 are calculated as the indicating position of the information input person 10.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a pointing position determining apparatus and method, and more particularly, to a pointing position determining method for picking up an image of a recognition target person and determining a pointing position or a direction according to a pointing operation performed by the recognition target person, and The present invention relates to a pointing position determination device to which the pointing position determination method can be applied.

[0002]

2. Description of the Related Art Conventionally, a display for displaying predetermined information and an image pickup means for picking up a recognition target person (user) who has arrived in the vicinity of the display from different directions are provided. The situation in which the pointing operation for pointing an arbitrary position with a finger or the like is performed is imaged from a plurality of directions by the imaging unit, the recognition target person is recognized based on the plurality of images obtained by the imaging, and the pointing operation is performed. The position on the display is determined, a cursor, etc. is displayed at the indicated position on the display, and when it is detected that the person to be recognized performs a click operation of raising the thumb, it is recognized that the indicated position on the display has been clicked. There is known a hand pointing input device for performing a predetermined process according to a known method (for example, Japanese Patent Laid-Open No. 4-27).
1423, Japanese Patent Laid-Open No. 5-195957, Japanese Patent Laid-Open No. 5-324181, etc.).

According to the above hand pointing input device, a person to be recognized can give various instructions to the information processing device or input various information without touching an input device such as a keyboard or a mouse. Because it will be possible
It is possible to simplify the operation for using the information processing device.

[0004]

By the way, as a method of determining the pointing position (or pointing direction) in the pointing operation by the recognition target person, a position where the position does not change with respect to the recognition target person regardless of the change of the pointing position or the like. Is set as a reference point, and a position whose position changes according to a change in the indicated position etc. is set as a characteristic point, and a method of determining the indicated position etc. based on the positional relationship between the reference point and the characteristic point is adopted. There is. In this determination method, the reference point is, for example, a location corresponding to the center of the back of the person to be recognized along the left-right direction, and the feature point is, for example, the location corresponding to the fingertip performing the pointing action. .

However, since the recognition target person's pointing operation is usually performed by using one of the left and right arms, the position of the feature point is offset in the left-right direction with respect to the body of the recognition target person. Therefore, the pointing position or pointing direction determined by the hand pointing device often differs greatly from the pointing position or pointing direction assumed by the person to be recognized, and improvement in the determination accuracy of the pointing position or pointing direction has been demanded. .

The present invention has been made in consideration of the above facts, and an object of the present invention is to provide a pointing position determination device and a pointing position determination method that can improve the accuracy of determination of a pointing position or a pointing direction in a pointing operation. .

[0007]

In order to achieve the above-mentioned object, a pointing position determination device according to the invention of claim 1 has an image pickup means for picking up a recognition target person from a plurality of different directions, and a recognition target person. Based on the image obtained by the imaging means capturing the situation in which the pointing operation indicating the specific position or the specific direction is performed, the recognition target person determines the right and left of the arm used for the pointing operation. The reference point is a position that does not change regardless of the change of the pointing position or the pointing direction in the pointing operation, and is offset to the center of the recognition target person in the left-right direction according to the determination result of the left and right of the arm. Based on an image portion corresponding to the recognition target person extracted from the image obtained by the image capturing means capturing an image, the change of the pointing position or the pointing direction in the pointing operation. Based on the feature point recognition means for obtaining the position of the feature point whose position changes according to the position of the reference point set by the reference point setting means and the feature point recognition means, based on the position of the feature point, Determination means for determining the pointing position or direction by the pointing operation,
It is configured to include.

According to the first aspect of the invention, the person to be recognized is imaged by the imaging means from a plurality of different directions,
Based on the image obtained by capturing the situation in which the recognition target person is performing the pointing operation pointing to a specific position or a specific direction, the reference point setting means uses the recognition target person for the pointing operation. Determine left and right arm.

When the image pickup means is configured to image the person to be recognized from above, the reference point setting means determines the right and left of the arm by the image pickup means as described in claim 2, for example. This can be performed based on the contour shape of the image portion corresponding to the recognition target person extracted from the image obtained in (1). Further, the discrimination between the left and right of the arm by the reference point setting means is described in more detail, for example, as described in claim 3.
This can be performed based on the offset direction of the position of the convex portion corresponding to the recognition target person's arm in the contour of the image portion corresponding to the recognition target person with respect to the entire contour of the image portion.

Since the pointing motion by the person to be recognized is performed by using one of the left and right arms, a characteristic point whose position changes in accordance with a change in the pointing position or the pointing direction in the pointing motion (for example, this characteristic point is recognized) The fingertips of the hand of the arm used by the target person for the pointing motion, the tip of the indicator held by the hand of the arm, etc. can be applied) to the left or right of the body of the recognition target person. Offset in either direction. Whether the position of the feature point is offset to the left or right of the person to be recognized can be determined from the left and right discrimination results of the arm.

Therefore, the reference point setting means does not change the position regardless of the change of the pointing position or the pointing direction in the pointing operation by the recognition target person, and the left and right sides of the recognition target person are determined according to the discrimination result of the left and right arms. The reference point is set at a position offset from the center of the direction. Specifically, for example, as described in claim 4, a reference point may be set at or near the base of the arm used by the recognition target person for the pointing motion, or the height of the recognition target person, etc. It is also possible to set an offset amount according to the above and set the reference point at a position offset by the set offset amount from the center along the left-right direction of the body of the person to be recognized. As a result, the position of the reference point is also offset according to the offset of the feature point with respect to the body of the person to be recognized.

According to the first aspect of the invention, the feature point recognizing means causes the feature point recognizing means to perform the pointing position in the pointing operation based on the image portion corresponding to the recognition target person extracted from the image obtained by the image pickup means. Alternatively, the position of the feature point whose position changes according to the change of the pointing direction is obtained, and the determination means is based on the position of the reference point set by the reference point setting means and the position of the feature point obtained by the feature point recognition means. Since the pointing position or direction by the pointing operation is determined, it is possible to eliminate the influence of the position of the feature point being offset in the left-right direction, and improve the accuracy of determining the pointing position or pointing direction in the pointing operation. Can be realized.

In the invention according to claim 1, the determination of the designated position or the designated direction by the determination means is, for example, as described in claim 5, the intersection of the straight line connecting the reference point and the feature point and the designated object. Can be determined to be the position designated by the pointing motion, or the direction in which the straight line connecting the reference point and the feature point extends can be determined to be the pointing direction designated by the pointing motion.

According to a sixth aspect of the present invention, in the pointing position determining method, the recognition target person is imaged from a plurality of different directions, and the recognition target person performs a pointing operation indicating a specific position or a specific direction. Based on the image obtained by capturing the image, the left and right of the arm used by the recognition target person for the pointing motion is determined, and the position does not change regardless of the change in the pointing position or the pointing direction in the pointing motion. , And the recognition target person extracted from the image obtained by the imaging while setting a reference point at a position offset with respect to the center of the recognition target person in the left-right direction in accordance with the result of discrimination between the left and right sides of the arm. Based on the image portion corresponding to, the position of the characteristic point whose position changes in accordance with the change of the designated position or the designated direction in the pointing operation, the position of the set reference point and the obtained characteristic point Based on the position, since it is determined pointing position or direction by the instruction operation, similarly to the first aspect of the present invention, it is possible to realize the indicated position or the improvement of the pointing direction of the determination accuracy of the instruction operation.

[0015]

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings. As shown in FIG. 1, a large-screen display 12 is embedded on the wall surface of a location where an information input person 10 as a recognition target person according to the present invention arrives. As the display 12, a liquid crystal display (LCD), a plasma display, a cathode ray tube,
Known display means such as an optical fiber display can be applied.

The display 12 is connected to an information processing device 14 including a personal computer (see FIG. 2).
Information), the information processing device 14 displays various kinds of information on the display surface in various display forms such as figures, tables, characters, and images. In the present embodiment, the information input person 10 has a portion (information input space) shown in FIG. 1 in front of the display 12.
When various information is displayed on the display surface of the display 12 and a click operation (details will be described later) is performed, various instructions are given to the information processing device 14 and various information is given. Let the process run.

As shown in FIG. 2, the information processing device 14 includes a hand pointing input device 2 according to the present embodiment.
0 controller 22 is connected. The display 12, the information processing device 14, and the hand pointing input device 20 described above have a function as a pointing position determination device according to the present invention to which the pointing position determination method according to the present invention is applied.

The controller 22 includes a CPU 22A and RO
M22B, RAM22C, input / output interface 22D
And are connected to each other via a bus. The information processing device 14 is connected to the input / output interface 22D, and a storage device (for example, a hard disk drive (HDD)) 24 having a built-in non-volatile storage medium capable of rewriting the stored contents, for displaying various information Display 26, a keyboard 28 for an operator to input various instructions and data, an illumination control device 30, an imaging control device 34, a mark plate driving device 3
8 and a CD-ROM driver 40 are connected respectively.

A program (hereinafter, simply referred to as "program") for causing the controller 22 to function as the hand pointing device according to the present invention is, for example, a CD-
It can be stored in the CD-ROM 42 loaded in the ROM driver 40. When the execution of the program is instructed, the program is read from the CD-ROM 42, and the read program is executed by the controller 22. Accordingly, each device including the controller 22 functions as a hand pointing device.

The illumination control device 30 includes a plurality of near-infrared light illuminating devices 32 which emit light in the near-infrared wavelength region into a beam.
A and 32B are connected. As shown in FIG. 1, the near-infrared light illuminators 32A and 32B are arranged at different positions above the information input space, and the information input person 10 who has arrived in the information input space is seen from different directions. The irradiation range is adjusted to illuminate. Lighting control device 30
Controls the turning on / off of the lighting devices 32A and 32B.

A plurality of video cameras 36A and 36B arranged at different positions above the information input space (see FIG. 1) are connected to the image pickup control device 34. Although not shown, the video cameras 36A and 36B are provided with an area sensor made of a CCD or the like having sensitivity to near-infrared light, and the light of an imaging lens that forms an image of incident light on the light receiving surface of the area sensor. A filter that transmits only light having a wavelength in the near infrared region is provided on the incident side. A far-infrared camera is used as the imaging means (camera), and far-infrared rays caused by the body temperature of a human (information input person) and a temperature separated from the human body temperature by a predetermined value or more (at a temperature higher or lower than the body temperature It may be present), and a method of discriminating a human from a background by far infrared light emitted from the floor whose temperature is set or adjusted. In this case, lighting devices 32A, 32B
It is also possible to omit the lighting control device 30.

As shown in FIG. 3, the video camera 36A,
The direction of 36B is adjusted so that the information input person 10 who has arrived in the information input space is imaged from different directions. Further, in the video camera 36A, the information input person 10 who has arrived in the information input space enters the imaging range, and the light emitted from the illumination device 32A does not directly enter the imaging lens,
In addition, the center of the imaging range intersects with the center of the illumination range by the lighting device 32A in the information input space at a predetermined height from the floor surface (the illumination range on the floor surface by the lighting device 32A deviates from the imaging range). ), The orientation (imaging range) is adjusted. Similarly, in the video camera 36B, the information input person 10 who has arrived in the information input space enters the imaging range, the light emitted from the illumination device 32B does not directly enter the imaging lens, and the center of the imaging range is Direction (imaging range) so that the center of the illumination range of the illuminating device 32B intersects with the floor surface at a predetermined height in the information input space (so that the illumination range on the floor surface of the illuminating device 32A is outside the imaging range). Has been adjusted.

The video cameras 36A and 36B, the image pickup control device 34, the illumination control device 30, and the near infrared light illumination devices 32A and 32B are the image pickup means of the present invention (specifically, the image pickup means according to claim 2). It corresponds.

Further, as shown in FIG. 4, the hand pointing input device 20 is provided with a mark plate 40 arranged near the information input space. The mark plate 40 is composed of a large number of marks 40A recorded in a matrix on a transparent flat plate at equal intervals, and information is input along a direction orthogonal to the arrangement direction of the marks 40A (direction of arrow A in FIG. 4). It is supposed to be movable across the space. Mark 40A
Is colored in a color (for example, red) that can be easily recognized on the image. The mark plate drive device 38 connected to the input / output interface 22D moves the mark plate 40 in the direction of arrow A in FIG. 4 according to an instruction from the controller 22.

Next, as the operation of this embodiment, first, the lattice point position information initial setting process executed by the controller 22 when the hand pointing input device 20 is installed will be described with reference to the flowchart of FIG. .

In step 100, the mark plate driving device 3
8, the mark plate 40 is moved to a predetermined position (a position corresponding to the end of the moving range of the mark plate 40). In the next step 102, at the current position of the mark plate 40,
Three-dimensional coordinates (x, y, z) in the information input space of many marks 40A recorded on the mark plate 40 are calculated. In step 104, the information input space is imaged by the video cameras 36A and 36B via the imaging control device 34, and in the next step 106, an image obtained by the video camera 36A imaging the information input space (hereinafter referred to as image A and (Referred to) is taken in via the imaging control device 34.

At step 108, the marks 40A existing in the image A captured at step 106 are recognized (extracted), and at the next step 110, all the recognized marks 40A are moved to the position (X _A on the image A). , Y _A ) is calculated. In step 112, for all the marks 40A present in the image A, 3-dimensional coordinates of the information input space (x, y, z) and the position on the image _A (X A, Y A) and Correspondence is made to be stored in the storage device 24 as grid point position information of the video camera 36A.

In the next steps 114 to 120, the video camera 36 is processed in the same manner as steps 106 to 112 described above.
Perform processing for B. That is, in step 114, the image obtained by the video camera 36B capturing the information input space (hereinafter referred to as image B) is captured through the image capturing control device 34, and in step 116, the image B captured in step 114 is captured. The existing marks 40A are recognized (extracted), and in the next step 118, the positions (X _B , Y _B ) on the image B are calculated for all the recognized marks 40A. And in step 120,
For all the marks 40A existing in the image B, the three-dimensional coordinates (x, y, z) in the information input space and the position (X _B , Y _B ) on the image B are made to correspond to each other, and the video camera is used. 36
The lattice point position information of B is stored in the storage device 24.

In the next step 122, the mark plate 40 is moved to the final position (step 10 in the moving range of the mark plate 40).
It is determined whether or not it has moved to a position corresponding to the end on the opposite side of the predetermined position at 0). When the determination in step 122 is negative, the process proceeds to step 124, and the mark plate driving device 38 moves the mark plate 40 in a predetermined direction by a predetermined distance (specifically, a distance corresponding to the distance between the marks 40A on the mark plate 40). ) Only after moving
Return to.

As described above, steps 102 to 124 are repeated until the mark plate 40 moves to the final position, so that a large number of marks 40A recorded on the mark plate 40 are spaced at a constant interval in the information input space. Is moved to a position corresponding to a large number of grid points lined up in a grid with the video camera 3
As the grid point position information of 6A, the three-dimensional coordinates of each grid point in the information input space and the position on the image A are stored in the storage device 24 in association with each other, and the grid point position information of the video camera 36B is stored. As a result, the three-dimensional coordinates of each grid point in the information input space and the position on the image B are stored in the storage device 24 in association with each other.

Since the mark plate 40 and the mark plate driving device 38 are used only in the above-mentioned lattice point position information initializing process and are not used in the process to be described later, the mark plate 40 and the mark plate 40 are not used in the process described later. The mark plate drive device 38 may be removed.

The mark plate 4 on which the mark 40A is recorded
Instead of 0, a mark plate in which a large number of light emitting elements such as LEDs are arranged in a matrix on a transparent flat plate is used, and the mark plate is moved stepwise by a constant distance, and a large number of light emitting elements are provided at each position in the step movement. The above process may be performed by repeatedly turning on the lights. Further, a robot arm device provided with a hand to which the light emitting element is attached and capable of moving the hand to an arbitrary position in the information input space is used, and the light emitting device is attached to each position corresponding to each lattice point by the robot arm device. It is also possible to carry out the above processing by repeating moving and lighting.

Next, with reference to the flow chart of FIG. 6, an instruction determination process which is routinely executed by the controller 22 after the above-mentioned lattice point position information initial setting process is performed will be described. It should be noted that this instruction determination processing is to determine an instruction from the information input person 10 who has arrived in the information input space.

In step 150, the video camera 36A
The image data representing the image A output from the video camera 36B and the image data representing the image B output from the video camera 36B are respectively captured, and information is input into the information input space based on the captured image data of the images A and B. It is determined whether the person 10 has arrived (is present), and the process waits until the determination is affirmed.

When the controller 22 is executing the instruction judging process, as shown in FIG.
Alternately turns on the lighting devices 32A and 32B, and the imaging control device 34 performs imaging of the information input space by the video camera 36A when the lighting device 32A is turned on, and imaging of the information input space by the video camera 36B. Control is performed when the lighting device 32B is turned on.

Since the image pickup range of the video camera 36A is adjusted so that the illumination range on the floor surface by the illumination device 32A is out of the image pickup range, the information input person 10 is within the illumination range on the floor surface by the illumination device 32A. Even if there is a non-recognition target 50A (see FIG. 3) such as luggage or dust, the non-recognition target 50A does not fall within the imaging range of the video camera 36A. Also, the video camera 36A on the floor
The unrecognized object 50B (FIG.
Even if the non-recognition target 50B exists, the non-recognition target 50B is out of the illumination range of the lighting device 32A, and thus the brightness of the image portion corresponding to the non-recognition target 50B existing in the image A is very low. Become.

Similarly, since the imaging range of the video camera 36B is adjusted so that the illumination range on the floor surface by the lighting device 32B deviates from the imaging range, the non-recognition target on the floor surface illuminated by the lighting device 32B. Even if the object 50B exists, the non-recognition target object 50B does not fall within the imaging range of the video camera 36B. Further, even if the non-recognition target object 50A is present within the range captured by the video camera 36B on the floor surface, the brightness of the image portion corresponding to the non-recognition target object 50B present in the image B is very high. Get lower.

Therefore, the determination in step 150 can be performed by a very simple determination process such as whether or not the image A and the image B have an image portion having a high brightness and an area of a predetermined value or more. . If the determination in step 150 is negative, the process waits until the determination is positive.

When the information input person 10 arrives at the information input space, the determination at step 150 is affirmed and step 152 is performed.
Then, various flags and processing modes are initialized. In the next step 154, it is determined whether or not the information input person 10 has left the information input space. When the information input person 10 leaves the information input space, the determination is affirmative and the process returns to step 150 to wait until the information input person 10 arrives again in the information input space, but when the determination is denied. In step 156, reference point / feature point coordinate calculation processing is performed. Hereinafter, the reference point / feature point coordinate calculation processing will be described with reference to the flowchart in FIG. 7.

At step 210, the video camera 36A,
Data of image A and image B are respectively captured from 36B,
In the next step 212, image portions corresponding to the entire image of the information input person 10 are extracted from the captured images A and B, respectively. The image portion corresponding to the entire image of the information input person 10 can also be easily extracted by determining a continuous region having an area of a predetermined value or more that is composed of high-luminance pixels.

In step 214, the height of the information input person 10 is obtained based on the image portion corresponding to the entire image of the information input person 10. As shown in FIG. 9, the focal length of the imaging lens of the video camera 36 located at the point O is f, and the distance between the point Q and the intersection Q of the vertical line passing through the point O and the floor of the information input space. Is H, a point Q, and a point P on the floor on which the information input person 10 stands.
Is R, and the distance h between the point P ′ corresponding to the crown of the information input person 10 and the point P is the height of the information input person 10.
Further, the angle formed by the point POQ is θ, the angle formed by the point P′OQ is θ ′, the length of the image of the information input person 10 formed on the light receiving surface of the area sensor of the video camera 36 is h ′, and the point P is Is the image forming point on the light receiving surface corresponding to the point p, the image forming point on the light receiving surface corresponding to the point P ′ is point p ′, the distance between the center o of the light receiving surface and the point p is r, the center of the light receiving surface is If the distance between o and the point p'is r ',
The angles θ and θ ′ and the distances r and r ′ can be obtained from the following equations (1) to (4).

Θ = tan ⁻¹ (R / H) (1) θ ′ = tan ⁻¹ {R / (H−h)} (2) r = fθ (3) r ′ = fθ ′ (( 4) Therefore, the height h and the distance R of the information input person 10 can be obtained by the following equations (5) and (6).

H = H {1-tan (r / f) / tan (r '/ f)} (5) R = Htan (r / f) (6) The distance H and the focal length f are known. So step 21
4, the distances r and r ′ are obtained from either the image A or the image B obtained by the video cameras 36A and 36B, and the distances r and r ′ are substituted into the equation (5) to obtain the information input person 1
A height h of 0 can be obtained. Also, step 21
4, the distance r is obtained from each of the image A and the image B, and this is substituted into the equation (6) to obtain the distance R, respectively, thereby obtaining the position of the information input person 10 on the floor surface (two-dimensional coordinate). Ask for.

In the next step 216, step 214
Based on the height h of the information input person 10 obtained in step 1, the height corresponding to the shoulder area of the information input person 10 when the height of the floor is set to 0 is estimated and calculated, and the estimated height is input to the information. Person 10
Of the reference point P ₀ (see FIG. 13A) from the floor surface z ₀
To decide. Note that, instead of the height corresponding to the shoulder area of the information input person 10, the height corresponding to the chest of the information input person 10 and the elbow of the information input person 10 in the state in which the information input person 10 is performing the instruction operation. The height corresponding to may be estimated and calculated.

In step 218, based on the shape of the image portion corresponding to the entire image of the information input person 10 on the image A and the image B, the information input person 10 uses the finger or the like to display the display 1.
It is determined whether or not an action (instruction action) indicating the second side is being performed. Since the direction of the display 12 as seen from the information input person 10 is known, the determination in step 218 can be determined to be equivalent to the hand of the information input person 10 in the image portion corresponding to the entire image of the information input person 10. In position
This can be realized by determining whether or not there is a portion protruding toward the display 12 viewed from the information input person 10.

As a result, the information input person 10 can
From the upright state shown in (A), if the arm is raised and the hand is directed to the display 12 side as shown in FIG. 10 (B) or (C), it is determined that the information input person 10 is performing the instruction operation. It will be. When the determination in step 218 is negative, the reference point / feature point coordinate calculation process is ended without performing the calculation of the three-dimensional coordinates of the reference point and the feature point (details will be described later).

If the information input person 10 is performing the instruction operation, the determination at step 218 is affirmative and step 2
Move to 20. In step 220, the instruction operation by the information input person 10 is performed using the left arm of the information input person 10 based on the shape of the image part corresponding to the entire image of the information input person 10 on the images A and B. It is determined whether the information input person 10 is using the right arm.

When the instructing operation by the information input person 10 is performed using the left arm, as shown in FIG. 11A as an example, the information input person 10 is imaged from above. The image portion corresponding to the whole image is the information input person 1
The part corresponding to the arm (left arm) of the information input person 10 is projected from the left side (the left arm side of the information input person 10) of the part corresponding to the body of 0. When the instruction operation by the information input person 10 is performed using the right arm, the information input person 10 when the information input person 10 is imaged from above as shown in FIG. In the image portion corresponding to the entire image of, the portion corresponding to the arm (right arm) of the information input person 10 projects from the right side of the portion corresponding to the body of the information input person 10 (the right arm side of the information input person 10). It becomes a shape.

Therefore, the right / left discrimination in step 220 is performed such that the shape of the image portion corresponding to the entire image of the information input person 10 is the shape shown in FIG. 11 (A) or the shape shown in FIG. 11 (B). It can be performed by determining whether or not. More specifically, this determination is performed by, for example, determining a portion corresponding to the arm of the information input person 10 included in the image portion corresponding to the entire image of the information input person 10, and determining whether the portion is left or right with respect to the image portion. This can be done by determining whether the direction is biased. Alternatively, for example, the image portion corresponding to the entire image of the information input person 10 may be divided at the central portion along the left-right direction, and the thicknesses of both image portions along the direction orthogonal to the left-right direction may be compared. However, it is also possible to determine whether the center of gravity of the image portion corresponding to the entire image of the information input person 10 is biased to the left or right.

In the next step 222, step 220
Information input person 1 based on the result of discrimination between the left and right arms in
0 arm side is used to instruct operation shoulders as the reference point P ₀ in the vicinity (left shoulder if left arm, right shoulder if right arm) is located, horizontal plane coordinates (e.g. x ₀ of the reference point P ₀ , Y ₀ values).

As a result, if the arm used by the information input person 10 for the pointing motion is the left arm, the position of the reference point P _{0 in} the horizontal plane is the position shown in FIGS. 11 (C) and 13 (B). If the arm used by the information input person 10 for the pointing operation is the right arm, the position of the reference point P _{0 in} the horizontal plane is set to the position shown in FIG. 11D and FIG. 13C. together with the height z ₀ of the reference point P ₀ determined in step 216, the 3-dimensional coordinates of the reference point _{P 0 (x 0, y 0} , z 0) so that is determined.

Steps 214 to 222 described above
Corresponds to the reference point setting means of the present invention (specifically, the reference point setting means according to claims 2 to 4).
The processing after 24 corresponds to the feature point recognition means of the present invention. It should be noted that instead of setting the position of the reference point P _{0 in} the horizontal plane near the shoulder on the arm side used for the pointing motion, the vicinity of the chest on the arm side used for the pointing motion or the arm used for the pointing motion. It may be near the side elbow.

At step 224, the video camera 36A
The feature point P _X of the information input person 10 existing in the image A is extracted based on the image data representing the image A captured from the image A, and the position (X _A , Y _A ) of the feature point P _X on the image A is extracted. Is calculated. As the characteristic point P _X of the information input person 10, a point corresponding to the tip of the finger performing the operation of pointing the display 12 side can be used. In this case, the display 12 is placed at a height position that can be determined to correspond to the hand of the information inputting person 10 in the image portion showing the entire image of the information inputting person 10.
The position of the tip of the portion projecting in the direction of can be calculated as the position of the feature point P _X.

As a result, when the hand of the information input person 10 is imaged by the video camera 36A as shown in FIG. 12A, the characteristic point P _X shown in FIG. 12B is set as the position of the characteristic point P _X. The coordinates (X _A , Y _A ) are calculated.

In step 226, the position on the image A is in the range of (X _A ± dX, Y _A ± dY) based on the grid point position information of the video camera 36 A stored in the storage device 24 (FIG. 12 ( Search all grid points that fall within the hatched area in B)). The sizes of dX and dY are set so that at least one or more grid points are extracted based on the space between the grid points (the space between the marks 40A).

Further, in this embodiment, a wide-angle lens is used as the image forming lens of the video camera, and it is assumed that dX and d
Assuming that Y is constant, as the distance between the video camera and the grid points increases, more grid points become (X _A ± dX, Y _A ± d
The range of Y) is entered, which leads to a reduction in the accuracy of calculation of the three-dimensional coordinates of the feature point P _X described later. Therefore, dX and dY
Is set so that the value becomes smaller as the distance from the video camera on the three-dimensional coordinates increases. Therefore, 3
The range corresponding to (X _A ± dX, Y _A ± dY) on the dimensional coordinates is a quadrangular pyramid whose bottom surface is located on the video camera side.

In step 228, the previous step 224
Similarly, based on the image data representing the image B captured from the video camera 36B, extracts feature points P _X of the information inputting person 10 present in the image B, the characteristic point P _X on the image B and
The position (X _B , Y _B ) of is calculated. In step 230,
Similar to step 226 above, based on the grid point position information of the video camera 36B stored in the storage device 24,
All the grid points whose positions on the image B fall within the range of (X _B ± dX, Y _B ± dY) are searched.

At the next step 232, the grid points commonly extracted from the image A and the image B are determined. This allows
Only a plurality of grid points existing at positions close to the feature point P _X in the information input space will be extracted. In step 234, the three-dimensional coordinates of the grid points commonly extracted from the images A and B are fetched from the grid point position information.

In the present embodiment, the feature point P will be described later.
_XThe three-dimensional coordinates of the feature point P in the information input space_XClose to
Interpolation from 3D coordinates of multiple grid points existing at the position
(Specifically, 3 of the plurality of grid points)
The feature point P is obtained by weighted averaging of the coordinate values of the three-dimensional coordinates._XOf 3
Find the coordinate values of the dimensional coordinates). Therefore, the feature point P_Xof
Prior to the calculation of the three-dimensional coordinates, in the next step 236,
Image A of each grid point extracted in common from image A and image B
And the position on the image B, the feature point P on the image A _XPosition of
(X_A, Y_A), Feature point P on image B_XPosition (X_B, Y
_B) Based on
Interpolation ratio from 3D coordinates of each grid point (3 for each grid point
The weight for the coordinate value of the dimensional coordinate) is determined. This interpolation
Of the feature points P on the image A and the image B, for example._XAnd near
Coordinate values of the three-dimensional coordinates of the grid points existing at the touched position
Can be determined to have a larger weight.

Then, in step 238, the characteristic point P is calculated based on the three-dimensional coordinates of the grid points commonly extracted from the images A and B and the interpolation ratio determined in step 236.
Three-dimensional coordinates of _{X (X X, Y X,} Z X) calculates a. When the three-dimensional coordinates of the feature point P _X are calculated as described above, the reference point
The feature point coordinate calculation process is terminated, and the process proceeds to step 158 of the instruction determination process (FIG. 6). Since the reference point / feature point coordinate calculation process described above is repeatedly executed, the values of the three-dimensional coordinates of the reference point P ₀ and the feature point P _X calculated by the process are the posture and movement of the information input person 10. Will be sequentially updated according to the change of.

In step 158, it is determined whether or not the information input person 10 is performing the pointing operation based on the determination result of step 218 of the reference point / feature point coordinate calculation processing (FIG. 7) described above. In the reference point / feature point coordinate calculation process, the feature point P is obtained only when the information input person 10 is performing the pointing operation (when the determination in step 218 is affirmative).
_Since the three-dimensional coordinates of _X are calculated, the determination in step 158 can be made by, for example, determining whether the three-dimensional coordinates of the feature point P _X have been calculated. If this determination is denied, the process returns to step 154, and steps 154 to 158 are repeated while the information input person 10 exists in the information input space.

On the other hand, if the determination in step 158 is affirmative, the routine proceeds to step 160, where the reference point P ₀ and the characteristic point P _X calculated in the previous reference point / characteristic point coordinate calculation processing are 3
Based on the dimensional coordinates, as the direction pointed to by the information input person 10, the direction in which the virtual line connecting the reference point P ₀ and the characteristic point P _X (see the virtual line 54 in FIG. 13) extends is obtained, and the information input person is determined. As the position pointed by 10 (pointing position),
The coordinates (plane coordinates) of the intersection of the plane including the display surface of the display 12 and the virtual line (see point S in FIG. 13) are calculated.

Here, when the information input person 10 is performing the pointing operation with the left arm, the feature point P _X is offset to the left side with respect to the position of the information input person 10, but the reference point / feature In step 222 of the point coordinate calculation processing, the reference point P ₀ is set near the left shoulder of the information input person 10 as shown in FIG. 13B, and the position of the reference point P ₀ is also offset to the left. In step 160, the coordinates of the position (point S shown in FIG. 13B) that accurately matches the designated position assumed by the information input person 10 can be obtained as the coordinate of the designated position.

When the information input person 10 is performing the pointing operation with the right arm, the feature point P _X is offset to the right with respect to the position of the information input person 10. In step 222 of the coordinate calculation process, as shown in FIG.
Since the reference point P ₀ is set near the right shoulder of the information input person 10 as shown in FIG. _3C , the position of the reference point P ₀ is also offset to the right side. As a result, it is possible to obtain the coordinates of the position (point S shown in FIG. 13C) that accurately matches the designated position assumed by the information input person 10. Thus, step 160
Corresponds to the determination means of the present invention together with the reference point / feature point coordinate calculation processing described above.

In the next step 162, step 160
Based on the coordinates of the indicated position calculated in
Determines whether is pointing within the display surface of the display 12. If the determination in step 162 is negative, the process returns to step 154. On the other hand, if the determination in step 162 is affirmative, the process proceeds to step 164, and the coordinates of the position designated by the information input person 10 calculated in the previous step 160 is used as the coordinates of the position where the cursor should be displayed. Output to and instruct to display the cursor. As a result, the information processing device 14 displays the cursor at the designated position on the display surface of the display 12.

Instead of displaying the cursor at the position designated by the information input person 10 as described above, Japanese Patent Laid-Open No.
000-222098 and JP 2000-2221.
Needless to say, by applying the technique described in Japanese Patent No. 17 or the like, the fluctuation of the display position of the cursor may be suppressed regardless of the fluctuation of the pointing position by the information input person 10.

At the next step 166, it is determined whether or not the click operation has been performed by the information input person 10. Various operations can be adopted as the click operation. For example, the information input person 10 quickly moves his or her hand forward (see FIG. 14A, hereinafter referred to as “forward click”), and the information input person 10 An operation of quickly moving the hand backward (see FIG. 14B, hereinafter referred to as "reverse movement click") can be performed as a click operation. This click operation is an extremely natural operation as an operation of pointing and selecting a specific location on the display surface of the display 12, so that the person to be recognized can perform the click operation without feeling any discomfort.

Whether the forward click motion or the backward click motion is performed is determined by, for example, the distance k between the reference point P ₀ and the feature point P _X while the information input person 10 is performing the instruction motion.
This can be performed by repeatedly calculating (see FIG. 13) and determining in step 166 whether the difference between the distance k calculated this time and the distance k calculated last time is a predetermined value or more. If the determination in step 166 is negative, the process returns to step 154 without performing any processing, but if the determination in step 166 is affirmative, the process proceeds to step 168, and the click made by the information input person 10 is performed. The information processing device 14 is notified of information related to a motion (for example, information indicating a designated position (click position) at the time of a click motion, information indicating whether a click motion performed by the information input person 10 is a forward click or a backward click). .

As a result, the information processing device 14 executes a predetermined process based on the notified information about the click operation. As this predetermined process, any process can be adopted. For example, when a position where an icon representing a specific application is displayed on the display surface of the display 12 is forward-clicked, the specific application is displayed. When, for example, the location displaying the icon is clicked backward, special processing related to the specific application (for example, setting of various parameters related to the application, application version display, icon display parameter (Settings etc.) are displayed in a list, and when a specific special process is clicked forward, the clicked special process can be executed.

Further, a plurality of types of processing modes such as a normal processing mode, a file transfer mode, a file search mode, etc. are prepared in advance, and the information input person 10 switches to one of the plurality of types of processing modes. Mode switching instructing means (a foot switch may be used, or a mechanism (algorithm) for detecting whether or not the information input person 10 has performed a processing mode switching instructing operation such as changing the shape of a hand. May be provided), and different processing may be performed for each processing mode when the click operation is performed.

It is natural for the information inputting person 10 to extend his elbow when his arm is lowered, but when he raises his arm to indicate a desired position on the display surface of the display 12, the elbow is naturally stretched. Bends slightly (even if you think about performing a click action later, raising the arm while slightly bending the elbow indicates a desired position, rather than raising the arm with the elbow extended. It is rational because the elbow is naturally slightly bent when it is struck). Therefore, the reference point P during the movement of the information input person 10 to raise his arm.
The distance k between ₀ and the feature point P _X changes. Considering this, whether the information input person 10 starts to determine whether or not the click operation is performed after the state in which the information input person 10 is pointing to the position corresponding to the display surface of the display 12 continues for a relatively short time or more. Alternatively, it is preferable to prohibit notification that the click operation has been performed on the information processing device 14.
Thus, when the position designated by the information input person 10 changes from outside the display surface of the display 12 to inside the display surface, it is possible to prevent erroneous determination that the click operation has been performed.

Considering that the pointing position may shift during the click operation by the information input person 10, the pointing position when the information input person 10 starts the clicking operation is used as the click position. Is preferred. Accordingly, even if the designated position is displaced during the click operation, the process corresponding to the position initially pointed to by the information input person 10 can be surely executed.

In the above, two video cameras 36 are used.
Although the example in which A and 36B are provided has been described, the present invention is not limited to this, and the information input space may be imaged by a larger number of video cameras and the instruction from the information input person 10 may be determined. .

In the above description, the characteristic point P _{X according} to the present invention has been described by taking the point corresponding to the fingertip as an example, but the characteristic point may be any point whose position changes when the person to be recognized moves his or her arm. For example, a point corresponding to the wrist of the recognition target person or a point corresponding to the back of the recognition target person may be used as the feature point.

When the posture of the information input person 10 changes from a state where the arm is raised (see FIG. 12B or C) to a state where the arm is lowered (see FIG. 12A). To
The cursor may be continuously displayed at the designated position immediately before the arm is lowered. Thus, even when the information inputter 10 wants to continuously display the cursor at a certain position (for example, a presentation at a conference), the information inputter 1
Since 0 does not need to keep raising the arm continuously, the burden on the information input person 10 can be reduced.

Further, in the above, as the click action, only the forward click action in which the information input person 10 quickly moves his hand forward and the backward click action in which the information input person 10 quickly moves his hand backward are determined. But
The present invention is not limited to this. For example, by monitoring the transition of the distance k between the reference point P ₀ and the characteristic point P _X over a predetermined time, a double-click operation (hand pointing device) that is common in mouse operations is performed. Then, for example, a forward click operation or a backward click operation can be substituted by two consecutive operations, or a drag-and-drop operation (in the hand pointing device, for example, a forward click operation → instructed position moving operation → backward click operation). May be determined by the operation).

Further, as the display 12, a well-known display device such as a plasma display, a CRT, an optical fiber display or the like may be applied, or liquid crystal goggles or the like may be applied. Alternatively, a three-dimensional image representing a pseudo three-dimensional space may be displayed on the display. Specifically, the flat display as described above displays an image according to the one-point perspective method or the two-point perspective method,
An image may be displayed on a three-dimensional display using a liquid crystal shutter or a lenticular lens, or a holographic technique may be applied to display a stereoscopic image.

Needless to say, arbitrary information can be displayed on the display 12, for example, by connecting the information processing device 14 to a network such as the Internet,
A web page or the like having the URL designated by the user may be displayed in the display area. Further, when the hand pointing input device to which the present invention is applied is arranged in front of an automatic warehouse in which various products are stored, and the user specifies a desired product and completes payment (payment), the specified product It is also possible to use the hand-pointing input device to which the present invention is applied as an operation unit of a large-scale vending machine, such as when taking out from a warehouse and delivering it to a user.

[0079]

As described above, the present invention is based on an image obtained by capturing from a plurality of directions a situation in which a person to be recognized is performing a pointing operation pointing to a specific position or a specific direction. , The left and right of the arm used for the pointing motion is determined, the position does not change regardless of the change in the pointing position or the pointing direction in the pointing motion, and the left and right direction of the recognition target person is determined according to the result of the left / right determination of the arm. A reference point is set at a position offset with respect to the center, and the position of the feature point whose position changes according to the change in the designated position or the designated direction in the pointing motion is obtained, and the position of the reference point and the position of the feature point are set. Based on this, the pointing position or direction by the pointing action is determined,
This has an excellent effect that it is possible to improve the accuracy of determination of the pointing position or the pointing direction in the pointing operation.

[Brief description of drawings]

FIG. 1 is a perspective view showing the periphery of an information input space.

FIG. 2 is a block diagram showing a schematic configuration of a hand pointing input device according to the present embodiment.

FIG. 3 is a schematic diagram showing an example of a relationship between an illumination range of a lighting device and an imaging range of a video camera.

FIG. 4 is a perspective view of an information input space showing an example of a mark plate.

FIG. 5 is a flowchart showing a grid point position information initial setting process.

FIG. 6 is a flowchart showing the contents of an instruction determination process.

FIG. 7 is a flowchart showing the contents of reference point / feature point coordinate calculation processing.

FIG. 8 is a timing chart showing the timing of turning on / off the lighting devices A and B and outputting (capturing) an image obtained by capturing an image with a video camera.

FIG. 9 is a side view of the information input space for explaining the calculation of the height of the information input person and the position on the floor.

10A to 10C are image diagrams showing an example of an operation of an information input person.

FIG. 11 is an image diagram for explaining the left / right discrimination of the arm used by the information input person for the pointing motion and setting of the reference point and the feature point based on the discrimination result.

FIG. 12A is an image diagram showing a hand of an information input person imaged by a video camera, and FIG. 12B is a concept showing a search range of grid points for obtaining coordinates of feature points and three-dimensional coordinates of the feature points. It is a figure.

13A and 13B are plan views of the information input space, and FIGS. 13B and 13C are side views of the information input space for explaining the determination of the position on the display pointed by the information input person. .

FIG. 14A is an image diagram for explaining a forward click operation, and FIG. 14B is an image diagram for explaining a backward click operation.

[Explanation of symbols]

10 Information Entry Person 12 large screen display 14 Information processing equipment 20 Hand pointing input device 22 Controller 36A video camera 36B video camera 54 virtual line

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Hisashi Hanzawa             Chiba Prefecture Inzai City 1-5 Otsuka 1 Stock Association             Takenaka Corporation Technical Research Institute (72) Inventor Masayuki Uraya             8-21-21 Ginza, Chuo-ku, Tokyo Stock market             Takenaka Corporation Tokyo Main Store (72) Inventor Tetsufumi Kawashima             8-21-21 Ginza, Chuo-ku, Tokyo Stock market             Takenaka Corporation Tokyo Main Store F term (reference) 5B057 AA20 BA13 BA15 CA13 CA16                       DA07 DA08 DC05 DC08 DC16                 5B087 AA02 BC06 BC12 BC13 BC26                       BC32

Claims (6)

[Claims] 1. An image pickup means for picking up an image of a recognition target person from a plurality of directions different from each other, and the image pickup means picking up an image of a situation in which the recognition target person performs a pointing operation pointing to a specific position or a specific direction. Based on the obtained image, the recognition target person determines the right and left of the arm used for the pointing motion, the position does not change regardless of the change of the pointing position or the pointing direction in the pointing motion, and Reference point setting means for setting a reference point at a position offset with respect to the center of the recognition target person in the left-right direction according to the result of discrimination between the left and right, and the image extracted by the image capturing means. A feature point recognizing unit that obtains a position of a feature point whose position changes according to a change in a designated position or a designated direction in the pointing operation based on an image portion corresponding to a recognition target person, and the reference point setting unit. A pointed position determination device including: a determination unit configured to determine a pointed position or a direction by the pointing operation based on the position of the reference point set by the above and the position of the characteristic point obtained by the characteristic point recognition unit. 2. The image pickup means images the recognition target person from above, and the reference point setting means extracts an image corresponding to the recognition target person extracted from the image obtained by the image pickup means. The pointing position determination device according to claim 1, wherein the left and right of the arm used by the recognition target person for the pointing operation is determined based on the shape of the contour of the part. 3. The arm according to the reference point setting means, based on an offset direction of a position of a convex portion of the contour of the image portion corresponding to the arm of the recognition target person with respect to the entire contour of the image portion. The pointed position determination device according to claim 2, wherein the left and right sides of the position are determined. 4. The reference point setting means sets a reference point at or near the base of the arm used by the person to be recognized for the pointing motion based on the result of discrimination between the left and right of the arm. The pointing position determination device according to claim 1. 5. The determination means determines an intersection of a straight line connecting the reference point and the feature point and a pointed object as a pointed position by the pointing motion, or connects the reference point and the feature point. The pointing position determination device according to claim 1, wherein the direction in which the straight line extends is determined as the pointing direction by the pointing operation. 6. Based on an image obtained by capturing an image of a recognition target person from a plurality of directions different from each other, and capturing a situation in which the recognition target person is performing a pointing operation indicating a specific position or a specific direction. , The right and left of the arm used by the recognition target person for the pointing motion is determined, the position does not change regardless of the change in the pointing position or the pointing direction in the pointing motion, and according to the result of the left / right determination of the arm. A reference point is set at a position offset with respect to the center of the recognition target person in the left-right direction, and based on an image part corresponding to the recognition target person extracted from the image obtained by the imaging, in the pointing operation, The position of the characteristic point whose position changes according to the change of the designated position or the pointing direction is obtained, and the designated position by the pointing operation is based on the position of the set reference point and the obtained position of the characteristic point. Or, a pointing position determining method for determining the direction.