JP3860550B2 - Interface method, apparatus, and program - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an interface device that detects an object in an imaging space pointed to by a user using a body part (such as a finger or an arm).
[0002]
[Prior art]
Up to now, regarding the interface between a computer and a human, there are the following methods as an interface method and apparatus based on a human operation.
[0003]
As a first conventional technique, there is an apparatus that detects a user's movement from sensor information by attaching a sensor capable of measuring the movement of a hand or finger to the body. For example, there are commercially available products such as a device using a magnetic sensor (such as “MotionStar” manufactured by ASCENSION) or a device using a mechanical sensor (“Gypsy” manufactured by Spice, “CyberGlove” manufactured by Immersion). Further, there is a method described in Non-Patent Document 1. This is to recognize a user's movement by wearing a glove with an acceleration sensor or the like attached to his / her hand.
[0004]
As a second conventional technique, there is a technique described in Non-Patent Document 2. This method uses the arm constraints and the actual arm length of the user to extract the arm region from the input image of one camera, and the arm movement 3 from the temporal change in the length. Estimate dimensional change.
[0005]
As a third conventional technique, there is a technique described in Non-Patent Document 3. This method connects the coordinate position obtained in the body ("virtual projection center") and the coordinate position of the detected fingertip, and the extended pointing axis is the screen (screen of the display device displaying the input image) In this method, the intersecting point is set as a cursor position (instructed position). Since the virtual projection center position is obtained from the intersection of straight lines extending from the four corner positions of the screen to the fingertip, the position that can be indicated is only on the screen. The extraction of the fingertip position is realized by detecting the object closest to the screen by using two cameras and installing one at the position where the image is taken from above.
[0006]
As a fourth conventional technique, there is a technique described in Non-Patent Document 4. This method uses the distance information generated using a multi-lens stereo camera to detect the object closest to the screen as the fingertip, and uses the color information and distance information to detect the position of the eyebrows (eyes). In this method, the point where the extension line connecting the lines intersects the screen is set as the cursor position (indicated position).
[0007]
[Non-Patent Document 1]
Tsukada et al., “Ubi-Finger: Prototype of Mobile Oriented Gesture Input Device”, Workshop on Interactive and Software (WISS2001), pp. 119-124, 2001
[Non-Patent Document 2]
Abe et al., “Three-dimensional tracking of arm movement using optical flow and color information”, Image Recognition and Understanding Symposium (MIRU2002), pp. I267-I272, 2002
[Non-Patent Document 3]
Fukumoto et al., “Non-contact hand reader by moving image processing”, Proceedings of 7th Human Interface Symposium, pp. 427-432, 1991
[Non-Patent Document 4]
Kaneji et al., “Specifying method of cursor position with pointing pointer”, IEICE Technical Committee on Image Engineering, 2002.1
[0008]
[Problems to be solved by the invention]
However, the conventional methods described above have the following problems.
[0009]
In the first conventional method, the movement of the hand or finger can be recognized, but it is necessary to always attach some sensor to the part to be recognized, so that it is not convenient as a practical interface device.
[0010]
The second conventional method can recognize the movement of the arm without wearing anything on the body, but since it uses information from only one camera, information in the depth direction cannot be obtained directly, so a three-dimensional user The movement of the arm cannot be accurately extracted.
[0011]
The third and fourth conventional methods are pointing methods that allow a user to recognize a three-dimensional motion and indicate a position on a screen screen without wearing and non-contacting. It is limited to the screen and cannot point to a real object or position in any other direction.
[0012]
The object of the present invention is to solve the problems of these conventional methods that are not convenient due to the wearing type, and the problem that the depth accuracy is poor in the method using one camera, and is easy to associate the position with the real space. Provided is an interface method, apparatus, and program that allow a user to register the position and information of a real object visually or register real object information at a position that cannot be directly seen from a reference camera. There is.
[0013]
[Means for Solving the Problems]
  In order to achieve the above object, the interface device of the present invention provides:
  Means for inputting an image of a real space using a plurality of video cameras including a reference camera;
  Means for generating distance information from a reference camera to a photographed object by a stereo method using the photographed image;
  Voxel using distance information from the reference image, which is an image taken with the reference cameraAn initial voxel space is generated, and the initial voxel space is divided hierarchically until a voxel having a predetermined size is obtained or a predetermined number of divisions is reached, and the voxels at which the distance information intersects at that time Only by subdividingMeans for generating a space;
  Means for detecting two predetermined parts of a user's body as a start point and an end point on a photographed image from a reference camera, and calculating their two-dimensional coordinates on the image;
  Means for calculating the three-dimensional coordinates of the start point / end point on the real space from the distance information and the two-dimensional coordinates on the image of the start point / end point, and calculating direction information on the real space indicated by the user;
  Information on real objects existing in the real space at the generated three-dimensional position in the voxel space, that is,3Means for registering the dimension position information and the additional information;
  The direction information in the real space pointed to by the user is compared with the three-dimensional position information of the registered object, and the extension line in the direction indicated by the user, which is the three-dimensional pointed position information pointed by the operator, is registered. Means for detecting information about the intersection with the detected object;
  Generated voxel space3Dimension direction information, andShootingMeans for superimposing and displaying shadow imagesWhen
  have.
[0014]
Here, main terms will be described.
1. Voxel space
It is a set of “voxels” which are unit three-dimensional spaces having a cubic shape. FIG. 15 shows the relationship between the voxel and the voxel space, and an image of its hierarchical division.
2. Stereo method
This is a technique for measuring the distance from a reference camera to two or more input images including the image from the reference camera, using the principle of stereo vision (triangulation).
3. Reference camera
It is a camera that becomes a reference for generating distance information when performing the stereo method. An image taken with the reference camera is called a reference image.
[0015]
According to the present invention, it is not attached, depth information can be obtained with high accuracy, and three-dimensional pointing to a real space can be realized. In addition, by generating a voxel space viewed from the reference camera and displaying it visually superimposed on the real image, it is possible to easily register real object information, and real objects that are invisible from the reference image Can also be registered.
[0016]
In the embodiment of the present invention, the interface device includes means for generating an image obtained by horizontally inverting the reference image, and the voxel space is also generated in a state of being horizontally reversed in accordance with the inverted image. Display information is also superimposed on the inverted image.
[0017]
Therefore, in addition to the above-described advantages, it is possible to provide an interface that is more direct and intuitive, such as performing a pointing operation while looking at a mirror in which a self-image is reflected.
[0018]
In another embodiment of the present invention, the interface device has means for generating distance information by an active stereo method using a video camera and a light projector instead of the means for generating the distance information. .
[0019]
Therefore, similarly, the above-described problem can be solved, and an interface having high convenience and convenience for registering object information can be provided.
[0020]
DETAILED DESCRIPTION OF THE INVENTION
Next, embodiments of the present invention will be described with reference to the drawings.
[0021]
[First Embodiment]
FIG. 1 is a configuration diagram of an interface apparatus according to the first embodiment of the present invention, and FIG. 2 is a flowchart showing a flow of the processing.
[0022]
The interface apparatus according to the present embodiment uses an image captured by a plurality of cameras as an input image, and detects an actual space position (or object) pointed to by a user using a body part (such as a finger or an arm). In the device, the device can recognize the pointing position in the three-dimensional space based on the user's direct and intuitive three-dimensional pointing operation, and the user can perform the interface operation while looking at the self-image at the time of the operation. It is a device that can do it.
[0023]
This interface apparatus has an image input unit 11.₁, 11₂, Distance information generation unit 12, voxel space generation unit 13, start / end point two-dimensional coordinate calculation unit 14, start / end point three-dimensional coordinate calculation unit 15, three-dimensional indication position detection unit 16, spatial information registration unit 17, and inversion information generation unit 18. And an information display unit 19 and spatial information data 20.
[0024]
Image input unit 11₁, 11₂As shown in FIG. 1, two (or three or more) video cameras are used. The camera may be a commonly used video camera or CCD camera, and may be monochrome or color. However, a color camera is required when using a method using color information described later.
[0025]
The distance information generation unit 12 generates distance information from the reference camera to the photographing object using image processing by a stereo method from two or more input images including a reference image (an image of the reference image is shown in FIG. 3). (Step 21). As an example of a specific image processing method for generating the distance information, there is a method using a commercially available product, Digigreys (trinocular camera type), Bumblebee (binocular camera type), etc. of Point Gray Research. These are image input devices each incorporating three or two cameras, and can generate distance information as an output. In addition, since the method using the stereo method is common in the field of image processing (many publications), it is possible to make it by using any two or more cameras. FIG. 4 shows an image of distance information obtained along the x-axis direction on the line of one pixel on the reference image of FIG. In the figure, the distance axis direction represents the distance from the reference camera, and if it is far away, it takes a larger value, that is, it is in a position farther from the reference camera.
[0026]
The voxel space generation unit 13 creates the voxel space while dividing the hierarchy according to the distance information obtained by the distance information generation unit 12 (step 22). For example, in FIG. 5, an initial voxel space composed of 4 × 3 × 3 voxels is generated for the obtained distance information. Here, the initial voxel space is hierarchically divided up to the designated hierarchy, and a flag is set for each voxel whether or not a real object can exist based on the distance information. Hereinafter, in order to simplify the description, a method of setting a flag on a voxel and a two-dimensional hierarchical division method will be described using a graph (rectangular graph) representing distance information in FIG. FIG. 6 shows an example of division. Each voxel is given one of the three flags of white, gray, and black. That is, as shown in FIG. 6, when the graph representing the distance information intersects with the target voxel, it is gray, when it passes completely above, it is black, when it completely passes below, and the flag of that voxel To do. That is, it is determined whether or not the distance graph intersects with each voxel, and a flag is determined according to the result. Of these, only the voxels determined to be gray are further subdivided into four equal parts as shown in FIG. 6 (in the case of three-dimensional voxels, subdivided into eight equal parts as shown in FIG. 15). And give the flag again. The above is repeated until the size of the given voxel is reached or a predetermined number of divisions is reached. When recursive division is performed in this way, voxels that intersect the distance graph are recursively divided, and other areas that do not make sense in distance are represented by undivided voxels with a white or black flag. The Rukoto. The voxel space generated in this way has an octree structure as shown in FIG. 7, whereby the relationship between the parent voxel and the child voxel can be extracted efficiently and at high speed. As a fourth voxel flag, there is also a registration flag. This will be described later.
[0027]
The start / end point two-dimensional coordinate calculation unit 14 uses a reference image among the input images used in the distance information generation unit 12 to perform two-dimensional display on the screen of two predetermined parts (start point and end point) of the user's body. The coordinates are calculated (step 23). For example, the start point and the end point may be the start point of the user's shoulder and the end point of the hand. As a result, in this case, the direction of the tip of the hand with the arm extended becomes a three-dimensional instruction direction to be described later. A specific detection method on the screen when the position of the right shoulder and the right hand is set as the start point / end point is shown below.
[0028]
As a method for detecting the position of the right hand by image processing, for example, when the input image is a color image, the color information in the color image is used, and the skin color region (the range that the skin color can take is arbitrarily given a width. (Specify in the range of color values.) From the obtained skin color areas, constraint information such as size and position (for example, specify the area range of the skin color area that may be inferred from the size of the hand, or near the ceiling or floor on the screen. A candidate such as a hand skin color region is selected by using a restriction such as excluding a place where a hand is unlikely to exist or a distance information. In addition, 1) a normal user can be considered to be wearing clothes, and a skin color area that is likely to be a candidate for the skin color area is considered to be both hands and face 2) the area with the largest area is considered to be a face The skin color region having the second and third largest areas is used as a candidate for the hand. The right hand is an area on the right side of the face, and the center of gravity is the right hand position. When it is desired to obtain the left-hand position, it can be considered to be reversed left and right, and in the case of both hands, the two cases can be combined.
[0029]
In addition, as a method for detecting the position of the right shoulder by image processing, for example, there is a method of calculating the shoulder position after extracting the face position by the above-described means. Specifically, first, from the result of performing the skin color extraction process, since the skin color area having the largest area is highly likely to be a face, it is determined that the skin color area is a face, and the center of gravity is obtained. Next, it can be assumed that the position of the right shoulder is shifted from the center of gravity of the face by a certain distance downward and by a certain distance to the right. Therefore, the value of the right shoulder can be calculated from the center of gravity of the face. As a result, the two-dimensional coordinates on the screen of the start point and end point can be obtained. Here, the position of the shoulder is used as the starting point, but the center of gravity of the face obtained as described above may be used as the starting point. In that case, an extension line connecting the position of the face and the position of the hand is the user instruction direction.
[0030]
The start / end point three-dimensional coordinate calculation unit 15 calculates a three-dimensional coordinate value of the start point / end point from the generated distance information and the two-dimensional coordinates of the start point / end point (step 24). As a specific method, for example, a distance value corresponding to a two-dimensional coordinate on the image of the start point on the reference image is set as the distance value of the start point. The same applies to the end point. In the three-dimensional real space, the conversion formula between the two-dimensional coordinate system and the three-dimensional coordinate system of the image can generally be easily calculated in advance, so that the two-dimensional start point and end point on the screen obtained based on it can be calculated. A three-dimensional coordinate value in the three-dimensional space of the start point and the end point can be obtained from the coordinate value and each distance value thereof. Furthermore, by obtaining a three-dimensional straight line connecting the two points from the obtained two three-dimensional coordinate values of the start point and the end point, it is possible to obtain the user instruction direction.
[0031]
The three-dimensional indication position detection unit 16 detects the three-dimensional position in the actual photographing space pointed to by the user (step 25). As a specific method, first, a three-dimensional straight line connecting the start point and the end point obtained by the start / end point three-dimensional coordinate calculation unit 15 is extended in the hand (end point) direction. At this time, using the voxel space, it is detected whether or not the extension line intersects with a voxel in which three-dimensional position information of an object or the like in a previously registered space is registered. Such voxels have a registration flag as will be described later, and can be easily discriminated. Here again, a method of obtaining the intersecting voxels two-dimensionally based on the example of the distance information in FIG. 4 will be described. Now, assume that the black voxel in FIG. 8 is a voxel (registered voxel) in which object information is registered, and the user is pointing in that direction. At this time, in order to determine whether or not the voxel crosses the extension line in the designated direction, whether or not the extension line passes through the registered voxel is calculated from the voxel in the upper layer. When a registered voxel that passes through is found, whether or not the voxel in the lower layer of the registered voxel passes through the registered voxel is calculated. This passage determination is repeated until a voxel in which information is actually registered is found. Whether or not the extension line passes through the voxel can be calculated very easily because the voxel takes a regular cubic shape, and further, it is determined from the upper layer voxel toward the lower layer. It is possible to exclude a large area where information is not registered at an early stage, and it is possible to find out three-dimensional indication position information where registered information exists at extremely high speed. Here, information on objects in the space will be described in the description of the space information registration unit 17. The obtained three-dimensional indication position information is accumulated as spatial information data 20 and displayed on the display 10 in the information display unit 19.
[0032]
The spatial information registration unit 17 registers information such as an object in the actual photographing space that the user may instruct in the spatial information data 20 (step 26). For example, when the user is in a room, the object in the real space is home appliances in the room (TV, air conditioner, computer, clock, window, shelf, chair, desk, drawer, document , Audio equipment, lighting equipment, etc.) objects, and walls, floors, ceilings, windows, etc. of the room itself can be considered as objects. Here, using the information display unit 19, the real object information is registered three-dimensionally by conversation processing while displaying the voxel space superimposed on the reference image. FIG. 9 shows an image diagram in which the initial voxel space is superimposed and displayed on the reference image. In response to this result, a voxel representing a position where information is to be registered is selected, and information is registered in it. At this time, as shown in FIG. 10, only a certain voxel of interest is displayed, the adjacent voxel is displayed by moving the mouse, or the lower layer voxel is displayed by clicking the mouse. However, it is conceivable to narrow down the target voxels. Note that this display method, voxel selection, and movement method are not limited to this method. For example, it is conceivable to perform a pointing operation in the present invention in which a voxel space is designated as an indication target instead of a real space. It is also conceivable to register the same information in a plurality of voxels. In addition, information can be registered in a space where there is no real object or a place hidden from the reference image by adding an operation that can forcibly divide the white flag and black flag voxels during the conversation process. Finally, a registration flag is set as a fourth flag for the voxel in which information is registered and all voxels from the highest level to the lowest level to which the information belongs.
[0033]
Such information (coordinate information of a three-dimensional position, information on other objects, etc.) may be registered and stored in advance in the spatial information data 20 using real coordinates and automatically assigned to voxels. . Regarding the registration of information, a sensor capable of recognizing a position for each target real object (commercially available magnetic sensor, ultrasonic sensor, infrared tag, wireless tag) is not used as fixed three-dimensional position coordinates in advance. Etc.), the position of each object can be recognized in real time. Therefore, the object information is generated from the three-dimensional position information obtained from the object, and the three-dimensional position coordinates of the object are always generated. It is also possible to update the information. In this case, even if the object is moved, the three-dimensional position information and the like can be updated in real time.
[0034]
The display information reversing unit 18 horizontally inverts the reference image and also horizontally inverts the voxel space and the three-dimensional position information (step 27). FIG. 11 shows an image of a reference image and an image obtained by reversing the reference image. In this case, an image close to a user performing a pointing operation while looking at a mirror can be obtained. The left-right reversal of the reference image can be executed in real time by commercially available general-purpose image processing software (for example, HALCON) for the input image taken into the computer. Alternatively, it can also be realized by a commercially available device that inputs an input image and generates a reverse image in real time (for example, a screen horizontal reversing device UPI-100LRF of Sakae Co., Ltd., or SONY's EVI-D100 with an integrated camera). Further, an embodiment without the information reversing unit 18 is also conceivable. In that case, the coordinates are displayed as they are.
[0035]
For example, during the pointing operation, the information display unit 19 superimposes and displays on the display 10 CG representing 1) a reference image that is horizontally reversed, 2) the same voxel space, 3) the three-dimensional indication direction, and position (step) 28). FIG. 12 shows an example of each image in that case, and FIG. The display 10 may be a general-purpose display used for a computer as long as it can display a computer screen and a camera image. Note that the display method is not limited to this example. For example, it is conceivable that the display of the voxel space is limited to only some voxels of which attention is focused, or not performed at all.
[0036]
[Second Embodiment]
FIG. 13 is a block diagram of the interface apparatus according to the second embodiment of the present invention, and FIG. 14 is a flowchart showing the processing flow.
[0037]
In the second embodiment, the interface device according to the present embodiment is an active device that uses the camera 11 and the light projector 31 instead of using the passive stereo method generated from input images from two or more cameras. Distance information is generated by a stereo method. It is also possible to mix both.
[0038]
The passive stereo method generated from two or more images is, for example, searching for corresponding points between two camera input images close to the line-of-sight direction (corresponding point search is performed), and the deviation of the coordinate values. This is a method for obtaining the distance of the point from the size (parallax). The triangulation principle is used to calculate the distance. This method has a problem that it is difficult to search for corresponding points and it is difficult to obtain accurate distance information. However, there is no need for an active operation or device such as irradiating light, and there is an advantage that it is not affected by the photographing environment. For example, there are commercially available products such as Digigreys (trinocular camera type) and Bumblebee (binocular camera type) of Point Gray Research.
[0039]
On the other hand, the active stereo method using the light projector device replaces one of the two cameras with a light source that projects light, and uses information as a clue for searching for corresponding points as a target. This is a direct projection method. As the light, methods or products using various kinds of light such as slit light, spot light, and various kinds of pattern light are proposed or commercially available. This method requires a complex device that projects light, and has the problem of being affected by the shooting environment. However, since the corresponding point search can be performed stably, there is an advantage that the distance image can be obtained with high accuracy. have. For example, there are commercially available products such as Danae-R (non-contact type three-dimensional shape measurement range finder) manufactured by NEC Engineering.
[0040]
Since these two stereo methods can obtain distance information, they can be replaced with each other. Therefore, by using not only two or more cameras but also an active stereo method using one or more cameras and a projector, the available methods and commercially available devices become wide and versatile. Can be increased and the application destination can be expanded.
[0041]
In addition to what is implemented by dedicated hardware, the present invention records a program for realizing the function on a computer-readable recording medium, and the program recorded on the recording medium is stored in a computer system. It may be read and executed. The computer-readable recording medium refers to a recording medium such as a floppy disk, a magneto-optical disk, a CD-ROM, or a storage device such as a hard disk device built in a computer system. Furthermore, a computer-readable recording medium is a server that dynamically holds a program (transmission medium or transmission wave) for a short period of time, as in the case of transmitting a program via the Internet, and a server in that case. Some of them hold programs for a certain period of time, such as volatile memory inside computer systems.
[0042]
【The invention's effect】
As described above, the present invention has the following effects.
[0043]
Since the first, fourth, and seventh aspects of the present invention are non-wearing interfaces, user convenience is improved. In addition, since the voxel space using distance information obtained by the stereo method is used, registration of real object information and detection of a user's three-dimensional pointing position can be performed efficiently. Further, as a destination for three-dimensional pointing, not only the position on the screen but also the position in the real space can be pointed, and the application destination can be expanded.
[0044]
In addition to the effect of the first aspect, the inventions of the second, fifth, and seventh aspects can perform an interface operation using a mirror metaphor, thereby further improving user convenience.
[0045]
In addition to the effects of claims 1 and 2, the inventions of claims 3, 6 and 7 utilize not only two or more cameras but also an active stereo method using one or more cameras and a projector. Since methods or commercially available equipment can be used, versatility can be enhanced and application destinations can be expanded.
[Brief description of the drawings]
FIG. 1 is a configuration diagram of an interface apparatus according to a first embodiment of this invention.
FIG. 2 is a flowchart illustrating a processing flow of the interface apparatus according to the first embodiment.
FIG. 3 is a diagram illustrating an image of a reference image.
4 is a diagram illustrating an image of distance information obtained along one line on the reference image in FIG. 3;
FIG. 5 is a diagram illustrating an image of a voxel space generated based on distance information.
FIG. 6 is a diagram illustrating an image of how to divide a voxel space and set a flag.
FIG. 7 is a diagram illustrating an image when the generated voxel space is held as an octree structure.
FIG. 8 is a diagram illustrating an image of an algorithm for searching for a voxel in which information is registered from an instruction direction.
FIG. 9 is a diagram illustrating an image in which an initial voxel space is superimposed and displayed on a reference image.
FIG. 10 is a diagram illustrating an image when a neighboring voxel is highlighted or a lower-layer voxel is highlighted from a state in which one voxel is highlighted in the spatial information registration unit.
FIG. 11 is a diagram illustrating an image of the reference image of FIG.
FIG. 12 is a diagram illustrating an image of a configuration image with respect to a display result in the first embodiment.
FIG. 13 is a configuration diagram of an interface apparatus according to a second embodiment of this invention.
FIG. 14 is a flowchart illustrating a processing flow of the interface apparatus according to the second embodiment.
FIG. 15 is a diagram illustrating a relationship between a voxel and a voxel space and a concept of hierarchical division thereof.
[Explanation of symbols]
10 display
11, 11₁, 11₂    Image input section
12 Distance information generator
13 Voxel space generator
14 Start / end 2D coordinate calculation part
15 Start / end 3D coordinate calculation unit
16 3D pointing position detector
17 Spatial Information Registration Department
18 Display information reversing part
19 Information display section
20 Spatial information data
21 to 28 steps
31 Emitter
I, I₁, I₂    Input image

Claims (7)

An interface method for recognizing a position in a photographing space or a real object, which is taken by a video camera and pointed to by the user using a body part,
Using a plurality of video cameras including a reference camera to input an image of real space;
Generating distance information from the reference camera to the photographed object by a stereo method using the photographed image;
An initial voxel space is generated using the distance information from a reference image that is an image captured by the reference camera, and the initial voxel space is divided until a voxel having a predetermined size is obtained or predetermined division Generating a voxel space by subdividing only the voxels where the distance information intersects, subdividing hierarchically until reaching a number ,
Detecting two predetermined parts of the user's body on the captured image from the reference camera as a start point and an end point, and calculating their two-dimensional coordinates on the image;
Calculating the three-dimensional coordinates of the start point / end point in real space from the distance information and the two-dimensional coordinates on the image of the start point / end point, and calculating direction information in the real space indicated by the user When,
Registering three- dimensional position information and its additional information , which is information of a real object existing in real space, in an arbitrary three-dimensional position on the generated voxel space;
The direction information in the real space pointed to by the user is collated with the three-dimensional position information of the registered object, and the extension line in the direction indicated by the user, which is the three-dimensional pointed position information pointed by the operator, and registration Detecting information about intersections with the detected object;
The generated voxel space, before Symbol 3-dimensional pointing direction information, interface method having the steps of superimpose and shadow image shooting beauty Oyo. Generating a left-right inverted image of the reference image;
The step of generating the voxel space is generated in a state where the voxel space is horizontally reversed in accordance with the inverted image, and the step of superimposing and displaying various display information is also superimposed on the inverted image. The interface method according to 1.   The interface method according to claim 1, further comprising a step of generating distance information by an active stereo method using a video camera and a light projector instead of the step of generating the distance information. An interface device for photographing a user with a video camera and recognizing a position or real object in the photographing space, which the user points to using a body part,
Means for inputting an image of a real space using a plurality of video cameras including a reference camera;
Means for generating distance information from the reference camera to the photographed object by a stereo method using the photographed image;
An initial voxel space is generated using the distance information from a reference image that is an image captured by the reference camera, and the initial voxel space is divided until a voxel having a predetermined size is obtained or predetermined division Means for generating a voxel space by subdividing hierarchically until reaching a number, and subdividing only voxels where the distance information intersects ,
Means for detecting two predetermined parts of a user's body as a start point and an end point on a photographed image from the reference camera, and calculating their two-dimensional coordinates on the image;
Means for calculating the three-dimensional coordinates of the start point / end point in real space from the distance information and the two-dimensional coordinates on the image of the start point / end point, and calculating direction information in the real space indicated by the user When,
In any three-dimensional position on the generated voxel space is information on the real object existing in a real space, and the three-dimensional position information and means for registering the additional information,
The direction information in the real space pointed to by the user is compared with the three-dimensional position information of the registered object, and the extension line in the direction indicated by the user, which is the three-dimensional pointed position information pointed by the operator, is registered. Means for detecting information about the intersection with the detected object;
The generated voxel space, before Symbol 3-dimensional pointing direction information, the interface device comprising means for displaying superimposing the shadow image shooting beauty Oyo. Means for generating an image obtained by horizontally inverting the reference image;
The voxel space generating means also matches the voxel space with the inverted image,
5. The interface apparatus according to claim 4, wherein the interface device is generated in a horizontally reversed state, and the superimposing display unit also superimposes and displays various display information on the reversed image.   6. The interface apparatus according to claim 4, further comprising means for generating distance information by an active stereo method using a video camera and a light projector instead of the means for generating the distance information.   An interface program for causing a computer to execute the interface method according to claim 1.

Images