JP4053903B2 - Pointing method, apparatus, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a pointing device allowing operation even in a position apart from an interface device, allowing the operation without restraint from a specific remote controller, and having recognition accuracy not differing by peripheral illumination conditions or the like. <P>SOLUTION: An image including a color tag 5 held by a hand of an operator 4 is photographed by a camera, and is sent to a position detection part 11 through a directing tag registration part 16. The directing tag registration part 16 designates an area of a window W1 to start the position detection part 11, and detects an object position. When a unique object is decided as a detection result thereof, the position of the object is periodically tracked by use of a vicinity search part 12, and a temporary cursor position is transmitted to an interaction interface part 15. When the object is present inside the window W1 for a prescribed time T1 sec or above, comes out of the window W1, comes into a window W2 within a prescribed time T2 sec, and remains inside the window W2 for a prescribed time T3 sec or above, the object is recognized as a directing tag. <P>COPYRIGHT: (C)2004,JPO&amp;NCIPI

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an interface device that performs interactions such as device control and monitoring command execution.
[0002]
[Prior art]
Typical conventional techniques for pointing on the screen of a display device are: 1. Using a specific pointing device connected to the system, such as a mouse or tablet pen, 2. Using the remote control terminal, the interface device side Means for performing pointing by communicating from a remote location; 3. means for using gesture motion recognition of an operator based on an operator image acquired from a camera; 4. superimposing the operator image on a display screen and performing the operation There is a pointing method by detecting an overlap between a part of a person's body and a pointing object.
[0003]
The method using a pointing device such as the first mouse or tablet pen is a traditional method, and is a movement amount for moving the pointing device connected to the interface device on a plate-like object, or an electronic means. Pointing is realized by associating the detected x and y coordinates with the screen and displaying them as a cursor. A touch panel that directly touches the display screen is a kind of this method in that x and y coordinate values are obtained using a device connected to the interface device. As an extension of this method, there is a wireless mouse using infrared rays.
[0004]
In the second remote control terminal method, a command is transmitted using infrared light emission or the like based on an operator's button operation, and pointing is performed by the interface device receiving the command. The operator can operate even a few meters away from the interface device.
[0005]
In the third method based on gesture recognition, an operator's action is analyzed and a command is selected (see Non-Patent Documents 1 and 2). The motion of the operator is acquired in two dimensions or three dimensions using a camera, and is changed from a specified command set by a pattern recognition process of a motion locus of a body part specified by a change in the feature amount of the screen or image processing. You can choose one.
[0006]
In the fourth method using the overlap between the operator image and the object image, the operator or the operator's hand part is extracted from the camera image, and the overlap between the extracted area and the pointing object is detected. , Pointing is realized (see Non-Patent Documents 3 to 5).
[0007]
In the third and fourth methods, in recognition of an operator's action and image, a method of facilitating recognition by attaching a marker or sensor using a magnetic sensor, a light emitting element, or a reflective material to the body, and background difference -Some perform recognition processing by image processing such as motion difference processing and skin color recognition.
[0008]
[Non-Patent Document 1]
Kirishima, Sato, and Chihara, “Real-Time Image Recognition of Gestures by Learning of Gaze and Selection Control” IEICE Technical Report, PRMU2001-203, Jan., 2002.
[Non-Patent Document 2]
Konno, Yagi, Yamauchi, Yasumura, “Prototype of multi-modal interactive system by gesture input”, Information Processing Research Reports, HI, No. 60, pp. 65-72, 1995.
[Non-Patent Document 3]
Vivid Group Gesture Xtreme Software related products,
http://www.vividgroup.com/
[Non-Patent Document 4]
Saki et al., "A trial of action interface using personal computer video input", Human Interface Group, Society of Instrument and Control Engineers, Proceedings of the 12th Human Interface Symposium, pp. 213 -218, 1996.
[Non-Patent Document 5]
Kitabata, Ikenaga, Nojima, Uchimura, Yamashita, “Examination of Interaction with Character Agent: Evaluation of Interface Using Self-Image Display”, Human Interface Society, 5th Nonverbal Interface Study Group (SIGNOI), pp. 11- 16, 2002.
[0009]
[Problems to be solved by the invention]
In an instruction (pointing) operation for specifying an object in real space and acquiring information about the object or operating the object, it is assumed that the operator cannot directly touch the object to be operated. . Further, when using the object, it is desirable that the object is not constrained near the interface device (computer, operation panel, etc.) and can be operated without using a special device. In this regard, the conventional pointing method has the following problems.
[0010]
In the first method using a pointing device such as a mouse or a tablet pen, the operator is restricted to the closest position of the interface device and needs to use a pointing device designated by the interface. Even in the case of the touch panel, the operator needs to operate with a hand or a pen, and the operator cannot leave the interface device.
[0011]
In the pointing operation by the second remote control terminal, in many cases, the remote control terminal is specially made for the operation of a specific device, so it is necessary to carry the remote control terminal for each operation target object (device), If the necessary remote control terminal cannot be found at hand, it will take time to search for it. In addition, since many remote control devices do not like dirt, there are restrictions on the state of the operator's hand, such as not touching with a wet hand.
[0012]
[Means for Solving the Problems]
The pointing device of the present invention is a pointing device that recognizes an operator's pointing operation in an interface device that points to a specific instruction or operation by pointing an image of an object acquired by a camera and displayed on a display screen. ,
Location detection in a given area in a camera image of a given object with a color tag that emits or reflects a given range of hue, saturation, brightness, or light primary colors in real space Position detecting means for performing,
In the camera image immediately before in the camera image, a vicinity instructing position detection to the position detection unit using a region of a predetermined size including the object position output by the position detection unit as a position detection range Search means;
Cursor generating means for converting the color tag coordinates on the displacer screen coordinates,
Interaction interface means for transmitting the converted display screen coordinates as cursor data to the interface device;
Cursor tracking means for tracking the color tag using the proximity search means and transmitting the color tag coordinates to the cursor generation means;
When the position detecting unit detects the object in a predetermined area on the screen, the proximity tracking unit is used to periodically detect the position of the object, and the object is in the predetermined area for a predetermined time or more. An instruction tag registration means for registering the object as an instruction tag and transmitting an instruction tag tracking instruction to the cursor tracking means ,
After the instruction tag is registered by the instruction tag registering means, a color range including a high frequency range using a light emission or reflection frequency of a color range in which the instruction tag actually emits or reflects at the time of registration; Thus, a color range obtained by reducing a predetermined color range is obtained, and in the search for the instruction tag, a neighborhood search is performed by the neighborhood search means using the reduced color range.
[0013]
An object of the present invention is to provide a pointing method, apparatus, and program that can be operated at a position away from an interface apparatus or can be operated without being bound by a specific remote control apparatus, and whose recognition accuracy is less likely to deteriorate due to surrounding illumination conditions. There is.
[0014]
[Means for Solving the Problems]
The pointing device of the present invention includes:
Position detecting means for detecting the position of a predetermined object in real space within a given area;
Proximity search means for instructing the position detection means as a position detection range in the vicinity of the object position on the previous screen in time;
A cursor generating means for converting the instruction tag coordinates into display screen coordinates;
Interaction interface means for transmitting cursor data to the interface device;
Cursor tracking means for tracking an instruction tag using a proximity search means and transmitting to the cursor generation means;
It has an instruction tag registration means for recognizing an object as an instruction tag at an arbitrary timing using a position detection means, a proximity search means, and a cursor tracking means.
[0015]
In the present invention, an arbitrary object that can be touched by the operator is simply characterized in advance, and the object held by the operator during the pointing operation is presented to the system on the spot, thereby being used as a pointing device. I can do it. The operator can arbitrarily select a handy object at hand to be a pointing device, and by re-presenting it to the system, the object to be used for the pointing device can be replaced according to the situation on the spot be able to. For this reason, it is possible to easily operate a computer, a home appliance, and the like without moving to the computer each time the operation is performed, washing the hands each time, losing the remote control device and searching the room.
[0016]
DETAILED DESCRIPTION OF THE INVENTION
Next, embodiments of the present invention will be described with reference to the drawings.
[0017]
(First embodiment)
FIG. 1 is a configuration diagram of a pointing device according to a first embodiment of the present invention. The pointing device of the present embodiment is a device that creates cursor information and sends the cursor information to an interaction device that executes icon selection / execution processing.
[0018]
The color tag 5 held by the operator 4 is an object that emits or reflects a predetermined color. The camera 3 is a camera that captures an image of a room, and acquires a color image that allows the color to be identified. The pointing device 1 includes a position detection unit 11, a proximity search unit 12, a cursor generation unit 13, a cursor tracking unit 14, an interaction interface unit 15, an instruction tag registration unit 16, and a display 2 for display. On the display 2, an instruction tag registration window W1 and a window W2 are provided.
[0019]
The interaction device is included in the display 2. However, the camera image must be shared between the pointing device 1 and the interaction device. Actually, it is realized as a program that operates while communicating with each other on the same PC. In addition, a processing process capable of referring to the camera video from both operates simultaneously.
[0020]
In this embodiment, the position detection unit 11 detects a position based on the color in the region given from the instruction tag registration unit 16. Color signals in the hue range, brightness range, and saturation range determined as the color are detected from the image acquired from the camera 3, and the image is divided into continuous regions. A region having an area that is too small is ignored as noise, and then the barycentric coordinates and area column of the detected region set are output. The center of gravity is used as the position of the object itself. The area indicates the size of the object, and the purpose is to compare the size with the object detected by the coordinate neighborhood search. Appropriate thresholds are set, and it is estimated that objects that differ greatly in size are not objects being tracked. When the object disappears, the center of gravity coordinates and area are treated as disappearance.
[0021]
The proximity search unit 12 detects the position by specifying the vicinity of the object (object with the color tag 5) position on the previous screen (the screen immediately before the screen acquired at the time of processing). Position detection is performed by the unit 11. If an object is detected in the vicinity, a new position is output. When a plurality of objects are detected, only one position is selected using the distance from the object position on the previous screen, the area of the area, and the like as evaluation values. If no object is detected in the vicinity, the search failure is reported to the instruction tag registration unit 16. As for “multiple objects”, there is a possibility that a plurality of objects of that color will enter the specified area, and some parts of one object will be in that color, so it looks like multiple objects there is a possibility. In this case, although accurate object recognition cannot actually be performed, it is assumed that the corresponding color region having the largest area is estimated as an “object” to be aimed at, or the color regions close to a distance equal to or less than the threshold are treated as a lump. It is uniquely determined by a method such as estimating the largest of the lump of objects as an object.
[0022]
The cursor generation unit 13 converts the instruction tag coordinates given from the cursor tracking unit 14 into display screen coordinates. The image is converted by rotating / enlarging / reducing / moving processing in consideration of the camera installation position, the display installation position, and the angle of view of the camera. The conversion parameters are set in advance by calibration so that an appropriate range on the display can be instructed from the average standing position of the operator.
[0023]
The interaction interface unit 15 transmits cursor data to the interaction device. The cursor has two states, a main cursor and a temporary cursor, and the interaction work in the interaction device is valid only in the main cursor state, and only the cursor position is displayed in the temporary cursor state.
[0024]
In order to detect the position of the object to be added first, the instruction tag registration unit 16 activates the position detection unit 11 by designating a region of the window W1, which is a specific region on the screen at a certain point in time, The position detection unit 11 is caused to detect the object position (detection is performed by color). Once the position of the object is detected (determined as the only object), the cursor tracking unit 14 is used to repeat the operation of detecting where the object has moved next to track the object. On the screen next to the object position detected screen, position detection is performed using the vicinity of the detected position (neighboring) as a search range, thereby attempting to detect the moved object position at the next time point (screen). The instruction tag registration unit 16 causes the cursor tracking unit 14 to track the object. On the other hand, when a new object is detected along the flow on the left side of FIG. 2 (instruction tag registration unit 16), the window W1, Following the object in W2, the temporary cursor is detected and the change to the main cursor is determined. When a new temporary cursor or real cursor is detected, it is transmitted to the cursor tracking unit 14.
[0025]
When activated by the instruction tag registration unit 16, the cursor tracking unit 14 keeps tracking the cursor position (two tags (temporary cursor and main cursor) at the same time) independently of the instruction tag registration unit 16. The cursor tracking unit 14 tracks the instruction tag received from the instruction tag registration unit 16, converts the result into cursor coordinates by the cursor generation unit 13, and outputs the result from the interaction interface unit 15. When a new position is received from the instruction tag registration unit 16, tracking is performed from that position. If tracking fails, the tracking is stopped until a new tracking position is received from the instruction tag registration unit 16.
[0026]
FIG. 2 is a flowchart showing an outline of the operation flow of each of the above parts. Each unit operates as an independent processing process, and is configured to advance processing while transmitting information to each other.
[0027]
The instruction tag registration unit 16 generates the position of the object as a temporary cursor when an object with the color tag 5 exists in the window W1 for a predetermined time T1 seconds or longer. When an object exits the window W1 and enters the window W2 within a predetermined time T2 seconds and stays in the window W2 for a predetermined time T3 seconds or more, the object is recognized as an instruction tag and is determined to be used as a cursor. (The temporary cursor becomes the real cursor.) If the condition is not satisfied, the temporary cursor transmission is stopped and the initial state is restored.
[0028]
By doing this, the operator 5 holds the object of the color (that is, the object with the color tag 5 of the color) in his hand, moves the object while looking at the screen, and then moves to the window W1. By putting it in W2, the object can then be made an object that acts like a normal mouse or tablet pen. As long as the system successfully tracks the object, the function of the pointing device is valid (the color tag can be used as a cursor) and the operator 4 can interact with the interaction device. If tracking an object fails along the way, the cursor disappears. However, the cursor can be restored by holding the object again over the windows W1 and W2. Furthermore, even when the cursor is alive, a temporary cursor is generated when another object is entered into the window W1, and another object becomes the main cursor when moved to the window W2. When another object becomes the cursor, the position of the first object is ignored. Since the temporary cursor can be generated during the tracking of the cursor, the cursor tracking unit 14 tracks two tags at the same time.
[0029]
In this way, any color can be used as a pointing device such as a mouse or tablet pen. Even in the case of using an ID tag, the ID tag of the object appropriately selected by the user becomes the pointing device as it is. Therefore, as long as the ID tag is present, it is possible to grasp anything around that and make any pointing device. It becomes unnecessary to attach to the system in advance or to assign the ID fixedly like a normal mouse or tablet pen. The instruction tag registration unit 16 activates the cursor tracking unit 14 when the cursor tracking unit 14 is not activated, and notifies the cursor tracking unit 14 of the position of a new designated object when the cursor tracking unit 14 is already activated.
[0030]
In the present embodiment, since the instruction tag registration unit 16 operates in parallel with the cursor tracking unit 14, even when the operator 4 leaves the color tag 5 during the pointing operation, the same color is displayed at the destination. If a separate color tag is prepared, the pointing operation can be resumed by executing the registration operation again.
[0031]
By preparing a plurality of color tags 5 at any appropriate position in the room, the operation can be continued even if the place is moved without being bound by a registered object.
[0032]
Also, if the device fails to track the color tag during operation, it can be known by the disappearance of the cursor. In that case, the pointing operation can be resumed by performing the registration operation again, and the use becomes easy even in an environment where the recognition accuracy cannot be increased. If a mechanism for issuing a warning to the operator when the cursor disappears is provided, the usability can be further improved.
[0033]
As a modification of the present embodiment, it is possible to use the operator's hand as a color tag by using skin color recognition. For hand recognition, it is possible to improve the accuracy by applying the existing invention. Furthermore, even a sticker of a prominent color can be used as a color tag if it is sufficiently bright.
[0034]
In addition, after the instruction tag is registered by the instruction tag registration unit 16, a color range including a high frequency range is used by using the light emission or reflection frequency of the color range in which the instruction tag actually emits or reflects at the time of registration. Thus, a color range obtained by reducing a predetermined color range may be obtained, and a neighborhood search by the neighborhood search unit 12 may be performed using the reduced color range only in the search for the instruction tag.
[0035]
If a predetermined color range is originally divided into small areas in a color space such as HSV or RGB, and the color of each pixel actually detected on the screen is plotted in that space, the frequency of use of the color is increased. . By using a color range surrounding the color set including the top n of the usage frequencies, or a color set in which the cumulative frequency value is x% or more selected from the top colors, a new color is used for neighborhood search. Can be squeezed.
[0036]
This is because if a certain color range is kept wide, and something like that actually enters the window, whatever it is, it will be used as an instruction tag, and if another color comes in It is a method of applying the tracking with the color of the other one. The advantage is that you can reduce the possibility of misrecognizing other things by narrowing down the color once you have selected a tag after widening the color range of things that can be used for the instruction tag (more can be used as instruction tags). Is Rukoto.
[0037]
(Second Embodiment)
The pointing device of FIG. 3 is an example in which the present invention is applied to a self-image display interface device. The self-image display type interface device is an interface that superimposes and displays a mirror image of a camera image taken by an operator on the display 2 and selects and operates an object that is simultaneously photographed in the display 2, a virtual object by CG, an icon, and the like. . In order to produce a mirror image, the mirror image processing unit 17 is required.
[0038]
The pointing device can be integrated into the self-image display interface device by transmitting mirror image data together with cursor information to the interaction device.
[0039]
(Third embodiment)
The pointing device of FIG. 4 is an embodiment using a tag with an ID. In this embodiment, a tag using infrared (IR) is assumed as the tag 6 with ID. The tag transmits its ID by infrared signal using a signal such as a Morse code, the image is acquired by an infrared camera, and information about the ID number and the tag is distributed by image processing. It has a function to do. There is another method in which another infrared ray transmission source is placed on the camera side, an ID is sent from the outside, and an Acknowledge signal is returned to the ID of the tag to identify the position. In any case, the present embodiment can be used by detecting the position of the ID tag on the screen and distributing it as an ID + coordinate value. Other tags with IDs may be used as long as sufficient accuracy can be obtained when matching on the display screen, such as a radio wave tag (RF tag) or a tag using ultrasonic waves. The ID tag recognizing apparatus may be realized by another PC because the camera does not necessarily have to be shared.
[0040]
Existing technology can be used for tag position sensing. In FIG. 4, a set 20 of tag ID / coordinate value pairs is obtained by the infrared camera 3 ′ and the IR tag control / ID / coordinate detection unit 18. Unlike the first embodiment, the position detection unit 11 detects a tag in the designated area from the coordinate group output from the IR tag control / ID / coordinate detection unit 18. Further, the neighborhood searching unit 12 searches for a neighborhood of a tag having the same ID as the tracking tag.
[0041]
(Fourth embodiment)
The fourth embodiment shown in FIG. 5 integrates the second embodiment and the third embodiment to configure a self-image display interface by tag recognition. The tag recognition result can be used as information for object registration when constructing an interface based on real-world object recognition. As shown in FIG. 5, by sharing both the ID / coordinate information of the IR ID tag and the mirror image data of the camera with the interaction device, the object with the IR ID tag attached can be used as an instruction tag. It is possible to construct a self-display interface that can be used as a real-world object to be processed.
[0042]
Further, as in the third and fourth embodiments, when selecting by ID tag, the first detection (not by the proximity search unit 12 but by starting the position detection unit 11 directly from the instruction tag registration unit 16 and selecting) A plurality of ID-added objects may be present in the area. In this case, it is possible to envisage methods such as selecting at random, selecting one that has been selected in the past history, or selecting the one with the smallest number. However, once the ID is determined, the proximity search unit 12 tries to search only the object with the ID, so even if there are a plurality of ID-attached objects, the one to be selected is determined uniquely (unless the IDs overlap).
[0043]
(Fifth embodiment)
The pointing device of the fifth embodiment shown in FIG. 5 is obtained by adding a microphone 7 and a voice recognition command unit 21 to the pointing device of the first embodiment.
[0044]
It operates in the same manner as in the first embodiment until the provisional cursor is generated after an object is detected in the window W1 for more than T1 seconds, and thereafter, from the voice command recognition unit 21 within the T4 seconds, “tag registration” and the like If the predetermined keyword is detected, the temporary cursor is moved to the main cursor, and if the keyword cannot be detected within T4 seconds, the instruction tag registration unit 16 returns to the object detection process in the window W1 and starts again.
[0045]
(Sixth embodiment)
The pointing device of the sixth embodiment shown in FIG. 6 is obtained by adding a movement trajectory analysis unit 22 to the pointing device of the first embodiment.
[0046]
After the operation is performed in the same manner as in the first embodiment until the temporary cursor is generated after an object is detected in the window W1 for T1 seconds or longer, the instruction tag registration unit 16 stores the object position information as a result of the proximity search. The time series is transmitted to the movement trajectory analysis unit 22. The movement trajectory analysis unit 22 analyzes the movement trajectory of the object. For example, when a reciprocating motion of the object having a predetermined amplitude or more is detected more than a predetermined number of times, the fact is reported to the instruction tag registration unit 16. Instruction tags can be registered. If the report from the movement trajectory analysis unit 22 does not come for T5 seconds or more, the instruction tag registration unit 16 notifies the movement trajectory analysis unit 22 of the end of tracking, and returns to the object detection processing in the window W1 and starts again.
[0047]
In addition to what is implemented by dedicated hardware, the present invention records a program for realizing the function on a computer-readable recording medium, and the program recorded on the recording medium is stored in a computer system. It may be read and executed. The computer-readable recording medium refers to a recording medium such as a floppy disk, a magneto-optical disk, a CD-ROM, or a storage device such as a hard disk device built in the computer system. Furthermore, a computer-readable recording medium is a server that dynamically holds a program (transmission medium or transmission wave) for a short period of time, as in the case of transmitting a program via the Internet, and a server in that case. Some of them hold programs for a certain period of time, such as volatile memory inside computer systems.
[0048]
【The invention's effect】
As described above, the present invention has the following effects.
[0049]
By using an arbitrary color tag or ID tag as an instruction tag, the operator can perform an operation even from a remote location without being constrained near the interface device or while moving in the room. .
[0050]
Furthermore, since any instruction tag can be registered, the operation can be performed without being tied to a specific remote control device, and therefore, the operation can be performed without wasting the operator's time by searching for the remote control. By combining with human hand recognition technology, it is possible to point with bare hands, using hands instead of tags, and to build man-machine interfaces with a high degree of freedom.
[0051]
Further, as shown in the embodiment, it is possible to switch the instruction tag by a re-registration operation even during pointing. Even when resuming work after an act such as letting go of the instruction tag and standing up for another work during the operation, the operation can be resumed with the newly registered tag without being bound by the previously registered instruction tag. .
[0052]
Furthermore, even if the tag tracking fails during the operation, it is possible to easily resume the work by re-registering the tag.
[0053]
Since color tags and ID tags used for instruction tags can be manufactured in small sizes, they can be attached to various types of tags. It can be used with pens and rulers used in the office, paper knives and kitchen utensils at home.
[0054]
As described above, according to the present invention, it is possible to use an interface with a computer in many situations, and a pointing device for constructing an interface that uses calculation power can be constructed under various daily situations. .
[Brief description of the drawings]
FIG. 1 is a configuration diagram of a pointing device according to a first embodiment of the present invention.
FIG. 2 is a flowchart illustrating the operation of each unit of the pointing device according to the first embodiment.
FIG. 3 is a configuration diagram of a pointing device according to a second embodiment of the present invention.
FIG. 4 is a configuration diagram of a pointing device according to a third embodiment of the present invention.
FIG. 5 is a configuration diagram of a pointing device according to a fourth embodiment of the present invention.
FIG. 6 is a configuration diagram of a pointing device according to a fifth embodiment of the present invention.
FIG. 7 is a configuration diagram of a pointing device according to a sixth embodiment of the present invention.
[Explanation of symbols]
1 Pointing device 2 Display 3 Camera 3 'Infrared camera 4 Operator 5 Color tag 6 IR ID tag 7 Microphone W1, W2 Window
C Cursor 11 Position detection unit 12 Neighborhood search unit 13 Cursor generation unit 14 Cursor tracking unit 15 Instruction interface unit 16 Instruction tag registration unit 17 Mirror image processing unit 18 Image data 19 IR tag control ID / coordinate detection unit 20 Tag ID / coordinate data 21 voice command recognition unit 22 movement trajectory analysis unit

Claims (9)

Are acquired by a camera in the interface device to instruct the image specific instructions or operations by pointing a thing body displayed on the display screen, a recognizing pointing method of the pointing operation of the operator,
The position of a given object in a given area in a camera image with a color tag that emits or reflects a given range of colors in hue, saturation, lightness, or light in the three primary colors in real space. A position detection step for performing detection;
The vicinity instructing position detection to the position detection step in the camera image immediately before in the camera image with a region of a predetermined size including the object position output by the position detection step as a position detection range A search step;
A cursor generation step for converting color tag coordinates into display screen coordinates;
An interaction interface step of transmitting the converted display screen coordinates as cursor data to the interface device;
A cursor tracking step of tracking the color tag using the proximity search step and transmitting the color tag coordinates to the cursor generation step;
When the object is detected in a predetermined area on the screen by the position detecting step, the proximity tracking step is used to periodically detect the position of the object, and the object is detected in the predetermined area for a predetermined time or more. the presence within, registers the object as instructed tag, possess an indication tag registration step of transmitting the tracking instruction instruction tag on the cursor tracking step,
After the instruction tag is registered by the instruction tag registration step, a color range including a high-frequency range using a light emission or reflection frequency of a color range in which the instruction tag actually emits or reflects at the time of registration; A color range obtained by reducing the predetermined color range is obtained, and in the search for the instruction tag, a neighborhood search is performed by the neighborhood search step using the reduced color range.
A pointing method characterized by the above . A pointing method for recognizing an operator's pointing operation in an interface device for pointing a specific instruction or operation by pointing an image of an object acquired by a camera and displayed on a display screen,
An ID tag position detecting step for detecting a position in a camera image of a predetermined object with an ID tag capable of detecting the position and ID in the real image in the real space ; and
A tracking step of tracking an ID tag having an ID detected by the ID tag position detection step in a camera image one time earlier in the camera image;
A cursor generation step for converting ID tag coordinates into display screen coordinates;
An interaction interface step of transmitting the converted display screen coordinates as cursor data to the interface device;
A cursor tracking step of tracking the ID tag using the tracking step and transmitting the ID tag coordinates to the cursor generating step ;
When the object is detected in a predetermined area on the screen by the ID tag position detection step, the tracking step is used to periodically detect the position of the object, and the object is detected for a predetermined time or more. An instruction tag registration step of registering the object as an instruction tag by being present in the region, and transmitting an instruction tag tracking instruction to the cursor tracking step ;
After the instruction tag is registered by the instruction tag registration step, the ID of the ID tag at the time of registration is tracked as a pointer.
A pointing method characterized by the above . In the instruction tag registration step, the proximity search step or the tracking step is performed when the object is in a first area designated in advance on the screen and detected in the position detection step or the ID tag position detection step. with periodically detects the position of the object using the and transmitted to the cursor tracking step the coordinates as the cursor position of the temporary, the object enters the second region out of the first region The pointing method according to claim 1, wherein the pointing method is a step of recognizing an object as an instruction tag and transmitting an instruction tag tracking instruction to the cursor tracking step. Further comprising a voice command recognition step, the instruction tag registration step, when the object is detected in the predetermined area on the screen, using the neighboring tracking step or said tracking step, position the regular And a step of recognizing the object as an instruction tag and transmitting an instruction tag tracking instruction to the cursor tracking step when a tag recognition command is given by a voice recognition command. The method according to any one of 1 to 3. Has a movement locus analysis step further, the instruction tag registration step, when the object is detected in the predetermined area on the screen, using the neighboring tracking step, with periodically detecting position, 5. The step of recognizing the object as an instruction tag by detecting a simple gesture in the moving locus analysis step and transmitting an instruction tag tracking instruction to the cursor tracking step. The method in any one of. The method according to any one of claims 1 to 5, further comprising a mirror image processing step of generating a mirror image of a camera image obtained by photographing an operator and transmitting the image to the interface device together with cursor information. A pointing device for recognizing an operator's pointing operation in an interface device for pointing a specific instruction or operation by pointing an image of an object acquired by a camera and displayed on a display screen,
The position of a given object in a given area in a camera image with a color tag that emits or reflects a given range of colors in hue, saturation, lightness, or light in the three primary colors in real space. Position detecting means for performing detection;
In the camera image immediately before in the camera image, a vicinity instructing position detection to the position detection unit using a region of a predetermined size including the object position output by the position detection unit as a position detection range Search means;
Cursor generating means for converting the color tag coordinates on the displacer screen coordinates,
Interaction interface means for transmitting the converted display screen coordinates as cursor data to the interface device;
Cursor tracking means for tracking the color tag using the proximity search means and transmitting the color tag coordinates to the cursor generation means;
When the position detecting unit detects the object in a predetermined area on the screen, the proximity tracking unit is used to periodically detect the position of the object, and the object is in the predetermined area for a predetermined time or more. An instruction tag registration means for registering the object as an instruction tag and transmitting an instruction tag tracking instruction to the cursor tracking means ,
After the instruction tag is registered by the instruction tag registering means, a color range including a high frequency range using a light emission or reflection frequency of a color range in which the instruction tag actually emits or reflects at the time of registration; A color range obtained by reducing the predetermined color range is obtained, and in the search for the instruction tag, a neighborhood search is performed by the neighborhood search means using the reduced color range.
A pointing device. A pointing device for recognizing an operator's pointing operation in an interface device for pointing a specific instruction or operation by pointing an image of an object acquired by a camera and displayed on a display screen,
An ID tag position detecting means for detecting a position in the camera image of a predetermined object with an ID tag capable of detecting the position on the camera image and the ID in the real space ;
A tracking means for tracking an ID tag having an ID detected by the ID tag position detecting means in a camera image one time earlier in the camera image;
Cursor generation means for converting ID tag coordinates into display screen coordinates;
An interaction interface for transmitting the converted display screen coordinates as cursor data to the interface device;
Cursor tracking means for tracking the ID tag using the tracking means and transmitting the ID tag coordinates to the cursor generating step ;
When the object is detected in a predetermined area on the screen by the ID tag position detection means, the tracking means is used to periodically detect the position of the object, and the object is detected for a predetermined time or more. Registering the object as an instruction tag by being present in the region, and having an instruction tag registration means for transmitting an instruction tag tracking instruction to the cursor tracking means ,
After the instruction tag is registered by the instruction tag registration means, the ID tag ID at the time of registration is tracked as a pointer.
Pointin device characterized by that . Pointing program for executing a pointing method according to the computer in any one of claims 1 to 6.

Images