JP2004246578A - Interface method and device using self-image display, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a highly convenient interface device capable of detecting an operation in a non-contact status and in an arbitrary place. <P>SOLUTION: An image including an operator is designated by an image inputting part 11. An image(inverted image) photographed by the image inputting part 11, and inverted right and left is generated by an inverted image generating part 12. A position(user instructed position) shown by a user with the site of the body such as the finger is detected on the inverted image by an instructed position detection part 13. A command selecting part 14 collates the user instructed position obtained by the instructed position detecting part 13 with the display position of a plurality of command information displayed at an information display part 15, and retrieves and selects the command information to be instructed by the user from their positional relation. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an interface method and an apparatus for inputting an image from a camera and selecting a menu on a screen based on a human operation.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, in an interface between a computer and a human, as an interface device that receives an image from a camera and selects a menu on a screen based on a human operation, there is a device using a pointing device such as a keyboard and a mouse. There are also a method of attaching a marker to the body and detecting movement from image information from a camera, and a method of attaching a sensor to the body and directly detecting the movement. Further, there is a method of detecting a human motion from information such as an input image from a camera without wearing anything on the body.
[0003]
As a first conventional example of an interface device using such a method, there are a keyboard, a mouse, and a touch panel display as devices that can be connected to a general computer and are commercially available. These are all for detecting the movement of the user's hand and converting it into input information.
[0004]
In addition, as a second conventional example, there is a device for attaching a marker to a body and detecting a motion by image processing in a product related to a motion capture system. See, for example, Vicon Motion Systems Ltd. "3D motion analysis system Vicon" (sold by NAC Image Technology Co., Ltd.) pastes spherical reflective markers on multiple parts of the body and processes each marker by image processing from input images from multiple cameras. This is a method of calculating three-dimensional coordinates.
[0005]
Further, as a third conventional example, a device that attaches a sensor capable of measuring a motion to a body and detects a motion from the sensor information, for example, a device using a magnetic sensor (such as “Motion Star” manufactured by ASCENSION) or a machine There are devices using a type sensor ("Gypsy" by Spice, "Cyber Glove" by Immersion), and the like.
[0006]
In addition, as a fourth conventional example, there is a device that detects movement only from a camera input image without wearing anything on the body (Non-Patent Document 1). This method is a method of recognizing a gesture of a person without attaching a marker or a sensor by image processing, and extracts a feature related to a motion of the person from an image to perform a motion recognition.
[0007]
As a fifth conventional example, there is a product related to the Gesture Xstream Software of Vivid Group (for example, Soccer GX, Volleyball, etc.). This product superimposes the self-image on the CG background image, determines the contact between the CG ball etc. and the user's body on the image, and realizes the interface between the CG software such as games and the user based on the result. are doing.
[0008]
As a sixth conventional example, there is a method described in Non-Patent Document 2. In this method, a CG icon is displayed on a self-portrait, and a desired icon is selected (executed) based on an operation performed toward the camera.
[0009]
[Non-patent document 1]
Kirishima et al., “Real-time Image Recognition of Gestures by Learning Gaze Points and Selecting and Controlling,” PRMU 2001-203, 2002.1.
[Non-patent document 2]
Saki et al., “A trial of action interface using personal computer video input”, Proceedings of the 12th Human Interface Symposium, pp. 21-218, 1996.
[0010]
[Problems to be solved by the invention]
However, the conventional method described above has the following problems.
[0011]
In the first conventional example, it is necessary to contact the device when detecting the movement of the hand of the operator (user), and to require a fixed and limited place for using the device. That is, it is a contact type, and it is difficult to detect an operation at an arbitrary place.
[0012]
The second and third conventional examples are non-contact type and can detect an operation at an arbitrary position within a range shown in a camera image. However, since it is necessary to always attach some marker or sensor to a body part, Lacks the convenience as an interface device in normal life such as work and work.
[0013]
In the fourth conventional example, the movement can be detected without wearing anything on the body, but it is not convenient for the user because it is impossible to visually check whether the movement of the user is recognized.
[0014]
In the fifth conventional example, the position of the user can be detected by touching the CG displayed on the same display while watching the self-portrait. However, since contact determination is performed on the entire body of the user, portions other than the fingers (head, Even if the user touches an icon (CG to be touched) without any distinction, the recognition accuracy is low because the possibility of erroneous recognition is high. In addition, since an icon (CG to be touched) cannot be placed behind the user, an interface icon cannot be placed at an arbitrary position, which limits the icon position and the degree of freedom of user operation.
[0015]
In the fifth and sixth conventional examples, a user's movement can be detected by touching (overlapping) a finger or the like of a CG displayed on the same display while watching a self-portrait. There is a limitation in the application range because it cannot be instructed.
[0016]
SUMMARY OF THE INVENTION An object of the present invention is to provide an interface method and an apparatus that solve the problem of a contact type, the problem of difficulty in detecting motion at an arbitrary place, and the problem of lack of convenience as an interface apparatus. .
[0017]
It is another object of the present invention to provide an interface method and apparatus in which the position of icons and the degree of freedom of the user's operation are not limited and the application range is not limited.
[0018]
[Means for Solving the Problems]
In order to achieve the above object, an interface device of the present invention comprises:
Image input means for taking an image (self image) including the operator;
Means for generating an inverted image obtained by horizontally inverting the input image;
Means for detecting a position specified by the operator on the inverted image;
Means for selecting a command specified by the operator on the inverted image;
Means for displaying information on the inverted image based on the designated position and the command.
[0019]
Therefore, since the user can select a command to be specified without contacting the apparatus, the above problems (1), (2), and (3) can be solved.
[0020]
Further, in the embodiment of the present invention, the means for selecting a command includes, in an image obtained by photographing the real space displayed on the reverse image, a means for registering an arbitrary real object position in advance, and: Means for detecting a real object designated by the operator from a plurality of real objects registered in advance.
[0021]
Therefore, not only the CG displayed on the image but also the real object in the image obtained by photographing the real space can be designated, so that the problems (4) and (5) can be solved.
[0022]
BEST MODE FOR CARRYING OUT THE INVENTION
Next, embodiments of the present invention will be described with reference to the drawings.
[0023]
Referring to FIG. 1, an interface device according to a first embodiment of the present invention uses an image (self image) including an operator (user) as an input image and allows the user to view the self image while watching the self image. The command selection is performed on the image based on the image input unit 11, the inverted image generation unit 12, the designated position detection unit 13, the information display unit 14, and the command selection unit 15.
[0024]
As the image input unit 11, one camera is used as shown in FIG. The camera may be a commonly used video camera, which may be black and white or color. However, when a method using color information described below is used, a color camera is required.
[0025]
The inverted image generation unit 12 receives the image (input image) input by the image input unit 11, and generates an image (inverted image) that is inverted left and right (Step 21). An inverted image can be generated in real time by general-purpose image processing software (eg, HALCON) for an input image taken into the computer. Alternatively, it can be realized by a device that inputs an input image and generates a reverse image in real time (eg, a screen left / right reversing device UPI-100LRF of FOR-A Co., Ltd., or a Sony EVY-D100 with a camera integrated type).
[0026]
The designated position detecting unit 13 detects a position (user designated position) at which the user points at a body part such as a finger on the inverted image obtained by the inverted image generating unit 12 (step 22).
[0027]
As a first example of the designated position detection unit 13, there is a method in which a user has a marker and detects the marker in the inverted image by image processing. For example, first, an object of red (although other colors may be used, but a small number of colors are preferable in the background image) is held or attached to the hand, and the object is shaken and moved in front of the camera 11. A red component is extracted from the values of the color information such as RGB of the obtained color image (step 21), and a labeling process (a general image processing method of grouping each block) is performed (step 22). One red portion having the largest area is selected (step 23), the position of the center of gravity (x, y) at that position is obtained, and the obtained (x, y) coordinates are used as the user-specified position (step 24). Here, instead of the red object, the user holds or wears the red light emitting diode (LED) in the state where the light emitting diode (LED) may emit light in the same manner as the above-mentioned object, and extracts the color in the same manner. A method of doing this is also possible. (X, y) is not the center of gravity position (the average value of the sum of the x-coordinates of all the pixel values of the extracted part and the average of the sum of the y-coordinates), but the center position (the maximum of the x-coordinate of the extracted part) (The center value between the value and the minimum value, and the center value between the maximum value and the minimum value of the y coordinate).
[0028]
As a second example of the pointing position detection unit 13, there is a method in which the user does not have a marker and directly detects the position of the hand in the inverted image by image processing. For example, first, a skin color component is extracted from the values of the RGB color matching information of the obtained color image (step 31), and a labeling process is performed (step 32). A target hand is selected from the obtained plurality of skin color portions using constraint information such as the size and position of the hand. As a specific method of selection, if the user usually wears clothes, it is considered that both hands and the face are likely to be candidates for the skin color portion, and that the face having the largest area is the face. Therefore, the second and third large skin color portions are selected as hand candidates (step 33). If the user-specified positions are both hands, the center of gravity positions for the two candidates for the skin color portion may be the user-specified positions of the left and right hands. In addition, if the user specifies one position, it is necessary to select one from two candidates. For example, if the right hand is specified in advance, a candidate on the right hand side of the body (because the right hand is likely) is selected. The right-hand skin color portion is selected, and the position of the center of gravity may be set as the user-specified position of the right hand (step 34). The same is true for the left hand.
[0029]
In these examples of the pointing position detection unit 13, the accuracy can be improved by adding a process using a difference image before performing the above-described detection process. The process using the difference image is, for example, a process of comparing a background image (an image captured without a user) and a current image currently captured (an image with a user) on a reverse image, for example. Create a difference image between them. The difference image can be created, for example, by obtaining the absolute value of the difference between the pixel value of the background image and the pixel value of the current image (at the same pixel position) for each pixel. By using this difference image, only a moving object such as a body is extracted in advance, and then the image processing as described above (for example, a method of extracting a marker attached to the body, a method of extracting a skin color, or the like) is performed. If a user-specified position is detected and its coordinates are acquired, it can be easily estimated that false detection of a color similar to the color of a marker or a skin color included in the background image can be reduced.
[0030]
With such an indicated position detection unit 13, only a marker or a hand can be used as a contact determination site without selecting the entire body for command selection, so that an icon or the like for command selection can be set at an arbitrary position. A high degree of freedom is obtained for its position and user movement.
[0031]
The command selection unit 14 collates the user-specified position obtained by the specified-position detection unit 13 with the display positions of a plurality of pieces of command information displayed on the information display unit 15, and the user attempts to specify based on the positional relationship between the two. Search and select command information (step 23). For example, if there is a command displayed within a certain distance range from the user-specified position, it may be selected. Since the designated position detecting unit 13 performs the contact determination at the body part desired by the user without performing the contact determination on the whole body, the display position of the command information displayed on the information display unit 15 is the contact position at the user's body part. Since it is permissible to hide behind a part for which no determination is made, both the display position of the command information and the position of the user himself can be arranged at arbitrary positions on the screen.
[0032]
The information display unit 15 superimposes the inverted image obtained by the inverted image generation unit 12 and the command information selected by the command selection unit 14 and displays the same on the display (step 24). The display may be a general-purpose display used for a computer, as long as it can display a computer screen and a camera image. As the command information, commands that can be selected by a CG created by a computer may be displayed.
[0033]
The user can select a command simply by superimposing his / her finger on the command information displayed in the CG on the same screen while watching the screen on which the self-image is reflected, so that it is possible to give an easy-to-understand instruction with few errors. .
[0034]
Referring to FIG. 4, the interface device according to the second embodiment of the present invention uses an image (self image) including an operator (user) as an input image, and allows the user to view the self image while watching the self image. A real object in a real space included in an image is selected based on the image input unit 11, the inverted image generation unit 12, the designated position detection unit 13, the real object detection unit 16, the real object registration unit 19, and information display. It comprises a unit 15.
[0035]
In the present embodiment, the command selection unit 14 shown in the first embodiment is provided with a mechanism capable of detecting a real object. Specifically, the command selection unit 14 of the first embodiment includes a real object detection unit 16 and a real object registration unit 19. The real object registration unit 19 includes a real object registration processing unit 17 and real object position data 18. The other image input unit 11, inverted image generation unit 12, indicated position detection unit 13, and information display unit 15 are the same as in the first embodiment.
[0036]
The real object registration processing unit 17 is an arbitrary real object (for example, in a house, a television, an air conditioner, a computer, a clock, a window, a shelf, a chair, a desk, a drawer, a document, an audio device) , Lighting equipment, etc.), display positions on the inverted image are obtained in advance, and registered and stored in the real object position data 18 (step 25). Alternatively, instead of obtaining a fixed position in advance, by attaching a sensor capable of recognizing the position of each target real object (a commercially available magnetic sensor or ultrasonic sensor, etc.), Since the position can be recognized in real time, the real object position data can be calculated from the position information obtained thereby, and the real object position data can be constantly updated. In this case, even if the object is moved, its position can be moved.
[0037]
The real object position data 18 may be one piece of center coordinates (x, y) of the object, but information indicating a range (for example, starting point coordinates (x1, y1) and ending point coordinates (x2, y2) of the rectangular frame, center The coordinates (x, y) and the radius value r, or a coordinate list of all the dots where the object is present may be stored. The center coordinate may be the barycentric coordinate similarly to the processing of the designated position detecting unit 13 of the first embodiment.
[0038]
The real object detection unit 19 collates the plurality of position data stored in the real object position data 18 with the user-specified position obtained by the specified position detection unit 13, and checks the command selection unit 14 according to the first embodiment. Similarly to the above, a real object to be pointed by the user is detected and selected based on the positional relationship between them (step 26). For example, if there is position data of a real object within a certain distance range from the user-specified position, it may be selected. Alternatively, when a range is provided as the real object position data, a real object whose coordinates specified by the user is included in the range is detected and selected.
[0039]
FIG. 6 shows an example of the interface operation according to the present embodiment.
[0040]
The display position of the real object reflected on the self image viewed by the user is obtained in advance and registered in the real object position data 18 (when the position recognition sensor or the like is used as described above, the display position is automatically determined). The actual object position can be detected and registered and updated in the object position data 18). When the user does not perform the instructing operation, there is no reaction, but when the user performs an operation of superimposing a hand (or a marker) on the real object that the user wants to access, and determines that the real object detecting unit 16 has detected the real object. , The position of the detected real object is specified. As shown in the figure, for example, as shown in the figure, an icon of the same shape or substantially the same size as the real object (a color changed, a translucent or transparent, a flashed, or the like as necessary) is displayed. A method of superimposing on a real object and displaying a marker by CG (color-changed, translucent or transparent, blinking, etc.) around the center of the real object or around the real object as necessary The display method can be easily considered. That is, the user actually touches a virtual icon on the position of the real object or a displayed CG icon (transparent, translucent, or another color). Further, related information (name, state, etc.) may be displayed around the detected real object. Further, as shown in the figure, a menu list to be subsequently selected is displayed by CG around the position of the detected real object, and the user selects a similar selection method (that is, a menu for selecting a hand or the like) from the menu list. It is also possible to select and execute a menu by, for example, a method of superimposing and selecting an icon.
[0041]
Thus, the user can not only select a command by specifying the CG icon displayed on the image, but also directly select a real object in the real space image. When combined with CG command selection as in the first embodiment, for example, when pointing to and selecting a television or the like, a CG menu for selecting a command for controlling a switch, channel, or the like of the television is displayed on the real object (television). Displayed in the vicinity, the user can continuously select a command and control a television or the like.
[0042]
In FIG. 6, when the user does not perform an instruction operation, nothing is reacted. For example, as shown in FIG. 7, when information to be presented to the user on the real object side is acquired, The information is displayed on or around the real object, or icons (colored, translucent, blinking, etc.) are clearly displayed on the real object by superimposition. It is also conceivable to notify the user who is in charge. Thereafter, as described above, when the user selects the real object, necessary information and a menu list are displayed.
[0043]
In addition, the present invention records a program for realizing the function other than that realized by dedicated hardware on a computer-readable recording medium, and stores the program recorded on the recording medium in a computer system. It may be read and executed. The computer-readable recording medium refers to a recording medium such as a floppy disk, a magneto-optical disk, a CD-ROM, or a storage device such as a hard disk device built in a computer system. Further, the computer-readable recording medium is one that dynamically holds the program for a short time (transmission medium or transmission wave), such as a case where the program is transmitted via the Internet, and serves as a server in that case. It also includes those that hold programs for a certain period of time, such as volatile memory inside a computer system.
[0044]
【The invention's effect】
According to the first, third and fifth aspects of the present invention, the user can select a command to be issued by the user without wearing anything or directly touching anything. Motion detection can be performed. In addition, since the user can select a command displayed on the self-image while directly looking at an arbitrary part (finger or the like) of his or her own body, an interface device intuitive for the user can be realized, which is convenient. The performance is improved.
[0045]
According to the invention of claims 2, 4, and 5, since not only the CG displayed on the image but also the real object in the image obtained by photographing the real space can be specified, not only command selection by CG but also real object It is possible to directly select the body, and the range of application can be expanded.
[Brief description of the drawings]
FIG. 1 is a block diagram of an interface device according to a first embodiment of the present invention.
FIG. 2 is a flowchart illustrating a flow of an entire process according to the first embodiment.
FIG. 3 is a flowchart showing a flow of processing of a designated position detecting unit 13;
FIG. 4 is a block diagram of an interface device according to a second embodiment of the present invention.
FIG. 5 is a flowchart illustrating a flow of a process according to a second embodiment.
FIG. 6 is a diagram illustrating an example of an interface operation according to the second embodiment.
FIG. 7 is a diagram illustrating an example of information display of a real object according to the second embodiment.
[Explanation of symbols]
Reference Signs List 11 Image input unit 12 Inverted image generation unit 13 Pointed position detection unit 14 Command selection unit 15 Information display unit 16 Real object detection unit 17 Real object registration processing unit 18 Real object position data 19 Real object registration units 21 to 26, 31 to 34 Steps

Claims (5)

An interface method for inputting an image from a camera and recognizing human motion,
An image input step of capturing an image including the operator;
Generating an inverted image by inverting the input image horizontally,
Detecting a position specified by the operator on the inverted image;
Selecting a command specified by the operator on the inverted image;
Displaying information on the inverted image based on the designated position and the command. The step of selecting the command includes:
A step of arbitrarily registering a real object position in advance in an image obtained by photographing the real space displayed on the inverted image,
Detecting the real object specified by the operator from the plurality of pre-registered real objects on the inverted image. An interface device that receives an image from a camera and recognizes human motion,
Image input means for capturing an image including the operator;
Means for generating an inverted image obtained by horizontally inverting the input image;
Means for detecting a position specified by the operator on the inverted image;
Means for selecting a command specified by the operator on the inverted image;
Means for displaying information on the inverted image based on the designated position and the command. The means for selecting the command includes: means for registering an arbitrary real object position in advance in an image obtained by photographing the real space displayed on the inverted image; 4. An interface device according to claim 3, further comprising means for detecting a body from a plurality of real objects registered in advance. An interface program for causing a computer to execute the interface method according to claim 1.

Images