JP3777650B2 - Interface equipment - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an interface device that performs input / output of information devices such as computers and word processors, and devices having a display such as a television.
[0002]
[Prior art]
As a conventional interface device of this kind, the coordinate position detected by the mouse on the display screen can be used to add other information to the information in the display device, to change the display information, or to make a selection. Some things are done by displaying a cursor.
[0003]
FIG. 30 shows an outline of this type of conventional interface device. In FIG. 30, 501 is a host computer, 502 is a display, and virtual operation buttons 503, 504, and 505 are displayed on the display 502 by the host computer 501. Reference numeral 506 denotes a mouse cursor, which is controlled by the host computer 501 so as to move in the screen in synchronization with the movement of the mouse 507 based on the movement amount of the mouse 507 detected by the mouse 507. The user moves the mouse 507 to move the mouse cursor 506 to the position of an arbitrary virtual operation button on the display screen, and presses the switch 508 on the mouse 507 to select the operation button. The operation instruction can be given to.
[0004]
[Problems to be solved by the invention]
However, the above-described conventional configuration requires an input device called a mouse separately from the main body of the device, and requires a base such as a table for operating the mouse, and is not suitable for use in a portable information device or the like. In addition, since the operation is performed through the mouse, the interface is not always direct and easy to understand.
[0005]
In view of this point, the present invention is intended to provide an interface device that does not require an input device such as a keyboard and a mouse, can be easily operated, and has better operability and is easy to understand. Is.
[0006]
[Means for Solving the Problems]
    The present invention provides an operator's hand shape,The hand shape continues for a predetermined timeRecognizing means for recognizing hand movements, and hand shape features recognized by the recognizing meansSelectable byDisplay menu in imageThen, the hand shape feature recognized by the recognition means is displayed on the image as an icon representing a number.A display means to display and a menu displayed on the image based on a hand shape feature recognized by the recognition means are selected, and control is performed based on a predetermined hand movement recognized by the recognition means after selection of the menu. Interface having control means for giving instructionsIDevice.
[0007]
  Preferably, saidThe recognition means recognizes the number of fingers of the hand, the display means displays the number of fingers on the image, and the control means selects a menu based on the number of fingers of the hand recognized by the recognition means.
[0008]
  Preferably, the display meansDisplays an icon representing a number and highlights a part of the menu corresponding to the number based on the number of fingers of the hand recognized by the recognition means.indicate.
[0009]
    The shape of the operator's handWhenRecognizing means for recognizing a predetermined hand movement, and hand shape features recognized by the recognizing meansSelectable byDisplay the menu on the screenRuDisplay means;An icon generation unit that generates an icon corresponding to the recognition result of the hand shape by the recognition unit; and the icon generated by the icon generation unit is displayed on the screen,Based on the predetermined hand movement recognized by the recognition means,Move the icon on the image and select a menuInterface with control meansIDevice.
[0012]
DETAILED DESCRIPTION OF THE INVENTION
(First embodiment)
In the first embodiment of the present invention, a recognition unit such as an imaging device for recognizing the shape of an operator's hand, and features of the shape of the hand recognized by the recognition unit are displayed on a screen as a special shape using an icon or the like. Display means for displaying, and control means for controlling the information displayed in the screen by changing the shape of the hand by operating a special shape such as an icon displayed on the screen by the display means as a so-called cursor. It is an interface device.
[0013]
(Second Embodiment)
The second embodiment of the present invention displays at least an imaging unit, a motion recognition unit that recognizes the shape and / or motion of an object in the captured image, and the shape and / or motion of the object recognized by the motion recognition unit. A frame memory configured to store an image captured by the imaging unit, and a reference image memory for storing an image captured at a time before the image stored in the frame memory as a reference image. The motion recognition unit is an interface device provided with an image change extraction unit that extracts a difference between an image in the frame memory and a reference image stored in the reference image memory.
[0014]
(Third embodiment)
The third embodiment of the present invention includes at least an imaging unit, a motion recognition unit that recognizes the shape and / or movement of the user's hand in the captured image, and the user's hand recognized by the motion recognition unit. A contour extracting means for extracting a contour of a user who has captured the shape and / or movement, tracking the extracted contour, and the relationship between the angle of the contour and the length of the contour, that is, the contour An interface apparatus comprising the contour recognition unit configured to calculate a waveform, and a shape filter that filters the contour waveform calculated by the contour waveform calculator and generates a shape waveform representing a predetermined shape. It is.
[0015]
When the user faces the imaging unit of the interface device configured as described above and gives an instruction by, for example, a hand gesture, the imaging unit captures an image of the user. The contour extracting means extracts the contour of the user's image, and the contour is calculated by the contour waveform calculation unit with the angle of the contour line with respect to the horizontal line, with the length of the contour line starting from the reference point on the contour as the horizontal axis, That is, it is converted as a contour waveform. This contour waveform is converted into a shape waveform representing the uneven shape of the finger by a bandpass filter of a predetermined band, for example, a shape filter configured by a bandpass filter corresponding to the unevenness of the finger, and simultaneously calculates the position where the hand is located. By simply counting the number of pulses present in this shape waveform, the number of fingers that have been put out, that is, the shape of the hand can be determined accurately. Based on the position and shape of the hand, the given hand gesture is displayed on the display screen of the display unit.For example, the virtual switch displayed on the display screen can be selected by hand gesture, and an input device such as a mouse is required. Therefore, it is possible to operate a very simple device.
[0016]
In addition, a plurality of shape filters may be configured by a plurality of band pass filters having different bands, and the user's operation may be determined based on the shape waveforms generated by the plurality of shape filters. Thereby, a plurality of shapes can be recognized.
[0017]
In addition, a plurality of shape filters may be configured by a bandpass filter having a contour waveform shape corresponding to at least unevenness of the hand and a bandpass filter having a contour waveform shape corresponding to the unevenness of the finger. As a result, the waveform is converted into a smooth shape waveform that reflects only the unevenness of the hand portion, and is converted into a shape waveform that reflects only the unevenness of the finger.
[0018]
Further, a coordinate table for storing the correspondence between the coordinates of the contour shape of the imaged user and the contour waveform calculated by the contour waveform calculation unit is provided, and the captured image is obtained using the wave height existing position of the shape waveform and the coordinate table. The motion recognition unit may be configured by providing a coordinate calculation unit that calculates coordinates in which a predetermined shape exists. Thereby, the coordinates of the contour shape are calculated, and the coordinates are output.
[0019]
In addition, a shape determination unit that counts the number of pulses in the shape waveform generated by the shape filter may be provided to configure the motion recognition unit, and the shape of the object may be determined based on the output value of the shape determination unit. This makes it easy to determine whether the shape is a two-finger shape or a hand-held shape based on the number of pulses.
[0020]
Further, the action recognition unit may be configured by providing a differentiator that differentiates the shape waveform generated by the shape filter. By differentiation, the waveform becomes more pulsed and the number of pulses can be easily counted.
[0021]
(Fourth embodiment)
In the previous embodiments, an example of performing an operation on a secondary screen displayed on a display screen has been described. However, the present embodiment relates to an operation when virtual three-dimensional images are displayed on a two-dimensional display screen. Is. In general, assuming an operation in which a virtual object in a virtual space is grasped using a cursor in a displayed virtual three-dimensional space, the configuration is as follows.
[0022]
In FIG. 21, A1 is an input device, A2 is a cursor coordinate storage unit, A3 is an object coordinate storage unit, A4 is a display device, and A5 is a contact determination unit. FIG. 22 shows a two-fingered manipulator-shaped cursor that can be displayed from the shape of the operator's hand, as in the embodiment described above. 22 (a), (c) and (e) show a state where the finger is opened, and FIGS. 22 (b), (d) and (f) show a state where the finger is closed. FIG. 23 shows an example of a virtual object in the virtual space. Here, it is assumed that the operator uses a two-finger cursor to grab a virtual object in the virtual three-dimensional space. 24A and 24B show the arrangement of the cursor and the virtual object in the virtual space when the virtual object is grabbed using the cursor. FIG. 25 shows the display of the display device A4.
[0023]
When the operator's operation is given to the input unit A1, the cursor coordinates stored in the cursor coordinate storage unit A2 and the interval between the two fingers of the cursor are updated based on the operation. The display device A4 draws a virtual space including the cursor and the virtual object using the information stored in the cursor coordinate storage unit A2 and the position information of the virtual object stored in the object coordinate storage unit A3. Here, the contact determination unit A5 uses the information stored in the cursor coordinate storage unit A2 and the position information of the virtual object stored in the object coordinate storage unit A3 to calculate whether the cursor and the virtual object are in contact with each other in the virtual space. To do. Specifically, the distance in the virtual space between a plurality of surfaces constituting the cursor and the virtual object is calculated for each surface, and if the virtual object touches between the two fingers of the cursor, the cursor is It is determined that the virtual object is grabbed, and thereafter the coordinates of the object are changed according to the movement of the cursor.
[0024]
However, in such a method, in the case of the arrangement as shown in FIGS. 24A and 24B, the display by the display device is as shown in FIG. 25, and the positions of the cursor and the virtual object do not completely match in the virtual space. Even in this case, the operator may erroneously determine that the coordinates match. Further, even when a three-dimensional display device is used or a display such as those shown in FIGS. 24A and 24B, a smooth operation becomes difficult due to the difference in distance sense between the real space and the virtual space.
[0025]
As described above, the cursor and the virtual space are different depending on the difference between the sense of distance in the virtual space that is the display space and the sense of distance in the real space, and the difference between the cursor movement intended by the operator and the actual cursor movement. In an interaction with a virtual object (for example, when a virtual object is grabbed by a virtual manipulator), an interaction according to the intention of the operator (grabbing the virtual object in the above example) cannot be realized smoothly.
[0026]
In the present embodiment, the operator can perform cursor control with good operability in a virtual space with non-contact by hand gestures, etc., and whether or not an interaction with a virtual object in the virtual space has occurred Rather than being determined only by the distance between the components of the object (or the surface if the virtual space is three-dimensional), the interaction determining means does not necessarily interact with objects that are not close to each other in the virtual space. By causing the action, the determination as to whether the cursor interacts with the virtual object can be made closer to the intention of the operator, and the operability of the interface can be further improved. Furthermore, it is possible to cause an interaction even with an object that is not necessarily close in the virtual space.
[0027]
The first configuration of the present embodiment stores display means, input means for changing the position and shape of the cursor displayed on the display means, representative point coordinates representing the cursor position, and the shape of the cursor. Cursor storage means, object point storage means for storing representative point coordinates representing the position of a display object other than the cursor and the shape of the display object, cursor position and shape stored in the cursor storage means, and the object storage And means for determining an interaction between the cursor and the display object using the position and shape of the display object stored in the means, the interaction determination means comprising at least one point of the Distance calculating means for calculating a distance between a representative point of the cursor and at least one representative point of the display object; and a movement of the cursor or a change in shape is recognized. And an overall determination means for determining an interaction between the cursor and the display object using a distance calculated by the distance calculation means and a recognition result of the motion recognition means. Device.
[0028]
With this configuration, whether or not an interaction between the cursor operated by the operator in the virtual space and the virtual object in the virtual space has occurred is determined based on the components of the cursor and the virtual object in the virtual space (the virtual space is a three-dimensional In this case, the overall determination means determines whether or not an interaction has occurred, based on the distance between the representative points calculated by the distance calculation means and the movement of the cursor recognized by the action recognition means. Thus, it is possible to cause an interaction even with an object that is not necessarily close in the virtual space.
[0029]
In the first configuration, when the motion recognition unit recognizes a motion registered in advance, the interaction determination unit generates an interaction with a display object whose distance calculated by the distance calculation unit is equal to or less than a predetermined reference. A second configuration may be adopted.
[0030]
Further, in the first and second configurations, a movement vector calculation unit that calculates the movement direction and movement amount of the cursor in the display space is provided to constitute an interaction determination unit, and the cursor direction calculated by the movement direction calculation unit is configured. A third configuration may be adopted in which the interaction determining means determines the interaction between the cursor and the display object based on the moving direction and the moving amount of the cursor.
[0031]
Further, in the third configuration, a fourth configuration may be employed in which the interaction determination unit generates an interaction when the amount of movement of the cursor calculated by the movement vector calculation unit is equal to or less than a predetermined reference value.
[0032]
Further, in the third and fourth configurations, as a fifth configuration in which the interaction determining unit generates an interaction with respect to a display object existing in the vicinity of the extension line in the movement direction of the cursor calculated by the movement vector calculating unit. Good.
[0033]
Further, in the first to fifth configurations, a sixth configuration may be employed in which the interaction determination unit generates an interaction when the cursor shape and the display object shape are registered in advance.
[0034]
Further, in the first to sixth configurations, a shape determining means for recognizing the cursor shape and the shape of the display object is provided to constitute an interaction determining means, and the shape of the cursor recognized by the shape recognition means and the shape of the display object It is good also as a 7th structure in which an interaction determination means produces | generates an interaction about the case where and match.
[0035]
Further, in the first to seventh configurations, a line-of-sight input means for detecting a line-of-sight direction is provided, and the movement recognition means recognizes an action registered in advance for a display object in the vicinity of the extension of the line of sight detected by the line-of-sight input means In this case, the interaction determination means may be an eighth configuration for generating the interaction.
[0036]
Further, in the eighth configuration, when a cursor is present near the extension line of the line of sight detected by the line-of-sight extension detected by the line-of-sight input unit, and the motion recognition unit recognizes a motion registered in advance In addition, the interaction determination unit may have a ninth configuration that generates an interaction.
[0037]
In the first to ninth configurations, when an interaction is generated, a learning means is provided for learning the positional relationship between the cursor and the target display object, the shape of the cursor, and the shape of the display object. A tenth configuration may be adopted in which the action determination means determines the interaction based on the learning result of the learning means.
[0038]
Further, in the tenth configuration, when the positional relationship between the cursor and the target display object or the shape of the cursor and the shape of the display object are similar to the positional relationship or the shape learned in the past by the learning means, the interaction is performed. The eleventh configuration in which the determination means generates an interaction may be adopted.
[0039]
Further, in the first to eleventh configurations, as a twelfth configuration in which an interaction determining unit is configured by providing a coordinate converting unit that performs coordinate conversion on the input from the cursor storage unit and the object storage unit to the distance calculating unit. Good.
[0040]
Further, in the twelfth configuration, when an interaction is generated, a thirteenth configuration may be employed in which coordinate conversion is performed so that the positional relationship between the cursor and the target display object approaches.
[0041]
(First embodiment)
FIG. 1 shows the appearance of a first embodiment of an interface device according to the present invention. In FIG. 1, 1 is a host computer, 2 is a display used for display, and 3 is a CCD camera for capturing an image. The CCD camera 3 has an imaging surface arranged in the same direction as the display direction of the display 2, and the user's hand shape can be imaged when the user faces the display surface. On the display 2, menus 201, 202 and an icon 200 for displaying the hand shape can be displayed.
[0042]
FIG. 2 shows a detailed block diagram of this embodiment. An image input from the CCD camera 3 is stored in the frame memory 21. The reference image memory 25 stores a background image that does not include a person imaged in advance as a reference image. The reference image can be updated as needed.
[0043]
The shape identifying means 22 removes the background image from the image by extracting the difference between the image stored in the frame memory 21 and the image stored in the reference image memory 25, and corresponds to the user's hand, for example. The part is extracted, and the shape is, for example, the shape of a single finger as shown in FIG. 3 (a), the shape of a two finger as shown in FIG. 3 (b), or otherwise It is determined whether this is the case of FIG.
[0044]
FIG. 4 shows a detailed embodiment of the shape discriminating means 22 and shows an example constituted by the image difference calculating unit 221, the contour extracting unit 222, and the shape discriminating unit 223. As described above, the image difference calculation unit 221 calculates the difference between the image stored in the frame memory and the image stored in the reference image memory. Thereby, an object to be extracted, for example, a user and a background portion can be separated. For example, when the image difference calculation unit 221 is configured by a simple subtraction circuit, only the hand portion of the user in the image in the frame memory can be extracted as shown in FIG. The contour extracting unit 222 extracts the contour shape of the object existing in the image obtained as a result of the calculation by the image difference calculating unit 221. As a specific method example, a contour shape can be easily extracted by extracting an edge of an image.
[0045]
The shape identifying unit 223 identifies in detail the contour shape of the hand extracted by the contour extracting unit 222. For example, whether the shape is a one-finger shape as shown in FIG. It is determined whether the shape is two fingers as shown in (b). As a shape identification method, for example, template matching, a matching method with a shape model, a neural network, or the like can be used.
[0046]
The icon generation unit 24 generates an icon image as a special shape to be displayed on the display based on the result of the hand shape identification by the shape identification unit 223. For example, if the identification result of the hand shape is a one-finger shape, for example, the number “1” icon as shown in FIG. 6A is generated. An icon of the number “2” as shown in 6 (b) is generated. As the icon shape, if the identification result of the hand shape is a one-finger shape, a one-finger type icon as shown in FIG. A two-finger icon as shown in (d) may be generated. The display control unit 23 controls the display based on the hand shape identification result by the shape identification unit 223. For example, the icon based on the identification result is displayed, and a menu previously associated with the identification result is highlighted based on the hand shape identification result.
[0047]
An operation example according to this embodiment configured as described above will be described below. As shown in FIG. 7 (a), when the user faces a device equipped with the interface device of the present embodiment and the hand is in the shape of one finger, the number “1” icon is displayed on the display, The first menu, the television display, is highlighted. At this time, it is possible to further alert the operator by outputting sound or voice from the display device in accordance with the highlighting. Here, when the hand is formed into a two-finger shape as shown in FIG. 7B, the number “2” icon is displayed on the display and the display of the network as the second menu is highlighted. . In this state, for example, by maintaining the same hand shape for a certain period of time, the second menu is selected, and a command can be given to the host computer to display the network terminal. Audio may be used in combination for menu selection. In cases other than a predetermined hand shape as shown in FIG. 7C, icons and menus are not displayed on the display, and no command is sent to the host computer.
[0048]
As described above, according to the present embodiment, it is possible to identify the shape of the hand in the captured image, and to control devices such as a computer according to the identification result, without using devices such as a keyboard and a mouse. It is possible to operate remotely without contact. Further, by reflecting the result of identifying the shape of the hand on the screen, the user can perform an operation while confirming the identification result, and an easy-to-use and reliable operation is possible.
[0049]
In the present embodiment, an example of simple application to menu selection is shown. However, according to the shape of the hand, the icon display can be displayed in advance by setting the icon display to replace the picture or text. Needless to say, writing control is also easy.
[0050]
(Second embodiment)
FIG. 8 shows the external appearance of a second embodiment of the interface device according to the present invention. In FIG. 8, components having the same functions as those in the first embodiment are represented by the same numerical values. That is, 1 is a host computer, 2 is a display used for display, and 3 is a CCD camera for capturing an image. The CCD camera 3 has an imaging surface arranged in the same direction as the display direction of the display 2, and the user's hand gesture can be imaged when the user faces the display surface. On the display surface of the display 2, virtual switches 204, 205, 206, and an arrow cursor icon 203 for selecting the virtual switch can be displayed.
[0051]
FIG. 9 shows a block diagram of a detailed configuration of the present embodiment. An image input from the CCD camera 3 is stored in the frame memory 21. In the reference image memory 25, an image captured in advance is stored as a reference image. The reference image update unit 26 includes a timer 261 and an image update unit 262.The reference image update unit 26 transfers the latest image stored in the frame memory 21 to the reference image memory 25 at a predetermined time interval indicated by the timer 261, and Update the image.
[0052]
The motion recognition unit 22 removes the background image from the image by extracting the difference between the image stored in the frame memory and the image stored in the reference image memory, and for example, removes the portion corresponding to the user's hand. It is extracted, and it is determined whether the shape is, for example, the shape of a single finger as shown in FIG. 10A or the shape of a fist as shown in FIG.
[0053]
FIG. 11 shows the details of the motion recognition unit 22 and shows an example constituted by the image difference calculation unit 221, the contour extraction unit 222, the shape change identification unit 225, and the position detection unit 224.
[0054]
The image difference calculation unit 221 calculates the difference between the image stored in the frame memory 21 and the image stored in the reference image memory 25 as described above. As a result, an object to be extracted as a motion, for example, a user's hand portion and a background portion can be separated, and at the same time, only a moving object image can be extracted. For example, when the image difference calculation unit 221 is configured by a simple subtraction circuit, only the hand portion in the reference image and the hand portion in the latest image in the frame memory are extracted as shown in FIG. It is possible to identify only the part of the moving hand. The contour extraction unit 222 extracts an object existing in the image obtained as a result of the calculation by the image difference calculation unit 221, that is, the contour shape of the hand part before and after the movement. As a specific method example, a contour shape can be easily extracted by extracting an edge of an image.
[0055]
The shape change identification unit 225 identifies in detail the contour shape of the hand part after movement extracted by the contour extraction unit 222. For example, the shape is the shape of a single finger as shown in FIG. And whether the shape of the fist is as shown in FIG. 10 (b). At the same time, the position detection unit 224 calculates the barycentric coordinates of the contour shape of the user's hand after the movement.
[0056]
The icon generation unit 24 generates an icon image to be displayed on the display based on the result of the hand shape identification by the shape change identification unit 225. As an example of the icon image, for example, if the identification result of the hand shape is the shape of one finger, an arrow icon as shown in FIG. 13A is generated. An icon with an X mark as shown in FIG. 13B is generated. If the identification result of the hand shape is a two-finger shape, for example, an icon having a shape imitating a two-finger shape as shown in FIG. 13C is generated. For example, an icon having a shape imitating the shape of a fist as shown in FIG. 13D may be generated.
[0057]
The display control unit 23 controls the display position of the icon generated by the icon generation unit 24 on the display 2 and includes a coordinate conversion unit 231 and a coordinate inversion unit 232. The coordinate conversion unit 231 converts the coordinates of the captured image into the display coordinates of the display 2, and the coordinate inversion unit 232 inverts the left and right positions of the converted display coordinates.
[0058]
That is, the coordinates of the center of gravity in the image corresponding to the user's hand detected by the position detection unit 224 are converted into the display coordinates of the display 2, and the left and right coordinates are inverted to display icons on the display 2. It is. As a result of this operation, if the user moves his hand to the right, the icon moves to the right on the display screen in the same manner as when this action is reflected in the mirror.
[0059]
An operation example according to this embodiment configured as described above will be described below. As shown in FIG. 8, when the user faces a device equipped with the interface device according to the present embodiment and moves the hand in the shape of a single finger, the arrow cursor displayed on the display is an arbitrary one corresponding to the movement of the hand. Move to the position. Next, by moving the hand over any of the virtual switches 204, 205, 206 displayed on the display 2, the arrow cursor is moved, and when the hand is gripped and formed into a fist shape, the virtual switch 204, 205, 206 is selected and commanded to the host computer 1. Can be given.
[0060]
In this embodiment, the configuration and the movement of the object in the captured image are recognized. However, it goes without saying that either the configuration or the movement of the object in the captured image may be recognized.
[0061]
As described above, according to the present embodiment, the motion recognition unit that recognizes the shape and / or motion of an object in the captured image, the display unit that displays the shape and / or motion of the object recognized by the motion recognition unit, and the imaging A frame memory for storing an image captured by the image capturing unit, and a reference image memory for storing an image captured at a time before the image stored in the frame memory as a reference image. By extracting the difference between the image and the reference image stored in the reference image memory, when the user turns to the imaging unit and gives an instruction by hand gesture, for example, the given hand gesture is displayed on the display screen. A virtual switch or the like displayed on the display screen can be selected by hand gesture, and an input device such as a mouse is not required, and a very simple device operation is possible.
[0062]
(Third embodiment)
The external appearance of the third embodiment of the interface device according to the present invention is the same as that of the second embodiment shown in FIG. 8. The same parts as those of the second embodiment will be described with reference to FIGS. Only the different parts are shown in FIG.
[0063]
FIG. 14 shows a detailed block diagram of the third embodiment. An image input from the CCD camera 3 is stored in the frame memory 31. The motion recognition unit 32 extracts, for example, a part corresponding to the user's hand from the image stored in the frame memory 31, and the shape thereof is, for example, the shape of a single finger as shown in FIG. It is determined whether the shape of the fist is as shown in FIG.
[0064]
FIG. 15 shows the details of the motion recognition unit 32, and the detailed operations will be described with reference to FIGS. First, the contour extraction unit 321 extracts the contour shape of an object present in the image. As a specific method example, a contour shape can be easily extracted by binarizing an image and extracting its edges. FIG. 17A is an example of the extracted contour line, and the hand portion indicates the shape of one finger.
[0065]
Next, the contour waveform calculation unit 322 uses the contour shape of the object extracted by the contour extraction unit 321 as shown in FIG. 17A as the starting point s in the figure, and the direction of the arrow in the figure (counterclockwise) As shown in FIG. 19, the angle θ from the horizontal line of the contour line at each point x on the contour line is extracted as a function with respect to the distance l from the start point s, as shown in FIG. The distance l is converted into a waveform shape that is regarded as a time axis, and at the same time, the coordinates of each point on the contour line corresponding to the distance l are stored in the conversion table 324 as a table. The shape filter 1 and the shape filter 2 shown in 3231 and 3232 are filters that pass bands corresponding to the unevenness of the hand portion and the unevenness of the finger portion in the contour waveform shown in FIG.
[0066]
As shown in FIG. 17C, the shape filter 1 converts FIG. 17B into a smooth shape waveform reflecting only the unevenness of the hand portion, and the shape filter 2 converts FIG. 17D. As shown in FIG. 17, the waveform is converted into a shape waveform reflecting only the unevenness of the finger, and differentiated by differentiators 3251 and 3252, respectively, and finally the shape differential waveform shown in FIGS. 17 (e) and 17 (f) is obtained. It is done. The shape determination unit 3262 determines whether the contour shape of the hand portion is a two-finger shape as shown in FIG. 10 (a) or a fist shape as shown in FIG. 10 (b). At the same time, the coordinate calculation unit 3261 calculates the barycentric coordinates of the contour shape of the user's hand. The coordinate calculation unit 3261 obtains the positions lc1 and lc2 where the large pulse waveform exists in the shape differential waveform shown in FIG. 17E, and converts them into the points c1 and c2 shown in FIG. Then, the center of gravity of the hand part is calculated from the contour line of the hand part from the point c1 to the point c2 and output as hand coordinates.
[0067]
In addition, the shape determining unit 3262 counts and outputs the number of pulse waveforms corresponding to the finger portion in the shape differential waveform of FIG. That is, in the case of FIG. 17 (f), since there are two large pulse waveforms corresponding to the finger portion, it is determined that the shape is a two-finger shape as shown in FIG. 10 (a) and output. That is why. Further, as shown in FIG. 18 (a), in the case of a hand-held shape, there is almost no unevenness of the finger part, and the output of the shape filter 2 is a shape without unevenness as shown in FIG. 18 (c). Therefore, the output of the differentiator 3262 also becomes a shape differential waveform with no pulse waveform as shown in FIG. 18 (d), the pulse waveform count becomes 0, and the gripping fist as shown in FIG. 10 (b). Therefore, it can be determined that the shape is the shape and output. As a specific configuration example of the shape determination unit 3262, a simple threshold processing method, a neural network, or the like can be used.
[0068]
The icon generation unit 34 in FIG. 14 generates an icon image to be displayed on the display based on the result of hand shape identification by the shape determination unit 3262 in FIG. For example, if the identification result of the shape of the hand is the shape of a single finger, for example, an arrow icon as shown in FIG. 16A, and if it is the shape of a fist, for example, as shown in FIG. The X icon is generated. The display control unit 33 controls the display position of the icon generated by the icon generation unit 34 on the display, and includes a coordinate conversion unit 331 and a coordinate inversion unit 332. The coordinate conversion unit 331 converts the coordinates of the captured image into the display coordinates of the display, and the coordinate inversion unit 332 inverts the left and right positions of the converted display coordinates. That is, the coordinates of the center of gravity in the image corresponding to the user's hand output by the coordinate calculation unit 3261 in FIG. 15 are converted to the display coordinates of the display, and the icons are displayed on the display by inverting the left and right coordinates. That is why. As a result of this operation, if the user moves his hand to the right, the icon moves to the right in the form of a display screen, similar to the case where this action is reflected in the mirror.
[0069]
An operation example of the present embodiment configured as described above will be described below. As shown in FIG. 8, when the user faces a device equipped with the interface device according to the present embodiment and moves his / her hand in the shape of a single finger, the arrow cursor of the icon 203 displayed on the display 2 indicates the movement of the hand. Move to any position corresponding to. Next, when the arrow cursor is moved by moving the hand over any of the virtual switches 204, 205, and 206 displayed on the display 2, and when the hand is gripped and formed into a fist shape, the virtual switch is selected and a command is sent to the host computer 1. It can be given.
[0070]
Further, as an example of the icon to be displayed, as shown in FIGS. 16C and 16D, if the hand shape itself is converted into an icon, it can correspond to the actual hand movement and is easy to understand. Specifically, images such as those shown in FIGS. 16C and 16D may be registered in advance, or the hand contour shape data extracted by the contour extracting unit may be reduced or enlarged to an arbitrary size. However, it can also be used as an icon image.
[0071]
As described above, in the present embodiment, when the user faces the imaging unit of the interface device and gives an instruction by hand gesture, for example, the imaging unit captures the user image, extracts the contour of the user image, and extracts the reference on the contour. Using the length of the contour line starting from the point as the horizontal axis, the angle of the contour line with respect to the horizontal line, that is, the contour waveform is converted. This contour waveform is converted into a shape waveform representing the uneven shape of the finger by a bandpass filter of a predetermined band, for example, a shape filter configured by a bandpass filter corresponding to the unevenness of the finger, and simultaneously calculates the position where the hand is located. By simply counting the number of pulses present in this shape waveform, the number of fingers that have been put out, that is, the shape of the hand can be determined accurately. Based on the position and shape of the hand, the given hand gesture is displayed on the display screen. For example, the virtual switch displayed on the display screen can be selected by hand gesture, and no input device such as a mouse is required. This makes it possible to easily operate the equipment.
[0072]
(Fourth embodiment)
A fourth embodiment of the present invention will be described with reference to the drawings. FIG. 26 shows a block diagram of a fourth embodiment of the interface apparatus.
[0073]
In FIG. 26, 41 is an input means, 42 is a cursor storage means, 43 is an object storage means, 44 is a display means, 45 is an interaction determination means, 45a is a distance calculation means, 45b is an action recognition means, 45c is a comprehensive determination means. 45d is a movement vector calculation means, 45e is a shape determination means, 45f is a learning means, 45g is a coordinate conversion means, and 46 is a line-of-sight input means.
[0074]
In FIG. 26, the operator operates the input unit 41, and based on the operation result, the cursor storage unit 42 changes and stores the representative point coordinates and shape representing the position of the cursor in the virtual space. The display means 44 is representative point coordinates and shape representative of the position of the cursor stored in the cursor storage means 42 in the virtual space, and representative point coordinates representative of the position of the virtual object stored in the object storage means 43 in the virtual space. The cursor and the virtual object are displayed two-dimensionally or three-dimensionally based on the shape and the shape.
[0075]
The line-of-sight input means 46 detects the position of the operator's line of sight on the display. The distance calculation unit 45 a calculates the distance between the cursor and the virtual object in the virtual space based on the coordinates of the representative points stored in the cursor storage unit 42 and the object storage unit 43. The motion recognition unit 45 b recognizes the operation motion based on the contents stored in the cursor storage unit 42 and the object storage unit 43. The movement vector calculation means 45d calculates the movement direction and movement distance of the cursor in the virtual space. The shape determining means 45e determines whether or not the shape of the cursor and the shape of the virtual object are appropriate for causing interaction. The learning means 45f stores the relationship between the position and shape of the cursor and the virtual object when the comprehensive determination means 45c causes an interaction between the cursor and the virtual object, and the current state has caused an interaction in the past. Outputs whether the state is similar.
[0076]
The comprehensive determination unit 45c includes a distance between the cursor and the virtual object output from the distance calculation unit 45a, a recognition result recognized by the motion recognition unit 45b, a movement direction and a movement distance of the cursor calculated by the movement vector calculation unit 45d, Whether the cursor and the virtual object interact based on the degree of similarity between the line-of-sight position detected by the line-of-sight input unit 46, the determination result of the shape determining unit 45e, and the past interaction output by the learning unit 45f It is determined whether or not the representative point coordinates and shape of the cursor and the virtual object are changed according to the result of the interaction. The coordinate conversion means 45g is arranged such that when the interaction determination means 45 causes an interaction, the distance calculation means 45a approaches the coordinates in the virtual space between the cursor used by the distance calculation and the target virtual object in the virtual space. Perform coordinate transformation.
[0077]
22 (a) and 22 (b) show a two-finger manipulator in the first embodiment of the cursor used in the interface device of the present invention. In FIG. 22, (a) shows a state where the finger is opened, and (b) shows a state where the finger is closed. 22 (c) and 22 (d) show a second embodiment of a cursor used in the interface apparatus of the present invention, which has a two-finger / two-joint manipulator shape. In the figure, (c) shows a state where the finger is opened, and (d) shows a state where the finger is closed. FIGS. 22 (e) and 22 (f) show a third embodiment of the cursor used in the interface device of the present invention, which has a five-fingered hand shape. In FIG. 22, (e) shows a state where the hand is opened, and (f) shows a state where the hand is closed. FIGS. 23A and 23B are examples of objects in the virtual space used in the interface device of the present invention ((a) is a cube, and (b) is a plane).
[0078]
The operation of the interface device configured as described above will be described. In this embodiment, it is assumed that the operator moves a cursor as shown in FIG. 22 in a three-dimensional virtual space, and moves by holding a virtual object as shown in FIG. 23 existing in the virtual space. The operator's operation is performed on the input means 41.
[0079]
Here, as an input device for inputting information for changing the position or shape of the cursor for input, means as shown in FIGS. 27A to 27C, command input by a camera, keyboard, voice recognition, or the like, etc. Can be used. FIG. 27A shows a mouse, which is operated by moving the mouse body, clicking a button, and the like. FIG. 27B shows a data glove, which is worn on a person's hand. The cursor is operated by reflecting the joint angle of the finger and the position of the data glove in the real space on the position and shape of the cursor. FIG. 27C shows a joystick that operates a cursor by operating a lever and using the operation button together. When a camera is used, the body or a part of the body (such as a hand) is imaged with the camera, and the shape and position of the hand are read.
[0080]
FIG. 28 shows an embodiment of shape extraction when only the hand is imaged by the camera. FIG. 28A shows an example in which a hand is imaged by a camera. FIG. 28B shows the binarized luminance of each pixel in the image of FIG. In FIG. 28 (b), the degree of opening and closing of the hand can be determined by the ratio of the length of the vertical side and the length of the horizontal side of the rectangle circumscribing the black area, and the barycentric coordinates and area of the entire black pixel can be determined. It is possible to input the position and distance. The input means 41 outputs the operation content (cursor movement amount, cursor shape change amount, etc.) to the cursor storage means 42.
[0081]
The cursor storage means 42 stores the coordinates and shape of the representative point of the cursor stored in the cursor storage means in the virtual space based on the operation content output by the input means 41. As the representative point, the barycentric coordinates (X0, Y0, Z0) of the cursor are used as the representative point.
[0082]
As the representative point, the center coordinates or vertex coordinates of each surface constituting the cursor may be used. As for the shape, in FIG. 22 (a), the distance d between two fingers, and in FIGS. 22 (b) and 22 (c), the internal angle θn of each finger joint (n is the joint number: θn becomes smaller as the joint number is reduced). The storage information). As the shape, the fingertip of each finger or the coordinates of each joint in the virtual space may be used.
[0083]
The object storage means 43 stores the coordinates and shape of the representative point of the virtual object in the virtual space shown in FIG. As a representative point
Center of gravity coordinates of the virtual object (cube: (X1, Y1, Z1), plane: (X2, Y2, Z2))
Is used as a representative point. As the representative points, the center coordinates and vertex coordinates of each surface constituting the virtual object may be used. As a shape, a parameter α representing a predetermined shape is stored as a shape (here, a cube is defined as α = 1 and a plane body is defined as α = 2). As the shape, vertex coordinates or the like may be used.
[0084]
The display unit 44 two-dimensionally displays an image viewed from a viewpoint designated in advance in the virtual space based on the cursor and the position and shape information of the virtual object stored in the cursor storage unit 42 and the object storage unit 43. FIG. 29A shows a display example of the display means. When the operator performs an operation, the display position or shape of the cursor changes, and the operator continues the operation based on the display.
[0085]
Whenever the cursor position changes, the interaction determination means 45 determines whether or not the cursor has grabbed the object (the presence or absence of interaction). Move the coordinates of the object. The distance calculation means 45a includes the center-of-gravity coordinates (X0, Y0, Z0) of the cursor stored in the cursor storage means 42 and the center-of-gravity coordinates (X1, Y1, Z1), (X2, Y2,) of the virtual object stored in the object storage means 43. Z2) is calculated.
[0086]
The action recognition means 45b means recognizes the action of “grabbing” as a previously registered action by using a change in the shape of the cursor. In the case of the cursor in FIG. 22 (a), the state where the distance d between two fingers continues to decrease is recognized as a “grabbing” operation, and in the case of the cursors in FIGS. 22 (b) and 22 (c), the angle θn of all fingers. Is recognized as a “grab” action. As a motion recognition method, time-series pattern recognition methods (table matching, DP matching, Hidden Markov model (HMM), recurrent neural network, etc.) After a specific action is learned in advance using, it may be used as a recognition method.
[0087]
The movement vector calculation means 45d calculates the movement direction and the movement amount of the cursor in the virtual space using the change in the center of gravity coordinates (X0, Y0, Z0) of the cursor. For example, the direction and magnitude of the difference vector between the centroid coordinates (X0, Y0, Z0) t at the current time t and the centroid coordinates (X0, Y0, Z0) t-1 one hour before are expressed as the movement direction and movement amount of the cursor. To do.
[0088]
The shape determining means 45e determines whether or not the cursor shape stored in the cursor storage means is an appropriate shape for grasping the virtual object having the shape stored in the object storage means (appropriate for causing the cursor shape to interact with the virtual object). Whether or not). Here, when the value of the parameter α representing the shape of the object is 1, the state in which the finger of the cursor is open is set to an appropriate state. For example, in the case of the cursor shown in FIG. 22 (a), it is determined that the value of d is larger than the intermediate value between the maximum value dmax and 0 of d in the case of the cursor shown in FIG. In the cases of b) and (c), it is assumed that the joint angles θn are all larger than the intermediate value between the respective maximum values θn max and 0.
[0089]
When the value of the parameter α representing the shape of the object is 0, a state where the distance between the fingertips of the cursor is narrow is set to an appropriate state. For example, in the case of the cursor shown in FIG. 22 (a), it is determined that the value of d is smaller than the intermediate value between the maximum value dmax of d and 0 in the case of the cursor shown in FIG. 22 (b). In the case of (c), the joint angles θn are all smaller than the intermediate values between the maximum values θn max and 0, respectively. As a method for determining the shape, parameters (d or θn) representing the cursor shape when the cursor is in contact with the virtual object in the virtual space are stored in advance, and the value of each parameter is ± 30. It can also be judged that it is appropriate for the movement to grasp if they match within the range of%.
[0090]
The line-of-sight input means 46 detects the line of sight of the operator and calculates the coordinates (gaze point coordinates) that the operator gazes on the screen displayed by the display means 44. As a means for detecting the line of sight, a method of detecting the orientation of the operator's pupil using a CCD camera, an optical sensor, or the like is used, and the position of the operator's head is measured by using the camera or the like. Calculate the gaze point.
[0091]
The learning means 45f includes a parameter (d or θn) indicating the shape of the cursor when the comprehensive determination means 45c determines that the cursor has grasped the virtual object, the parameter α representing the shape of the grasped virtual object, the position of the cursor, Stores the relative positional relationship (the vector connecting the center of gravity of the cursor and the center of gravity of the virtual object) with the position of the virtual object, and the parameter indicating the shape of the current virtual object and the shape of the surrounding virtual objects and the current When the positional relationship between the center of gravity of the cursor and the center of gravity of the surrounding virtual objects is close to the case where the object has been grabbed in the past (for example, the values of the elements of each dimension of the vectors representing the parameters and positional relationships are the past values In the case of matching within a range of ± 30%, etc., it is determined that the situation is similar to the past situation, and 1 is output, and 0 is output in other cases. It should be noted that the parameter representing the shape of the cursor when the object is grasped in the past as another learning means, the parameter α representing the shape of the grasped virtual object, and the relative positional relationship between the position of the cursor and the position of the virtual object May be learned using a neural network or the like. Moreover, it is also possible to learn including items such as the positional relationship between the gaze point coordinates on the screen detected by the line-of-sight detection means 46 and the coordinates of the cursor on the display screen as items to be learned.
[0092]
When the coordinate conversion means 45g grasps an object (when an interaction occurs), the coordinates used by the distance calculation means for the distance calculation are coordinated so that the distance between the cursor in the virtual space and the target virtual object approaches. Convert. For example, when the respective coordinates when the cursor has grasped the virtual object are (100, 150, 20) and (105, 145, 50), the coordinate conversion means uses the Z coordinate having the largest difference among the coordinate axes ( 1) Perform conversion as shown in equation (1).
[0093]
Z ′ = 0.8 × z (1)
Here, z represents the Z coordinate of the center of gravity of the cursor and virtual object input by the coordinate conversion means, and Z ′ represents the Z coordinate output by the coordinate conversion means.
[0094]
In this case, the value of the X coordinate and the value of the Y coordinate are not changed. Further, since the values stored in the cursor storage means and the object storage means are not changed, the screen displayed by the display means does not change. By performing the above-mentioned conversion, if the operator grasps the movement even when the distance in the virtual space is far, the distance between the cursor and the virtual object when performing the distance calculation is reduced thereafter. Thus, the distance calculation means can calculate a distance close to the distance sense perceived by the operator.
[0095]
When the distance between the cursor output by the distance calculation unit 45a and the virtual object is equal to or less than a predetermined reference, the comprehensive determination unit 45c “grabs” when the motion recognition unit 45b recognizes the “grab” motion registered in advance. It is determined that the interaction “has occurred”, and thereafter the value of the barycentric coordinate of the grasped virtual object stored in the object storage unit 43 is matched with the barycentric coordinate of the cursor until the “grabbing” interaction is completed. Here, the predetermined reference value may be a value larger than the distance that the cursor and the object can actually contact in the virtual space. For example, in the case of FIG. 25 (arrangement in FIG. 24), if the distance between the virtual object and the cursor is equal to or less than the reference value of the distance, the operation that the operator grasps is input to the input unit 1 and the operation that is recognized by the operation recognition unit 45b. If it is recognized, it becomes possible to move by grabbing the virtual object.
[0096]
Further, when there are a plurality of virtual objects below the distance reference, the comprehensive determination unit 45c includes a line segment (dashed line) connecting the cursor and the virtual object and a movement vector calculation unit 45d as shown in FIG. It is possible to determine the interaction in consideration of the cursor moving direction in the operation of the operator only for an object whose angle formed with the cursor moving direction (arrow) to be calculated is a reference (for example, 90 degrees) or less. (In the figure, the top one of the three objects is selected).
[0097]
As for the movement distance of the cursor, no interaction occurs when the movement distance is larger than a predetermined movement distance reference. As a result, it is possible to prevent interaction that is not intended by the operator when the cursor is simply moved.
[0098]
In addition, as shown in FIG. 29 (c), when a plurality of virtual objects satisfy the above-mentioned criteria, the comprehensive determination unit 45c "grabs" a virtual object close to the position of the gazing point detected by the line-of-sight input unit 46 (In the figure, the object on the left side close to the “+” mark, which means a gazing point, is selected). Thereby, it becomes possible to easily select a target using the operator's line of sight.
[0099]
Also, as shown in FIG. 29 (d), when there is an object close to the screen, a virtual object that matches the shape of the cursor determined by the shape determining unit 45e is set as a target to be grasped by the comprehensive determining unit 45c ( In the figure, since the distance between the cursor fingers is narrow, it is determined that the plane body is appropriate as the target of the “grab” operation, and the plane body is selected). This makes it possible to select a virtual object intended by the operator according to the shape of the cursor, and the operation is facilitated by associating a cursor shape that is easily associated when the operator grasps the virtual object.
[0100]
Further, the comprehensive determination unit 45c preferentially selects a virtual object determined to be similar to the case where the learning unit 45f has grabbed the object in the past. As a result, it is possible to reproduce the judgment close to the operation performed by the operator in the past and improve the operability.
[0101]
As described above, in this embodiment, whether or not the interaction between the cursor operated by the operator in the virtual space and the virtual object in the virtual space occurs is not determined only by the distance between the cursor and the virtual object. It is possible to improve the operability in the interface that interacts with the virtual object using the cursor in the virtual space by determining the movement based on the user's operation, line of sight, or past cases.
[0102]
In this embodiment, the description has been made using the operation of grasping the virtual object using the cursor as the interaction. However, the same applies to other operations such as an instruction (pointing) to the virtual object, collision, friction, hitting, and remote operation It can be handled. The same effect can be obtained when the virtual space is a two-dimensional space or when the display means uses a three-dimensional stereoscopic display. Moreover, as an implementation means, you may implement | achieve using hardware or the software on a computer.
[0103]
As described above, in this embodiment, the presence or absence of the interaction between the cursor operated by the operator in the virtual space and the virtual object in the virtual space is determined, and the cursor and the virtual object in the virtual space are Is not necessarily determined by the distance alone, but the comprehensive determination means determines whether or not an interaction has occurred, based on the distance between the representative points calculated by the distance calculation means and the movement of the cursor recognized by the action recognition means. It is possible to interact with an object whose distance in the space is not close, and it is possible to provide an input / output interface with good operability. In addition, unlike the conventional contact determination method, it is not necessary to calculate all the distances between the components of the cursor and the virtual object in the virtual space, so the calculation amount can be reduced and the processing speed can be increased. It becomes.
[0104]
【The invention's effect】
As described above, the present invention recognizes the shape of the operator's hand or its movement, displays the recognized hand shape characteristics as a special shape on the screen, and displays the information displayed in the screen. The information displayed on the screen can be easily controlled with good operability by the shape and movement of the hand.
[0105]
Furthermore, the hand shape feature is displayed on the screen as a special shape and displayed as a cursor, and the interaction with the display object other than the cursor display is automatically determined sequentially according to the operator's intention. As a result, it is possible to realize an interface that further improves the operability of instructing and grasping the display object.
[Brief description of the drawings]
FIG. 1 is an external view of an interface device according to a first embodiment of the present invention.
FIG. 2 is a detailed block diagram of the interface device according to the first embodiment of the present invention.
FIGS. 3A to 3C are diagrams showing examples of hand shapes determined by the interface device. FIGS.
FIG. 4 is a diagram showing a detailed example of shape identifying means of the interface device according to the embodiment;
FIG. 5 is a diagram illustrating an example of calculation performed by an image difference calculation unit according to the embodiment.
FIGS. 6A to 6D are diagrams showing examples of icons generated by the icon generation unit of the embodiment;
FIGS. 7A to 7C are external views showing an operation example of the interface device according to the embodiment.
FIG. 8 is an external view of an interface device according to a second embodiment of the present invention.
FIG. 9 is a detailed block diagram of an interface device according to a second embodiment of the present invention.
FIGS. 10A and 10B are diagrams showing examples of hand shapes determined by the interface device. FIGS.
FIG. 11 is a diagram showing a detailed embodiment of an operation recognition unit of the interface device of the embodiment.
FIG. 12 is a diagram illustrating an example of calculation performed by an image difference calculation unit according to the embodiment.
FIGS. 13A to 13D are diagrams showing examples of icons generated by the icon generation unit of the embodiment;
FIG. 14 is a detailed block diagram of an interface device according to a third embodiment of the present invention.
FIG. 15 is a diagram showing an operation recognition unit of the interface device according to the third embodiment of the present invention;
FIGS. 16A to 16D are diagrams showing examples of icons displayed on the display screen by the interface device according to the embodiment;
FIGS. 17A to 17F are diagrams showing the operation of the operation recognition unit of the interface device according to the embodiment;
FIGS. 18A to 18F are diagrams showing the operation of the operation recognition unit of the interface device according to the embodiment;
FIG. 19 is a diagram showing the operation of the operation recognition unit of the interface device according to the embodiment;
FIG. 20 is a diagram showing the operation of the operation recognition unit of the interface device according to the embodiment;
FIG. 21 is a block diagram showing an interface device according to a fourth embodiment of the present invention.
FIG. 22A is a diagram showing a state in which the cursor is opened in an example of the cursor used in the interface device of the embodiment;
(b) Diagram showing the closed state
(c) The figure which shows the state which the cursor opened in the other example of the cursor used for the interface apparatus of the Example.
(d) Diagram showing the closed state
(e) The figure which shows the state which the cursor opened in the further another example of the cursor used for the interface apparatus of the Example.
(f) Diagram showing the closed state
FIG. 23A is a diagram showing an example of a virtual object used in the interface device of the embodiment;
(b) A diagram showing the shape of another example of the virtual object
FIG. 24A is a front view showing the placement of the cursor and virtual object in the virtual space.
(b) Side view showing placement of cursor and virtual object in virtual space
FIG. 25 is a view showing a display example of a virtual space for explaining the embodiment;
FIG. 26 is a block diagram showing an example of the interface device of the embodiment.
FIG. 27A is a diagram showing an example of an input device in the input means used in the interface device of the embodiment.
(b) The figure which shows the other example of the input device in the input means used for the interface apparatus of the Example
(c) The figure which shows the further another example of the input device in the input means used for the interface apparatus of the Example
FIG. 28A is a diagram showing an example of an image obtained by capturing a hand using the camera of the embodiment.
(b) The figure which shows an example which binarized the image which imaged the hand using the camera of the Example
FIG. 29A is a diagram showing an example of a screen displayed by display means used in the interface device of the present embodiment.
(b) Diagram showing a second example of the display screen
(c) Diagram showing a third example of the display screen
(d) Diagram showing a fourth example of the same display screen
FIG. 30 is a diagram showing a conventional interface device.
[Explanation of symbols]
1 Host computer
2 display
3 CCD camera
21 frame memory
22 Shape identification means
23 Display controller
24 Icon generator
25 Reference image memory
26 Reference image update unit
31 frame memory
32 Motion recognition unit
33 Display controller
34 Icon generator
41 Input means
42 Cursor storage means
43 Object storage means
44 Display means
45 Interaction judging means
45a Distance calculation means
45b Motion recognition means
45c Total judgment means
45d Movement vector calculation means
45e Shape determination means
45f learning means
45g coordinate conversion means
200 icons
201, 202 Menu
203 icon
204, 205, 206 Virtual switch
221 Image difference calculation unit
222 Contour extraction unit
223 Shape identification part
224 Position detector
225 Shape change identification part
231 Coordinate converter
232 Coordinate reversal unit
261 timer
262 Image update unit
321 Outline extraction unit
322 Contour waveform calculator
324 coordinate table
331 Coordinate converter
332 Coordinate reversal part
3231 Shape Filter 1
3232 Shape Filter 2
3251 Differentiator
3252 Differentiator
3261 Coordinate calculator
3262 Shape determination unit

Claims (4)

Recognizing means for recognizing the shape of the operator's hand and the movement of the hand in which the shape of the hand continues for a predetermined time ;
A menu that can be selected according to the hand shape feature recognized by the recognition means is displayed on the image ,
Display means for displaying on the image the shape feature of the hand recognized by the recognition means as an icon representing a number ;
Control means for selecting a menu displayed on the image based on a hand shape feature recognized by the recognition means, and for giving a control instruction based on a predetermined hand movement recognized by the recognition means after selection of the menu; interferon y scan apparatus having a. The recognition means recognizes the number of fingers of the hand, the display means displays the number of fingers on the image, and the control means selects the menu based on the number of fingers of the hand recognized by the recognition means. interferon y scan apparatus according to claim 1. The display means displays the icon representing the digits, interferon y scan apparatus according to claim 2, wherein the stressed part of the menu corresponding to the numeric display based on the number of fingers of the hand which the recognition unit recognizes . A recognition means for recognizing the shape of the operator's hand and a predetermined hand movement;
Display means for displaying a selectable menu on the screen according to the hand shape feature recognized by the recognition means;
An icon generation unit that generates an icon corresponding to the recognition result of the hand shape by the recognition unit, and an icon generated by the icon generation unit is displayed on the screen, and a predetermined hand movement recognized by the recognition unit is displayed. based on, interferon y scan and control means for moving and selecting a menu on the image of the icon.

Images