JP2006018856A - User interface device and operating range presenting method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a user interface device for performing input by image processing, which is easier to use to reduce the operation burden on a user, and an operation range presenting method. <P>SOLUTION: The user interface device for inputting by image processing is provided with a means for discriminating whether an object as an image processing object is within a proper range wherein image processing is possible or not and a means for presenting at least a part of prescribed visual information and auditory information in the case that it is discriminated that the object is not within the proper range. For example, a cursor is displayed small and/or thin when the object is far from the proper range, and the cursor is displayed large when the object is in the vicinity of the proper range, and the cursor is displayed with its left end distorted when the object is off to the left, and the cursor is displayed with its right end distorted when the object is off to the right. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

The present invention relates to a user interface device that performs input through image processing and an operation range presentation method for a target object of image processing.

The mouse is overwhelmingly used as a computer input device. However, what can be operated with the mouse is movement of the cursor, selection of a menu, and the like, and is merely a role as a two-dimensional pointing device. Two-dimensional information can be handled with a mouse, and it is difficult to select an object having a depth such as an object in a three-dimensional space. In addition, when creating an animation, it is difficult to apply a natural movement with an input device such as a mouse to add a movement of the character.

To compensate for the difficulty of pointing in a three-dimensional space, a device that inputs information in six axes by pushing or turning the ball in a desired direction, a so-called data glove or data suit,
Devices such as cyber glove have been developed for use in hands. However, these devices are not as popular as originally expected due to poor operability.

On the other hand, a direct instruction type input device has recently been developed in which a user can input information intended by the user by hand gesture or gesture without handling a special device.
For example, irradiate light, receive light reflected by the user's hand, image it, perform feature extraction and shape recognition processing, execute control according to the shape of the hand, or according to the amount of hand movement Some of them move the cursor as much as possible and change the viewpoint in the 3D model.

Alternatively, there is one that performs a process similar to the above by taking a video of the movement of the user's hand and analyzing the video image.
With such a device, the user can easily input by gesture without wearing a special device.

However, in this type of device, since the light receiving device for detecting an object is fixedly installed, the range in which a user's hand can be normally detected is limited. Therefore, depending on the position of the user's hand or the like, there is a problem that the shape and movement cannot be accurately detected, and the control desired by the user cannot be realized. In addition, since it is difficult for the user to immediately recognize the detectable range in the three-dimensional space, the user has to master the operation within the detectable range by experience, and the user is burdened with the operation. There was a problem to force.

The present invention has been made in consideration of the above circumstances, and provides a user interface device and an operation range presenting method that reduce the burden on the user's operation in a user interface device that performs input by image processing. With the goal.

According to the present invention, there is an appropriate range for image input in a three-dimensional space with a predetermined image input device as a reference. The image input device irradiates light on an object to be imaged within the appropriate range and reflects the light. Based on the result of the image processing, the image is input by receiving light, and the image processing including the acquisition of the information about the distance from the image input device to the object to be processed is performed based on the input image. In the user interface device that recognizes that the specified input is specified, information about the distance whether or not the object to be image-processed is in an appropriate range at least in the perspective direction with reference to the image input device. And a means for presenting at least one of predetermined visual information and auditory information for notifying that when it is determined that it is not within the appropriate range. Characterized by comprising and.

According to the present invention, when a target object such as a user's hand deviates from the appropriate range, for example, a notification is given using a display device or an audio playback device, so that the user can easily perform the appropriate range in the three-dimensional space. Can be recognized, and a desired instruction or the like can be input easily by performing a gesture or the like within an appropriate range.

In addition, the present invention has an appropriate range for image input in a three-dimensional space with a predetermined image input device as a reference, and the image input device irradiates light on an object to be imaged in the appropriate range. An image is input by receiving the reflected light, and image processing including acquisition of information on the distance from the image input device to the object to be processed is performed based on the input image. A method for presenting an operation range of a target object in a user interface device that recognizes that a predetermined input specified based on the image input is performed, wherein the object to be image processed is appropriate in at least a perspective direction based on the image input device Whether it is in the range or not is determined based on the information on the distance, and when it is determined that it is not in the proper range, predetermined visual information and auditory information for notifying that Characterized by presenting at least one of the broadcast.

In addition, the invention which concerns on each above apparatus is materialized also as description which concerns on a method. The above-described invention can also be realized as a machine-readable medium recording a program for causing a computer to execute a corresponding procedure or means.

According to the present invention, when the target object such as the user's hand deviates from the appropriate range, this is notified, so that the user can easily recognize the appropriate range in the three-dimensional space and easily A desired instruction or the like can be input by performing a gesture or the like within the range.

Hereinafter, embodiments of the invention will be described with reference to the drawings.
FIG. 1 is a diagram illustrating a configuration example of a user interface device according to an embodiment of the present invention. FIG. 2 shows an operation procedure example of the user interface device of the present embodiment.

This user interface device is suitable when applied to a computer having a graphic user interface, for example. In other words, icons such as cursor, slider bar, scroll bar, pull-down menu, box, link, application, etc. are displayed on the display screen, and the user inputs instructions such as cursor movement, icon selection, application activation by input device System in which an input device does not require a dedicated instrument such as a mouse and receives an input by processing an object such as a user's hand

Applies to
This user interface device is roughly divided into an input function part and a feedback function part. As the input function part, a publicly known one can be used, which emits light and captures the reflected light from the target object such as the user's hand as an image (or images the reflected light from the background light target object. The shape, movement, distance information, etc., and performs predetermined control according to the shape, etc. (for example, input / output device control or application software activation). It provides a function that allows an intended input to be performed by a motion or the like. In the present embodiment, the image storage unit 11,
It is assumed that a shape interpretation unit 12, an interpretation rule storage unit 13, and a presentation unit 14 are included.

The feedback function part is a part according to the present invention, and evaluates whether or not an image detection target object such as a user's hand is within an appropriate detection range, and presents the evaluation result to the user. In the present embodiment, the image storage unit 11, the aptitude range evaluation unit 15, and the evaluation result reflection unit 16 are assumed to be included.

First, an image storage unit 11 common to both functional parts and an image input device (not shown) will be described. The image storage unit 11 receives an image input device (not shown) every predetermined time (for example, every 1/30 seconds,
2D images of the object to be detected that are output every 60 seconds, every 1/100 seconds, etc.) are sequentially stored (step S1).

An image input device (not shown) receives, for example, a light emitting unit that irradiates a target object with light such as near infrared rays using a light emitting element such as an LED, and a light receiving element arranged in a two-dimensional array. And a light receiving portion. Normally, the background is corrected by taking the difference between the amount of light received when the light emitting unit is emitting light and the amount of light received when the light emitting unit is not emitting light. Only the component of the reflected light from the target object of light from is extracted. Note that the image input device may have only a light receiving unit such as a CCD camera without having a light emitting unit.

For example, when the user's hand is brought in front of the image input device, a reflected light image from the hand can be obtained. At this time, each pixel value of the reflected light image is affected by the properties of the object (specularly reflecting, scattering, absorbing, etc.), the orientation of the object surface, the distance of the object, etc. In the case of an object that uniformly scatters light, the amount of reflected light is closely related to the distance to the object.
Since the hand and the like have such properties, the reflected light screen when the hand is held out reflects the distance of the hand, the inclination of the hand (partly different distance), and the like. Therefore, various information can be input and generated by extracting these pieces of information.

Next, the input function part will be described. The shape interpretation unit 12 sequentially captures the two-dimensional image stored in the image storage unit 11 as an N × N (for example, 64 × 64) dot matrix (step S2). Each pixel has a gradation (for example, 8 bits = 256 gradations). FIG.
An example of a dot matrix with the target object as a hand is shown in a).

Next, the shape interpretation unit 12 extracts a predetermined feature amount from the dot matrix, and interprets the shape based on the interpretation rules stored in the interpretation rule storage unit 13 (step S3). Then, an instruction corresponding to the compatible interpretation rule is output as an interpretation result (steps S4 and S5). If there is no matching interpretation rule, change the method of extracting a predetermined feature amount from the dot matrix as necessary (for example, change the threshold value when performing dot matrix threshold processing) The matching process may be performed again. If there is no finally matching interpretation rule (step S4), it is assumed that no input has been made.

The interpretation rule storage unit 13 stores interpretation rules for shape interpretation. For example, the instruction content corresponding to the feature amount such as the shape of the user's hand is stored as an interpretation rule. The instruction contents include icon selection, application activation, cursor movement, and the like. In the case of cursor movement, the amount of movement of the cursor according to the movement direction / distance of the hand is also instructed. For example, the state where only the index finger is raised corresponds to the cursor movement (in this case, for example, the movement distance of the tip of the index finger,
(The direction is related to the distance and direction of movement of the cursor) With the index finger only raised, move the thumb to correspond to the selection of the icon where the cursor is located, and with the index finger only raised, palm up A rule may be considered such that the reverse operation corresponds to the activation of an application corresponding to the icon where the cursor is located.

Typical examples of feature amount extraction from a dot matrix in shape interpretation by the shape interpretation unit 12 are extraction of distance information and region extraction. If the object has a uniform and homogeneous scattering surface,
The reflected light image can be regarded as a distance image. Therefore, the three-dimensional shape of the object viewed from the light receiving unit can be extracted. If the object is a hand, the tilt of the palm can be detected.
Palm tilt appears as a partial distance difference. If the pixel value changes when the hand is moved, it can be regarded that the distance has moved. In addition, since there is almost no reflected light from a distant object like the background, the shape of the object can be easily cut out by a process of cutting out an area of a certain threshold value or more from the reflected light image. For example, if the object is a hand, it is very easy to cut out the silhouette image. Even in the case of using a distance image, it is often the case that a region is extracted once by a threshold value and then distance information in the region is used.

There are various methods for matching the feature amount extracted from the dot matrix and the interpretation rule. For example, vectorization for extracting a vector from an image, extraction of a deformation state of a shape based on a shape model, spectrum analysis based on a distance value on a scanning line, and the like.

If there is no suitable shape, the matching process may be performed again, for example, by changing the threshold value. If there is no finally matching shape, it is considered that there was no input.

When the interpretation result by the shape interpretation unit 12 instructs the user to present visual information, the presentation unit 14 presents the display device with the interpretation result reflected by the shape interpretation unit 12. For example, the cursor is moved, the shape of the cursor is changed, or a message is presented as necessary. Note that an audio playback device may be used or used together to present a message.

Next, the feedback function part will be described. The aptitude range evaluation unit 15 takes in the two-dimensional image stored in the image storage unit 11 as a dot matrix as shown in FIG.
Step S2) evaluates whether or not the target object is in the appropriate range, and generates feedback information according to the evaluation result (steps S6 to S16).

Based on the feedback information, the evaluation result reflecting unit 16 instructs the display device and / or the audio reproduction device to present the reflection of the evaluation result (step S17).

First, the aptitude range evaluation unit 15 will be described in detail. When the target object is in the appropriate range, for example, an appropriate dot matrix as shown in FIG. 3A is obtained, and a desired input by the user's hand or the like is possible. However, when the target object is in the appropriate range, more specifically,
When the target object is too close to the light receiving unit, too far away, protrudes from the left, or protrudes from the right, an instruction desired by the user cannot be input.

Therefore, the aptitude range evaluation unit 15 analyzes the dot matrix and determines whether the target object is in the aptitude range (step S6), is too close (step S8), is too far (step S10), or protrudes from the left. Whether it protrudes from the right (step S14) is evaluated. When the target object is in the aptitude range, user feedback information = NULL (or a code indicating NULL; for example, 0) is set (step S7).
If too close, user feedback information = near (or a code indicating near; for example, 1) (step S9). If too far, user feedback information = far (or a code indicating far; for example, 2) (step S11). User feedback information when protruding from the left

= Left (or a code indicating left; for example, 3) (step S13). When protruding from the right, user feedback information = Right (or a code indicating right; for example, 4) (step S15), otherwise , User feedback information = inappropriate (step S1
6).

FIG. 4 shows an example of this evaluation procedure. In addition, FIG. 3A, FIG. 7A, FIG. 8A, FIG. 9A, and FIG. 10A each show an example of a dot matrix when the target object is in the appropriate range. An example of a dot matrix, an example of a dot matrix when it is too far, an example of a dot matrix that protrudes from the left, and an example of a dot matrix that protrudes from the right are shown.

Various methods can be considered to determine whether or not the target object is in the appropriate range. In this example, the area s of the target object image, the distance d of the closest point in the target object image, the target object Let us use the length l of the vertical line in the image shape. The area s of the image of the target object can be represented by the number of pixels corresponding to the target object in the dot matrix or the proportion of all the pixels. Further, the distance d of the closest point in the image of the target object can be expressed by the reciprocal of the maximum density of the pixel corresponding to the target object in the dot matrix or the highest density that the pixel can take-the maximum value of the density. The length l of the vertical line in the image shape of the target object can be represented by the maximum number of pixels that are vertically continuous in the outer shape of the image of the target object in the dot matrix as shown in FIG.

Then, a lower limit value r and an upper limit value α of the area s, and a lower limit value β and an upper limit value δ of the distance d of the closest point in the image.
The upper limit value ε of the vertical line length l in the image shape is set. If r ≦ area s ≦ α
And if β ≦ nearest point distance d ≦ δ and the length of the vertical line l ≦ ε (step S21),
The range is appropriate (step S22). If the area s> α and the distance d <β between the closest points (step S23), it is assumed that the distance is too close (step S24). If area s <r and closest point distance d> δ (step S25), it is assumed that the distance is too far (step S26).

If the vertical line length l> ε and the vertical line position is on the right (step S27), it is assumed that the line protrudes from the right (step S28). If the length of the vertical line l> ε and the vertical line position is on the left (step S29), it is assumed that it protrudes from the left (step S30).

In other cases, the range is inappropriate (step S31). Next, the evaluation result reflection unit 1
6 will be described in detail.
FIG. 6 shows an example of this processing procedure. In this process, the evaluation result reflection unit 16 notifies the evaluation result by changing the shape of the cursor displayed on the display screen based on the user feedback information given as the evaluation result from the aptitude range evaluation unit 15. To do.

If user feedback information = NULL (step S41), the cursor shape is not changed (step S42). An example of this state is shown in FIG. If user feedback information = near (step S43), for example, the cursor is enlarged as shown in FIG. 7A (step S44). If user feedback information = far away (
Step S45), for example, the cursor is made small and thin as shown in FIG. 8A (step S46).

If user feedback information = left (step S47), for example, the left end of the cursor is deformed as shown in FIG. 9A (step S48). If user feedback information = right (step S49), for example, the left end of the cursor is deformed as shown in FIG. 10A (step S50).

If the user feedback information is inappropriate, an error message is displayed (step S51). FIG. 11 shows another example of this processing procedure. In this process, the evaluation result reflection unit 1
6, based on the user feedback information given as the evaluation result from the aptitude range evaluation unit 15, the voice reproduction apparatus notifies the evaluation result by voice.

If the user feedback information = NULL (step S61), nothing is presented or a sound of moving the cursor is presented (step S62). If user feedback information = near (step S63), for example, a sound such as “It is too close” is output (step S64).

If user feedback information = far (step S65), for example, a voice such as “too far” is output (step S66). If the user feedback information = left (step S67), for example, a sound such as “I am protruding to the left” is output (step S68).

If the user feedback information = right (step S69), for example, a sound such as “I protrude to the right” is output (step S70). If the user feedback information is inappropriate, an error message is presented by voice (step S71).

Note that the processing in FIG. 6 and the processing in FIG. 11 may be used together to notify by image and sound.
Alternatively, the function of notifying by image and the function of notifying by sound may be facilitated so that the user can individually set the function on / off.

As described above, according to the present embodiment, when a target object such as a user's hand deviates from the appropriate range, a notification to that effect is given, so that the user can easily recognize the appropriate range in the three-dimensional space. It is possible to easily input a desired instruction or the like by performing a gesture or the like within an appropriate range.

In the procedure shown in FIG. 2, the processing of the input function portion and the processing of the feedback function portion are executed independently. However, the procedure is modified so that the processing of the feedback function portion is performed prior to the processing of the input function portion. The processing of the input function portion may be executed only when it is determined that the target object is within the appropriate range.

The above functions can also be realized as software. Further, the present invention can also be implemented as a machine-readable medium in which a program for causing a computer to execute each procedure or means described above is recorded. The present invention is not limited to the embodiment described above, and can be implemented with various modifications within the technical scope thereof.

The figure which shows the structural example of the user interface apparatus which concerns on embodiment of this invention. The flowchart which shows an example of the process sequence of this embodiment. The figure which shows an example of a dot matrix when a target object exists in a suitable range, and the example of a screen display in that case. The flowchart which shows an example of the evaluation procedure whether a target object exists in a suitable range. The figure for demonstrating the length l of the perpendicular line in the image shape of a target object. The flowchart which shows an example of an evaluation result reflection procedure. The figure which shows an example of a dot matrix when a target object is too close, and the example of a screen display in that case. The figure which shows an example of the dot matrix in the suitable range when a target object is too far, and the example of a screen display in that case. The figure which shows an example of a dot matrix in case the target object protrudes from the left, and the example of a screen display in that case. The figure which shows an example of a dot matrix in case the target object protrudes from the right, and the example of a screen display in that case. The flowchart which shows the other example of an evaluation result reflection procedure.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 ... Image memory | storage part 12 ... Shape interpretation part 13 ... Interpretation rule memory | storage part 14 ... Presentation part 15 ... Aptitude range evaluation part 16 ... Evaluation result reflection part

Claims (2)

An appropriate range for image input exists in a three-dimensional space with a predetermined image input device as a reference, and the image input device irradiates light to the object to be image processed and receives the reflected light. The image is input, the image processing including the acquisition of information about the distance from the image input device to the object to be processed is performed based on the input image, and is specified based on the result of the image processing. In the user interface device that recognizes that the input of
Means for determining, based on the information about the distance, whether or not the object to be image-processed is in an appropriate range for at least a perspective direction based on the image input device;
A user interface device, comprising: means for presenting at least one of predetermined visual information and auditory information for notifying that when it is determined not to be in an appropriate range. An appropriate range for image input exists in a three-dimensional space with a predetermined image input device as a reference, and the image input device irradiates light to the object to be image processed and receives the reflected light. The image is input, the image processing including the acquisition of information about the distance from the image input device to the object to be processed is performed based on the input image, and is specified based on the result of the image processing. A method for presenting an operation range of a target object in a user interface device that is recognized as having been input,
Whether or not the object to be image-processed is in an appropriate range for at least a perspective direction based on the image input device is determined based on the information on the distance,
An operation range presenting method characterized by presenting at least one of predetermined visual information and auditory information for notifying that when it is determined that it is not within an appropriate range.

Images