JP3588527B2 - User interface device and instruction input method - Google Patents

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a user interface device and an input method for performing input by image processing.
[0002]
[Prior art]
Mice are overwhelmingly used as input devices for computers. However, what can be operated with the mouse is the movement of the cursor and the selection of menus, and is merely useful as a two-dimensional pointing device. The mouse can handle two-dimensional information, and it is difficult to select an object having a depth such as an object in a three-dimensional space. In addition, when creating an animation, it is difficult to make a natural movement with an input device such as a mouse to make the movement of the character.
[0003]
In order to compensate for the difficulty of pointing in three-dimensional space, push or rotate the ball in the desired direction to input information in six axes, or fit into so-called data gloves, data suits, cyber gloves, etc. Devices have been developed. However, these devices are not as widely used as initially expected due to poor operability and the like.
[0004]
On the other hand, recently, a direct instruction type input device has been developed in which a user can input information intended by the user by hand or gesture without handling a special device.
[0005]
For example, by irradiating light, receiving light reflected by the user's hand, imaging the light, performing feature amount extraction and shape recognition processing, and executing control according to the shape of the hand, and controlling the amount of movement of the hand. Some of them move the cursor by a certain amount or change the viewpoint in the three-dimensional model.
[0006]
Alternatively, there is a method in which the same processing as described above is performed by capturing a video of a user's hand motion and analyzing the video image.
[0007]
With such a device, the user can easily perform input by gesture without wearing a special device.
[0008]
[Problems to be solved by the invention]
However, in this type of device, each mode such as a cursor movement mode, a selection mode, and a double-click mode is fixedly used, and an explicit operation for changing the mode is required to change the mode. Therefore, there is a problem that the user is burdened with the operation.
[0009]
SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and provides a user interface device for performing input by image processing, which provides a user interface device and an instruction input method that are easier to use and reduce the burden on the user. Aim.
[0010]
[Means for Solving the Problems]
The present invention is a user interface device for performing input by image processing of an input image, comprising a means for switching between a pointing mode and other modes based on an image processing result of the input image. And
[0011]
Further, the present invention is a user interface device for performing input by image processing of an input image, wherein at least a mode of moving a cursor, a mode of selection, and a mode of double-clicking are set in an image processing result of the input image. And means for switching based on the information.
[0012]
Preferably, the apparatus further comprises means for designating a recognition method (recognition engine) that restricts image processing contents for each selectable object in the selection mode, and in performing image processing of the input image for the selected object, Image processing may be performed according to a recognition method specified for the object.
[0013]
Preferably, for each selectable object in the selection mode, means for designating a recognition method (recognition engine) that restricts image processing contents, and a method for designating the object in the vicinity of the displayed object indicated by the cursor. And means for presenting information indicating the recognition method.
[0014]
Preferably, the apparatus may further include means for presenting an image processing result of the input image in a predetermined shape on a cursor.
[0015]
The present invention is also a user interface device comprising a first device for inputting a reflection image and a second device for inputting the input image by image processing, wherein the second device is the first device. A means for designating a recognition method (recognition engine) restricting image processing content of an input image, wherein the first device performs predetermined image processing based on the designation of the recognition method; Means for returning the input image and the image processing result to the second device.
[0016]
Preferably, when the first device does not have an image processing unit (recognition engine) suitable for a required recognition method, the second device stores information (a recognition engine) required for image processing suitable for the recognition method. ) May be further provided, and the second device may further comprise means for transferring the requested information to the first device.
[0017]
Preferably, each of the first device and the second device is activated by another device when information necessary for image processing suitable for a predetermined recognition method in the own device is activated first. Means for notifying the information to be inactive, and means for inactivating the information when notified from another device to deactivate information necessary for image processing suitable for the predetermined recognition method. May be further provided.
[0018]
The present invention is also directed to a method for inputting an instruction by a user interface device for performing input by image processing of an input image, wherein a mode for performing image processing on an input image of a target object and performing pointing based on a result of the image processing is provided. , And other modes.
[0019]
The present invention also provides an instruction input method using a user interface device including a first device for inputting a reflection image and a second device for inputting the input image by image processing. The first device specifies a recognition method (recognition engine) that restricts the image processing content of the input image, and the first device performs predetermined image processing based on the specification of the recognition method. The input image and the image processing result are returned to the second device.
[0020]
According to the present invention, an explicit operation by the user for changing each mode such as a cursor movement mode, a selection mode, and a double-click mode becomes unnecessary.
[0021]
In addition, since the point designated by the user is read by the recognition process and reflected on the movement of the cursor on the screen, the calibration by the user's operation is unnecessary.
[0022]
Furthermore, if an image processing means (recognition engine) suitable for a necessary recognition method is used, improvement in input accuracy and operability for the user can be expected.
[0023]
Further, if the input image is made translucent and displayed over the cursor, the operation status can be fed back to the user.
[0024]
As described above, according to the present invention, it is possible to provide an easier-to-use user interface device in which the operational burden on the user is reduced.
[0025]
Further, according to the present invention, since a certain degree of recognition processing is performed on the first device side (device side), the load can be distributed, and the overall speed of the recognition processing can be improved.
[0026]
Further, it is possible to enhance the functionality of a device having an image input function.
[0027]
It should be noted that the inventions relating to the respective devices described above are also valid as descriptions relating to methods.
[0028]
Further, the above-described invention is also realized as a machine-readable medium storing a program for causing a computer to execute a corresponding procedure or means.
[0029]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of the invention will be described with reference to the drawings.
[0030]
First, a first embodiment will be described.
[0031]
FIG. 1 is a diagram illustrating a configuration example of a user interface device according to an embodiment of the present invention.
[0032]
The present user interface device is preferably applied to, for example, a computer having a graphic user interface. That is, icons such as a cursor, a slider bar, a scroll bar, a pull-down menu, a box, a link, and an application are displayed on a display screen, and a user inputs an instruction such as moving a cursor, selecting an icon, or starting an application using an input device. In this system, an input device receives an input by performing image processing on an object such as a user's hand without requiring a dedicated device such as a mouse.
[0033]
In the present embodiment, the shape, the motion, and the like of the reflected light by the target object such as the user's hand are roughly captured (or the reflected light of the background light by the target object is captured as an image). It detects distance information and the like, and performs predetermined control (for example, control on an input / output device or activation of application software) in accordance with the shape or the like. The user can perform an intended input by hand motion or the like. It provides functions. In addition, since each mode such as cursor movement, icon selection, and application startup is switched according to the image processing result, the user does not need to perform an explicit operation for switching each mode.
[0034]
The user interface device includes an image input unit 10, an image storage unit 11, a shape interpretation unit 12, an interpretation rule storage unit 13, a presentation unit 14, and a cursor switching unit 15.
[0035]
FIG. 2 shows a configuration example of the image input unit 10.
[0036]
The image input unit 10 receives, for example, a light-emitting unit 101 that irradiates a target object with light such as near-infrared light by a light-emitting element such as an LED, and receives light reflected from the target object by a light-receiving element arranged in a two-dimensional array. It has a reflected light extraction unit 102 and a timing control unit 103 that controls the operation timing of the light emitting unit 101 and the reflected light extraction unit 102. By calculating the difference between the amount of light received by the reflected light extraction unit 102 when the light emitting unit 101 is emitting light and the amount of light received by the reflected light extraction unit 102 when the light emitting unit 101 is not emitting light Then, the background is corrected, and only the component of the light from the light emitting unit 101 reflected by the target object is extracted. Note that the image input unit 10 may have only a light receiving unit such as a CCD camera without having a light emitting unit.
[0037]
FIG. 3 shows the relationship between the display device 20, the housing 8 of the image input unit 10, and the target object 22. For example, when the user's hand 22 is brought in front of the image input unit 10, a reflected light image from the hand is obtained. At this time, each pixel value of the reflected light image is affected by the properties of the object (specular reflection, scattering, absorption, etc. of the light), the direction of the object surface, the distance of the object, and the like. When an object scatters light uniformly, the amount of reflected light has a close relationship with the distance to the object. Since a hand or the like has such a property, the reflected light screen when the hand is extended reflects the distance of the hand, the inclination of the hand (partially different distance), and the like. Therefore, by extracting such information, it is possible to input and generate various information.
[0038]
The image storage unit 11 sequentially stores two-dimensional images of the image detection target object output from the image input unit 10 at predetermined time intervals (eg, every 1/30 second, every 1/60 second, every 1/100 second, etc.). I do.
[0039]
The shape interpretation unit 12 sequentially takes in the two-dimensional images stored in the image storage unit 11 as an N × N (for example, 64 × 64) dot matrix. Each pixel has a gradation (for example, 8 bits = 256 gradations).
[0040]
Further, the shape interpretation unit 12 extracts a predetermined feature amount from the dot matrix and interprets the shape based on the interpretation rules stored in the interpretation rule storage unit 13. Then, an instruction corresponding to the matching interpretation rule is output as an interpretation result. If there is no matching interpretation rule, the method of extracting a predetermined feature amount from the dot matrix is changed as necessary (for example, when performing threshold processing of the dot matrix, the threshold value is changed). Alternatively, the matching process may be performed again. If there is no finally applicable interpretation rule, there is no input.
[0041]
The interpretation rule storage unit 13 stores interpretation rules for shape interpretation. For example, a feature amount of a target object such as a user's hand in a dot matrix, for example, a predetermined value such as a shape, an area, an uppermost point, and a center of gravity, and an instruction corresponding thereto are stored as interpretation rules. Instructions include icon selection, application activation, cursor movement, and the like. In the case of cursor movement, the movement amount of the cursor according to the movement direction and distance of the hand is also instructed. For example, a state in which the thumb and the index finger are opened and set up corresponds to the cursor movement (in this case, for example, the movement distance / direction of the tip of the index finger corresponds to the movement distance / direction of the cursor), and the thumb and the index finger are closed and set up Corresponds to the selection of the icon at which the cursor is located, the thumb and forefinger are set up, and the palm is inverted in the case of cursor movement, and the state where the palm is inverted corresponds to the activation of the application corresponding to the icon at which the cursor is located Such rules are conceivable.
[0042]
Representative examples of extracting a feature amount from a dot matrix in the shape interpretation by the shape interpretation unit 12 include extraction of distance information and region extraction. If the object has a uniform and uniform scattering surface, the reflected light image can be regarded as a distance image. Therefore, the three-dimensional shape of the object viewed from the light receiving unit can be extracted. If the object is a hand, the inclination of the palm can be detected. The tilt of the palm appears as a partial difference in distance. If the pixel value changes when the hand is moved, it can be regarded that the distance has moved. Further, since there is almost no reflected light from a distant object such as the background, the shape of the object can be easily cut out by processing to cut out an area of a certain threshold or more from the reflected light image. For example, if the object is a hand, it is extremely easy to cut out a silhouette image thereof. Even when a distance image is used, it is often the case that an area is extracted once using a threshold, and then the distance information within the area is used.
[0043]
There are various methods for matching the feature quantity extracted from the dot matrix with the interpretation rule. For example, vectorization for extracting a vector from an image, extraction of a deformed state of a shape based on a shape model, spectrum analysis based on a distance value on a scanning line, and the like.
[0044]
If there is no suitable shape, the matching process may be performed again, for example, by changing the threshold value. If there is no final conforming shape, it is assumed that there was no input.
[0045]
When the shape interpreting unit 12 recognizes that the instruction is to activate an application or a function of an OS, the software is activated.
[0046]
The presentation unit 14 performs presentation on the display device reflecting the interpretation result by the shape interpretation unit 12. For example, the cursor is moved, and a message is presented as necessary.
[0047]
The cursor switching unit 15 controls cursor switching based on the result of interpretation by the shape interpretation unit 12.
[0048]
4 to 6 show an example of an operation procedure of the user interface device according to the present embodiment.
[0049]
First, the control state C of the cursor is initialized (C ← 0), the selection state S is initialized (S ← 0), the cursor information I is initialized (I ← 0), and the recognition engine flag R is initialized. (R ← 0) is performed (step S1).
[0050]
Next, the reflection image is read and written into the image storage unit 11 (step S2).
[0051]
Next, a dot matrix is read into the shape interpretation unit 12 (step S3).
[0052]
Next, the mode indicated by the gesture is determined from the feature amount extracted from the dot matrix and the interpretation rule by the shape interpretation unit 12 (step S4).
[0053]
Thereafter, the processing is divided depending on the result of the determination and the parameter value.
[0054]
If the gesture of the cursor control and C = 0 and S = 0 and R = 0 (step S5), the process enters the cursor control, C is set to 1 (step S11), and the process returns to step S2.
[0055]
The gesture of the cursor control is, for example, a case where the hand shape illustrated in FIG. 7 is recognized.
[0056]
If the gesture of the cursor control and C = 1, S = 0 and R = 0 (step S6), the processing is during the movement of the cursor. In this case, first, the coordinates (x, y) of the neighboring point are calculated from the dot matrix (step S12), and the cursor is moved to the calculated coordinates (x, y) (step S13). Then, the calculated coordinates (x, y) are held. It is assumed that Cp = (x, y) (step S14). If there is an object in Cp (step S15), the state of the object is set (I ← object state) (step S16). If there is no object (step S15), I = O is set (step S17). Then, the process returns to step S2.
[0057]
If the gesture of the cursor control and C = 1 and S = 1 (step S7), the process returns to the cursor control. S ← 0, R ← 0, I ← 0 (step S18), and the process returns to step S2.
[0058]
If the gesture of selection and C = 1, S = 0, and R = 0 (step S8), the process is to select an object. In this case, first, S ← 1 is set (step S19), an object closest to Cp is searched (step S20), and the searched object is selected (step S21). Here, if the selected object has the specified recognition engine (step S22), R ← 1 is set (step S23), and the process returns to step S2. If there is no specified recognition engine, if the object is a link object (step S24), jump to the link destination (step S25), set C ← 0, S ← 0, I ← 0 (step S26), and return to step S2. If it is not a link object (step S24), the process returns to step S2.
[0059]
The gesture of selection is, for example, when the hand shape shown in FIG. 11 is recognized.
[0060]
Here, the recognition engine extracts a predetermined feature amount from the dot matrix as described in detail later. For example, the recognition engine extracts a vertical movement amount of the highest point of the target object shape in the dot matrix. There are a variety of things, such as upper point vertical engines.
[0061]
If the gesture for selection and R = 1 (step S9), this is a process for moving the selected object, and recognition is performed along the recognition object (step S27), and the process returns to step S2.
[0062]
If the gesture of the double click and C = 1 (step S10), the process is a double click. In this case, the object closest to Cp is opened (step S28), C ← 0, S ← 0, I ← 0, R ← 0 (step S29), and the process returns to step S2.
[0063]
The double-click gesture is, for example, when the hand shape shown in FIG. 14 is recognized.
[0064]
Otherwise, another recognition process is performed (step S30), and the process returns to step S2.
[0065]
Hereinafter, the present embodiment will be described using a specific example.
[0066]
First, it is assumed that the cursor movement is instructed when the shape of the hand with the thumb and forefinger extended as shown in FIG. 7 is recognized.
[0067]
For example, in the state of FIG. 8A, the hand shape of FIG. 7 is recognized, and the cursor movement is accepted. When the user moves the hand shape in the state of FIG. 7, the cursor 201 is displayed accordingly. Move around on the screen. In this case, a certain point in the dot matrix of the shape of the user's hand, for example, the highest point where the vertical position in the image is the highest (for example, corresponding to the tip of the index finger), or the most light receiving portion in the image The movement amount / direction at the nearest point (for example, the point with the highest gradation) close to is extracted.
[0068]
As shown in FIG. 8B, the input image 202 may be displayed translucently on the cursor 201, and the recognition status may be fed back to the user.
[0069]
When the cursor is over various objects such as a slider bar and a link node, the shape of the cursor is changed so as to exhibit its function.
[0070]
For example, as shown in FIG. 9A, when the slider bar is located where the cursor has moved, the shape of the cursor is deformed as indicated by 203 in the figure. Note that, as shown in FIG. 9B, the input image may be displayed as a translucent 205 on the cursor.
[0071]
Here, the arrow indicated by 204 in FIG. 9A indicates that the operation of the slider bar to be operated is restricted in the vertical direction. That is, a vertical engine is specified for the slider bar. In this case, no matter how the user moves his hand, only the movement in the vertical direction is recognized.
[0072]
Further, for example, when a link node is located at a position where the cursor has moved as shown in FIG. 10A, the shape of the cursor is deformed as indicated by 206 in the figure. Note that, as shown in FIG. 10B, the input image 207 may be displayed translucently on the cursor.
[0073]
Next, after the cursor is moved to a desired position with the hand shape of FIG. 7, the index finger is extended as shown in FIG. 11, and when the shape of the hand with the thumb closed is recognized, the object indicated by the cursor is displayed. A selection shall be indicated. This is equivalent to a single mouse click.
[0074]
For example, in the state shown in FIG. 12A, if the hand is shaped as shown in FIG. 11, the slider bar is selected. The arrow indicated by 208 in the figure indicates that the operation of the slider bar to be operated is restricted in the vertical direction. As shown in FIG. 12B, the input image 209 may be displayed semi-transparently on the cursor.
[0075]
For example, in the state of FIG. 10A, if the hand is shaped as shown in FIG. 11, the link node “company profile” is selected, and the display content changes as shown in FIG. 13A. As shown in FIG. 13B, the input image 210 may be displayed translucently on the cursor.
[0076]
Next, after the cursor is moved to a desired position with the hand shape of FIG. 7 and the state where the wrist is rotated by 180 degrees without changing the hand shape itself as shown in FIG. A double click is instructed for the object indicated by.
[0077]
For example, in FIG. 15A, the cursor on the “Internet” icon is moved in the shape of the hand in FIG. 7, and then the hand is rotated as shown in FIG. Then, a double click of “Internet” is accepted. FIG. 15B shows a state immediately before an icon is selected and the icon is opened. As shown in FIG. 16, the input image 212 may be displayed translucently on the cursor.
[0078]
For example, first, the cursor is moved to “file” as shown in FIG. At this time, the input image 213 may be displayed translucently on the cursor as shown in FIG. 17B.
[0079]
Next, when "File" is selected, a pull-down menu is displayed as shown in FIG. Here, the arrow indicated by 214 in the figure indicates that the operation of the pull-down menu to be operated is restricted in the vertical direction.
[0080]
Next, as shown in FIG. 18A, the cursor is positioned on “save” and the hand is shaped as shown in FIG. 14, so that a double-click of “save” is accepted. Here, as shown in FIG. 18B, the input image 213 may be displayed translucently on the cursor.
[0081]
When "save" is double-clicked, the shape of the cursor is changed to, for example, 216 in FIG. This indicates that the document is being saved by selecting save. Here, as shown in FIG. 19B, the input image 217 may be displayed translucently on the cursor.
[0082]
Further, for example, first, as shown in FIG. 20A, the cursor is moved to "file", "file" is selected, further moved to print, and "print" is selected. Then, a menu corresponding to “print” is displayed. At this time, as shown in FIG. 20B, the input image 219 may be displayed translucently on the cursor.
[0083]
Next, the recognition engine will be described.
[0084]
FIG. 21 shows a flowchart of processing relating to the recognition engine.
[0085]
In the present embodiment, a recognition engine is specified for each object as needed.
[0086]
The recognition engine extracts a predetermined feature amount from the dot matrix. That is, if there is a recognition engine specified for the selected object (step S31), the shape interpretation unit 12 extracts a feature amount from the dot matrix according to the specified recognition engine. If no recognition engine is specified (step S31), normal recognition is executed (step S32).
[0087]
Recognition engines include, for example,
A nearest point vertical engine 121 for extracting the vertical movement amount of the nearest point of the target object shape in the dot matrix, a nearest point horizontal engine 122 for extracting the horizontal movement amount of the nearest point, an oblique direction of the nearest point Nearest point diagonal engine 123 for extracting the amount of movement, centroid vertical engine 124 for extracting the amount of vertical movement of the center of gravity, centroid horizontal engine 125 for extracting the amount of horizontal movement of the center of gravity, centroid Center-of-gravity point diagonal engine 126 for extracting the amount of movement of the center of gravity in the diagonal direction, center-of-gravity point vertical engine 127 for extracting the amount of vertical movement of the center of gravity, and center-of-gravity point horizontal engine for extracting the amount of horizontal movement of the center of gravity 128, a center-of-gravity point oblique direction engine 129 for extracting the amount of movement of the center-of-gravity point in an oblique direction; An extraction engine 130, an area calculation engine 131 for calculating the area of the target object shape in the dot matrix, and a nearest point x-axis rotation angle engine 132 for extracting the x-axis rotation angle of the nearest point of the target object shape in the dot matrix. A closest point y-axis rotation angle engine 133 for extracting the closest point y-axis rotation angle, a closest point z-axis rotation angle engine 134 for extracting the closest point z-axis rotation angle, and a x-axis rotation of the center of gravity Center of gravity point x-axis rotation angle engine 135 for extracting angles, center of gravity point y-axis rotation angle engine 136 for extracting the center of gravity point y-axis rotation angle, center of gravity point z-axis for extraction of the center of gravity point z-axis rotation angle A rotation angle engine 137, a centroid point x-axis rotation angle engine 138 for extracting the x-axis rotation angle of the center of gravity point, a centroid point y-axis rotation angle for extracting the y-axis rotation angle of the center of gravity point Engine 139, the center of gravity z-axis rotation angle engine 140 for extracting z-axis rotation angle of the center of gravity, and other, such as recognition engine 141 by weighting combination predetermined engine, considered various ones.
[0088]
FIG. 22 shows an example of a description of a recognition engine for each object.
[0089]
As described above, according to the present embodiment, an explicit operation by the user for changing each mode such as the cursor movement mode, the selection mode, and the double-click mode becomes unnecessary.
[0090]
In addition, since the point designated by the user is read by the recognition process and reflected on the movement of the cursor on the screen, the calibration by the user's operation is unnecessary.
[0091]
Furthermore, if a recognition engine is used, an improvement in input accuracy and an improvement in user operability can be expected.
[0092]
Further, if the input image is made translucent and displayed over the cursor, the operation status can be fed back to the user.
[0093]
As described above, according to the present embodiment, it is possible to provide an easier-to-use user interface device in which the operational burden on the user is reduced.
[0094]
Next, a second embodiment will be described.
[0095]
The second embodiment is basically the same as the first embodiment, but in the present embodiment, a part of the recognition processing is performed in an image input unit side housing (hereinafter, referred to as a device side). The image input unit 10 transfers a dot matrix of an input image and a predetermined recognition result to the main body. It is desirable that the recognition process performed on the device side is a light load process.
[0096]
FIG. 23 shows a configuration example of the interface device according to the present embodiment.
[0097]
In FIG. 23, a main body control unit 32, a presentation unit 14, and a cursor switching unit 15 are arranged in the main body, and an image input unit 10, an image storage unit 11, a recognition engine control unit 30, an active list 31, The predetermined recognition engines 121, 122, 142, 143, 144 are arranged.
[0098]
The main body control unit 32 corresponds to, for example, the shape interpretation unit 12 and the interpretation rule storage unit 13 (including the recognition engine) in FIG. 1, but is not limited thereto. The recognition processing may be performed as long as the recognition engine is used.
[0099]
FIG. 24 shows an example of the description of the active list storage unit when the vertical slider bar is selected. In this case, it is indicated that the cursor engine 142 and the closest point vertical direction engine 121 are designated.
[0100]
In such a configuration, a list of recognition engines to be activated or a list of recognition engines to be deactivated are transmitted from the main body side to the device side. On the device side, this list is stored in the active list storage unit 31, a predetermined feature amount is extracted from the input image as a recognition result by the recognition engine control unit 30 according to the specified recognition engine, and the input image and the recognition result are stored in the main unit. return.
[0101]
According to the present embodiment, since a certain degree of recognition processing is performed on the device side, the load can be distributed, and the overall speed of the recognition processing can be improved.
[0102]
Further, it is possible to enhance the functionality of a device having an image input function.
[0103]
FIG. 27 shows still another configuration example of the interface device according to the present embodiment.
[0104]
In FIG. 27, the configuration in the main body is almost the same as that in FIG. 25. However, in the present configuration example, when the recognition engine exists in a plurality of places such as the main body and the device side, the recognition engine activated first Is configured to communicate with the other side having the same recognition engine so as to deactivate the recognition engine.
[0105]
FIG. 28 shows an operation procedure example of the active notification in this case, and FIG. 29 shows an operation procedure example of the active notification reception.
[0106]
First, image input is performed on the device side (step S39), and recognition is performed on the main body side and / or device side (step S40). Then, the side that has performed the recognition sends the recognition result, the image matrix, and the active list (or the inactive list) to the notification partner (step S41).
[0107]
Next, the side receiving the notification receives the recognition result, the image matrix, and the active list (or inactive list) (step S42), and recognizes the recognition engine that is ON in the received active list (or OFF in the inactive list). The recognition engine is rewritten to OFF in the active list stored therein (step S43). Then, other processing is executed as needed (step S44).
[0108]
Note that each of the above functions can be implemented as software. Further, the present invention can be embodied as a machine-readable medium storing a program for causing a computer to execute the above-described procedures or means.
[0109]
The present invention is not limited to the above-described embodiment, and can be implemented with various modifications within the technical scope thereof.
[0110]
【The invention's effect】
According to the present invention, since each mode such as a cursor movement mode, a selection mode, and a double-click mode is switched based on the image processing result, an explicit operation by the user to change the mode is not required.
[0111]
In addition, since the point designated by the user is read by the recognition process and reflected on the movement of the cursor on the screen, the calibration by the user's operation is unnecessary.
[0112]
As described above, according to the present invention, it is possible to provide an easier-to-use user interface device in which the operational burden on the user is reduced.
[0113]
Further, according to the present invention, since a certain degree of recognition processing is performed on the first device side (device side), the load can be distributed, and the overall speed of the recognition processing can be improved.
[Brief description of the drawings]
FIG. 1 is a diagram showing a configuration example of an interface device according to a first embodiment of the present invention.
FIG. 2 is a diagram illustrating a configuration example of an image input unit;
FIG. 3 is a diagram illustrating a relationship between a display device, a housing of an image input unit, and a target object.
FIG. 4 is an exemplary flowchart illustrating an example of an operation procedure of the user interface device according to the embodiment.
FIG. 5 is an exemplary flowchart illustrating an example of an operation procedure of the user interface device of the embodiment.
FIG. 6 is an exemplary flowchart illustrating an example of an operation procedure of the user interface device of the embodiment.
FIG. 7 is a view showing an example of an input image showing a gesture of cursor control;
FIG. 8 is a diagram showing a screen display example.
FIG. 9 is a diagram showing a screen display example.
FIG. 10 is a diagram showing a screen display example.
FIG. 11 is a diagram showing an example of an input image showing a gesture of selection
FIG. 12 is a diagram showing a screen display example.
FIG. 13 is a diagram showing a screen display example.
FIG. 14 is a diagram showing an example of an input image showing a double-click gesture.
FIG. 15 is a diagram showing a screen display example.
FIG. 16 shows a screen display example.
FIG. 17 is a diagram showing a screen display example.
FIG. 18 is a diagram showing a screen display example.
FIG. 19 is a diagram showing a screen display example.
FIG. 20 is a diagram showing a screen display example.
FIG. 21 is a diagram for explaining processing according to a specified recognition engine;
FIG. 22 is a diagram illustrating an example of a description of a recognition engine for each object.
FIG. 23 is a diagram illustrating a configuration example of an interface device according to a second embodiment of the present invention;
FIG. 24 is a diagram showing an example of a description in an active list storage unit when a vertical slider bar is selected.
FIG. 25 is a diagram showing another configuration example of the interface device according to the second embodiment of the present invention.
FIG. 26 is an exemplary flowchart illustrating an example of the operation procedure of the user interface device of the embodiment.
FIG. 27 is a diagram showing still another configuration example of the interface device according to the second embodiment of the present invention;
FIG. 28 is an exemplary flowchart illustrating an example of the operation procedure of the user interface device of the embodiment.
FIG. 29 is an exemplary flowchart illustrating an example of the operation procedure of the user interface device of the embodiment.
[Explanation of symbols]
10. Image input unit
11 image storage unit
12 ... Shape interpreter
13. Interpretation rule storage
14 ... Presentation part
15 Cursor switching unit
101: Light emitting unit
102: reflected light extraction unit
103 timing control unit
20 ... Display device
8 ... housing
30 ... Recognition engine control unit
31, 34: Active list storage unit
32 Body control unit
33 ... Recognition engine storage
121: closest point vertical engine
122: nearest point horizontal engine
142 ... Cursor engine
143… Selection engine
144: Double click engine
151 to 15N: Recognition engine

Claims (6)

A user interface device including a first device that inputs a reflection image as an input image and a second device that performs input for operating an operation target on a display screen based on a recognition processing result of the input image,
The second device comprises:
A means for designating a predetermined recognition method for the operation target for the first device;
The first device comprises:
Means for performing a recognition process corresponding to the recognition method specified by the second device on the input image, and returning a recognition result obtained as a result and the input image to the second device;
A user interface device comprising: A user interface device including a first device that inputs a reflection image as an input image and a second device that performs input for operating a plurality of operation targets on a display screen based on a recognition processing result of the input image. hand,
The second device comprises:
Means for selecting one of a plurality of operation targets displayed on the display screen based on the input image;
For the first device, further comprising means for specifying a predetermined recognition method for the selected operation target,
The first device comprises:
A recognition process corresponding to a recognition method specified by the second device is performed on an input image input after the operation target is selected, and a recognition result obtained as a result and the input image are compared with the second image. Means for returning to the device of
A user interface device comprising: The first device further includes means for requesting the second device to transfer the recognition engine when the first device does not hold a recognition engine required for performing the recognition processing ,
3. The user interface device according to claim 1, wherein the second device further includes a unit configured to transfer a recognition engine requested by the first device to the first device. Each of the first device and the second device holds a plurality of recognition engines that perform different recognition processes, and the same recognition engine is held by the first device and the second device. ,
Each of the first device and the second device,
In one of the first device and the second device, when the same recognition engine is activated, the same recognition engine is deactivated with respect to the other device. Means for notifying,
Means for deactivating the same recognition engine when notified from the one device to deactivate the same recognition engine;
The user interface device according to claim 1 or 2, further comprising: An instruction input method in a user interface device including a first device for inputting a reflection image as an input image and a second device for inputting an operation target on a display screen based on a recognition processing result of the input image And
When the second device specifies a predetermined recognition method for the operation target to the first device, the first device specifies the input image from the second device for the input image. Performing a recognition process corresponding to the recognized recognition method, and returning the obtained recognition result and the input image to the second device. Instructions in a user interface device including a first device that inputs a reflection image as an input image and a second device that performs input for operating a plurality of operation targets on a display screen based on a recognition processing result of the input image An input method,
The second device selects one of a plurality of operation objects displayed on the display screen based on the input image, and executes a recognition method predetermined for the selected operation object. When specified for the first device, the first device performs recognition processing corresponding to the recognition method specified by the second device on an input image input after the operation target is selected. Returning the recognition result and the input image obtained as a result to the second device.

Images