WO2017013805A1 - Input device, input support method and input support program - Google Patents

Abstract

This input device is mounted on an indicator body and is provided with a switch which receives input. The input device detects motion information outputted from a motion sensor. Further, from a first component of the motion information, the input device extracts information relating to a first mode, and, from a second component of said motion information, extracts information relating to a second mode. Thereafter, the input device uses the input pattern inputted to the switch to output information relating to the first mode and/or information relating to the second mode.

Description

Input device, input support method, and input support program

The present invention relates to an input device, an input support method, and an input support program.

In recent years, wearable terminals such as smart watches and smart glasses have been introduced into the market, and new services different from conventional personal computers and smartphones have become widespread. Since such a wearable device is worn and used, it is difficult to input by touching the screen.

For example, as an input to the wearable device, a method of inputting characters by voice is known. There is also known a method in which a sensor is attached to a finger and a simple command is input by a gesture of the finger.

JP 2006-79221 A JP 07-271506 A JP2015-35103A

However, with the above technology, operability is not good when inputting to a wearable device. For example, since a character input or a gesture input is set manually before operating a sensor attached to a finger, there is a lot of trouble and an input error due to a setting error increases. Further, it is difficult to automatically determine whether a character input or a gesture input, and an input different from the user's intention may be executed.

An object of one aspect is to provide an input device, an input support method, and an input support program that can improve operability.

In the first proposal, the input device is provided with a switch for receiving input and is attached to the indicator. The input device extracts information relating to the first mode from a first component of the motion information, a detection unit that detects motion information output from the motion sensor, and a second component of the motion information. And an extraction unit that extracts information on the second mode from the components. The input device includes an output unit that outputs one or both of the information related to the first mode and the information related to the second mode using an input pattern for the switch.

According to one embodiment, operability can be improved.

FIG. 1 is a diagram illustrating an example of the overall configuration of a system according to the first embodiment. FIG. 2A is a diagram illustrating an example of a wearable device. FIG. 2B is a diagram illustrating an example of a wearable device. FIG. 2C is a diagram illustrating an example of a wearable device. FIG. 2D is a diagram illustrating an example of a wearable device. FIG. 3 is a diagram illustrating an example of a head mounted display. FIG. 4 is a functional block diagram of the functional configuration of the system according to the first embodiment. FIG. 5 is a diagram illustrating an example of a rotation axis of a finger. FIG. 6 is a diagram illustrating an output example from the wearable device. FIG. 7 is a diagram illustrating an example of a menu screen. FIG. 8 is a diagram illustrating an example of a display result of a locus of characters input by handwriting. FIG. 9 is a diagram illustrating an example of the result of character recognition. FIG. 10 is a flowchart showing the flow of processing. FIG. 11 is a diagram illustrating the filtering process. FIG. 12 is a diagram illustrating an example of filtering. FIG. 13 is a diagram for explaining a correction example of the trajectory. FIG. 14 is a flowchart for explaining the flow of processing using the direction of the wearable device. FIG. 15 is a diagram for explaining a storage target based on a score. FIG. 16 is a diagram illustrating an example of issuing a gesture ID. FIG. 17 is an explanatory diagram illustrating an example of a computer that executes an input support program.

Hereinafter, embodiments of an input device, an input support method, and an input support program according to the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the embodiments. In addition, the embodiments can be appropriately combined within a consistent range.

[overall structure]
FIG. 1 is a diagram illustrating an example of the overall configuration of a system according to the first embodiment. The input system shown in FIG. 1 includes a wearable device 10, a head mounted display 30, and an input device 50. The wearable device 10, the head mounted display 30, and the input device 50 are communicably connected via a network and can exchange various types of information. As an aspect of such a network, any type of communication network such as mobile communication such as a cellular phone, the Internet (Internet), a LAN (Local Area Network), a VPN (Virtual Private Network), or the like, regardless of wired or wireless. Can be adopted. In the present embodiment, a case where the wearable device 10, the head mounted display 30, and the input device 50 communicate by wireless communication will be described as an example.

The input system is a system that supports user input. For example, the input system is used for user work support in a factory or the like, and is used when the user takes notes or issues commands to other devices. A user may work while moving in various places. For this reason, by enabling various inputs using the wearable device 10 instead of a fixed terminal such as a personal computer, the user can input while moving in various places.

Wearable device 10 is a device that a user wears and uses to detect a user's finger operation and gesture. In the present embodiment, the wearable device 10 is a device worn on a finger. The wearable device 10 detects a change in the posture of the finger and transmits information related to the change in the posture of the finger to the input device 50.

2A to 2C are diagrams illustrating an example of a wearable device. The wearable device 10 has a ring shape like a ring, and can be attached to the finger by passing the finger through the ring as shown in FIG. 2C. In the wearable device 10, a part of the ring is formed thicker and wider than the other part, and is a component built-in part that incorporates main electronic components. Further, the wearable device 10 has a flat inner surface of a ring-containing component part. That is, the wearable device 10 has a shape that easily fits with a finger when the component built-in part is placed above the finger.

As shown in FIG. 2C, the wearable device 10 is attached to the finger in a substantially similar orientation with the component built-in part on the upper side of the finger. Further, as shown in FIG. 2A, the wearable device 10 is provided with a switch 14 on the side surface side of the ring. As shown in FIG. 2C, the switch 14 is disposed at a position corresponding to the thumb when the wearable device 10 is worn on the index finger of the right hand. The wearable device 10 is formed in a shape in which the peripheral portion of the switch 14 is raised to the same height as the upper surface of the switch 14. As a result, the wearable device 10 does not turn on the switch 14 simply by placing a finger on the switch 14 portion.

FIG. 2D is a diagram illustrating an example of an operation on a wearable device switch. The example of FIG. 2D shows a case where the wearable device 10 is attached to the index finger and the switch 14 is operated with the thumb. As shown in the left side of FIG. 2D, the wearable device 10 does not turn on the switch 14 just by placing the thumb on the switch 14, and the switch 14 is turned on by pushing the thumb as shown in the right side of FIG. 2D. . At the start of input, the user places the finger on the input position, and starts input by pushing the finger with the finger when performing input.

The switch 14 is energized so that it is turned on when it is contracted, turned off when it is extended, and stretched by an elastic body such as a built-in spring. As a result, the switch 14 is turned on when it is pressed with a finger, and is turned off when the finger force is released. With this configuration, the wearable device 10 cannot start input unless it is worn in a normal state, and when it is worn on the finger, the wearing position is naturally corrected to a normal position. Further, the user can control the input / non-input section without releasing the finger from the switch 14.

Referring back to FIG. 1, the head mounted display 30 is a device that is worn by the user on the head and displays various types of information so as to be visible to the user. The head mounted display 30 may be compatible with both eyes or may be compatible with only one eye.

FIG. 3 is a diagram showing an example of a head mounted display. In the present embodiment, the head mounted display 30 has a glasses shape corresponding to both eyes. The head-mounted display 30 has transparency in the lens portion so that an external real environment can be visually recognized even when the user is wearing the head-mounted display 30. In addition, the head mounted display 30 includes a display unit having transparency in a part of the lens portion, and various types of information such as images can be displayed on the display unit. As a result, the head mounted display 30 realizes augmented reality in which the actual environment is expanded by visually recognizing various information in a part of the field of view while allowing the wearing user to visually recognize the actual environment. FIG. 3 schematically shows a display unit 30B provided in a part of the field of view 30A of the user who wears it.

The head mounted display 30 has a built-in camera between two lens portions, and the camera can shoot an image in the line of sight of the user wearing the head mounted display 30.

Referring back to FIG. 1, the input device 50 is a device that supports various inputs by user's finger operation. The input device 50 is a portable information processing device such as a smartphone or a tablet terminal. The input device 50 may be implemented as one or a plurality of computers provided in a data center or the like. That is, the input device 50 may be a cloud computer as long as it can communicate with the wearable device 10 and the head mounted display 30.

Such an input system detects motion information output from a motion sensor included in the wearable device 10. The input system extracts information related to the first mode from the first component of the exercise information, and extracts information related to the second mode from the second component of the exercise information. Thereafter, the input system outputs one or both of the information related to the first mode and the information related to the second mode using the input pattern for the switch 14.

That is, the input system extracts a character trajectory and a gesture from finger movement information output from a motion sensor worn on the finger, and outputs both or one according to the on / off state of the switch 14. As described above, the input system can simultaneously process character recognition and gesture events, and can improve the operability of input using the wearable device 10.

[Configuration of each device]
Next, the functional configuration of the input system shown in FIG. 1 will be described. FIG. 4 is a functional block diagram of the functional configuration of the system according to the first embodiment. Here, functional configurations of the wearable device 10, the head mounted display 30, and the input device 50 will be described.

(Functional configuration of wearable device)
As illustrated in FIG. 4, the wearable device 10 includes a wireless unit 11, a storage unit 12, an attitude sensor 13, a switch 14, and a control unit 20. The wearable device 10 may have other devices other than the above devices.

The wireless unit 11 is an interface that performs wireless communication control with other devices. As the wireless unit 11, a network interface card such as a wireless chip can be adopted. The wireless unit 11 is a device that communicates wirelessly, and transmits and receives various information to and from other devices wirelessly. For example, the wireless unit 11 transmits finger operation information and posture change information to the input device 50.

The storage unit 12 is a storage device that stores programs executed by the control unit 20 and various types of information, such as a hard disk or a memory. For example, the storage unit 12 stores various information generated by the control unit 20, intermediate data of processing executed by the control unit 20, and the like.

The posture sensor 13 is a device that detects a user's finger operation. For example, the attitude sensor 13 is a triaxial gyro sensor or the like. As shown in FIG. 2C, the posture sensor 13 is built in the wearable device 10 so that the three axes correspond to the rotation axis of the finger when the wearable device 10 is correctly attached to the finger. FIG. 5 is a diagram illustrating an example of a rotation axis of a finger. In the example of FIG. 5, three axes of X, Y, and Z that are orthogonal to each other are shown. In the example of FIG. 5, the rotation axis in the movement direction for bending the finger is the Y axis, the rotation axis in the movement direction for swinging the finger left and right is the Z axis, and the rotation axis in the movement direction for turning the finger is the X axis. Yes. The posture sensor 13 detects the rotation of each of the X, Y, and Z rotation axes in accordance with control from the control unit 20, and uses the detected three-axis rotation as posture change information indicating a posture change of the finger. 20 output.

The switch 14 is a device that accepts input from the user. The switch 14 is provided on the side of the ring of the wearable device 10 as shown in FIG. 2C. The switch 14 is turned on when pressed, and turned off when released. The switch 14 receives an operation input from the user. For example, when the wearable device 10 is worn on the user's index finger, the switch 14 receives an operation input by the user's thumb. The switch 14 outputs operation information indicating the received operation content to the control unit 20. The user performs various inputs by operating the switch 14. For example, the user turns on the switch 14 when starting input by finger operation.

The control unit 20 is a device that controls the wearable device 10. As the control unit 20, an integrated circuit such as a microcomputer, an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array) can be employed. The control unit 20 transmits operation information of the switch 14 to the input device 50 via the wireless unit 11. Further, when the switch 14 is turned on, the control unit 20 controls the posture sensor 13 to detect a posture change. The control unit 20 transmits posture change information detected by the posture sensor 13 to the input device 50 via the wireless unit 11.

Such a control unit 20 includes a detection unit 21, an extraction unit 22, a locus generation unit 23, a gesture recognition unit 24, and an output unit 25.

The detection unit 21 is a processing unit that detects a sensor value output from the attitude sensor 13, that is, motion information. Specifically, the detection unit 21 sets the position at the time when the switch 14 is turned on as a reference, and continues to detect the movement history from the reference until the switch 14 turns from on to off, and the detected movement. The history is output to the extraction unit 22 as exercise information. In addition, as a movement history, a coordinate and a triaxial gyroscope can also be used, and the information which combined a coordinate and a triaxial gyroscope can also be used. Moreover, the detection part 21 can also output the stop time etc. which a motion stopped, matched with the applicable location of exercise | movement information, when the switch 14 was turned on, when it was turned off.

The extraction unit 22 is a processing unit that extracts various types of information to be output from the exercise information detected by the detection unit 21. Specifically, the extraction unit 22 is a processing unit that extracts information about a trajectory characterizing a character and information about a gesture performed by a user.

For example, the extraction unit 22 extracts a series of exercise information from the exercise information received from the detection unit 21 until the switch 14 is turned off to output it to the locus generation unit 23. That is, the extraction unit 22 extracts a series of movement information about the movement of the finger in one segment until the switch 14 is turned on.

Further, the extraction unit 22 extracts a period from the start to the end of the finger movement from the exercise information received from the detection unit 21, extracts the exercise information within each period, and outputs it to the gesture recognition unit 24. That is, the extraction unit 22 extracts a plurality of segments, and extracts movement information that specifies a finger operation in each segment.

The trajectory generation unit 23 is a processing unit that generates a movement trajectory of the finger from a series of motion information until the switch 14 extracted by the extraction unit 22 is turned on. For example, in the exercise information received from the extraction unit 22, the trajectory generation unit 23 sets the position where the switch 14 is turned on as a reference, and sets the position where the switch 14 is turned off as the end position. Then, the trajectory generation unit 23 sets the reference to the start of the trajectory, and generates a trajectory connecting from the start to the end position. Thereafter, the trajectory generation unit 23 stores the generated trajectory information in the storage unit 12 in association with the generation time. In addition, the trajectory generation unit 23 outputs the generated trajectory information to the output unit 25.

The gesture recognition unit 24 is a processing unit that recognizes a gesture performed by a user using a finger from the exercise information extracted by the extraction unit 22. For example, the gesture recognition unit 24 sets the position where the switch 14 is turned on as a reference, and sets the position where the finger first stops for a predetermined time as the first end position. Then, the gesture recognition unit 24 identifies the trajectory from the reference to the first end position as the first trajectory.

Subsequently, the gesture recognition unit 24 sets the first end position as the second start position, and then sets the position where the finger has stopped for a predetermined time as the second end position. Then, the gesture recognition unit 24 specifies a trajectory from the second start position to the second end position as the second trajectory. As described above, the gesture recognition unit 24 identifies each period from the start to the end of the exercise, and generates a trajectory for each period. Then, the gesture recognition unit 24 identifies each trajectory as a gesture, and stores the identified gesture and time in the storage unit 12 in association with each other. In addition, the gesture recognition unit 24 outputs each trajectory to the output unit 25 as gesture information.

The output unit 25 is a processing unit that transmits various types of information generated by the control unit 20 to the input device 50. For example, the output unit 25 transmits the trajectory information generated by the trajectory generation unit 23 and the gesture information generated by the gesture recognition unit 24 to the input device 50. The output unit 25 also transmits on / off information of the switch 14 to the input device 50.

Here, the trajectory information and the gesture information generated from the finger operation (movement information) when the wearable device 10 detects a series of finger operations will be described. FIG. 6 is a diagram illustrating an output example from the wearable device.

As shown in (1) of FIG. 6, the detection unit 21 detects that the user's finger motion starts from a, pauses at b, and further moves to c, and this a → b → The motion of c is extracted as exercise information. Subsequently, the extraction unit 22 extracts a series of operations of a → b → c with reference to a, and outputs the extracted sequence to the trajectory generation unit 23. Further, the extraction unit 22 extracts the a → b trajectory and the b → c trajectory, and outputs them to the gesture recognition unit 24.

And the locus | trajectory production | generation part 23 specifies a user's finger movement as a locus | trajectory of "0" from the exercise | movement information of a-> b-> c, as shown to (2) of FIG. On the other hand, the gesture recognizing unit 24 specifies the a → b gesture and the b → c gesture. Such a gesture corresponds to an operation of turning a dial as shown in (3) of FIG. In the example of FIG. 6, the gesture of a → b corresponds to the operation of turning the dial “0”, and the gesture of b → c corresponds to the operation of turning the dial “1”. Then, the output unit 25 transmits operation information including “0” trajectory information shown in (2) of FIG. 6 and gesture information indicating dial rotation shown in (3) of FIG. 6 to the input device 50. .

(Functional configuration of head-mounted display)
As illustrated in FIG. 4, the head mounted display 30 includes a wireless unit 31, a camera 32, a display unit 33, and a control unit 34. The head mounted display 30 may have other devices other than the above devices.

The wireless unit 31 is a device that performs wireless communication. The wireless unit 31 transmits and receives various types of information to and from other devices wirelessly. For example, the wireless unit 31 receives image information of an image to be displayed on the display unit 33 and an operation command instructing shooting from the input device 50. In addition, the wireless unit 31 transmits image information of an image captured by the camera 32 to the input device 50.

The camera 32 is a device that captures an image. As shown in FIG. 3, the camera 32 is provided between two lens portions. The camera 32 captures an image under the control of the control unit 34.

The display unit 33 is a device that displays various types of information. As shown in FIG. 3, the display unit 33 is provided in the lens portion of the head mounted display 30. The display unit 33 displays various information. For example, the display unit 33 displays a menu screen described later, a virtual laser pointer, an input locus, and the like.

The control unit 34 is a device that controls the head mounted display 30. As the control unit 34, an electronic circuit such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit), or an integrated circuit such as a microcomputer, ASIC, or FPGA can be employed. The control unit 34 performs control to display the image information received from the input device 50 on the display unit 33. In addition, when receiving an operation command for instructing photographing from the input device 50, the control unit 34 controls the camera 32 to photograph an image. Then, the control unit 34 controls the wireless unit 31 to transmit the image information of the captured image to the input device 50.

(Functional configuration of input device)
As illustrated in FIG. 4, the input device 50 includes a wireless unit 51, a storage unit 52, and a control unit 60. Note that the input device 50 may include devices other than the above-described devices.

The wireless unit 51 is a device that performs wireless communication. The wireless unit 51 transmits / receives various information to / from other devices wirelessly. For example, the wireless unit 51 receives trajectory information, posture change information, and the like from the wearable device 10. In addition, the wireless unit 51 transmits image information of an image to be displayed on the head mounted display 30 and various operation commands to the head mounted display 30. In addition, the wireless unit 51 receives image information of an image captured by the camera 32 of the head mounted display 30.

The storage unit 52 is a storage device such as a hard disk, an SSD (Solid State Drive), or an optical disk. The storage unit 52 may be a semiconductor memory capable of rewriting data such as RAM (Random Access Memory), flash memory, NVSRAM (Non Volatile Static Random Access Memory).

Further, the storage unit 52 stores an OS (Operating System) and various programs executed by the control unit 60. For example, the storage unit 52 stores various programs used for input support. Further, the storage unit 52 stores various data used in programs executed by the control unit 60. For example, the storage unit 52 stores recognition dictionary data 53, memo information 54, and image information 55.

The recognition dictionary data 53 is dictionary data for recognizing handwritten characters. For example, the recognition dictionary data 53 stores standard trajectory information of various characters. The memo information 54 is data storing information input by handwriting. For example, in the memo information 54, an image of a character input by handwriting and character information obtained as a result of recognizing the character input by handwriting are stored in association with each other. The image information 55 is image information of an image taken by the camera 32 of the head mounted display 30.

The control unit 60 is a device that controls the input device 50. As the control unit 60, an electronic circuit such as a CPU or MPU, or an integrated circuit such as a microcomputer, ASIC, or FPGA can be employed. The control unit 60 has an internal memory for storing programs defining various processing procedures and control data, and executes various processes using these.

The control unit 60 functions as various processing units by operating various programs. For example, the control unit 60 includes an input detection unit 61, a display control unit 62, a calibration unit 63, an axis detection unit 64, a gesture acquisition unit 65, an event specification unit 66, a character recognition unit 67, and a recognition unit. A result display unit 68 and an operation command execution unit 69 are provided.

The input detection unit 61 detects various inputs based on operation information and posture change information received from the wearable device 10. For example, the input detection unit 61 detects an operation on the switch 14 based on the operation information. For example, the input detection unit 61 detects single click, double click, triple click, and long press operation of the switch 14 from the number of times the switch 14 is pressed within a predetermined time. Further, the input detection unit 61 identifies the rotation of the three axes and the movement to each axis from the posture change information received from the wearable device 10, and detects the posture change of the finger.

The display control unit 62 performs various display controls. For example, the display control unit 62 generates image information of various screens according to the detection result by the input detection unit 61, and transmits the generated image information to the head mounted display 30 via the wireless unit 51. As a result, an image of image information is displayed on the display unit 33 of the head mounted display 30. For example, when a double click is detected by the input detection unit 61, the display control unit 62 displays a menu screen on the display unit 33 of the head mounted display 30.

FIG. 7 is a diagram showing an example of the menu screen. As shown in FIG. 7, the menu screen 70 displays items of “1 calibration”, “2 memo input”, “3 memo browsing”, and “4 shooting”. The item “1 calibration” designates a calibration mode for calibrating the posture information of the detected finger. The item “2 Memo Input” specifies a memo input mode for inputting a memo by handwriting. The item “3 Browsing memos” designates a browsing mode for browsing inputted memos. The item “4 shooting” is for designating a shooting mode in which an image is shot by the camera 32 of the head mounted display 30.

For example, the input device 50 according to the present embodiment is capable of selecting an item on the menu screen 70 by handwriting input, a cursor, a finger gesture, or the like. For example, when the handwritten input locus is recognized as a number “1” to “4”, the display control unit 62 determines that the item mode corresponding to the recognized number has been selected. Further, the display control unit 62 displays a cursor on the screen, and moves the cursor in accordance with the finger posture change detected by the input detection unit 61. For example, when the rotation of the Y axis is detected, the display control unit 62 moves the cursor in the horizontal direction of the screen at a speed corresponding to the rotation. Further, when the rotation of the Z axis is detected, the display control unit 62 moves the cursor in the vertical direction of the screen at a speed corresponding to the rotation. When the cursor is positioned on any item on the menu screen 70 and a single click is detected by the input detection unit 61, the display control unit 62 determines that the mode of the item on which the cursor is positioned has been selected. When any item on the menu screen 70 is selected, the display control unit 62 deletes the menu screen 70.

The calibration unit 63 calibrates the detected posture information of the finger. For example, when the calibration mode is selected on the menu screen 70, the calibration unit 63 calibrates the detected finger posture information.

Here, the wearable device 10 may be worn in a misaligned state rotated in the circumferential direction with respect to the finger. When the wearable device 10 is worn in a shifted state with respect to a finger, there is a case where the posture change detected by the wearable device 10 is shifted by a rotation amount, and the detected motion may be different from the user's intention. In such a case, the user selects the calibration mode on the menu screen 70. When the user selects the calibration mode on the menu screen 70, the user opens and closes the hand on which the wearable device 10 is worn. The wearable device 10 transmits posture change information of the posture change of the finger when the hand is opened and closed to the input device 50.

For example, the calibration unit 63 detects the movement of the finger when the finger wearing the wearable device 10 is bent and stretched by opening and closing the hand based on the posture change information. The calibration unit 63 calibrates the reference direction of the finger movement based on the detected finger movement.

The axis detection unit 64 detects the axis indicating the posture based on the posture change of the finger detected by the input detection unit 61. For example, the axis detection unit 64 detects an axis that moves in the direction according to a change in the posture of the finger. For example, the axis detector 64 passes through the origin in a three-dimensional space, and moves in the X, Y, and Z directions according to the respective rotation directions and rotation speeds of the X, Y, and Z rotation axes. The direction vector of is calculated. In addition, when detecting a motion only by a posture, it is difficult to move the wrist as far as it goes away from the facing direction. Further, when the palm is horizontal, the degree of freedom in the vertical direction is high, but the degree of freedom in the horizontal direction may be low.

Therefore, the axis detection unit 64 may change the vertical and left pointing sensitivity from the center point of the axis direction corrected by the calibration unit 63. For example, the axis detection unit 64 calculates the axis direction vector by correcting the rotation in the left-right direction of the hand to be larger than the rotation in the up-down direction of the hand. That is, when the rotation amount is the same, the axis detection unit 64 corrects the movement amount due to the rotation in the left-right direction larger than the movement amount due to the rotation in the vertical direction. Further, the axis detection unit 64 may increase the sensitivity as the distance from the center point in the corrected axis direction increases. For example, the axis detection unit 64 calculates the axis direction vector by correcting the rotation to be greater as the distance from the center point in the axis direction increases. That is, when the rotation amount is the same, the axis detection unit 64 corrects the movement amount due to the rotation in the peripheral portion away from the center point in the axis direction larger than the movement amount due to the rotation near the center point. Thereby, since the sensitivity of rotation is set corresponding to the ease of movement of the wrist, accurate pointing can be facilitated.

The control unit 60 can also display a virtual laser pointer linked to the axis detected by the axis detection unit 64 on the display unit 33 of the head mounted display 30. For example, when the calibration mode is selected on the menu screen 70, the control unit 60 generates image information on a screen on which a virtual laser pointer linked to the axis detected by the axis detection unit 64 is arranged. Then, the control unit 60 controls the wireless unit 51 to transmit the generated image information to the head mounted display 30. As a result, a virtual laser pointer image is displayed on the display unit 33 of the head mounted display 30.

The gesture acquisition unit 65 is a processing unit that acquires gesture information from the wearable device 10. For example, the gesture acquisition unit 65 receives gesture information including a locus recognized by the wearable device 10 and outputs the gesture information to the event identification unit 66. When the gesture acquisition unit 65 receives a gesture identifier or the like for identifying a gesture from the wearable device 10, the gesture acquisition unit 65 outputs the received gesture identifier to the event specifying unit 66.

The event identification unit 66 is a processing unit that identifies an event corresponding to a gesture. For example, the event specifying unit 66 stores the gesture information or the gesture identifier and the gesture content in advance in association with each other. And the event specific | specification part 66 will specify the content of the gesture which the user performed according to the matching memorize | stored beforehand, if gesture information or a gesture identifier is received from the gesture acquisition part 65. FIG. Thereafter, the event specifying unit 66 outputs the specified gesture content to the operation command executing unit 69.

The character recognition unit 67 is a processing unit that executes character recognition in accordance with the trajectory information acquired from the wearable device 10. For example, the character recognition unit 67 compares the acquired trajectory with the standard trajectories of various characters stored in the recognition dictionary data 53, specifies the character with the highest similarity, and sets the character code of the specified character. Output. Further, the character recognition unit 67 outputs the recognized character and character code to the operation command execution unit 69.

In addition, if the recognized character is repelled, the character recognition unit 67 can perform character recognition again after correcting the locus specified from the locus information or the result of character recognition. For example, when there is “upper left splash” that is not frequently used in general Chinese characters, the character recognition unit 67 can also perform character recognition by deleting the locus corresponding to the hit.

Here, when the user performs handwritten input of characters, the user can also perform input by long pressing the switch 14 for each character. That is, the switch 14 is released once for each character. The character recognizing unit 67 can also record a handwritten input locus character by character and recognize characters from the locus character by character.

FIG. 8 is a diagram illustrating an example of a display result of a locus of characters input by handwriting. FIG. 8A shows an example in which “bird” is input by handwriting. FIG. 8B is an example in which “God” is input by handwriting. As shown in FIGS. 8A to 8B, the controller 60 makes it easy to recognize the character indicated by the trajectory by displaying the trajectory moving to the upper left in a lightly distinct manner. In the example of FIG. 8, a thin line portion is indicated by a broken line. The character recognition unit 67 performs character recognition on the trajectory indicated by the dark line in FIGS. 8A to 8B.

FIG. 9 is a diagram showing an example of the result of character recognition. In the example of FIG. 9, a thin line portion is indicated by a broken line. FIG. 9 shows the result of character recognition for “Tu” input by handwriting. FIG. 9 shows a candidate character and a score (score) indicating the degree of similarity when the character is recognized without deleting the upper left locus moved to the upper left and when the character is recognized with the upper left locus deleted. Has been. Candidate characters are shown after “code:”. The score indicates the degree of similarity. The larger the value, the higher the degree of similarity. For example, “Mori” input by handwriting has a score of 920 when the character is recognized without deleting the upper left locus, and the score is 928 when the character is recognized by deleting the upper left locus, and the upper left locus is deleted. If you do, the score will be higher. Moreover, erroneous conversion can be suppressed by deleting the upper left locus. For example, “Tu” entered by handwriting has “Ream”, “Slow”, and “Han” as the recognition candidate characters. However, deleting the upper left trajectory will increase the score, thus suppressing erroneous conversion. it can.

Also, the character recognition unit 67 stores the locus, recognition result, character code, recognition execution time, and the like in the storage unit 52 in association with each other. For example, the character recognition unit 67 stores the trajectory of the handwritten character and the recognized character in the memo information 54. For example, when the memo input mode is selected on the menu screen 70, the character recognizing unit 67 associates the trajectory information with the recognized character and stores it in the memo information 54 together with the date / time information. Information stored in the memo information 54 can be referred to. For example, when the memo input mode is selected on the menu screen 70, the character recognition unit 67 displays information stored in the memo information 54 of the storage unit 52.

For example, the character recognizing unit 67 displays the input date and time of memo input, the text of a text recognizing a handwritten character, and the text of a handwritten character image in association with each other. Thus, the user can confirm whether the handwritten character is correctly recognized by displaying the sentence of the text and the sentence of the handwritten character image in association with each other. In addition, by displaying a text sentence and a sentence based on a handwritten character image in association with each other, the user can refer to the corresponding character image even when the character is erroneously converted by the recognition of the trajectory. You can grasp the handwritten characters. In addition, the handwriting characteristics of the user are recorded in the handwritten character image. For this reason, the image of the handwritten character can also be stored and used for proof that the user has input, for example, like a signature.

The recognition result display unit 68 is a processing unit that displays the result of character recognition recognized by the character recognition unit 67. For example, when the character recognized by the character recognition unit 67 is repelled and corrected, the recognition result display unit 68 displays the recognition result before correction and the recognition result after correction on the head mounted display 30 or the like. In this way, the user can be inquired about the correctness of the recognition result.

The operation command execution unit 69 outputs an operation command to another device based on the recognized character or symbol, gesture or the like. For example, when shooting with the camera 32 of the head mounted display 30, the user selects a shooting mode on the menu screen 70. Then, the user presses and operates the switch 14 of the wearable device 10 at a timing when photographing is desired, and inputs a predetermined character by handwriting. The predetermined character may be any of a character, a number, and a symbol, for example, “1”.

When the shooting mode is selected on the menu screen 70, the operation command execution unit 69 is ready for shooting. The character recognizing unit 67 records a trajectory input by handwriting in a shooting preparation state, and recognizes a character from the trajectory. When the character recognition unit 67 recognizes a predetermined character, the operation command execution unit 69 transmits an operation command for instructing shooting to the head mounted display 30. When the head mounted display 30 receives an operation command for instructing photographing, the head mounted display 30 performs photographing by the camera 32 and transmits image information of the photographed image to the input device 50. The control unit 60 stores the image information of the captured image in the storage unit 52 as the image information 55.

Also, the operation command execution unit 69 can execute a process corresponding to the gesture specified by the event specification unit 66. For example, the operation command execution unit 69 can hold a correspondence table in which the gesture information and the command are associated with each other in the storage unit 52 and issue the operation command corresponding to the gesture according to the correspondence table. In addition to the gesture information, a gesture ID described later can be used. As described above, the input device 50 can perform an operation by outputting an operation command to another device.

[Process flow]
FIG. 10 is a flowchart showing the flow of processing. As shown in FIG. 10, the wearable device 10 is turned on (S101: Yes), and the detection unit 21 of the wearable device 10 turns on the fingertip via the posture sensor 13 when the switch 14 is turned on (S102: Yes). Is detected (S103).

Subsequently, the detection unit 21 extracts the motion component of the finger operation (S104). For example, the detection unit 21 detects a sensor value output from the posture sensor 13, that is, motion information.

And the extraction part 22 extracts the exercise | movement pattern after exercise | movement started from the extracted exercise | movement component (S105). The trajectory generation unit 23 generates finger movement transitions, that is, trajectory information, using a series of motion information from the on-off state to the switch-off state among the extracted motion patterns (S106). In parallel with S106, the gesture recognizing unit 24 uses the movement information for each period from the start to the end of the finger movement among the extracted movement patterns to generate a finger operation, that is, gesture information (S107). The output unit 25 outputs the gesture information to the input device 50 (S108). Thereafter, the gesture acquisition unit 65 of the input device 50 recognizes the gesture by the finger operation, and the operation command execution unit 69 executes the operation corresponding to the recognized gesture.

And while the switch 14 is on (S109: No), the steps after S103 are repeated. On the other hand, when the switch 14 is turned off (S109: Yes), the input detection unit 61 of the input device 50 identifies the state change of the switch 14 according to the information received from the wearable device 10 (S110). Thereafter, the input detection unit 61 generates an event for specifying an operation according to the state change of the switch 14 (S111). Then, the display control unit 62 and the operation command execution unit 69 output an operation corresponding to the identified event to the corresponding device to execute the operation (S112).

In parallel with the processing after S110, the character recognizing unit 67 specifies the locus operated by the finger using the locus information acquired from the wearable device 10 (S113). Thereafter, the character recognition unit 67 performs character recognition using the identified locus and the recognition dictionary data 53 (S114). In parallel with S114, the character recognition unit 67 corrects the identified locus or the result of character recognition (S115). Then, the recognition result display unit 68 displays information such as character recognition results, correction results, and trajectories on the head mounted display 30 and stores them in the memo information 54 (S116). Thereafter, the operation command execution unit 69 outputs an operation command corresponding to the recognition result (S117).

[effect]
As described above, the wearable device 10 and the input device 50 can simultaneously process a gesture event and character recognition, and output a character input, a gesture event, or both depending on the result. As a result, the input operation can be executed without manually switching the input, so that the user's trouble can be omitted and setting mistakes can be suppressed. Therefore, operability can be improved.

Although the above embodiment has been described with respect to the present invention, the present invention is not limited to this. Therefore, in the second embodiment, processing different from that in the first embodiment will be described.

[filtering]
The input device 50 can improve the character recognition accuracy by filtering the motion information using the cut-off frequency and suppressing the body shake component. Specifically, when the switch 14 is turned from on to off, the character recognition unit 67 of the input device 50 performs a finger movement pattern based on a result of filtering exercise information according to a cutoff frequency corresponding to a preset character type. And the trajectory information can be extracted from the extracted motion pattern.

For example, the character recognition unit 67 holds a correspondence table in which the character type and the cut-off frequency (fc (Hz)) are associated with each other in the storage unit 52 or the like. For example, “Default, 3”, “Number 2”, “Kana, 0.5”, “Kana 2,” “Alphabet, 1.5”, “Kanji, 0.2” as “Type, fc” Hold in association. Although the setting here is an example, fc is set high for numbers with short input times, and fc is set low for kanji characters with long input times.

Then, the character recognition unit 67 accepts the setting of the character recognition target from the user in advance and determines fc according to the correspondence table. In this state, when the character recognition unit 67 acquires the exercise information from the wearable device 10, the character recognition unit 67 filters the input exercise information using the determined fc, and executes character recognition from the filtering result.

FIG. 11 is a diagram for explaining the filtering process. As shown in FIG. 11, the character recognition unit 67 sets a character type for recognizing an input character (S201). Subsequently, the character recognition unit 67 searches the correspondence table (table) between the character type and fc for the fc corresponding to the set character type (S202). Here, when there are a plurality of set character types, the character recognition unit 67 determines the lowest fc as the setting target (S203). Then, the character recognition unit 67 sets the cutoff frequency fc corresponding to the input character type in the high pass filter (HPF) (S204).

After that, when receiving the motion information represented by the frequency from the wearable device 10, the character recognition unit 67 passes the low-pass filter (LPF) and performs spike noise cut. After that, the character recognition unit 67 passes the high-pass filter in which the cutoff frequency fc is set, suppresses the shake component of the body, and performs character recognition using the result.

FIG. 12 is a diagram for explaining an example of filtering. FIG. 12 is a diagram for explaining filtering in the HPF. As shown in FIG. 12, the character recognition unit 67 changes the cutoff frequency fc in accordance with the input target, that is, the character type to be determined. For example, the character recognizing unit 67 executes character recognition mainly based on a slow motion component mainly of body shaking by setting fc low for numbers and the like having a short input time. On the other hand, the character recognizing unit 67 executes character recognition based on a fast motion component mainly composed of handwritten components by setting fc high for kanji characters having a long input time.

In this way, the accuracy of character recognition can be improved, and the accuracy of operation events caused by finger movements can also be improved.

[Correction of locus]
By the way, with respect to the operation of the finger, there are a direction in which movement is easy and a direction in which movement is difficult. For example, when the wearable device 10 is attached to the index finger of the right hand and only the finger is moved without moving the wrist, the right direction is generally easier to move than the left direction, and the lower direction is moved more than the upper direction. Cheap. Therefore, in the second embodiment, an example will be described in which the trajectory is correctly recognized by changing the gain according to the direction.

FIG. 13 is a diagram for explaining an example of correction of a trajectory. As shown in FIG. 13, the posture at which the switch 14 is turned on is set as a posture 0 point, and the estimated value of the posture change of the finger from the posture 0 point calculated based on the data source after extracting the motion component of the finger operation Set the gain to convert to length. If this gain is larger, a long trajectory can be written with a slight change in posture. For example, the trajectory generation gain change profile is changed according to the posture change direction from the posture 0 point, the trajectory generation gain change rate is changed according to the posture change amount from the posture 0 point, and the gain C of the posture 0 point is matched. A continuous function satisfying the above is assumed.

By doing so, the character recognition unit 67 increases the gain with respect to the + Y-axis posture change with respect to the trajectory in the vertical axis direction, and increases the gain with respect to the −Z-axis posture change with respect to the trajectory in the horizontal axis direction. To do. As a result, a trajectory in a direction that is difficult to move can be processed in the same way as a trajectory in a direction that is easy to move, so that the trajectory can be accurately recognized.

[Consideration of gravity direction]
In Example 1, although the example which performs both a locus | trajectory (movement transition) and gesture recognition was demonstrated, it is not limited to this. Therefore, here, an example in which either one is output based on the posture when the switch 14 is turned on will be described.

FIG. 14 is a flowchart for explaining the flow of processing using the direction of the wearable device. As shown in FIG. 14, the power of the wearable device 10 is turned on (S301: Yes), and when the switch 14 is turned on (S302: Yes), the detection unit 21 of the wearable device 10 passes through the attitude sensor 13. It is specified whether the gravity axis direction when the switch 14 is turned on is the Y axis or the X axis, and the specified result is substituted into the variable “G” (S303). Subsequently, the detection unit 21 detects the movement of the fingertip via the posture sensor 13 (S304).

Subsequently, the detection unit 21 extracts the motion component of the finger operation (S305). And the extraction part 22 extracts the exercise | movement pattern after exercise | movement started from the extracted exercise | movement component (S306).

Thereafter, when G is not the Y axis but the X axis (S307: No), the trajectory generating unit 23 uses a series of motion information from the on to the off of the extracted motion pattern. Then, finger movement transition, that is, trajectory information is generated (S308).

On the other hand, when G is the Y axis (S307: Yes), the gesture recognition unit 24 uses the motion information for each period from the start to the end of the finger motion among the extracted motion patterns, The gesture information is generated (S309), and the output unit 25 outputs the gesture information to the input device 50 (S310).

Since the subsequent processes from S311 to S319 are the same as the processes from S109 to S117 described in FIG. 10, detailed description thereof will be omitted. However, in the case of FIG. 14, by considering the direction (value) of G, it is possible to determine in which axial direction the click operation is performed, such as a double click in the X-axis direction. Therefore, even when compared with FIG. 10, it is possible to subdivide the events caused by the operation of the switch 14 and execute more detailed command issuance.

[To be saved]
In the first embodiment, the example in which the character recognition result is stored has been described. However, the present invention is not limited to this, and the storage target can be dynamically changed according to the correction result. FIG. 15 is a diagram for explaining a storage target based on a score. As shown in FIG. 15, the character recognition unit 67 recognizes “dimensions” according to the trajectory information received from the wearable device 10 (S <b> 1). Then, the character recognition unit 67 calculates a score value obtained by comparing the recognition result and the dictionary as 250 points.

On the other hand, the character recognition unit 67 executes character recognition after removing the upper left splash from the trajectory information received from the wearable device 10 (S2). Then, the character recognition unit 67 calculates a score value obtained by comparing the corrected recognition result with the dictionary as 200 points.

Then, the character recognition unit 67 compares both score values (S3). Here, the character recognition unit 67 stores the corrected recognition result in the memo information 54 if the corrected score value is higher. On the other hand, if the score value before correction is higher, the character recognition unit 67 associates both recognition results and stores them in the memo information 54. By doing in this way, the correction method and the score value calculation method can be learned, and the recognition accuracy can be improved.

[Gesture output]
In the first embodiment, the wearable device 10 has described an example in which the gesture information including the recognized gesture is output to the input device 50, but the present invention is not limited to this. For example, the wearable device 10 can also output to the input device 50 a gesture ID associated with “rotation direction, number”.

FIG. 16 is a diagram showing an example of issuing a gesture ID. As illustrated in FIG. 16, when the gesture recognition unit 24 detects one rotation counterclockwise after the switch 14 is turned on, the gesture recognition unit 24 outputs a gesture ID (R, 1) to the input device 50. Subsequently, when the gesture recognition unit 24 further detects one rotation (second rotation) counterclockwise, the gesture recognition unit 24 outputs the gesture ID (R, 2) to the input device 50. Subsequently, when the gesture recognition unit 24 further detects one rotation (third rotation) counterclockwise, the gesture recognition unit 24 outputs the gesture ID (R, 3) to the input device 50.

As described above, the gesture recognition unit 24 issues a gesture ID every time it passes through a predetermined determination zone, and repeats until the switch 14 is turned off. The input device 50 can also specify a gesture according to the gesture ID. Further, the input device 50 can output this gesture ID to the application, and the application can specify the gesture.

The embodiments of the present invention have been described so far, but the present invention may be implemented in various different forms other than the above-described embodiments.

[Wearable device]
In the above-described embodiment, the example in which the wearable device 10 and the input device 50 are executed by different devices has been described. However, the present invention is not limited to this, and only the wearable device 10 having the same processing unit as the input device 50 is realized. You can also In addition, the processing units distributed to the wearable device 10 and the input device 50 can be arbitrarily changed according to the processing capability of the wearable device 10.

[system]
In addition, each component of each part illustrated does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each unit is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed / integrated in arbitrary units according to various loads or usage conditions. Can be configured. Furthermore, various processing functions performed by each device may be executed entirely or arbitrarily on a CPU (or a microcomputer such as an MPU or MCU (Micro Controller Unit)). In addition, various processing functions may be executed in whole or in any part on a program that is analyzed and executed by a CPU (or a microcomputer such as an MPU or MCU) or on hardware based on wired logic. Needless to say, it is good.

[hardware]
The various processes described in the above embodiments can be realized by executing a prepared program on a computer. Therefore, in the following, an example of a computer that executes a program having the same function as in the above embodiment will be described. FIG. 17 is an explanatory diagram illustrating an example of a computer that executes an input support program.

As shown in FIG. 17, the computer 300 includes a CPU 310, an HDD (Hard Disk Drive) 320, and a RAM (Random Access Memory) 340. These units 310 to 340 are connected via a bus 400.

The HDD 320 includes an input support program 320 a that exhibits the same functions as the detection unit 21, the extraction unit 22, the trajectory generation unit 23, the gesture recognition unit 24, and the output unit 25 of the wearable device 10, or the input detection unit 61 of the input device 50. A display control unit 62, a calibration unit 63, an axis detection unit 64, a gesture acquisition unit 65, an event identification unit 66, a character recognition unit 67, a recognition result display unit 68, and an operation command execution unit 69. An input program 320a that exhibits the same function is stored in advance. Note that the input support program 320a may be separated as appropriate. The HDD 320 stores various information.

Then, the CPU 310 reads out and executes the input support program 320a from the HDD 320, thereby executing the same operation as each processing unit of the embodiment. That is, the input support program 320a is similar in operation to the detection unit 21, the extraction unit 22, the trajectory generation unit 23, the gesture recognition unit 24, and the output unit 25, or the input detection unit 61, the display control unit 62, and the calibration. The same operations as the unit 63, the axis detection unit 64, the gesture acquisition unit 65, the event identification unit 66, the character recognition unit 67, the recognition result display unit 68, and the operation command execution unit 69 are executed.

Note that the input support program 320a is not necessarily stored in the HDD 320 from the beginning. For example, the program is stored in a “portable physical medium” such as a flexible disk (FD), a CD-ROM, a DVD disk, a magneto-optical disk, or an IC card inserted into the computer 300. Then, the computer 300 may read and execute the program from these. Furthermore, the program is stored in “another computer (or server)” connected to the computer 300 via a public line, the Internet, a LAN, a WAN, or the like. Then, the computer 300 may read and execute the program from these.

DESCRIPTION OF SYMBOLS 10 Wearable device 11 Radio | wireless part 12 Memory | storage part 13 Posture sensor 14 Switch 20 Control part 21 Detection part 22 Extraction part 23 Trajectory generation part 24 Gesture recognition part 25 Output part 30 Head mounted display 31 Radio | wireless part 32 Camera 33 Display part 34 Control part 50 Input device 51 Wireless unit 52 Storage unit 53 Recognition dictionary data 54 Memo information 55 Image information 60 Control unit 61 Input detection unit 62 Display control unit 63 Calibration unit 64 Axis detection unit 65 Gesture acquisition unit 66 Event identification unit 67 Character recognition unit 68 Recognition result display section 69 Operation command execution section

Claims (8)

1. An input device including a switch for receiving an input and attached to a pointer,
   A motion sensor,
   A detection unit for detecting motion information output from the motion sensor;
   An extraction unit for extracting information on the first mode from the first component of the movement information, and extracting information on the second mode from the second component of the movement information;
   Using an input pattern for the switch, an output unit that outputs one or both of the information related to the first mode and the information related to the second mode;
   An input device comprising:
2. The detection unit detects motion information output from the motion sensor while the switch is on,
   The extraction unit extracts gesture information performed by a user as information related to the second mode using the movement pattern of the indicator from the start to the end of exercise in the exercise information while the switch is on, The input device according to claim 1, wherein when the switch is turned off, trajectory information is extracted from a series of movement patterns from on to off of the switch.
3. The output unit outputs the gesture information extracted by the extraction unit when the gravity axis direction when the switch is turned on is the predetermined direction of the switch, and the gravity axis direction is the predetermined direction of the switch. The input device according to claim 2, wherein, when the direction is other than the direction, the trajectory information extracted by the extraction unit is output.
4. When the switch is turned from on to off, the extraction unit extracts the movement pattern of the indicator from the result of filtering the movement information according to a cutoff frequency corresponding to a preset character type, and extracts the pointer The input device according to claim 2, wherein trajectory information is extracted from the motion pattern.
5. The extraction unit extracts trajectory information from the motion pattern according to a trajectory gain that converts the posture change value of the indicator from the reference into the trajectory information, based on the posture at which the switch is turned on. The input device according to claim 2.
6. The input device according to claim 1, wherein the output unit further outputs an operation event specified in accordance with a time-series change of the switch on and off.
7. An input device that includes a switch for receiving input and is attached to the indicator,
   Detect motion information output from the motion sensor,
   Extracting information related to the first mode from the first component of the motion information, and extracting information related to the second mode from the second component of the motion information;
   Using the input pattern for the switch, output one or both of the information on the first mode and the information on the second mode,
   An input support method characterized by executing processing.
8. An input device equipped with a switch for receiving input and attached to the indicator,
   Detect motion information output from the motion sensor,
   Extracting information related to the first mode from the first component of the motion information, and extracting information related to the second mode from the second component of the motion information;
   Using the input pattern for the switch, output one or both of the information on the first mode and the information on the second mode,
   An input support program characterized by causing processing to be executed.

Images