JP7453450B2 - Method of operation - Google Patents

Description

The present invention relates to a small-sized input terminal device that is worn on the wrist or the like and an operation input method.

In recent years, mobile information terminals such as smartphones that perform not only phone calls but also searches for various information, sends and receives emails, and manages schedules have rapidly become popular. However, these portable information terminals are often stored in a bag or pocket when traveling, and it is necessary to take out the portable information terminal from the bag or pocket one by one in order to check received mail or the like. For this reason, smaller input terminal devices such as wristwatches that are connected to these portable information terminals by wireless communication have been developed.

On mobile information terminals such as smartphones, information can be displayed on a touch panel and input using a finger can be performed.However, on a small input terminal device, the touch panel is also small, making it difficult to perform detailed operation input by touching with a finger. . Therefore, non-contact input methods have been proposed that use the back of the user's hand as an input surface.

For example, in the wristband type input device described in Patent Document 1, the main body is provided with a light emitting part and a light receiving part, and characters can be input by detecting the position of the fingertips on the back of the hand or the palm without contact. The structure is as follows.

Japanese Patent Application Publication No. 2014-106765

In conventional character input methods using touch panels, etc., when the user's finger is in contact with the panel surface, characters are drawn according to the movement of the user's finger, and when the user's finger is away from the panel surface, the character is drawn. You can distinguish the valid period and invalid period of input to avoid errors. As a result, when inputting characters consisting of multiple "strokes" such as line segments and dots, the valid "stroke" part and the invalid part that moves between "strokes" By distinguishing the characters, you can input the desired characters correctly.

On the other hand, in a non-contact input method such as that disclosed in Patent Document 1, a plurality of grid points on the back of the hand through which the fingertip passes are detected, and the input character is recognized from the dot matrix information. However, regarding the passing position of the fingertip, it does not distinguish between a valid part for "strokes" forming a character and an invalid part that moves between "strokes". That is, there is a problem in that inputted characters are likely to be erroneously recognized by detecting invalid parts unrelated to "pictures" as part of the characters. In other words, the starting point and ending point of character drawing are unclear. Even if this invalid part is predicted and excluded in advance, the trajectory of the invalid part will not be constant if the stroke order is different when inputting, or if you input characters with voiced marks or characters with a large number of strokes. It seems that input errors are unavoidable.

An object of the present invention is to provide an input terminal device that, when performing an input operation by detecting the position of a finger in a non-contact manner, does not react to invalid finger movements and correctly performs processing according to the intended input operation. The goal is to provide the following.

The present invention provides an input terminal device that inputs a user's operation via a position input object, including a position detection section that detects the position of the position input object operated by the user in a non-contact manner; A display unit that displays a cursor based on the detected position; and an operation process control unit that executes a corresponding operation process based on the position detected by the position detection unit, and the operation process control unit controls the operation process. It has a plurality of operation modes related to execution, and one of the operation modes includes a state in which no operation process is executed other than movement of the cursor according to the position of the position input object, and the operation process control The unit switches the plurality of operation modes when the user performs a specific operation via the position input object.

The present invention also provides an operation input method for inputting a user's operation via a position input object, including a position detection step of detecting the position of the position input object operated by the user in a non-contact manner; a display step for displaying a cursor on a display unit based on the position detected in the position detection step; and an operation processing step for performing a corresponding operation process based on the position detected in the position detection step, the operation processing step It has a plurality of operation modes related to the execution of processing, and one of the operation modes includes a state in which no operation processing other than movement of the cursor according to the position of the position input object is performed, and the operation In the processing step, when the user performs a specific operation via the position input object, the plurality of operation modes are switched.

According to the present invention, when performing an input operation by detecting the position of a finger in a non-contact manner, the input terminal device does not react to invalid finger movements and correctly performs processing according to the intended input operation. can be provided.

FIG. 2 is a block diagram showing the configuration of an input terminal device (Embodiment 1). The figure which shows an example of the software structure of an input terminal device. FIG. 3 is a diagram showing the appearance of an input terminal device. The figure which shows the detection principle of finger operation mode. The figure which shows the example of a display of a display part in handwritten character input processing. 5 is a flowchart showing operation switching of the entire input terminal device. The figure which shows the example of an operation mode and the cursor display corresponding to it. The figure which shows the example of a screen display in handwritten character input processing. FIG. 8 is a diagram showing temporal changes in the operation mode and drawing position in FIG. 7; 10 is a flowchart showing handwritten character input processing in finger operation mode. 10 is a flowchart showing operation mode determination processing S605 in FIG. 9. FIG. 7 is a diagram showing the configuration of a display unit for key character input processing (Embodiment 2). The figure which shows the example of key character input processing by finger operation mode. 5 is a flowchart showing key character input processing in finger operation mode. 14 is a flowchart showing operation mode determination processing S1105 in FIG. 13. FIG. 7 is a diagram illustrating an example of an operation mode and a cursor display used in mouse operation input processing (Embodiment 3). The figure which shows the example of a screen display in mouse operation input processing. 5 is a flowchart showing mouse operation input processing in finger operation mode. 18 is a flowchart showing operation mode determination processing S1504 in FIG. 17. FIG. 4 is a diagram showing the detection principle of 3D finger operation mode (Example 4). The figure which shows the example of a screen display in 3D finger operation mode. 5 is a flowchart showing mouse operation input processing in 3D finger operation mode. The figure which shows the example of a screen display in key character input processing. FIG. 7 is a diagram showing an example of a screen display when a transition area is provided between layers (Example 5). 5 is a flowchart showing mouse operation input processing when a transition area is provided between layers.

Embodiments of the present invention will be described below with reference to the drawings.

In Example 1, the configuration of an input terminal device and a case where handwritten characters are input by detecting finger positions will be described.

FIG. 1A is a block diagram showing the configuration of the input terminal device 1 according to the first embodiment, and its internal configuration (hardware configuration) will be explained. The input terminal device 1 is used by being connected to a portable information terminal 2 such as a smartphone. The input terminal device 1 of this embodiment has a finger operation mode (non-contact detection method) using light reflection and a touch operation mode (contact detection method) as operation input functions.

The main control unit 101 is composed of a CPU (Central Processing Unit), etc., and controls the entire input terminal device 1 according to various operating programs and data stored in a ROM (Read Only Memory) 102 or a RAM (Random Access Memory) 103. Control. The system bus 100 is a data communication path for transmitting and receiving data between the control section 101 and each section within the input terminal device 1.

The ROM 102 is a memory that stores various programs for controlling the input terminal device 1, and uses a rewritable ROM such as an EEPROM (Electrically Erasable Programmable ROM) or a flash ROM. The RAM 103 is used as a temporary storage area during execution of a program stored in the ROM 102 or as temporary storage of finger position information detected by the finger position detection sensor 106. The storage unit 104 stores information such as operation setting values of the input terminal device 1, and uses a nonvolatile and rewritable device such as a flash ROM or an SSD (Solid State Drive). The ROM 102 and the RAM 103 may be integrated with the main control unit 101. Further, the ROM 102 may not be configured independently, but may be configured to use a part of the storage area within the storage unit 104.

Acceleration sensor 105 is a sensor that detects the acceleration of input terminal device 1 . The finger position detection sensor 106 is a sensor that detects the position of a position input object such as a user's finger or a pen using reflected light in a non-contact manner. The short-range wireless communication unit 107 connects to the mobile information terminal 2 by wireless communication, and transmits information input to the input terminal device 1. As the communication method, for example, Bluetooth (registered trademark) or WiFi (registered trademark) is used.

The display control unit 108 controls the display of input characters, various setting images of the input terminal device 1, etc. on the display unit 109a. The touch panel 109 includes a display section 109a and a touch operation input section 109b. The display section 109a is a display device such as a liquid crystal panel, and displays characters, images, etc. under the control of the display control section 108. The touch operation input section 109b is arranged on the display surface of the display section 109a.

The touch operation input unit 109b is, for example, a capacitive touch pad type input device, and detects a touch operation using a finger, a touch pen, or the like as an operation input. Specifically, for example, a gesture called a swipe involves touching the touch panel with a finger and then moving the finger in a specific direction while touching the touch panel, or a gesture called a swipe in which the finger touches the touch panel with a finger and then moves the finger at a speed higher than a predetermined value. It detects gestures such as a flick, which releases a finger, and a tap, which releases a finger quickly after touching the touch panel. This allows operation input to the input terminal device 1 using a touch method.

The clock unit 110 measures the elapsed time from the date and time set by the user, for example, using an RTC (Real Time Clock) circuit, and outputs date and time information. The main control unit 101 performs processing such as a timer by reading date and time information output from the clock unit 110. The operation input unit 111 is an instruction input unit for inputting operation instructions to the input terminal device 1, and has buttons for turning the power on/off and setting the operation mode.

In this way, the input terminal device 1 of the present embodiment includes a finger position detection sensor 106 that detects the position of a user's finger without contact and inputs characters, etc., and a touch sensor 106 that inputs an operation by touching the finger on the touch panel 109. The configuration includes an operation input section 109b and a display section 109a that displays characters, images, and the like. This enables two operation input functions: finger operation mode and touch operation mode.

FIG. 1B is a diagram showing an example of the software configuration of the input terminal device 1, and shows the software configuration in the ROM 102, RAM 103, and storage unit 104.

The main control unit 101 configures a basic motion processing control unit 1021, a finger position information acquisition unit 1022, and a touch position information acquisition unit 1023 by executing a program stored in the ROM 102. The RAM 103 also includes a temporary storage area 1031 that temporarily holds data as needed when executing the program stored in the ROM 102. The storage unit 104 stores setting information 1041 for executing the program stored in the ROM 102. Note that, in order to simplify the explanation, the description will be given below assuming that the main control unit 101 controls each operation block by executing a program stored in the ROM 102.

The basic operation processing control unit 1021 controls various settings and overall operation of the input terminal device 1. The finger position information acquisition unit 1022 acquires position information of a position input object such as a user's finger or a pen based on the output of the finger position detection sensor 106. The touch position information acquisition unit 1023 acquires position information where a finger, touch pen, or the like contacts the touch panel 109 based on the output of the touch operation input unit 109b.

The finger operation processing control unit 1024 determines what kind of operation has been input based on the information on the position of the position input object acquired from the finger position information acquisition unit 1022, and controls the process corresponding to the determination result. conduct. The touch operation processing control unit 1024 determines what kind of operation has been input based on the information about the touched position on the touch panel 109 acquired from the touch position information acquisition unit 1023, and controls the process corresponding to the determination result. conduct.

FIG. 2 is a diagram showing the appearance of the input terminal device 1. As shown in FIG. It is a wristwatch-type input terminal device that is worn on the user's wrist. A touch panel 109 (display section 109a) is arranged on the front side of the device. Two sensors 106a and 106b are arranged as the finger position detection sensor 106 on the back side of the user's hand on the side of the device. These sensors include, for example, light emitting/light receiving elements. In addition, an operation button is arranged as an operation input section 111 on the side of the device on the user's arm side.

FIG. 3 is a diagram showing the detection principle of the finger operation mode, and shows the relationship between the finger position and the display position of the display unit. The user's finger (position input object) PS moves, for example, within positions ABCD on the back side of the user's hand. The finger position detection sensors 106a and 106b are composed of an infrared light emitting element and a light receiving element. Infrared light emitted from the light emitting element is reflected by the finger PS, and the reflected light is detected by the light receiving element. The level of the detection signal at the light receiving element corresponds to the distance La, Lb from each sensor 106a, 106b to the finger PS.

The output signal of the finger position detection sensor 106 is sent to the finger position information acquisition unit 1022, which acquires the position F of the finger PS in the position input detection range Pa represented by the positions ABCD. Further, positions ABCD of the position input detection range Pa correspond to positions abcd of the display range Da of the display section 109a of the touch panel 109, respectively. From this, the display position f of the cursor or the like in the display range Da is determined in accordance with the position F of the finger PS in the position input detection range Pa.

FIG. 4 is a diagram showing a display example of the display unit 109a during handwritten character input processing. Area 501 displays the result of recognizing input handwritten characters, and area 502 displays handwritten characters input with a finger. The buttons 503 selected by the user for character input processing include a recognition processing instruction button 503a for selecting a process for recognizing handwritten characters, a character type switching instruction button 503b for switching the type of input characters such as hiragana and alphabets, and a character type switching instruction button 503b for selecting a process for recognizing handwritten characters. A deletion process instruction button 503c for erasing the display of characters, a character conversion process instruction button 503d for selecting a process for converting the character string of the recognition result displayed in the area 501 into kanji, etc., and a termination process instruction for terminating the handwritten character input process. A button 503e and the like are arranged.

FIG. 5 is a flowchart showing operation switching of the input terminal device 1 as a whole. Here, the power is turned on/off and the operation input mode (finger operation mode/touch operation mode) is switched by pressing a button on the operation input unit 111.

In S301, the output of the operation input unit 111 is captured, and in S302, it is determined whether or not the operation input unit 111 is pressed. If the operation input unit 111 is pressed (Yes), the process branches to S306. In S306, the time period during which the operation input unit 111 has been pressed is determined, and the process proceeds to power on/off switching or operation input mode switching processing.

If the operation input unit 111 has been pressed for a predetermined time Tp or more in the determination process S306 (Yes), the process branches to S313 and on/off switching processing is performed depending on whether the power is in the on state. If the power is on in S313 (Yes), the process branches to S314, the power is turned off, and the process returns to S301. When the power is turned off, the supply of power to parts other than the main control section 101, time measurement section 110, operation input section 111, etc. is stopped in order to reduce power consumption. If the power is off in S313 (No), the process branches to S315, the power is turned on, and the process returns to S301. When the power is turned on, power is supplied to each part constituting the input terminal device 1.

If the operation input unit 111 has not been pressed for a predetermined time Tp or more in the determination process S306 (No), the process branches to S307 and it is determined whether the power is on. If the power is off (No), the process returns to S301. If the power is on (Yes), the process branches to S308.

In S308, an operation input mode switching process is performed. That is, if the current operation input mode is set to the finger operation mode (Yes), the switch is set to the touch operation mode in S309, and the cursor is not displayed on the display section 109a of the touch panel 109 in S310. Set to . If the current operation input mode is not set to finger operation mode (No) (that is, if it is set to touch operation mode), the switch to finger operation mode is set in S311, and the touch panel 109 is set in S312. The cursor is set to be displayed on the display section 109a of the screen.

If the operation input unit 111 is not pressed in the determination process S302 (No), the process advances to S303 and processing is performed according to the set operation input mode. If it is the finger operation mode (Yes), the process advances to S304, where it is determined what kind of operation has been input based on the finger position information acquired from the finger position information acquisition unit 1022, and processing corresponding to the determination result is performed. Execute. If it is not the finger operation mode (No) (that is, if it is the touch operation mode), the process advances to S305 and determines what kind of operation was input based on the information on the touched position acquired from the touch position information acquisition unit 1023. is determined, and processing corresponding to the determination result is executed. After executing the finger operation execution process S304 or the touch operation execution process S305, the process returns to S301.

The handwritten character input process in the finger operation mode will be specifically described below. In this embodiment, by providing a plurality of operation states (operation modes) to finger operations, it is possible to correctly and easily input characters by hand.

FIG. 6 is a diagram illustrating an example of an operation mode and a cursor display corresponding to the operation mode. Operation mode M0 is a state in which drawing for character input other than cursor movement is not performed (drawing is stopped) even if the finger is operated. The operation mode M1 is a state in which when a finger is operated, drawing for character input is executed according to the position of the finger. Further, the display of the cursor (for example, white/black display) is switched depending on the operation mode so that the user can easily identify the current operation mode. As described later, the operation mode can be switched from M0 to M1 or vice versa by performing a specific operation of holding the operating finger at a certain position on the operation surface for a predetermined time t0 or more. It is possible to switch from M1 to M0.

From this, drawing is possible only when the operation mode is M1. Therefore, when inputting characters that are composed of multiple "strokes" such as line segments and dots, set the operation mode to M1 for the valid "stroke" part and check the difference between the "strokes" and "strokes". By setting the operation mode to M0 for invalid portions that move between characters, desired characters can be easily and correctly input.

FIG. 7 is a diagram showing an example of a screen display in handwritten character input processing. Here, we will take the case of inputting the hiragana character "a" as an example. The character input procedure is shown as (S1) to (S9), and the operating mode (M0/M1) at that time is also noted.

In the following explanation, it is expressed as "holding" the cursor at a certain position, but even if the cursor moves (shakes) within a predetermined distance r0, this is allowed and is considered to be in a "holding" state. shall be. This r0 is caused by unintentional movements of the user (such as hand shake), and is set to a value of, for example, several millimeters. Further, the time t0 for determining the "hold" state is, for example, a value of about several seconds. These threshold values may be set depending on the user and usage environment.

(S1): When the operation mode is M0 (drawing stopped), the cursor is moved to position 7a in the character input display area 502 by moving the finger PS within the position input detection range Pa.
(S2): Hold the cursor at position 7a for a predetermined time t0 or more. This switches the operation mode to M1 (drawing execution) and changes the cursor display.

(S3): When the cursor display changes, move the finger to move the cursor to position 7b in the character input display area 502. As a result, a line from position 7a to position 7b is drawn.
(S4): Hold the cursor at position 7b for a predetermined time t0 or more. As a result, the operation mode is switched to M0 (drawing stopped) and the cursor display is changed.

(S5): When the cursor display changes, move your finger to move the cursor to position 7c in the character input display area 502.
(S6): Hold at position 7c for a predetermined time t0 or more. This switches the operation mode to M1 and changes the cursor display.

Thereafter, the drawing operation continues in the same manner. That is, when drawing is performed, the operation mode is set to M1 and the cursor is moved, and when no drawing is performed, the operation mode is set to M0 and the cursor is moved. Furthermore, by holding the cursor at the moved position for a predetermined time t0 or more, the operation mode is switched, and handwritten characters are input.

(S7): When the drawing of the hiragana "a" is completed, it is held at the position 7d for a predetermined time t0 or more. This switches the operation mode to M0 and changes the cursor display.
(S8): Move the cursor to the recognition processing instruction button 503a.
(S9): The recognition process is executed by holding the cursor on the recognition process instruction button 503a for a predetermined time t0 or more, and the recognized result is displayed in the display area 501.

FIG. 8 is a diagram showing temporal changes in the operation mode and drawing position in FIG. 7. (a) shows the position of the position input object (user's finger) PS, (b) shows the operation mode, and (c) shows the display unit drawing position. (S1) to (S6) on the horizontal axis and symbols (7a) to (7c) correspond to the symbols in FIG. In this way, drawing is performed only during the period when the operation mode is M1, and the operation mode is switched by holding the position input object PS at the same position for more than time t0.

FIG. 9 is a flowchart showing handwritten character input processing in finger operation mode. The contents of each process will be explained in order below.

In S601, the finger position information acquisition unit 1022 acquires the position F of the finger PS, and in S602, the acquired position F of the finger PS is converted into a position f in the display range Da as described with reference to FIG. Next, in S603, it is determined whether the distance r between the position f and the previously acquired position fd stored in the temporary storage area of the RAM 103 is less than or equal to a predetermined distance r0.

In the determination process S603, if the distance r is greater than or equal to the predetermined distance r0 (No), the process branches to S617. In S617, it is determined whether the position f is within the display range of the character input display area 502. If the position f is within the range of the character input display area 502 (Yes), the process branches to S618. If the position f is outside the range of the character input display area 502 (No), the process branches to S620. In S620, the operation mode is set to M0, and the process advances to S615.

In S618, the set operation mode is determined. If the operation mode is M0, the process advances to S615 without performing drawing processing. If the operation mode is M1, the process branches to S619, where drawing processing is performed from position fd to position f in display range Da, and the process advances to S615.

On the other hand, in the determination process S603, if the distance r is less than or equal to the predetermined distance r0 (Yes), the process branches to S604. In S604, it is determined whether the position f is within the display range of a predetermined area 502 for inputting characters in the display range Da or within the display range of the processing instruction button 503. If the position f is within the display range of the character input display area 502 or the processing instruction button 503 (Yes), the process advances to S605. If the position f is outside the display range of the character input display area 502 or the processing instruction button 503 (No), the process advances to S615.

In S605, the operation mode is determined by a timer. That is, the holding time is compared with a predetermined time t0 and the operation mode is switched. Note that details of the determination process in S605 will be explained with reference to FIG. In S606, branch processing is performed depending on the determined operation mode. If the operation mode is M0, the process advances to S615, and if the operation mode is M1, the process advances to S607. In S607, it is determined whether the position f is within the display range of the processing instruction button 503. If the position f is within the display range of the processing instruction button 503 (Yes), the process advances to S608, and if the position f is outside the display range of the process instruction button 503 (No), the process advances to S615.

In S608, branch processing is performed depending on which processing instruction button from 503a to 503e the position f is in the display range. If the position f is within the display range of the recognition processing instruction button 503a, the process advances to S609 and recognition processing is executed. If the position f is within the display range of the character type switching processing instruction button 503b, the process advances to S610 and character type switching processing is executed. If the position f is within the display range of the erasure processing instruction button 503c, the process advances to S611 and the erasure processing is executed. If the position f is within the display range of the conversion process instruction button 503d, the process advances to S612 and the conversion process is executed.
After each process is executed, the process advances to S613 to reset and stop the timer, and in S614 the operation mode is set to M0. If the position f is within the display range of the end process instruction button 503e, the handwritten character input process is ended.

In S615, a cursor as shown in FIG. 6, for example, is displayed at position f in display range Da, depending on the set operation mode. Next, in S616, the current position f is replaced with the previous position fd and stored in the temporary storage area of the RAM 103, and the process returns to S601.

FIG. 10 is a flowchart showing the operation mode determination process S605 in FIG.

In S701, it is determined whether the timer is activated. If the timer has not been started (No), the process advances to S702, starts the timer, and ends the process. If the timer has been activated (Yes), the process advances to S703.

In S703, it is determined whether the timer has exceeded a predetermined time t0. If the predetermined time t0 has not been exceeded (No), the process ends. If the predetermined time t0 has been exceeded (Yes), the process advances to S704.

In S704, branch processing is performed depending on the operation mode. If the operation mode is M0, the process advances to S705 and the operation mode is switched to M1. If the operation mode is M1, the process advances to S706 and the operation mode is switched to M0. Thereafter, the process advances to S707, where the timer is reset and stopped, and the process ends.

As described above, in the first embodiment, when drawing is performed by switching the operation mode by holding the position of a position input target such as a user's finger on the display screen within a predetermined distance r0 for a predetermined time t0 or more. To do so, the operation mode is set to M1 and the cursor is moved, and when no drawing is to be performed, the operation mode is set to M0 and the cursor is moved. As a result, even when inputting characters with multiple stroke counts, the movement between strokes can be distinguished as a non-drawing section, making it easy to input the intended character correctly even if the stroke order is different. becomes possible. It is also possible to input more complex characters such as kanji.

In addition, by moving the cursor to the area where the processing instruction button is displayed and holding the position within a predetermined distance r0 from the moved position for a predetermined time t0 or more, the process corresponding to the processing instruction button can be performed. You can perform the same operation as moving the cursor using an input means such as a mouse to the position where the processing instruction button is displayed and clicking it.

In this embodiment, an example is shown in which the processing instruction buttons include a recognition processing instruction button 503a, a character type switching instruction button 503b, an erasure processing instruction button 503c, a character conversion processing instruction button 503d, and an end processing instruction button 503e. However, the present invention is not limited to this, and other processing instruction buttons may be provided.

Further, in this embodiment, switching between the operation input mode by the touch panel 109 (touch operation mode) and the operation input mode by the finger position detection sensor 106 (finger operation mode) is performed by the operation input unit 111, but other methods may be used. It may also be switched. For example, when the back of the hand is tapped with a finger, vibration is detected by the acceleration sensor 105 and the touch is not detected by the touch panel 109, so it is determined that the back of the hand has been tapped based on the detection results of the touch panel 109 and the detection results of the acceleration sensor 105. However, the operation input mode may be switched when the back of the hand is tapped with a finger.

Further, in this embodiment, as a specific operation for switching the operation mode, an operation is adopted in which the position of a position input target such as a user's finger on the display screen is held within a predetermined distance r0 for a predetermined time t0 or more. However, it is not limited to this. As another specific operation, for example, the cursor may be moved by moving the position input object, and the operation mode may be switched by tapping the back of the hand at the moved position.

In the second embodiment, a case will be described in which input is performed by selecting keys displayed on a screen, such as a keyboard. In a small portable information terminal, the keys displayed on the display section 109a are also small, making it difficult to input by touching the touch panel 109 with a finger. Therefore, key characters are input using the finger operation mode.

FIG. 11 is a diagram showing the configuration of the display section 109a for key character input processing. An area 1001 is a part for displaying characters inputted using a key, and a character input key 1002 is a key for selecting a character to be input. Keys 1003 are for instructing various processes, including a character type switching instruction key 1003a for changing the type of input characters such as hiragana and alphabets, and a deletion processing instruction key for deleting input characters displayed in area 1001. 1003b, a termination process instruction key 1003c for terminating the key character input process, and the like.

FIG. 12 is a diagram illustrating an example of key character input processing in the finger operation mode, and here, the case of inputting the alphabet "Y" is taken as an example. The input procedure will be explained using (S1) to (S4).

(S1): When the operation mode is M0, by moving the finger PS within the position input detection range Pa, the cursor is moved to the position 12a where the character input key "WXYZ" is displayed.
(S2): Hold the cursor at position 12a for a predetermined time t0 or longer (also in this case, movement within a predetermined distance r0 is allowed. The same applies below). As a result, the operation mode is switched to M1, and a display for selecting the characters "W", "X", "Y", and "Z" is performed.

(S3): When the character selection is displayed, move your finger to move the cursor to position 12b where the character "Y" is displayed.
(S4): Hold the cursor at position 12b for a predetermined time t0 or more. As a result, the character "Y" is determined as the input character, and the character "Y" is displayed in the input character display area 1001. When character input is completed, the operation mode switches to M0.

FIG. 13 is a flowchart showing key character input processing in finger operation mode.

In S1101, the finger position information acquisition unit 1022 acquires the position F of the finger PS, and in S1102, the acquired position F of the finger PS is converted into a position f in the display range Da. Next, in S1103, it is determined whether the distance r between the position f and the previously acquired position fd is less than or equal to a predetermined distance r0.

In determination processing S1103, if the distance r is greater than or equal to the predetermined distance r0 (No), the process branches to S1114. If the distance r is less than or equal to the predetermined distance r0 (Yes), the process branches to S1104. In S1104, it is determined whether the position f is within the display area of the character input key 1002 or the processing instruction key 1003. In determination processing S1104, if the position f is within the display area of the character input key 1002 or the process instruction key 1003 (Yes), the process branches to S1105, and if the position f is outside the display area of the character input key 1002 or the process instruction key 1003 (No). ), the process branches to S1114.

In S1105, the operation mode is determined by a timer. That is, the holding time is compared with a predetermined time t0 and the operation mode is switched. Note that details of the determination process in S1105 will be explained with reference to FIG. In S1106, branch processing is performed according to the determined operation mode. If the operation mode is M0, the process advances to S1114, and if the operation mode is M1, the process advances to S1107. In S1107, it is determined whether the character selection display is on.

In determination processing S1107, if the character selection display is on (Yes), the process advances to S1116, and if the character selection display is off (No), the process advances to S1108. In S1116, it is determined whether the position f is within the character selection display area. If the position f is within the character selection display area (Yes), the process advances to S1117, and if the position f is outside the character selection display area (No), the process advances to S1114. In S1117, the character displayed at position f is determined as an input character and displayed in input character display area 1001. In S1118, the character selection display is turned off, and the process advances to S1112.

In S1108, branch processing is performed depending on whether the position f is within the display area of either the character input key 1002 or the process instruction key 1003. If the position f is within the display area of the character input key 1002, the process advances to S1109, where the character selection display is turned on, and the character corresponding to the key displayed at the position f is selectively displayed. If the position f is within the display area of the character type switching process instruction key 1003a, the process advances to S1110 and character type switching process is executed. If the position f is within the display area of the deletion process instruction key 1003b, the process advances to S1111 and the deletion process is executed. After executing each process, the process advances to S1112 to reset and stop the timer, and in S1113 the operation mode is set to M0. If the position f is within the display area of the end process instruction key 1003c, the key character input process is ended.

In S1114, a cursor as shown in FIG. 6, for example, is displayed at position f in display range Da, depending on the set operation mode. Next, in S1115, the current position f is replaced with the previous position fd, stored in the temporary storage area of the RAM 103, and the process returns to S1101.

FIG. 14 is a flowchart showing the operation mode determination process S1105 in FIG. 13.

In S1201, it is determined whether the timer is activated. If the timer has not been activated (No), the process advances to step S1202, where the timer is activated and the process ends. If the timer has been activated (Yes), the process advances to S1203.

In S1203, it is determined whether the timer has exceeded a predetermined time t0. If the predetermined time t0 has not been exceeded (No), the process ends. If the predetermined time t0 has been exceeded (Yes), the process advances to S1204.

In S1204, the operation mode is switched to M1. In this case, in S1113 of FIG. 13, the operation mode is switched to M0 after the key character input is executed, so there is no need to switch from M1 to M0. Thereafter, the process advances to S1205, where the timer is reset and stopped, and the process ends.

As described above, in the second embodiment, input is performed by moving the cursor to the position where the character input key or processing instruction key is displayed, and holding the position within a predetermined distance r0 from the moved cursor position for a predetermined time t0 or more. It becomes possible to easily select a character to be used or to instruct a process to be executed. Thereby, it is possible to perform an operation similar to moving a cursor using an input means such as a mouse to a position where a character input key or a processing instruction key is displayed and clicking the same.

In this embodiment, an example is shown in which the processing instruction keys include a character type switching instruction key 1003a, a deletion processing instruction key 1003b, and an end processing instruction key 1003c. A corresponding key may be provided. Further, regarding the character input keys, a keyboard in which each letter of the alphabet or hiragana corresponds to one key, or a keyboard that corresponds to the input of numbers may be displayed for input.

In the third embodiment, operations such as mouse clicks, double clicks, and drag-and-drop operations (hereinafter referred to as mouse operation input processing) are realized in finger operation mode. In order to cope with this, in this embodiment, the number of types of operation modes is increased.

FIG. 15 is a diagram showing an example of the operation mode and cursor display used in mouse operation input processing. Here, six types of operation modes M0 to M5 are used. Operation mode M0 is a state in which no input operation other than cursor movement is performed even if the finger is operated. Operation modes M1 to M5 are states in which mouse operations are performed for click processing (M1), double-click processing (M2), start of drag processing (M3), end of drag processing (M4), and drop processing (M5), respectively. be. Further, the display of the cursor (shape and/or color display) is changed according to each operation mode. Note that the operation mode is switched by comparing the length of time for which the operating finger is held at a certain position on the operating surface with three threshold values t0, t1, and t2, as described later.

FIG. 16 is a diagram showing an example of a screen display in mouse operation input processing. On the display unit 109a, a group of icons 1401 corresponding to applications A to D are displayed as a window screen (home screen). The user can select a desired application by clicking on the icon, execute the application by double-clicking it, and move the display position of the icon by dragging and dropping it. The flow of a series of processes will be explained using (S1) to (S7).

(S1): When the operation mode is M0, by moving the finger within the position input detection range, the cursor is moved to the position 14a where the icon [D] is displayed.
(S2): Hold the cursor at position 14a for a predetermined time (range from t0 to t1). As a result, the operation mode is switched to M1 (click processing), and the icon "D" is selected as the mouse operation target. Further, the cursor display changes to a display corresponding to M1 in FIG. 15. After that, by moving your finger arbitrarily (in this case, moving the cursor from position 14a to a position more than a predetermined distance r0) until the predetermined time t1 has elapsed, you can move the selected icon "D". Click processing is executed.

(S3): When the cursor is held at position 14a for a predetermined time t1 or more, the operation mode is switched to M2 (double click processing), and the cursor changes to a display corresponding to M2 shown in FIG. 15. Thereafter, by moving the finger arbitrarily (moving the cursor by a predetermined distance r0 or more from the position 14a) until the predetermined time t2 elapses, a double-click process is executed on the selected icon "D". .

(S4): When the cursor is held at the position 14a for a predetermined time t2 or more, the operation mode is switched to M3 (drag processing start), and the cursor changes to a display corresponding to M3 shown in FIG. 15.
(S5): Thereafter, by moving the finger and moving the cursor from the position 14a to the position 14b that is a predetermined distance r0 or more away, the operation mode is switched to M4 (end of drag processing). As a result, the cursor changes to a display corresponding to M4 shown in FIG. 15. Further, the icon "D" selected as the mouse operation target is moved and displayed at the position 14b to which the cursor has been moved, and a drag process is executed to move the position of the selected icon.

(S6): Further move the finger to move the cursor and the selected icon [D] to position 14c, and hold the cursor at position 14c for a predetermined time t0 or more. As a result, the operation mode is switched to M5 (drop processing), and the drop processing is executed at position 14c.
(S7): The display position of the selected icon "D" is set to 14c by the drop process. After that, the operation mode is switched to M0, and the cursor becomes a display corresponding to M0.

FIG. 17 is a flowchart showing mouse operation input processing in finger operation mode.

In S1501, the finger position information acquisition unit 1022 acquires the position F of the finger PS, and in S1502, the acquired position F of the finger PS is converted to a position f in the display range Da. Next, in S1503, it is determined whether the distance r between the position f and the previously acquired position fd is less than or equal to a predetermined distance r0.

In determination processing S1503, if the distance r is less than or equal to the predetermined distance r0 (Yes), the process branches to S1504, and in S1504, the operation mode is determined by a timer. That is, the holding time is compared with predetermined times t0, t1, and t2 to switch the operation mode. Note that details of the determination process in S1504 will be explained with reference to FIG. Thereafter, in S1505, it is determined whether the operation mode determined in S1504 is M5. If the operation mode is M5 (Yes), the process branches to S1508, and if the operation mode is not M5 (No), the process branches to S1506. In S1508, a drop process is executed to set the display position of the icon selected by the drag process to position f, and the process advances to S1509.

In S1506, it is determined whether the position f is within the icon display area. If the position f is within the icon display area (Yes), the process branches to S1507, and if the position f is outside the icon display area (No), the process branches to S1509. In S1507, the icon whose position f is within the display area is selected as a mouse operation target, and the process advances to S1518. In S1509, the operation mode is set to M0, and the process advances to S1510. In S1510, the timer is reset and stopped, and the process advances to S1518.

On the other hand, in determination processing S1503, if the distance r is equal to or greater than the predetermined distance r0 (No), the process branches to S1511. In S1511, the following branch processing is performed depending on the operation mode.
If the operation mode is M0, the process advances to S1518 without performing any processing.
If the operation mode is M1, the process advances to S1515, where a click process is performed on the icon selected as the mouse operation target in S1507, and the process advances to S1516.
If the operation mode is M2, the process advances to S1514, double-clicking is performed on the icon selected as the mouse operation target in S1507, and the process advances to S1516.
If the operation mode is M3, the process advances to S1513, the operation mode is set to M4, and the process advances to S1517.
If the operation mode is M4, the process advances to S1512, and the icon selected as the mouse operation target in S1507 is displayed at position f. As a result, drag processing is performed to move the position of the selected icon, and the process then advances to S1518.

In S1516, the operation mode is set to M0, in S1517 the timer is reset and stopped, and the process advances to S1518. In S1518, a cursor as shown in FIG. 15 is displayed at position f in display range Da according to the current operation mode. Next, in S1519, the current position f is replaced with the previous position fd, stored in the temporary storage area of the RAM 103, and the process returns to S1501.

FIG. 18 is a flowchart showing the operation mode determination process S1504 in FIG. 17.

In S1601, it is determined whether the timer is activated. If the timer has not been started (No), the process advances to S1602, starts the timer, and ends the process. If the timer has been activated (Yes), the process advances to S1603.

In S1603, it is determined whether the operation mode is M4. If the operation mode is M4 (Yes), the process advances to S1604. In S1604, it is determined whether or not the timer has exceeded a predetermined time t0, and if the time has not exceeded the predetermined time t0 (No), the process ends. If the predetermined time t0 has been exceeded (Yes), the process advances to S1605, sets the operation mode to M5, and ends the process. In determination processing S1603, if the operation mode is not M4 (No), the process advances to S1606.

In S1606, the following branch processing is performed depending on the elapsed time of the timer. Note that the following time thresholds have a relationship of t0<t1<t2.
If the timer has not exceeded the predetermined time t0 (timer≦t0), the process ends.
If the timer is between the predetermined times t0 and t1 (t0<timer≦t1), the process advances to S1609, sets the operation mode to M1, and ends the process.
If the timer is between the predetermined times t1 and t2 (t1<timer≦t2), the process advances to S1608, sets the operation mode to M2, and ends the process.
If the timer has exceeded the predetermined time t2 (timer>t2), the process advances to S1607, sets the operation mode to M3, and ends the process.

As described above, in the third embodiment, the operation mode is switched from M0 to M5 depending on the length of time the cursor is held within the predetermined distance r0 from the moved position. After the operation mode has been switched, the cursor can be moved to a position a predetermined distance r0 or more, similar to the flicking action in touch operations on a smartphone, to perform mouse clicks, double-clicks, drag-and-drop operations, etc. It becomes possible to realize the operation in the finger operation input mode. Further, it is possible to confirm that the operation mode has been switched by a change in the display of the cursor, and it becomes possible for the user to easily perform the intended mouse operation.

In the fourth embodiment, a case will be described in which the position of a user's finger is detected three-dimensionally and an input operation is performed (hereinafter referred to as 3D finger operation mode).

FIG. 19 is a diagram showing the detection principle of the 3D finger operation mode, and shows the relationship between the finger position and the display position of the display unit. The position of the finger PS is detected not only in a direction parallel to the display unit 109a of the input terminal device 1 (XY axis direction) but also in a direction perpendicular to the display unit 109a (height Z axis direction), and an input operation is performed. The figure shows the relationship between the position F of the detected finger (position input object) PS and the display position f of the display unit 109a.

Three finger position detection sensors 106a, 106b, and 106c are provided to three-dimensionally detect the finger position. Each of the sensors 106a to 106c includes, for example, an infrared light emitting element and a light receiving element, and each sensor 106a to 106c outputs a signal at a level corresponding to the distance La, Lb, Lc to the finger PS. The finger position information acquisition unit 1022 determines a position input detection range (space) surrounded by positions A, B, C, D, A', B', C', and D' based on the output signals from each sensor 106a to 106c. The position F of the finger PS at Pa is obtained. Corresponding to the acquired position F in the XY axis directions, a position f in the display range Da surrounded by positions abcd of the display unit 109a is determined, and a cursor or the like is displayed, for example. On the other hand, in accordance with the acquired position F in the Z-axis direction, a plurality of display screens are switched as described later.

FIG. 20 is a diagram showing an example of a screen display in the 3D finger operation mode. Here, a window screen (home screen) on which icons are arranged is displayed as the operation screen. The position in the height Z direction with respect to the display section 109a is divided into a plurality of layers L0, L1, and L2 using Z0 to Z3 as boundaries. The window screen is switched and displayed depending on which layer the finger PS is located. Display states in each layer are shown as (S1) to (S3).

(S1): When the position F0 of the finger PS is in the range of Z0 to Z1 in the height direction (layer L0), icon groups 1901 corresponding to the applications A to D are displayed on the window screen.
(S2): If the position F1 of the finger PS is within the range of Z1 to Z2 in the height direction (layer L1), icon groups 1902 corresponding to the applications E to H are displayed on the window screen.
(S3): When the position F2 of the finger PS is in the range of Z2 to Z3 in the height direction (layer L2), icon groups 1903 corresponding to the applications I to L are displayed on the window screen.

Thereafter, the mouse operation input process described in the third embodiment is executed. That is, an application is selected by clicking the displayed icon, an application is executed by double-clicking the icon, and the display position of the icon is moved by dragging and dropping the icon.

In this way, according to the 3D finger operation mode, the screen display for scanning can be switched for each of a plurality of layers, so it is possible to easily perform mouse operation input processing for a large number of applications (icons).

FIG. 21 is a flowchart showing mouse operation input processing in the 3D finger operation mode. The same processes as in Example 3 (FIG. 17) are designated by the same reference numerals, and repeated explanations will be omitted.

In S1501, the finger position information acquisition unit 1022 acquires the three-dimensional position F of the finger PS, and in S1502, the acquired position F in the XY direction is converted into a position f in the display range Da as explained in FIG. . Next, in S1520, it is determined which layer the acquired position F in the Z direction falls within. In S1521, it is determined whether the determined layer LN is the same as the previously determined layer LNd stored in the temporary storage area of the RAM 103.
As a result of the determination, if the determined layer LN is the same as the previously determined layer LNd (Yes), the process advances to S1503. The processing after S1503 is the same as that in FIG. 17, and the explanation will be omitted. As a result of the determination, if the determined layer LN is different from the previously determined layer LNd (No), the process advances to S1522.

In S1522, it is determined whether the operation mode is M3 or M4. If the operation mode is M3 or M4 (Yes), the process advances to S1523; if the operation mode is not M3 or M4 (No), the process advances to S1525. In S1525, the operation mode is set to M0, and the process advances to S1526. In S1526, the timer is reset and stopped, and the process advances to S1523.

In S1523, the determined layer LN is replaced as the previously determined layer LNd, stored in the temporary storage area of the RAM 103, and the process advances to S1524. In S1524, the screen of the display unit 109a is updated to display a window screen (home screen) corresponding to the determined layer LN, and the process advances to S1511. The processes after S1511 are mouse operation input processes according to each operation mode M0 to M4, and are the same as those in FIG. 17, so the explanation will be omitted.

As described above, by detecting the three-dimensional position of the user's finger and performing operation input, the screen is divided into multiple layers according to the position in the height Z direction, and the operation window screen ( home screen). It is also possible to perform mouse operations such as click, double-click, and drag-and-drop on the icons displayed in each layer's window.

In the above example, icons corresponding to different applications are displayed on the window screen for each of a plurality of layers, but the present invention is not limited to this, and it is possible to display a different operation surface for each layer.

For example, FIG. 22 is a diagram showing an example of a screen display in key character input processing. In key character input processing, a keyboard of different character types is displayed on the display unit for each layer.

(S1): If the position F0 of the finger PS is within the range of the layer L0, a keyboard for inputting alphabets and symbols is displayed.
(S2): If the position F1 is within the range of the layer L1, a keyboard for inputting hiragana is displayed.
(S3): If the position F2 is within the range of the layer L2, a keyboard for inputting numbers is displayed.
Thereby, the user can easily change the character type to be input by changing the height position of the finger PS in the Z direction.

As described above, according to the fourth embodiment, the screen display and operation contents can be switched for each of a plurality of layers using the 3D finger operation mode, so that many types of operations can be easily realized. Note that the number of layers may be set by the user as appropriate depending on the type of operation to be used.

In Example 5, a configuration will be described in which a transition area is provided between layers in order to smoothly switch between a plurality of layers in the 3D finger operation mode.

FIG. 23 is a diagram showing an example of a screen display when a transition area is provided between layers. The position of the finger PS in the height Z direction is divided into a plurality of layers L0 to L2, and layer transition regions T01 and T12 are provided at the boundaries of each layer. Similar to Embodiment 4, different operation window screens (here, icon groups) are displayed corresponding to each layer L0 to L2, but in layer transition areas T01 and T12, a window screen for different operations is displayed corresponding to one of the adjacent layers. display the window screen. Hereinafter, changes in the screen display will be explained along with the movement of the finger PS in the height direction.

First, a case in which the position of the finger PS is moved from F0 toward F4 (that is, from layer L0 to L2) will be described in (S1) to (S5).

(S1): The position F0 of the finger PS is within the range of the layer L0, and the icon group 2101 corresponding to the applications A to D is displayed on the window screen.
(S2): When the position F1 of the finger PS moves to the layer transition area T01 between the layers L0 and L1, the window screen (icon group 2101) displayed on the immediately previous layer L0 is continuously displayed. At that time, the screen background is changed to a display 109T indicating a transition area.

(S3): The position F2 of the finger PS is within the range of the layer L1, and the icon group 2102 corresponding to the applications E to H is switched and displayed on the window screen.
(S4): When the position F3 of the finger PS moves to the layer transition area T12 between layers L1 and L2, the window screen (icon group 2102) displayed on the immediately previous layer L1 is continuously displayed. At that time, the screen background is changed to a display 109T indicating a transition area.

(S5): The position F4 of the finger PS is within the range of the layer L2, and the icon group 2103 corresponding to the applications I to L is switched and displayed on the window screen.

On the other hand, a case where the position of the finger PS is moved from F4 toward F0 (that is, from layer L2 to L0) will be described in (S5') to (S1').

(S5'): The position F4 of the finger PS is within the range of the layer L2, and the icon group 2103 corresponding to the applications I to L is displayed on the window screen.
(S4'): When the position F3 of the finger PS moves to the layer transition area T12 between layers L1 and L2, the window screen (icon group 2103) displayed on the immediately previous layer L2 is continuously displayed. At that time, the screen background is changed to a display 109T indicating a transition area.

(S3'): The position F2 of the finger PS is within the range of the layer L1, and the icon group 2102 corresponding to the applications E to H is switched and displayed on the window screen.
(S2'): When the position F1 of the finger PS moves to the layer transition area T01 between the layers L0 and L1, the window screen (icon group 2102) displayed on the immediately previous layer L1 is continuously displayed. At that time, the screen background is changed to a display 109T indicating a transition area.

(S1'): The position F0 of the finger PS is within the range of the layer L0, and the icon group 2101 corresponding to the applications A to D is switched and displayed on the window screen.

As described above, when the finger PS is located within the range of each layer, the window screen corresponding to each layer is displayed, and when the finger PS moves to the layer transition area, the window screen of the layer located immediately before is displayed. continues to be displayed, and its background is changed and displayed.

In this way, a transition area is provided between each layer, and when the finger PS moves to the transition area, the background of the window screen is changed to notify the user that the finger PS has moved to the transition area. At this time, the window screen that was displayed immediately before continues, so there is no problem with the operation. This prevents the user from unintentionally moving to another layer. That is, when the background of the window screen changes, it is sufficient to move the finger PS in the direction of the original layer.

FIG. 24 is a flowchart showing mouse operation input processing when a transition area is provided between layers. The same processes as in Example 3 (FIG. 17) and Example 4 (FIG. 21) are given the same reference numerals, and repeated explanations will be omitted.

In S1501, the three-dimensional position F of the finger PS is acquired, and in S1502, the position of the position F in the XY direction is converted to a position f in the display range Da. Next, in S1520, it is determined whether the position F in the Z direction is within the range of which layer or within the layer transition area. In S1530, branch processing is performed according to the layer determination result.

In the determination process S1530, if the determined layer LN is within the layer transition area (LN=between layers), the process branches to S1531. In S1531, the background of the window display screen is set to display 109T corresponding to the layer transition area, and the process advances to S1503.

In determination processing S1530, if the determined layer LN is the same as the previously determined layer LNd (LN=LNd), the process advances to S1503. The processing after S1503 is the same as that in FIG. 17, and the explanation will be omitted.

In determination processing S1530, if the determined layer LN is different from the previously determined layer LNd (LN≠LNd), the process advances to S1522. The processing after S1522 is the same as that in FIG. 21, and the explanation will be omitted.

As described above, according to the fifth embodiment, the three-dimensional position of the finger PS is detected, and in addition to displaying windows corresponding to multiple layers corresponding to the position in the height direction Z, there is also a window between each layer. Added a transition area. Then, when the finger PS moves to the transition area, the user is notified by changing the background of the window screen, and it is possible to prevent the user from moving to another layer unintentionally. This allows smooth switching between each layer.

Note that in this embodiment, the background of the window screen is changed when the finger PS moves to the layer transition area, but the present invention is not limited to this. For example, any method that can be recognized by the user may be used, such as changing the display of an icon, changing the display of a cursor, or generating sound or vibration. Furthermore, when located in the transition area, the direction (up/down) of the original layer may be displayed.

In the above description of each embodiment, an example was shown in which the finger position detection sensor 106 is composed of a light emitting/light receiving element. Any object may be used as long as the position of the position input object, such as a pen or pen, can be detected. In addition, although a wristwatch-type input terminal device has been described as an example, in smartphones, tablets, personal computers, etc., the position of a position input object such as a user's finger or pen is detected to input handwritten characters, operate a mouse, etc. You can do it like this.

Furthermore, in each of the above embodiments, as a condition for a specific operation for switching the operation mode, the position of a position input object such as a user's finger on the display screen is maintained within a predetermined distance for a predetermined time or longer. However, it is not limited to this. For example, the condition may be that the amount of movement of the position of the position input target on the display screen within a predetermined time is within a predetermined distance. Alternatively, the condition may be that the moving speed of the position of the position input object on the display screen is equal to or less than a predetermined speed.

Although the embodiments of the present invention have been described above using several examples, it goes without saying that the configuration for realizing the technology of the present invention is not limited to the above-mentioned examples, and various modifications can be made. . For example, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these belong to the scope of the present invention. Further, the numerical values, messages, etc. that appear in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different values are used. Furthermore, the programs described in each processing example may be independent programs, or a plurality of programs may constitute one application program. Furthermore, the order in which each process is performed may be changed.

Some or all of the functions of the present invention described above may be realized by hardware, for example, by designing an integrated circuit. Alternatively, the functions may be realized in software by having a microprocessor unit or the like interpret and execute operating programs for realizing the respective functions. Hardware and software may be used together.

Further, the control lines and information lines shown in the drawings are those considered necessary for explanation, and do not necessarily show all control lines and information lines on the product. In reality, almost all components may be considered to be interconnected.

1: Input terminal device, 2: Mobile information terminal, 101: Main control unit, 102: ROM, 103: RAM, 104: Storage unit, 105: Acceleration sensor, 106, 106a, 106b, 106c: Finger position detection sensor, 108 : Display control unit, 109: Touch panel, 109a: Display unit, 109b: Touch operation input unit, 110: Time measurement unit, 111: Operation input unit, 1022: Finger position information acquisition unit, 1023: Touch position information acquisition unit, 1024: Finger operation processing control unit, 1025: touch operation processing control unit, PS: user's finger (position input object).

Claims (10)

A terminal device capable of inputting user operations via a position input object,
a position detection unit that detects the three-dimensional position of the position input object in a non-contact manner;
a display section that displays the operation target;
an operation processing control unit that executes a corresponding operation process based on the three-dimensional position detected by the position detection unit,
The operation processing control unit includes:
When the position detection unit detects that the position input target is at a three-dimensional position corresponding to the operation target displayed on the display unit for a predetermined period of time or more, an operation process is performed on the operation target. control to switch from a first operation mode in which no operation is performed to a second operation mode in which operation processing is performed on the operation target,
Furthermore, among the three-dimensional positions of the position input target relative to the display unit, the vertical position relative to the display unit is divided into a plurality of layers, and the vertical position of the position input target detected by the position detection unit is divided into multiple layers. controlling the display content of the display unit to be changed depending on which layer of the plurality of layers is within the range of the display unit;
A transition area is further set at the boundary of the plurality of layers, and when the position of the position input object moves within the transition area, the operation screen that was displayed immediately before continues on the display section. A terminal device characterized in that the terminal device is controlled to notify that the position of the position input object is within the transition area while displaying the position input object. The terminal device according to claim 1,
The operation processing control unit controls the display to change the display of the operation target when the vertical position of the position input target with respect to the display unit is within a predetermined first range in the second operation mode. A terminal device characterized in that it controls a part. The terminal device according to claim 1,
The operation processing control unit divides the vertical position with respect to the display unit into a plurality of layers, and determines which of the plurality of layers the vertical position of the position input object detected by the position detection unit belongs to. A terminal device, characterized in that the terminal device is controlled to change the display content of the display unit depending on whether the content is within a range. The terminal device according to claim 3,
The operation processing control unit further sets a transition area at the boundary of the plurality of layers, and when the position of the position input object moves within the transition area, the operation processing control unit further sets a transition area on the boundary of the plurality of layers, and when the position of the position input object moves within the transition area, the display unit displays the information that was displayed immediately before. A terminal device characterized by continuously displaying an operation screen and changing the background of the screen. The terminal device according to any one of claims 1 to 4,
The terminal device according to claim 1, wherein the position detection section includes a light emitting element and a light receiving element. The terminal device according to any one of claims 1 to 4,
The terminal device according to claim 1, wherein the position detection section includes a sensor using ultrasonic waves. The terminal device according to any one of claims 1 to 4,
The terminal device, wherein the position detection section includes a camera. The terminal device according to claim 1,
Furthermore, it is equipped with a touch panel that detects touch operations.
a first operation input mode in which a corresponding operation process is executed based on a touch operation on the touch panel; and a second operation input mode in which a corresponding operation process is executed based on a three-dimensional position detected by the position detection section; A terminal device comprising: 9. The terminal device according to claim 8,
The terminal device further includes an operation input section capable of switching from the first operation input mode to the second operation input mode. 9. The terminal device according to claim 8,
A terminal device characterized in that when a predetermined operation by the position input object is detected, the terminal device switches to the first operation input mode or the second operation input mode.

Images