JP6033061B2 - Input device and program - Google Patents

Description

  The present invention relates to an input device and a program technique for detecting information on a finger touching a touch panel and operating an area at a position different from the finger contact position.

  Conventionally, an information terminal device provided with a touch panel is known. In the touch panel, a contact detection area is usually set for an object such as a button. When the user touches the contact detection area with a finger, the button or the like reacts. However, in such an operation method, since the detection area comes under the finger, there is a case where an accurate position cannot be grasped when touching. In addition, since it may be difficult to operate a distant object, operability is improved by a function of moving the screen itself, such as enlargement, reduction, and slide of the screen by gesture operation.

  Patent Document 1 discloses a user interface device that improves the operability by associating a plurality of finger movements with gestures and displaying a guide assumed from a trajectory during the operation. Patent Document 2 discloses an input device that switches operations based not only on the movement of a plurality of fingers, but also on the touch panel, how the palm and fingers open, the area of a polygon formed by connecting contact positions, and the like. Patent Document 3 discloses an apparatus that detects the position and shape of a hand, displays a virtual image corresponding to the shadow of the hand, and operates with the virtual hand.

  On the other hand, Non-Patent Document 1 proposes Finger-Specific Interaction that distinguishes a finger to be touched and gives a different meaning depending on the distinguished finger (Chapter 7). It proposes a painting tool for large screens, a software keyboard / music player that switches modes by operating fingers, a photo viewer for CSCW, and PIN personal authentication that also uses operating fingers (Section 7.6). Finger identification is based on image recognition using a visible light camera and color markers placed on top of the hand. Besides the above-mentioned patent documents, many input methods using gestures have been proposed.

JP 2011-077049 A JP 2011-003074 A JP 2012-022458 A

Yutaka Suzuki, University of Tsukuba Bachelor thesis, "Study on Interaction Method Based on Parallel Actions" March 2011

  However, Patent Documents 1 and 2 and Non-Patent Document 1 do not consider directly operating an object that is far away, and only associate a certain gesture with a command. Further, in Patent Document 3, when the hand is far from the screen, the shadow of the hand can be made far, which is effective for the operation of an object far from the hand. If there is, there is no difference from normal touch panel operation.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an input device and a program capable of improving the operability of an object far away from a hand when operating a touch panel. .

  (1) In order to achieve the above object, the present invention takes the following measures. That is, the input device of the present invention is an input device that detects information on a finger that has touched the touch panel and operates an area at a position different from the finger contact position, and is characterized by the detected finger information. A finger information feature amount extraction unit that extracts an amount; an instruction direction estimation unit that estimates an instruction direction from the extracted feature amount; an operation control unit that operates an area at a position different from a finger contact position; It is characterized by providing.

  In this way, information on the finger touching the touch panel is detected, a feature amount is extracted from the detected finger information, an instruction direction is estimated from the extracted feature amount, and an area at a position different from the finger contact position is detected. Since it is set as the operation target, it is possible to improve the operability of the area located away from the finger on the touch panel.

  (2) Moreover, the input device of the present invention includes an operation mode determination unit that determines an operation mode corresponding to the information on the finger and the estimated instruction direction among a plurality of predetermined operation modes, The operation control unit is characterized in that, based on the determination result of the operation mode, a region at a position different from a finger contact position is set as an operation target.

  As described above, an operation mode determination unit that determines an operation mode corresponding to the finger information and the estimated instruction direction among a plurality of predetermined operation modes is provided, and the operation control unit includes the operation mode. Based on the determination result, an area at a position different from the contact position of the finger is set as an operation target, so that the operability of the area at a position away from the finger on the touch panel can be improved.

  (3) The input device of the present invention further includes an operation mode storage unit that stores the information on the finger and the estimated finger orientation and the operation mode in association with each other.

  As described above, since the finger information and the estimated finger orientation and the operation mode are stored in association with each other, the processing speed can be improved, and as a result, the region in the position away from the finger on the touch panel can be improved. The operability can be improved.

  (4) In the input device according to the present invention, the operation mode is determined based on the feature amount, the number of detected finger contact positions, and the estimated directions.

  Thus, since the operation mode is determined based on the feature amount, the number of detected finger contact positions, and each estimated direction, even when the user operates using a plurality of fingers, It is possible to easily determine the operation mode.

  (5) In the input device of the present invention, the operation control unit determines a line segment based on the detected finger contact area and the estimated pointing direction, and operates a region corresponding to the end point of the line segment. It is a target.

  As described above, the operation control unit determines a line segment based on the detected finger contact area and the estimated instruction direction, and sets an area corresponding to the end point of the line segment as an operation target. Can be uniquely identified. As a result, it is possible to improve the operability of the area located away from the finger on the touch panel.

  (6) Further, in the input device according to the present invention, the operation control unit translates or rotates a line segment parallel to the indicated direction based on the operation mode and an arbitrary operation.

  As described above, the operation control unit translates or rotates the line segment parallel to the designated direction based on the operation mode and an arbitrary operation, so that it is possible to specify the region to be operated. As a result, it is possible to improve the operability of the area located away from the finger on the touch panel.

  (7) Further, in the input device according to the present invention, the operation control unit sets a region through which the line segment passes as an operation target.

  As described above, the operation control unit sets the region through which the line segment passes as the operation target, and thus it is possible to easily identify the region as the operation target. As a result, it is possible to improve the operability of the area located away from the finger on the touch panel.

  (8) In the input device of the present invention, the pointing direction estimation unit estimates the pointing direction based on line segments determined from a plurality of arbitrary points.

  As described above, the pointing direction estimation unit estimates the pointing direction based on a line segment determined from a plurality of arbitrary points, and thus it is possible to easily specify the region to be operated. As a result, it is possible to improve the operability of the area located away from the finger on the touch panel.

  (9) In the input device of the present invention, the operation control unit determines a line segment based on information input from a pressure sensor, a force sensor, or an external interface and an estimated instruction direction, and the line segment A region corresponding to the end point is an operation target.

  As described above, the operation control unit determines a line segment based on the information input from the pressure sensor, the force sensor, or the external interface and the estimated instruction direction, and sets the area corresponding to the end point of the line segment as the operation target. Therefore, it is possible to easily specify the region to be operated without detecting the finger contact area. As a result, it is possible to improve the operability of the area located away from the finger on the touch panel.

  (10) In the input device according to the present invention, the touch panel has a display function.

  As described above, since the touch panel has a display function, it is possible to provide a function of displaying an image and to reduce the size, and thus the present invention can be applied to a portable terminal or the like.

  (11) A program according to the present invention is a program for an input device that detects information on a finger that has touched the touch panel and operates an area at a position different from the contact position of the finger. A process for extracting a feature amount from the information, a process for estimating an instruction direction from the extracted feature amount, and corresponding to the finger information and the estimated finger orientation among a plurality of predetermined operation modes. A process for determining an operation mode to be performed, and a process for operating an area at a position different from a finger contact position based on the determination result of the operation mode, and causing the computer to execute a series of processes And

  As described above, information on the finger touching the touch panel is detected, a feature amount is extracted from the detected finger information, an instruction direction is estimated from the extracted feature amount, and a finger is selected from a plurality of predetermined operation modes. The operation mode corresponding to the information and the estimated finger orientation is determined, and based on the determination result of the operation mode, an area at a position different from the contact position of the finger is set as an operation target. It becomes possible to improve the operability of a region located at a distant position.

  According to the present invention, information on a finger touching the touch panel is detected, a feature amount is extracted from the detected finger information, a finger direction is estimated from the extracted feature amount, and a plurality of predetermined operation modes are determined. Among them, the operation mode corresponding to the finger information and the estimated instruction direction is determined, and based on the determination result of the operation mode, an area at a position different from the contact position of the finger is set as an operation target. It becomes possible to improve the operability of the region located away from the finger.

It is a block diagram which shows schematic structure of the input device which concerns on this embodiment. It is a figure which shows the detection angle of a finger | toe, and the direction of the intended finger | toe. It is a flowchart which shows operation | movement of the input device which concerns on 1st Embodiment. It is a flowchart which shows the operation example of operation mode discrimination | determination. It is a figure which shows the operation modes 1 and 2. FIG. It is a figure which shows an example of adjustment of the length in the operation method. It is a figure which shows the operation mode 3. FIG. It is a figure which shows the operation modes 4 and 5. It is a figure which shows area | region selection. It is a figure which shows an example of the operation mode which adapts to 3D. It is a figure which shows the example made into the direction which shows the direction which extended the line segment which two fingers comprise. It is a figure which shows the example applied to the character input selection and candidate character selection in a smart phone. It is a figure which shows the state which made the small display and the large display cooperate. It is a figure which shows the example which applied the input device which concerns on this embodiment to the large sized display. It is a figure which shows the example of a display at the time of applying this invention to a game. It is a figure which shows the example of a screen display of a keyboard.

  When operating the touch panel, the input device according to the embodiment of the present invention detects the contact surface of the first finger to improve the operability of an object far away from the hand, and the direction indicated by the first finger and the other It has a function that can be operated even at a position away from the contact point of the finger to the touch panel by the gesture operation of the finger. In addition, it has a function that enables a plurality of operations by changing the number of contact surfaces and switching.

  FIG. 1 is a block diagram illustrating a schematic configuration of the input device according to the present embodiment. In FIG. 1, the display unit 1 displays various operation screens. The horizontal direction of the display unit 1 is referred to as the X direction, and the vertical direction is referred to as the Y direction. The touch panel input unit 3 is input by a user using a touch panel. Touch coordinate information of one point or a plurality of points touched by the user is acquired. The finger shape information acquisition unit 5 acquires the shape information of the finger being operated. The acquisition method includes, for example, acquisition of a contact portion by FTIR (Frustrated Total Internal Reflection), but means is not limited.

  The finger information feature amount extraction unit 7 extracts feature amounts from the finger shape information acquired from the finger shape information acquisition unit 5. The finger direction estimation unit 9 estimates the direction of the finger from the extracted feature amount. The finger direction estimation unit constitutes an instruction direction estimation unit. The operation mode determination unit 11 determines the operation mode from the information of the finger information feature amount extraction unit 7. The operation control unit 13 controls the operation. In particular, an area at a position different from the contact position of the finger is set as an operation target based on the determination result of the operation mode. The display control unit 15 reflects the information from the operation control unit 13 on the display unit. The operation mode storage unit 17 stores the relationship between the information of the finger information feature amount extraction unit 7 and the operation mode.

[First Embodiment]
In the first embodiment, a case where finger shape information can be acquired will be described. In the first embodiment, a touch panel capable of detecting shape information of a finger contact surface based on FTIR is used in order to enable finger identification without an external camera and a color marker. Thereby, the direction of the user's finger (instruction direction) can be determined from the long axis of the ellipse indicated by the contact surface shown in FIG. Further, since the major axis of the ellipse does not necessarily match the orientation indicated by the user, correction may be applied.

  FIG. 3 is a flowchart illustrating the operation of the input device according to the first embodiment. First, the position of touch input is acquired (step S1). Further, at the time of user touch input, the time series information of the number of touched fingers, the touch coordinates of each finger, and finger shape information is acquired (step S2). Next, a finger information feature amount is extracted (step S3). Here, the feature amount is extracted from the finger shape information. The feature amounts include the contact area S of each finger, the circumference l, the major axis / minor axis lengths dmax and dmin of the approximate ellipse, the contact area ΔS, and the circumference change Δl.

  Next, operation mode discrimination / control is performed (step S4). The operation mode is discriminated from the number of touch points of the touched finger, the touch coordinates of each finger, the angle formed by the long axis of each finger, the coordinates and their distance, etc., and input control corresponding to the mode is performed. Discrimination is not only a method of switching modes based on certain conditions as shown in the flowchart described later, but also using pattern recognition such as SVM and HMM, and the model can be switched by the user or learned Good. In addition, user identification may be performed using a finger feature, a gesture, or the like, and the condition / model may be switched for each user.

  Next, the operation control of the operation mode N is performed (step S5), and the operation result on the display unit 1 is reflected (step S6). Next, it is determined whether or not the operation has been completed (step S7). If the operation has not been completed, the process proceeds to step S1. On the other hand, when the operation is finished in step S7, the process is finished.

FIG. 4 is a flowchart showing an operation example of the operation mode discrimination in FIG. First, the number of contact fingers is detected (step T1). Here, <when the contact point is one point>, the contact area determines the magnitude of S ₁ and S _alpha (step T2). When S ₁ <S _α, it is determined as a normal touch operation (step T3). On the other hand, when S ₁ <S _α is not satisfied, that is, when the operation is performed only in the direction indicated by the first finger, the operation mode is set to 1 (step T4), and the line segment is β ₁ × S ₁ pixel in the direction indicated by the finger. And the position of the end point is set as the operation target (step T5).

As shown in FIG. 5, a line segment extends in the direction indicated by the finger in the direction indicated by the finger, and the end point has a function to be an operation target. Other line segment length L shall be determined from the function according to the area S ₁ of the finger. For example, L = β ₁ × S ₁ .

Next, in step T1, <when there are two contact points>, θ ₁ > θ _α is determined for the angle formed by the intersection of the major axes of the ellipses indicated by the contact surfaces. If θ ₁ > θ _α , the operation mode is set to 2 (step T7). Operation mode 2 is a change in length depending on the distance between two fingers. The line segment is extended by β ₂ × d (distance between two points) in the direction indicated by the first finger, and the end point is set as the operation target (step T8). As shown in FIG. 5, it is assumed that the length L is determined by a function having the distance d between the first finger and the second finger as a variable. The variable may include the coordinates (xi, yi) of the first and second fingers in addition to the distance. For example, L = β2 × d. As shown in FIG. 6, the determination may be made not only within the screen but also by operating an interface outside the screen.

In step T6, if θ ₁ > θ _α is not satisfied, the operation mode is set to 3 (step T9). The operation mode 3 is a case where a gesture of pinching a finger is performed. The line segment is extended by βi × Si pixels in the direction indicated by each finger, and the end point is set as the operation target (step T10). In the case of a gesture of pinching a finger, as shown in FIG. 7, the line segment extends in the direction indicated by each of the first finger and the second finger, and the end point is a function to be operated. The length of the line segment is determined from a function depending on the area Si of each finger. For example, Li = βi × Si.

In step T1, <when there are three contact points>, θ ₂ > θ _α is determined for the angle formed by the intersection of the major axes of the ellipses indicated by the contact surfaces. When θ ₂ > θ _α, that is, when the start point of the operation mode 2 is operated with another finger as shown on the left side of FIG. In operation mode 2, the starting point is moved by the slide amount of the third finger (step T13). It is assumed that the movement distance is determined from a function based on the movement distance d3 of the third finger.

In step T11, when θ ₂ > θ _α is not satisfied, that is, as shown on the right side of FIG. 8, when the start point of operation mode 3 is operated with another finger, the operation mode is set to 5 (step T14). . In operation mode 3, the starting point is moved by the slide amount of the third finger (step T15). It is assumed that the movement distance is determined from a function based on the movement distance d3 of the third finger.

  In step T1, <when the number of contact points is 4 or more (N)> or <other operation mode>, for example, as shown in FIG. This is a mode for selecting a region where has passed.

  As another example of the operation mode, as shown in FIG. 10, in the operation of 3D display on the screen, in addition to the above-described operation mode, there is a mode in which the operation is performed in the depth direction.

  The function is switched by another finger operation or some button operation provided on the apparatus main body.

[Second Embodiment]
In the second embodiment, a case where finger shape information cannot be acquired will be described. In the second embodiment, the correspondence is defined by changing the correspondence between the number of finger contact points and the mode. For example, the definition of operation mode 1 (line segment expansion / contraction) is changed. When the direction of the finger cannot be detected, as shown in FIG. 11, the direction indicating the direction in which the line segment formed by the two fingers is extended is set. The line length is changed by the pressure at touch operation or other external operation.

  Also, the definition of the operation mode 2 is changed. Touch two points to determine the direction, and change the length by the distance between the two points.

(Distinction using other features)
In the operation of the first embodiment or the second embodiment, the finger area is used, but other feature amounts may be used. As the feature amount, for example, the contact area S, the circumference l, the major axis / minor axis lengths dmax and dmin of the approximate ellipse, the contact area ΔS and the change Δl in the circumference, touch coordinates, time series data thereof, or other sensors May be used. Examples of other sensors include a pressure sensor and a force sensor.

  In the present embodiment, the mode classification may be determined from the number of fingers, the angle of the fingers, and the distance between the fingers, not the number of fingers shown in the above conditions, and includes differences due to one-handed operation or two-handed operation. Classification is also acceptable.

  In addition, as shown in FIG. 11, in the case of a large display, there are cases where a plurality of users operate, and the number of users may be determined from the above feature amount or the like, and the operation of each user may be applied.

(Switching mode)
In the above embodiment, the feature amount is used to switch the mode. However, the mode may be selected using a device button or a selection screen on the screen.

(Application operation example)
FIG. 12 is a diagram illustrating an example of application to character input selection and candidate character selection in a smartphone. Normally, a position where character input becomes active is selected by touching that position. In this case, there are inconveniences such as having to touch the position on purpose and difficult to make fine adjustment because the active position is under the finger. Therefore, by applying the present invention, it is possible to set a position to be activated without largely moving a finger.

  Then, at the activated position, it becomes possible to perform functions such as drawing tools, screen arrangement, selection at once, deletion, arrangement, and dragging.

  FIG. 13 is a diagram illustrating a state in which a small display and a large display are linked. In this example, the positional relationship between the display edges is obtained from a camera image or the like using a barcode, and an operation corresponding to the operation mode is reflected on the operation of the large display on the small display. In this case, a communication unit and a position estimation unit are required. Existing techniques can be used to do this.

  Further, as shown in FIG. 14, a configuration may be adopted in which a certain area in the large display is set as an operation area, and the scale of the operation is changed using this operation area and reflected in the operation of the large display.

  FIG. 15 shows a display example when the present invention is applied to a game. FIG. 15 shows an enemy shooting-down game. Thus, the present invention can be applied even when a game operation is performed.

  FIG. 16 is a diagram illustrating a screen display example of the keyboard. As shown in FIG. 16, it can be applied to security by a password. In the input of the personal identification number, it is possible to include any one of the touch position of the user, the designated position, the passage position of the line segment, and the selection area.

  As described above, according to the present embodiment, information on a finger touching the touch panel is detected, a feature amount is extracted from the detected finger information, a finger direction is estimated from the extracted feature amount, and predetermined. The operation mode corresponding to the finger information and the estimated finger orientation is determined from the plurality of operation modes, and an area at a position different from the finger contact position is operated based on the operation mode determination result. Since it is the target, it becomes possible to improve the operability of the region located away from the finger on the touch panel.

DESCRIPTION OF SYMBOLS 1 Display part 3 Touch-panel input part 5 Finger shape information acquisition part 7 Finger information feature-value extraction part 9 Finger direction estimation part 11 Operation mode discrimination | determination part 13 Operation control part 15 Display control part 17 Operation mode memory | storage part

Claims (8)

An input device that detects information of a finger that has touched the touch panel and targets a region at a position different from the contact position of the finger,
A finger information feature amount extraction unit that extracts a feature amount from the detected finger information;
An instruction direction estimating unit for estimating an instruction direction from the extracted feature amount;
An operation mode determination unit for determining an operation mode corresponding to the information on the finger and the estimated instruction direction among a plurality of predetermined operation modes;
An operation mode storage unit for storing the finger information and the estimated instruction direction and the operation mode in association with each other;
An operation control unit that operates on a region that is specified from the feature amount and a function determined in accordance with the operation mode, and a region that is specified from the pointing direction and is different from the contact position of the finger. And an input device. The input device according to claim 1 , wherein the operation mode is determined based on the feature amount, the number of detected finger contact positions, and the estimated directions . The operation control unit defines a line segment based on the contact area of a finger detected and estimated indicated direction, according to claim 1, characterized in that the operation target region corresponding to the end points of the line segment Input device. The operation control part, based on the operating mode and any operation input device according to claim 1, wherein the moving or rotating translating the parallel line segments in the designated direction. The input device according to claim 4 , wherein the operation control unit sets a region through which the line segment passes as an operation target. The input device according to claim 1, wherein the pointing direction estimation unit estimates a pointing direction based on a line segment determined from a plurality of arbitrary points . The operation control unit determines a line segment based on information input from a pressure sensor, a force sensor, or an external interface and an estimated instruction direction, and sets a region corresponding to an end point of the line segment as an operation target. The input device according to claim 1, wherein: A program for an input device that detects information on a finger touching a touch panel and operates an area at a position different from the contact position of the finger,
A process of extracting feature amounts from the detected finger information;
A process of estimating a pointing direction from the extracted feature amount;
A process of determining an operation mode corresponding to the information of the finger and the estimated instruction direction among a plurality of predetermined operation modes;
A process for operating an area at a position different from the contact position of the finger, which is specified from the length of a line segment determined from the function and a function determined corresponding to the operation mode, and the pointing direction; A program characterized by causing a computer to execute a series of processes.

Images