JP2011209928A - Mobile terminal - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a mobile terminal excellent in the operability of a user, and enabled in control which is reduced in erroneous operation by the user even in a situation that the user cannot move a hand.SOLUTION: The mobile terminal 1 includes: a movement detection part 3 detecting the movement of a pupil and a face of the user; an operation command storage part 6 storing an operation command related to the movement of a cursor displayed on a display part 2 in association with the movement of the pupil of the user; a selection command storage part 7 storing a selection command related to the selection of execution/stopping of processing in association with the movement of the face of the user; a movement decision part 4 deciding whether or not the detected movement accords with the operation command or the selection command when the movement of the user is detected by the movement detection part 3; and a processing control part 5 controlling the selection of the execution/stopping of the processing or the movement of the cursor corresponding to the according command when it is decided by the movement decision part 4 that the movement accords with the operation command or the selection command.

Description

  The present invention relates to a portable terminal that performs control corresponding to the actions of a user's pupil and face.

  In recent years, there is a technology that allows a user to operate a terminal with a user's eye movement even in a situation where the user cannot move his / her hand. For example, Patent Literature 1 discloses a portable terminal that moves and selects an item selection unit in a display screen based on the movement of a user's eyeball with black eyes and the presence or absence of blinking.

JP 2007-102415 A

  However, the above-described mobile terminal can only perform terminal operations based only on the movement of the black eyes and the presence or absence of blinks, so that the user's operability is poor, and therefore there are problems that the user has many malfunctions.

  The present invention has been made to solve the above problems, and an object of the present invention is to provide a portable terminal that is excellent in user operability and can be controlled with few malfunctions by the user even when the user cannot move his / her hand. And

  In order to solve the above-described problem, a mobile terminal according to the present invention includes an operation detection unit that detects the movement of the user's pupil and face, and an operation related to the movement of the cursor displayed on the display unit in association with the movement of the user's pupil. When an operation command storage means for storing a command, a selection command storage means for storing a selection command related to selection of execution / stop of processing in association with a motion of the user's face, and a user motion detected by the motion detection means In addition, an operation determination unit that determines whether or not the detected operation matches the operation command or the selection command, and the operation determination unit determines that the operation matches the operation command or the selection command. And a process control means for controlling selection of movement of the corresponding cursor or execution / stop of the process.

  In this portable terminal, when the movement of the user's pupil and face is detected, it is determined whether or not the detected movement matches the operation command or the selection command. The movement of the cursor corresponding to or selection of execution / stop of the process is controlled. In this portable terminal, even if the buttons are not operated, the movement of the cursor and the selection of execution / stop of the process are performed by detecting the movement of the pupil and the face by the user, so that the user remains looking at the screen Can easily operate a portable terminal. Further, in this portable terminal, since the operation command based on the pupil motion and the selection command based on the face motion are separated, the operability for the user can be ensured and the malfunction can be prevented.

  The operation command storage means stores a plurality of operation commands with priorities, the selection command storage means stores a plurality of selection commands with priorities, and the operation determination means has priority It is preferable to determine whether or not the operation matches the operation command or the selection command in descending order.

  In this portable terminal, it is determined whether or not the operation matches the operation command or the selection command in descending order of priority. Accordingly, the user can preferentially execute the control intended by the user by increasing the priority of the operation command or the selection command. Therefore, this portable terminal is excellent in user operability, and therefore, malfunctions by the user can be further reduced.

  The operation commands preferably include a start command for starting the movement of the cursor by the movement of the user's pupil and an end command for ending the movement of the cursor by the movement of the user's pupil.

  In this case, the user can cause the mobile terminal to control the movement of the cursor only in a situation where it is desired to control the movement of the cursor by separating the start and end of the movement of the cursor using the start command and the end command. Therefore, malfunction can be prevented more reliably.

  Further, the operation command preferably includes a position return command for returning the cursor position to the cursor position when the cursor movement is started by the start command.

  In this case, even if the user accidentally moves the cursor position to an unintended position, the cursor position can be immediately restored to the cursor position when the cursor movement is started by the start command.

  The selection command includes a start command for starting selection of execution / stop of processing based on the user's face motion and an end command for ending selection of execution / stop of processing based on the user's face motion. Preferably it is.

  In this case, the user uses the start command and end command to separate the start and end of the selection of execution / stop of the process, so that the user can execute the process / The stop selection can be controlled. Therefore, malfunction can be prevented more reliably.

  ADVANTAGE OF THE INVENTION According to this invention, even in the situation where a user cannot move a hand, the portable terminal which is excellent in a user's operativity and can perform control with few malfunctions by a user can be provided.

It is a block diagram which shows the functional component of the portable terminal which concerns on one Embodiment of this invention. 2 is a hardware configuration diagram of the mobile terminal 1. FIG. It is a table which shows the example of the data which the operation command storage part 6 hold | maintains. It is a table which shows the example of the data which the selection command storage part 7 hold | maintains. It is a figure which shows an example of the screen displayed on the display part 2 at the time of performing control corresponding to a user's pupil and face operation | movement in the portable terminal 1. FIG. It is a figure which shows an example of the screen displayed on the display part 2 at the time of performing control corresponding to a user's pupil and face operation | movement in the portable terminal 1. FIG. It is a figure which shows an example of the screen displayed on the display part 2 at the time of performing control corresponding to a user's pupil and face operation | movement in the portable terminal 1. FIG. It is a figure which shows an example of the screen displayed on the display part 2 at the time of performing control corresponding to a user's pupil and face operation | movement in the portable terminal 1. FIG. FIG. 6 is a flowchart showing the operation of the mobile terminal 1. FIG. 6 is a flowchart showing the operation of the mobile terminal 1.

  FIG. 1 is a schematic diagram showing an embodiment of a mobile terminal according to the present invention. As shown in FIG. 1, the mobile terminal 1 includes, as functional components, an operation detection unit 3 (operation detection unit), an operation determination unit 4 (operation determination unit), a process control unit 5 (process control unit), an operation A command storage unit 6 (operation command storage unit) and a selection command storage unit 7 (selection command storage unit) are provided. Moreover, the portable terminal 1 is provided with the display part 2 (display means) as one of the physical components as shown in FIG.

  The portable terminal 1 is composed of hardware such as a CPU. FIG. 2 is a hardware configuration diagram of the mobile terminal 1. As shown in FIG. 2, the portable terminal 1 shown in FIG. 1 physically includes a CPU 51, a RAM 52 and a ROM 53 that are main storage devices, input devices 54 such as numeric keys and cameras that are input devices, a display, and the like. The computer system includes an output device 55, a communication module 56 that is a data transmission / reception device such as a network card, and an auxiliary storage device 57 such as a hard disk. The functions of the constituent elements shown in FIG. 1 are such that predetermined computer software is loaded on hardware such as the CPU 51 and RAM 52 shown in FIG. 2 to control the input device 54, output device 55, and communication under the control of the CPU 51. This is realized by operating the module 56 and reading and writing data in the RAM 52 and the auxiliary storage device 57.

  Hereinafter, each component of the portable terminal 1 is demonstrated based on the component shown in FIG.

  The portable terminal 1 employs a graphical user interface as a user interface, and the display unit 2 displays a graphic in the graphical user interface. Examples of graphics in a graphical user interface include icons, cursors, dialog boxes, buttons, and the like.

  The motion detection unit 3 detects the motion of the user's pupil and face. For example, based on the user's pupil and face images captured by the camera that is the input device 54 at regular time intervals, the motion detection unit 3 detects the user's pupil and face motions from the difference information of the captured images.

  The operation command storage unit 6 stores an operation command related to the movement of the cursor displayed on the display unit 2 in association with the movement of the user's pupil.

  The operation command storage unit 6 may store a plurality of operation commands with priorities. The operation determination unit 4 to be described later determines whether or not the operation of the user's pupil matches the operation command from the highest priority among the plurality of operation commands to which priority is given, and a process control unit to be described later 5 controls the movement of the cursor corresponding to the matched operation command. That is, when the user intends to preferentially execute the control intended by the user, the control is preferentially controlled by the processing control unit 5 when the user increases the priority of the operation command related to the control. .

  The operation command may include a start command for starting the movement of the cursor by the user's pupil movement and an end command for ending the movement of the cursor by the user's pupil movement. By using the start command and the end command, the start and end of the movement of the cursor can be separated, and the mobile terminal 1 can control the movement of the cursor only in a situation where it is desired to control the movement of the cursor.

  The operation command may include a position return command for returning the cursor position to the cursor position when the cursor movement is started by the start command. Thus, even when the user accidentally moves the cursor position to an unintended position, the cursor position can be immediately restored to the cursor position when the cursor movement is started by the start command.

  An example of the operation command stored by the operation command storage unit 6 is shown in FIG. 3 includes a “command No. (command number)” for identifying each operation command, a “motion content” that is a motion of the user's pupil associated with the operation command, the “priority order” described above, The “judgment criterion”, which is a criterion for determining whether or not the motion of the pupil detected by the motion determination unit 4 matches the operation command, and the motion of the pupil detected by the motion determination unit 4 described later is an operation. “Processing content” that is processing content related to movement of a cursor controlled by the processing control unit 5 described later when it is determined to match the command is included. The priority indicates that the smaller the number, the higher the priority.

  Of the operation commands shown in FIG. Corresponds to the start command described above, and the command No. Corresponds to the above-mentioned end command, and the command No. 4 corresponds to the position return command described above.

  The selection command storage unit 7 stores a selection command related to selection of execution / stop of processing in association with the movement of the user's face.

  The selection command storage unit 7 may store a plurality of selection commands with priorities. The operation determination unit 4 described later determines whether or not the user's face operation matches the selection command in descending order of priority among the plurality of selection commands assigned priority, and a processing control unit described later 5 controls selection of execution / stop of processing corresponding to the matched selection command. That is, when the user wants to preferentially execute the control intended by the user, the control is preferentially controlled by the processing control unit 5 when the user increases the priority of the selection command related to the control. .

  The selection command includes a start command for starting selection of execution / stop of processing based on the user's face motion and an end command for ending selection of execution / stop of processing based on the user's face motion. May be. By using the start command and end command, the start and end of the selection of execution / stop of the process can be separated, and the execution / stop of the process is performed on the portable terminal 1 only in a situation where it is desired to control the selection of execution / stop of the process. The selection can be controlled.

  An example of the selection command stored by the selection command storage unit 7 is shown in FIG. 4 includes a “command No. (command number)” for identifying each selected command, a “motion content” that is a motion of the user's face associated with the selected command, the “priority order” described above, The “judgment criterion”, which is a criterion for determining whether the motion of the face detected by the motion determination unit 4 coincides with the selection command, the motion of the face detected by the motion determination unit 4 described later is selected. “Processing content” that is processing content related to selection of execution / stop of processing controlled by the processing control unit 5 described later when it is determined that the command matches is included. The priority indicates that the smaller the number, the higher the priority.

  When the motion detection unit 3 detects a user motion, the motion determination unit 4 determines whether the detected motion matches an operation command or a selection command. The operation determination unit 4 may determine whether the operation matches the operation command or the selection command from the highest priority order.

  When the operation determination unit 4 determines that the operation matches the operation command or the selection command, the process control unit 5 controls the movement of the cursor corresponding to the matched command or the selection of execution / stop of the process.

  Hereinafter, a specific example of the operation command coincidence determination by the operation determination unit 4 and the operation command control by the processing control unit 5 will be described with reference to FIG. First, it is assumed that the motion detection unit 3 detects a motion of opening both eyes as the motion of the user's pupil. At that time, the operation detection unit 3 detects that the operation of opening both eyes is continued for, for example, 1.2 seconds. Next, the operation determination unit 4 determines whether or not the detected operation of opening both eyes matches the operation command stored by the operation command storage unit 6. The operation determination unit 4 includes a command No. among the operation commands in FIG. 3 stored by the operation command storage unit 6. It is determined that the operation content of the operation command with 1 corresponds. Further, it is determined that the operation of opening both eyes satisfies the “continuation of 0.5 seconds or more” which is the determination criterion of the operation command, and the command No. Is determined to match the operation command of 1. Then, the process control unit 5 receives the command No. The control corresponding to “start of cursor movement”, which is the processing content of the operation command 1 is performed.

  Next, a specific example of selection command matching determination by the operation determination unit 4 and selection command control by the processing control unit 5 will be described with reference to FIG. First, it is assumed that the motion detection unit 3 detects the motion of closing the right eye as the motion of the user's face. At that time, the operation detection unit 3 detects that the operation of closing the right eye is continued for, for example, 0.7 seconds. Next, the operation determination unit 4 determines whether or not the detected operation of closing the right eye matches the selection command stored by the selection command storage unit 7. The operation determination unit 4 includes a command No. of the selection commands stored in the selection command storage unit 7 in FIG. Is determined to correspond to the operation content of the selection command of No. 3. Further, it is determined that the operation of closing the right eye satisfies the “continuation of 0.5 seconds or more” which is the determination criterion of the selected command. It is determined that the selection commands of 3 match. Then, the process control unit 5 receives the command No. Performs the control corresponding to “Select Yes” which is the processing content of the selection command 3. Note that “selecting Yes” means, for example, selecting the “Yes” button included in the dialog box displayed on the display unit 2.

  In the specific example of the selection command, it is assumed that the motion detection unit 3 detects that the operation of “closing both eyes and tilting the face forward” is continued for 0.7 seconds. In this case, of the selection commands in FIG. Is a command No. The selection command of 7 also satisfies the matching condition. However, the command No. The priority of the selection command of 3 is 3, and the command No. 7 has a priority of 7, and the command No. In this case, the operation determination unit 4 determines whether the command No. 3 has a higher command priority. It is determined that the selection commands of 3 match.

  Note that the action of the user's face associated with the selection command, such as “close both eyes and tilt the face forward”, which is a specific example of the action content described above, may be a plurality of face actions. In that case, the motion determination unit 4 determines whether or not the corresponding selection commands match depending on whether or not the motions of the plurality of faces all match.

  Next, an example of a screen displayed on the display unit 2 when performing control corresponding to the movement of the user's pupil and face in the mobile terminal 1 will be described with reference to FIGS. FIG. 5 is a diagram showing a state in which icons A1 to A5 and a cursor B are displayed on the display unit 2. Next, the motion detection unit 3 detects an operation of opening both eyes of the user, and the motion determination unit 4 detects the command No. 1 in FIG. Are determined to match the operation command (start command), and the process control unit 5 executes control of “start movement of cursor” which is the processing content of the start command.

  Next, the movement detection unit 3 detects the movement of the user's black eye, and the operation determination unit 4 detects the command number of FIG. Are determined to match, and the processing control unit 5 executes control of “move the cursor in accordance with the movement of the black eye”, which is the processing content of the operation command. FIG. 6 is a diagram showing that the cursor B has moved along the locus C and moved onto the icon A2 in accordance with the movement of the black eye.

  Next, the operation detecting unit 3 detects an operation of closing both eyes of the user twice, and the operation determining unit 4 detects the command No. 2 in FIG. Are determined to match the operation command (end command) 2, and the process control unit 5 executes control of “end cursor movement” which is the processing content of the end command. Accordingly, it is assumed that a dialog box D for confirming activation of the electronic mail software corresponding to the icon A2 is displayed. FIG. 7 is a diagram showing a state in which the dialog box D is displayed.

  Next, the motion detection unit 3 detects the motion of tilting the user's face in the left and right order. Is determined to match the 1 selection command (start command), and the process control unit 5 executes control of “start selection of execution / stop of process” which is the process content of the start command. Next, the motion detection unit 3 detects the motion of the shape of “Ye” by raising the mouth angle of the user's lips by a certain angle or more. Are selected, and the process control unit 5 executes the control of “select Yes” which is the processing content of the selected command. FIG. 8 shows a state where the Yes button on the dialog box D is selected (selected state E).

  Next, the operation of the mobile terminal 1 configured as described above will be described. 9 and 10 are flowcharts showing the operation of the mobile terminal 1. In addition, although the next to S6 in FIG. 9 is M, this means that it follows M in FIG. That is, S6 in FIG. 9 follows S7 in FIG.

  First, the motion detection unit 3 detects the user's pupil motion (S1), the motion determination unit 4 determines that the detected pupil motion matches the start command, and the processing control unit 5 moves the cursor. Control to start is performed (S2). Next, the motion detection unit 3 detects the movement of the user's pupil (S3), and the motion determination unit 4 determines that the detected pupil motion matches the operation command related to the movement of the cursor in accordance with the pupil motion. The process control unit 5 moves the cursor in accordance with the movement of the pupil (S4). Next, the motion detection unit 3 detects the motion of the user's pupil (S5), the motion determination unit 4 determines that the detected pupil motion matches the end command, and the processing control unit 5 moves the cursor. Control to end is performed (S6).

  Next, the motion detection unit 3 detects the motion of the user's face (S7), the motion determination unit 4 determines that the detected facial motion matches the start command, and the process control unit 5 executes the process. Control for starting selection of stop is performed (S8). Next, the motion detection unit 3 detects the motion of the user's face (S9), and the motion determination unit 4 determines that the detected face motion matches the selection command related to selection of execution / stop of processing, and processing The control unit 5 selects whether to execute or stop the process (S10). Next, the motion detection unit 3 detects the motion of the user's face (S11), the motion determination unit 4 determines that the detected face motion matches the end command, and the process control unit 5 executes the process. -Control which complete | finishes selection of a stop is performed (S12).

  As described above, in the portable terminal 1, when the movement of the user's pupil and face is detected, it is determined whether or not the detected movement matches the operation command or the selection command. In this case, the movement of the cursor corresponding to the matched command or the selection of execution / stop of the process is controlled. In this portable terminal 1, even if the operation of buttons or the like is not performed, the movement of the cursor and the selection of execution / stop of the process are performed by detecting the movement of the pupil and the face by the user. The portable terminal 1 can be easily operated in the state. Further, in the portable terminal 1, since the operation command based on the pupil motion and the selection command based on the face motion are separated, the operability for the user can be ensured and the malfunction can be prevented.

  Moreover, in this portable terminal 1, it is judged whether operation | movement corresponds to an operation command or a selection command from an order with a high priority. Accordingly, the user can preferentially execute the control intended by the user by increasing the priority of the operation command or the selection command. Therefore, this portable terminal 1 is excellent in user operability, and therefore, malfunctions by the user can be reduced.

  In the portable terminal 1, the user uses the start command and the end command to separate the start and end of the cursor movement, so that the portable terminal 1 controls the movement of the cursor only in a situation where the cursor movement is desired to be controlled. be able to. Therefore, malfunction can be prevented more reliably.

  Further, even if the user accidentally moves the cursor position to an unintended position, the mobile terminal 1 can immediately return the cursor position to the cursor position when the cursor movement is started by the start command. it can.

  In this portable terminal 1, the user uses the start command and the end command to separate the start and end of the process execution / stop selection, and only in the situation where the user wants to control the process execution / stop selection. The selection of execution / stop of processing can be controlled. Therefore, malfunction can be prevented more reliably.

  DESCRIPTION OF SYMBOLS 1 ... Portable terminal, 2 ... Display part, 3 ... Motion detection part, 4 ... Motion judgment part, 5 ... Process control part, 6 ... Operation command storage part, 7 ... Selection command storage part.

Claims (5)

Motion detection means for detecting the motion of the user's eyes and face;
Operation command storage means for storing an operation command related to movement of the cursor displayed on the display means in association with the movement of the user's pupil;
A selection command storage means for storing a selection command related to selection of execution / stop of processing in association with the movement of the user's face;
An operation determination unit that determines whether the detected operation matches the operation command or the selection command when a user operation is detected by the operation detection unit;
A process control means for controlling selection of movement of the cursor corresponding to the matched command or execution / stop of the process when the action determination means determines that the action matches the operation command or the selection command;
A portable terminal characterized by comprising: The operation command storage means stores a plurality of operation commands with priorities,
The selection command storage means stores a plurality of selection commands with priorities,
2. The mobile terminal according to claim 1, wherein the operation determining unit determines whether the operation matches the operation command or the selection command in descending order of priority.   The operation command includes a start command for starting movement of the cursor by a user's pupil movement and an end command for ending movement of the cursor by a user's pupil movement. Item 3. A portable terminal according to item 1 or 2.   4. The operation command includes a position return command for returning the position of the cursor to the position of the cursor when the movement of the cursor is started by the start command. Mobile device.   The selection command includes a start command for starting selection of execution / stop of the process based on the user's face motion, and an end command for ending selection of execution / stop of the processing based on the user's face motion. It is contained, The portable terminal as described in any one of Claims 1-4 characterized by the above-mentioned.

Images