JP2023001310A - mobile terminal - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a mobile terminal, such as smartphone, configured to prevent operation error without impairing operability.

SOLUTION: A mobile terminal includes: a touch panel which displays an icon and detects a touch, the touch panel displaying a first region for displaying an icon to start an application and a second region different from the first region, the first region and the second region being displayed in predetermined positions on the touch panel; a camera for imaging the surroundings of the touch panel; and a processor. The processor detects a point of sight of eyes captured by the camera, outputs a position of the detected point of sight on the first region, controls a cursor to be displayed on the first region, determines whether a preset input condition is satisfied or not, for a touch input to the second region and an input of the point of sight on the first region, and starts an application program corresponding to the icon when a determination is made that the preset input condition is satisfied.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2023,JPO&INPIT

Description

The present invention relates to mobile terminals.

In recent years, communication and information processing technology represented by smartphones, camera imaging technology, and semiconductor technology have made remarkable progress. is being developed.

For example, an advanced man-machine interface system has been introduced, such as installing a voice recognition function in a smartphone and operating the smartphone by voice. In addition, touch pads that enable intuitive operation using icons are commonly installed in smartphones. The reason why touchpad operation is common is that it can be manufactured due to its stable operation and inexpensive parts.

However, even if these are excellent in operability, the voice may be affected by ambient noise and other people's voices, and the touchpad may slip and operate a different icon nearby. It is possible.

As a technique for preventing erroneous operations, for example, Patent Document 1 discloses a line-of-sight effective range for detecting whether or not the user's line-of-sight is within the effective line-of-sight range for operation. is not within the effective range, and a technique for preventing erroneous operations is described.

JP 2015-28734 A

By using the technique described in Patent Document 1, it is possible to prevent erroneous key operations. However, such technology may impair the operability of smartphones, and is difficult to apply to smartphones.

An object of the present invention is to provide a technique for preventing erroneous operations without impairing operability in mobile terminals such as smartphones.

A representative mobile terminal according to the present invention includes a mobile terminal that includes a touch panel that displays icons and detects touches, a camera that captures an image of the surroundings of the touch panel, and a first finger of a hand captured by the camera. a detection unit that detects the type of the finger including the second finger; an information management unit that outputs the type of the finger detected by the detection unit and the touch position of the touch panel; a cursor display unit for controlling to display a cursor on the touch panel according to the touch position on the touch panel by the first finger; an input of the touch position on the touch panel by the first finger output from the information management unit; a condition management unit for determining whether a preset input condition is satisfied for an input of a touch position on the touch panel by a finger; and an application program execution unit that launches an application program corresponding to the icon based on the icon.

Advantageous Effects of Invention According to the present invention, it is possible to provide a technique for preventing erroneous operations without impairing operability of a mobile terminal such as a smartphone.

It is a figure which shows the example of a smart phone and its operation. It is a figure which shows the example which displayed the icon on the smart phone. FIG. 10 is a diagram showing an example of text displayed on a smartphone; It is a figure which shows the example of a display of a tablet terminal. It is a figure which shows the example which displayed the map on the tablet terminal. It is a figure which shows the example which enlargedly displayed the map on the tablet terminal. It is a figure which shows the example which displayed the detailed information of the map on the tablet terminal. It is a figure which shows the example which displayed the map on the tablet terminal. It is a figure which shows the example which displayed the detailed information of the map on the tablet terminal. FIG. 4 is a diagram showing an example of calibration; FIG. 4 is a diagram showing an example of distance measurement with two cameras; 1 is a diagram showing an example of a glasses-type device and a mobile terminal; FIG. FIG. 10 is a diagram showing an example of an operation without using a camera; FIG. 10 is a diagram illustrating an example of a tablet terminal that detects touches of multiple fingers; It is a figure which shows the example of the screen of a tablet terminal. FIG. 3 shows an example of a smart phone that includes a special area on its touchpad; FIG. 10 is a diagram showing an example of operating a special area of a smart phone; 2 is a diagram illustrating an example of a hardware configuration of a mobile terminal; FIG. 2 is a diagram illustrating an example of a hardware configuration of a mobile terminal; FIG. FIG. 4 is a diagram showing an example of processing of a mobile terminal;

Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a diagram showing an example of a smartphone 100 and its operation. In the example of FIG. 1 , a user 160 who operates a smartphone 100 is in a situation where one hand is blocked by a strap 170 in a transportation system such as a train or a bus, and the smartphone 100 is held with one hand 163. .

User 160 faces smartphone 100 and looks at smartphone 100 with eyes 161 in order to operate smartphone 100 . The smartphone 100 is provided with a touchpad (display screen) 110 , a camera 120 , a camera 130 , a side button 140 and a microphone 150 , and the user 160 looks at the touchpad 110 .

The camera 120 and the camera 130 capture images of the face of the user 160 , particularly the eyes 161 of the user 160 . Then, the distance between the face and the touch pad 110 is calculated by imaging with the two cameras, the line of sight is specified from the positional relationship between the white of the eye and the pupil of the eye 161, and the viewpoint on the touch pad 110 is obtained. will be further explained later.

Although two cameras are provided in the example of the smartphone 100 shown in FIG. 1, one camera may be provided as long as the distance between the face and the touch pad 110 can be detected by another sensor. For example, instead of the camera 120 on the same plane as the touch pad 110, an infrared or ultrasonic distance sensor may be provided at the position of the camera 120 to measure the distance to the face of the user 160. FIG.

In addition, when it is difficult to identify the positional relationship between the white of the eye 161 and the pupil, such as when the user 160 is wearing sunglasses, and it is difficult to specify the viewpoint, another input may be used instead. However, to determine if the user's 160 face itself is oriented toward the touchpad 110, the outline of the face, such as the hair and ear outline 162, may be identified.

FIG. 2 is a diagram showing an example of icons 220 displayed on the smartphone 100. As shown in FIG. The example in FIG. 2 is a display called a home screen or a standby screen, and icons 220 are displayed on the touch pad 110 . Note that when the icons 220-1 to 220-9 are represented without specifying which icon they are, they are represented as the icon 220, and the other symbols are hereinafter the same representation.

Each of icons 220-1 to 220-9 is an icon preset corresponding to an application program, and the application program corresponding to icon 220 selected among them is executed. A cursor 210 is displayed at the point of view of the user 160 . That is, it indicates the cursor position that feeds back the viewpoint to the user.

First, the line of sight of the user 160 vaguely captures a point on the touchpad 110 . When the smartphone 100 receives an incoming call and the icon 220-8 is the startup icon for the application program that automatically responds to the fact that the smartphone 100 cannot move to the receiving state, the user 160 moves the cursor 210 over the icon 220-8. move your gaze.

When the smartphone 100 identifies that the viewpoint is on the icon 220-8 and detects a switch-on operation of the side button 140, that is, a so-called “determine” operation, the application program corresponding to the icon 220-8 is activated. This makes it possible to select and start an application with one hand.

In FIG. 1, the action of the user 160 holding the strap 170 is taken as an example. There are many situations where operation with only one hand is required, such as when a person has an injury, and in these cases, the operation described above is effective.

When smartphone 100 identifies that the viewpoint has moved onto icon 220-8, smartphone 100 not only moves cursor 210 onto icon 220-8, but also changes the color of the display of icon 220-8. You may change the size of or change the shape of the display to highlight it. This allows the user 160 to more reliably confirm that the icon 220-8 has been selected.

Further, when the smartphone 100 identifies that the viewpoint has moved onto one of the icons 220 without displaying the cursor 210, the smartphone 100 changes the display color of the icon 220, changes the display size, or changes the display. You may change the shape of .

Also, in the example of FIG. 2, the shape of the cursor 210 is an arrow, but it may be a target mark combining a circle and a cross in order to emphasize the meaning of indicating the part that the line of sight is aimed at.

In the above description, an example of starting an application program on a standby screen has been described, but the processing of operations described above can also be used for operations during execution of an application program. For example, the icons 220 shown in FIG. 2 are numeric keys for entering a telephone number, and numbers may be entered by selecting the icons 220 while executing a telephone-related application program.

Further, for example, when using an answering machine function, there is a case where the input of numbers is requested according to a synthetic voice message. In other words, the state in which numbers can be input from the touch pad and the state in which a synthetic voice message emitted from the smartphone is heard may overlap.

On the other hand, in smartphone 100, icons 220, which are numeric keys, are selected from the viewpoint of user 160 through cameras 120 and 130, so even if an ear touches touch pad 110 to listen to a message, the icons are numeric keys. Icon 220 is not selected and no unintended touchpad input is accepted.

In addition, during execution of the standby screen or application program, unintended input due to unintentional contact with touch pad 110 is not accepted while smart phone 100 is being put into the pocket. This is because there is no selection of the icon 220 by the viewpoint and no switch-on operation of the side button 140, and the user's decision (execution) intention condition is not satisfied.

As described above, by inputting one instruction such as application program or determination of numeric keys by a combination of two types of operations, even if one type of operation is erroneously generated, it will still be an erroneous instruction. not reach.

The point here is that as long as it has camera functions that are becoming standard built-in smartphones, such as user face recognition and line-of-sight and viewpoint detection, relatively easy software processing prevents erroneous input recognition and improves usability with only one-handed operation. It is a point that a big effect can be obtained.

An example of executing another application program on the smartphone 100 will be described. FIG. 3 is a diagram showing an example of text displayed on the smartphone 100. As shown in FIG. The text is composed of a sequence of words, and the user 160 reads the words 320 in order, for example, word 320-1, word 320-2, word 320-3, and word 320-4 while moving the viewpoint.

Here, when user 160 wants to copy word 320-4, the viewpoint of user 160 is at word 320-4, and smartphone 100 identifies this viewpoint. Therefore, while the side button 140 is being switched on, the microphone 150 accepts voice operations. This voice operation is, for example, "copy".

In response, smartphone 100 copies word 320-4 and pastes it to the clipboard. Also, the voice operation may be another type of operation, or may be an operation for text editing.

As described above, by inputting the two instructions of selecting and copying the word 320-4 in a combination of three types of operations, even if one type of operation is erroneously generated, the erroneous instruction I can't reach it.

Next, an example of a tablet terminal will be described. FIG. 4 is a diagram showing an example of display on the tablet terminal 400. As shown in FIG. The tablet terminal 400 is provided with a touch pad (display screen) 410, a camera 420, a camera 430, an enter button 440, and a microphone 490, and the user looks at the touch pad 410. FIG.

These correspond to the touch pad 110, the camera 120, the camera 130, the side button 140, and the microphone 150 of the smartphone 100 already described, and the cursor 470 corresponds to the cursor 210, and the icon 220 is displayed to start the application program. Also, the input of numbers during execution of the application program has already been explained, and the processing of words by displaying the text has already been explained.

However, in activating the application program with the icon 220 displayed, the microphone 490 may accept “execute” as a voice operation instead of pressing the enter button 440 .

Instructions by such operations can also be used in information retrieval. In the example of FIG. 4, when the user wants to know the information about the person 480-2 in the photograph information in which three persons 480-1, 480-2, and 480-3 are subjects, the user When the cursor 470 moves to the display of the person 480-2 with the line of sight directed to the display of the person 480-2, the decision button 440 is pushed or the microphone 490 is uttered "Search for information."

On the other hand, tablet terminal 400 identifies the viewpoint of the user, moves cursor 470, and either presses decision button 440 or accepts a voice operation "Search for information" through microphone 490 to identify the viewpoint. Face recognition is performed on the image of the position where the cursor 470 is displayed, information stored in the tablet terminal 400 or information on the Internet is searched, and the found result is output by speech synthesis or touched. Displayed on pad 410 .

The tablet terminal 400 identifies the user's viewpoint, moves the cursor 470, accepts an operation of pressing the enter button 440, and accepts a voice operation of "Search for information" with the microphone 490. good.

The target of information retrieval may be map information. FIG. 5A is a diagram showing an example of displaying map information on the tablet terminal 400. As shown in FIG. A tablet terminal 400 shown in FIG. 5A has a down button 450 and an up button 460 added to the tablet terminal 400 shown in FIG. The down button 450 and the up button 460 are used for adjusting the volume of the sound output, but are also used for adjusting the zoom of the image.

The user first operates to display “Map of Africa”, and the map of Africa is displayed on the touch pad 410 in FIG. 5A. The operation for displaying the "Africa map" may be performed by inputting the voice "Africa map" into the microphone 490, and pressing the decision button 440 along with this voice or touching a specific area of the touch pad 410. good.

Then, when the user presses the up button 460 while adjusting the viewpoint to a part of the map of Africa, the tablet device 400 responds to the user's operation by displaying the user's image obtained by the camera 420 and the camera 430. 5B, the map is zoomed up centering on the cursor 470, which is the viewpoint position, when it is detected that the up button 460 has been pressed. .

As a result, the user can zoom in on the part that the user wants to see in order to confirm a more detailed position, and can observe the part that the user wants to know in detail.

When the tablet terminal 400 detects that the down button 450 has been pressed instead of the up button 460, it zooms down the map centering on the cursor 470, which is the viewpoint position. Also, it detects that the up button 460 or the down button 450 has been pressed, and also accepts a voice operation of "zoom up" or "zoom down" through the microphone 490, and executes processing to zoom up or zoom down the map. You can

Furthermore, in the display shown in FIG. 5B, when tablet terminal 400 detects that decision button 440 has been pressed, tablet terminal 400 identifies the country indicated on the map by cursor 470, which is the viewpoint position, and displays information about the identified country. is retrieved from the storage device in tablet terminal 400 or the Internet, and displayed on touch pad 410 as shown in FIG. 5C.

Here, the tablet terminal 400 may detect that the enter button 440 has been pressed, and may also accept a voice operation of "detailed information" through the microphone 490 to execute search processing and display processing.

In information retrieval based on map information, display of a map and retrieval of peripheral information according to the position of tablet terminal 400 may be enabled based on GPS (Global Positioning System). When the map application program is activated, tablet terminal 400 displays a map centered on the current position based on GPS.

FIG. 6A is a diagram showing an example of displaying map information on the tablet terminal 400. FIG. The touch pad 410, camera 420, camera 430, enter button 440, down button 450, up button 460, cursor 470, and microphone 490 have already been described. The touchpad 410 also displays a four-way arrow for scrolling the map and a current location mark 620 .

With the current position mark 620 displayed in the center of the map, when the viewpoint is aligned with the up arrow 610 and the decision button 440 is pressed, the map scrolls downward, and the display screen shown in FIG. 6A is obtained. . In FIG. 6A, as the map scrolls down, the current location mark 620 also moves down.

In the display screen shown in FIG. 6A, when the user's viewpoint is aligned with the department store 630, the cursor 470 is displayed to point to the department store 630, and when the enter button 440 is pressed, the display screen shown in FIG. 6B is displayed. This enables the user to obtain in advance information on each floor of the department store that the user intends to visit.

Calibration and distance measurement for specifying a viewpoint will be described below with reference to FIGS. 7 and 8. FIG. When the calibration is started, as shown in FIG. 7, in the display screen 710 of the touch pad 110 or the touch pad 410, first, a blinking mark 711-1 is displayed in the center like the display screen 710-1. be.

An image 720-1 captured by the camera 120, the camera 130, the camera 420, or the camera 430 is acquired for the display on the display screen 710-1. Here, as will be described later with reference to FIG. 8, the distance to the subject 712, ie, the user, may be measured using the parallax of the two cameras 120, 130 or the cameras 420, 430. FIG. The magnification of the camera in the captured image 720-1 may be such that the upper body of the subject 712 can be accommodated.

Next, like the display screen 710-2, a blinking mark 711-2 is displayed at the lower left. At this time, a message such as "Pay attention to the blinking figure" may be output as synthesized voice. Also, the magnification of the camera is increased, the eye portion 713 of the subject 712 is taken in close-up, and the captured image 720-2 is acquired.

Next, like the display screen 710-3, a mark 711-3 blinking on the upper right is displayed. A captured image 720-3 in which the eye portion 713 of the subject 712 is taken in close-up is acquired at the same magnification as the camera that acquired the captured image 720-2, and the captured image 720-2 and the captured image 720-3 are obtained. Determine the correlation of eye patterns.

That is, as the line of sight moves away from the center of the display screen 710 , the positions 715 of the iris and the iris on the eye contour 714 are displaced away from the center of the eye contour 714 . A position 715-2 of the black eye and iris with respect to the eye contour 714-2 and a black eye and iris position 715-2 with respect to the eye contour 714-3 when the end points (marks 711-2 and 711-3) of the display screen 710 are viewed. Using the displacement from the position 715-3 as a reference, the ratio of the displacement amount of the captured image newly obtained for actually specifying the viewpoint after calibration to the displacement reference is calculated as the ratio from the end point of the display screen. , may be calculated as the position of the viewpoint.

Also, from the distance between mark 711-2 and mark 711-3 and the distance between smartphone 100 (or tablet terminal 400) and the user (subject 712) measured when captured image 720-1 was acquired, , the line-of-sight angle of the captured image 720-2 and the captured image 720-3 is calculated, and the ratio of the line-of-sight angle is calculated from the ratio of the displacement amount of the newly obtained captured image to the displacement reference. The position of the viewpoint may be calculated from the angle and the distance when the newly obtained captured image was obtained.

In the explanation using FIG. 7, the processing for one eye has been explained, but the processing may be performed for each eye and averaged. Alternatively, the mark 711-2 and the mark 711-3 may be displayed without displaying the mark 711-1. Furthermore, in addition to the display of marks 711-2 and 711-3 and the acquisition of captured images 720-2 and 720-3, marks are also displayed on the upper left and lower right of the display screen 710, and the respective captured images are displayed. may be obtained.

FIG. 8 is a diagram showing an example of distance measurement using two cameras. A point 821 that is the center position of both eyes of the subject 712 is detected by face recognition, and distance measurement using the detected point 821 will be described. In order to simplify the explanation, the distance measurement will be explained in a state in which the user, who is the subject 712, faces the front of one of the two cameras.

In FIG. 8, the positional relationship in which the mobile terminal 810 images the subject 712 is shown on the xy plane viewed from above the z axis. A mobile terminal 810 is the smartphone 100 or the tablet terminal 400 and is provided with a lens 811 , a lens 812 , an image sensor 813 , and an image sensor 814 .

An image of an object 712 including a point 821 is formed on the imaging element 813 through the lens 811 . Here, the point 821 becomes a point 824 on the image sensor 813 . An image of the subject 712 including the point 821 is also formed on the imaging element 814 through the lens 812 . Here, the point 821 becomes a point 825 on the image sensor 814 .

A difference between a point 826 on the image sensor 814 corresponding to the point 824 on the image sensor 813 and a point 825 corresponds to a parallax 834 . In this way, when one subject 712 is imaged by two cameras, the images are different between the imaging elements 813 and 814 of each camera.

A large triangle formed by points 821, 822, and 823 and a small triangle formed by points 823, 826, and 825 are similar in optical properties. Therefore, if the length 831 from the mobile terminal 810 to the object 712 is the distance L, the length 832 between the lenses is the distance D, the focal length 833 is the distance F, and the parallax 834 is the distance S, the distance between these distances is , the relationship L:D=F:S is established.

From this relationship, the distance L is represented by L=D×F/S. For example, if D=5 cm, F=3 mm, and S=0.5 mm, L=30 cm is obtained. In the smart phone 100 or the tablet terminal 400, which is the mobile terminal 810, since the relationship between the positions of the lenses of the two cameras and the positions of the imaging elements is fixed, the distance between the subject 712 and the mobile terminal 810 is calculated as described above. becomes possible.

As described above, if the eyes of the user, who is the subject, can be imaged, the viewpoint and the distance can be calculated. Since there may be a large difference in , it may be difficult to calculate the viewpoint and distance from the outline of the eye and the iris. Therefore, an example using a glasses-type device will be described below.

FIG. 9 is a diagram showing an example of a glasses-type device 900 and a mobile terminal 940. As shown in FIG. Glasses-type device 900 has the shape of glasses, is fixed to the user's face with the nose and ears in contact, and has cameras 920-1 and 920-2. Since the camera 920 is positioned at a short distance from the user's eyes, it is easy to capture the contours of the eyes and the iris. Note that the number of cameras 920 may be one.

The distance between the camera 920 and the user's eyes is substantially constant according to the shape of the spectacles-type device 900, although there are individual differences in the shape of the user's face, and is fixed when the shape of the spectacles-type device 900 is designed. It may be set as a value.

The mobile terminal 940 is the smartphone 100 or the tablet terminal 400 , is connected to the spectacles-type device 900 by a connection cord 930 , and obtains image data captured by the camera 920 from the spectacles-type device 900 . Although FIG. 9 shows an example of wired connection using the connection cord 930 , the portable terminal 940 and the glasses-type device 900 may be wirelessly connected without the connection cord 930 .

The mobile terminal 940 is provided with a touch pad (display screen) 950, an up button 960, a down button 970, and an enter button 980, and the operations of these are as already described. good. In order for the user to see the touchpad 950, the rim 910 of the glasses-type device 900 is fitted with a transparent or tinted lens.

One or both of the rims 910-1 and 910-2 of the glasses-type device 900 may be fitted with a liquid crystal screen. The portable terminal 940 may output image data to the spectacles-type device 900 via the connection cord 930 in order to project an icon or the like on the liquid crystal screen.

The portable terminal 940 in which the liquid crystal screen is fitted in the spectacles type device 900 calculates the user's point of view to the icon or the like displayed on the liquid crystal screen. In this case, the mobile terminal 940 may not have the touchpad 950, or may have a touchpad that is smaller than the touchpad 950 or has a lower resolution.

One or both of the temples 990-1 and 990-2 of the glasses-type device 900 may be provided with earphones. Portable terminal 940 may output synthesized speech data to glasses-type device 900 via connection cord 930 in order to convey information to the user by sound using earphones. Instead of these earphones, the glasses-type device 900 may be equipped with headphones. A headband may be provided that contacts over the user's head to provide headphones.

Also, the temple 990 may be equipped with a microphone like a hands-free phone. In order to obtain audio data from the microphone, the mobile terminal 940 may input audio data from the glasses-type device 900 via the connection cord 930 .

Furthermore, the spectacles-type device 900 may be fitted with a liquid crystal screen, provided with an up button 960 , a down button 970 , and an enter button 980 , and the spectacles-type device 900 may execute the processing of the mobile terminal 940 . Since the glasses-type device 900 in this case can operate independently, the connection cord 930 may be omitted.

In the above description, the camera is used to identify the user's viewpoint, but the camera may not be used. For example, when a smartphone is grabbed, the fingers of the user's hand may unconsciously come into contact with the touch pad, causing unintended input by the user. 10 may be detected.

FIG. 10 is a diagram showing an example of operation without using a camera. A smartphone 100 is provided with a touch pad 110 , a side button 140 and a microphone 150 . A camera may be provided, but is not used. Instead of the position of the viewpoint, the smartphone 100 detects the contact position of the one hand 164 on the touch pad 110, detects the action of pressing the side button 140 with the one hand 163, and executes processing corresponding to the detected operation. .

Further, the smartphone 100 may detect the contact position of the touch pad 110 with the one hand 164 , and instead of detecting the operation of pressing the side button 140 , detect the voice operation of the microphone 150 . By detecting other operations in addition to detecting the operation of the touch pad 110, the possibility of erroneously detecting an operation not intended by the user can be reduced.

In the above description, an operation other than the touchpad operation is detected, but an example of detecting an operation on the touchpad to reduce the possibility of erroneously detecting an operation not intended by the user will be described. FIG. 11A is a diagram showing an example of a tablet terminal 1100 that detects touches of multiple fingers. A tablet terminal 1100 is provided with a touch pad (display screen) 1110 and a camera 1120 .

The camera 1120 is installed tilted toward the touch pad 1110 so as to capture an image of a hand 1140 that operates the touch pad 1110, including the fingers. , including the periphery of the touch pad 1110 .

The tablet terminal 1100 detects the touch of the first finger (for example, the index finger) of the hand 1140, and when detecting the movement of the touched finger in that state, moves the cursor. When the second finger (for example, the thumb) of the hand 1140 detects a touch within the range of the execution (decision) button in the touch pad 1110, the position of the cursor, which is the position where the first finger is detected, is detected. perform or determine the action to take.

FIG. 11B is a diagram showing an example of the screen of the tablet terminal 1100. As shown in FIG. Icons 1150-1, 1150-2, and 1150-3 are displayed on touchpad 1110, each icon 1150 corresponding to an application program or an instruction to an application program.

Icon 1160 is an execution (decision) button. For example, when a touch on icon 1150-1 with a first finger is detected and a touch on icon 1160 with a second finger is detected, tablet terminal 1100 detects icon 1150-1. Launch application programs and perform actions based on instructions.

Icon 1160 need not be displayed on touchpad 1110 . The range of the execution (decision) button in this case is an area other than the icon 1150 . Further, when it is detected that the first finger continues to touch any of the icons 1150 for a preset time or longer while the icon 1160 is not displayed, the icon 1160 is displayed and the second finger to the icon 1160 is displayed. A touch of the second finger may be detected.

By analyzing the image captured by the camera 1120, the type of finger touching the touch pad 1110 is recognized. By recognizing the type of finger, when the touch of the second finger is recognized as the touch of the thumb, it may be detected as a so-called left click of the mouse and the application program corresponding to the icon 1150 may be started. .

Further, when the touch of the second finger is recognized as the touch of the middle finger, it is detected as a so-called right click of the mouse, and the menu of the application program corresponding to the icon 1150 is displayed, or the information of the application program is displayed. It may be displayed. Also, when the first finger is touched with the middle finger, it is detected as a left click of the mouse when the second finger is touched with the index finger, and the second finger is touched with the ring finger. May be detected as a right mouse click.

In order to allow the user to understand the recognition result of the type of finger, the periphery of the touched portion of the touch pad 1110 may be displayed in a specific color according to the type of the recognized finger. Then, the color display around the touched portion may be regarded as a cursor. As a result, it is possible to reduce the possibility of erroneous indication due to erroneous recognition of the type of finger.

Furthermore, an image of hand 1140 captured by camera 1120 may be displayed on touch pad 1110 while being superimposed on icon 1150 and icon 1160 .

Next, an example of using a touch pad instead of the side button 140 shown in FIG. 1 and the enter button 440 shown in FIG. 4 will be described, although the viewpoint is detected by the camera. FIG. 12A is a diagram showing an example of smartphone 100 in which part of touchpad 110 is a special area.

A smartphone 100 is provided with a touch pad 110 , a camera 120 and a camera 130 . Camera 120 and camera 130 have already been described. The smartphone 100 may be provided with side buttons and a microphone, but they are not used here. Smartphone 100 may be tablet terminal 400 .

In order to use the touchpad 110 instead of the side button 140 and the enter button 440 , a special area 1210 that replaces the side button 140 and the enter button 440 is set on the touchpad 110 . Special area 1210 is an area that is touched, for example, with a thumb when smartphone 100 is held with one hand.

When the user grips smartphone 100 with one hand, the gripping position differs depending on the user's habits, and the gripping position also differs depending on whether the user is right-handed or left-handed. Therefore, one special area 1210 is selected from special areas 1210-1 to 1210-6 as shown in FIG. 12A after the smartphone 100 is powered on or reset. accept.

In FIG. 12A, for example, special regions 1210-1, 1210-3, 1210-5 are for left-handed grasping, and special regions 1210-2, 1210-4, 1210-6 are for right-handed grasping. is for The special area 1210 touched by the hand holding the smartphone 100 is preferably an area in contact with the edge of the touchpad 110 .

FIG. 12B is an example of receiving a setting that selects the special area 1210-3. Point-of-view position specification is enabled on the touchpad 110 except for the special area 1210-3, which is dedicated to touch. Therefore, no icon 1230 is placed in the special area 1210-3.

Then, as already described, the user's viewpoint is specified based on the images captured by the cameras 120 and 130, and the cursor 210 is displayed. The position of cursor 210, which is the position of the viewpoint, can be moved from icon 1230-1 to icon 1230-8.

When the cursor 210 is positioned on the icon 1230-6 and it is detected that the special area 1210-3 has been touched by the thumb of one hand 1220, the application corresponding to the icon 1230-6 is started or the application instructions are sent to

Note that a character string or a special icon representing "determine" or "touch" may be displayed in special area 1210-3 so that the user can understand that it is a special area.

By detecting a plurality of operations such as the first operation with the viewpoint or the first finger and the second operation with a button or voice as described above, the validity of the operation is confirmed, and the user's intention is met. implement the action. The operation for confirming validity is not limited to the operations described above.

For example, to detect an operation using the camera, if the icon selection state by the viewpoint continues for several seconds, when executing or deciding, throw a message to the user such as "If you are okay, please close only your right eye for 3 seconds" and follow the instructions. If the action of is detected, the action of execution or determination may be performed.

In addition, if the user is wearing sunglasses and requires input other than the eyes, for example, in order to check the shape of the lips, a message such as "If you like, please make the shape of your mouth when pronouncing "o"". may be presented to the user, and it may be determined whether the exact action is detected.

As has already been described as a sensor of the smartphone 100 or the tablet terminal 400 for detecting an operation, an operation to be detected may be prioritized as an input condition. for example,
(1) Line of sight, viewpoint, or eye condition (2) Lip shape (3) Voice (4) Switch input (5) Priorities are set in advance for touch information on the touch pad, and arbitrarily combined. can be

An input condition management unit, which will be described later, selects an input condition with a high priority or an input condition with a low priority according to the surrounding environment. For example, when it is determined that the surrounding environment is dark, the input condition for using the camera may not be selected, and low priority switch input and touch information on the touch pad may be selected.

FIG. 13 is a diagram showing an example of a first hardware configuration of the mobile terminal 810. As shown in FIG. The mobile terminal 810 is the smart phone 100 or the tablet terminal 400 as already described. The camera unit 1301 corresponds to the cameras 120, 130 or the cameras 420, 430, etc., and the touch pad display unit 1308 and the touch pad sense unit 1316 correspond to the touch pad 110, the touch pad 410, or the like.

A microphone unit 1313 corresponds to the microphone 150 or the microphone 490, and a switch input unit 1315
corresponds to the side button 140, enter button 440, or the like. Hardware such as the original telephone function of the smartphone 100 is omitted because it is irrelevant to the description of this embodiment. Also, the activation of the application program will be particularly described.

Camera section 1301 outputs image information obtained by imaging with two or more cameras to face recognition section 1302 . A face recognition unit 1302 recognizes the face and extracts features of the face. Further, the face recognition unit 1302 may calculate the distance to the face, and perform an operation to issue a message to the user if the face cannot be recognized from the front or the eyes cannot be recognized.

A viewpoint detection unit 1303 receives the information on the eye part and the distance to the face from the face recognition unit 1302, and calculates the position of the viewpoint. Here, it may be determined whether the viewpoint is directed to the touch pad display unit 1308 . Since the viewpoint has the property of constantly causing fine movements, the low-pass filter unit 1304 cuts the fibrillation information and outputs the viewpoint information to the viewpoint information management unit 1305 .

The viewpoint information management unit 1305 outputs where the viewpoint is on the screen of the touchpad display unit 1308 to the cursor display unit 1307 and displays the cursor to the user via the touchpad display unit 1308 . Note that, as already explained, an icon may be highlighted instead of the cursor.

On the other hand, at least one or more inputs to be combined with viewpoint information include voice input input from the microphone unit 1313 and decoded through the voice recognition unit 1314, switch input from the switch input unit 1315 such as a side button, or For example, there is finger touch input from the touch pad sensing unit 1316 .

When these inputs match the viewpoint input from the viewpoint information management unit 1305, the input condition management unit 1309 causes the application program execution unit 1310 to start the application program specified by the input. As already explained, the input condition management unit 1309 outputs an instruction to the application program execution unit 1310 when it determines that a plurality of inputs satisfy a predetermined condition.

The input condition management unit 1309 has correspondence information between the display position of the icon and the identification information of the application program corresponding to the icon. You can output to

Alternatively, when the application program execution unit 1310 has correspondence information between the display position of the icon and the identification information of the application program corresponding to the icon, and the input condition management unit 1309 determines that the predetermined condition is satisfied, The viewpoint input from the viewpoint information management unit 1305 may be transferred to the application program execution unit 1310 .

Input condition management unit 1309 has correspondence information between areas such as touch pad 110 or touch pad 410 and instructions to application programs, and provides instructions to application programs according to touch position input from touch pad sense unit 1316 . You may output to the execution part 1310. FIG.

The input condition management unit 1309 has a plurality of information on the special area 1210 as the area of the touch pad 110 or the touch pad 410, and may have information on the selected special area 1210-3. It may have an area for buttons.

Although the input condition management unit 1309 is described as an independent control unit in the above description, it may be a part of the application program execution unit 1310 . Finally, the application program execution unit 1310 may cause the speaker unit 1311 to transmit information to the user by voice, or may send a command to the communication control unit 1312 for communication.

As already described with reference to FIG. 7, eye information captured by a camera and viewpoints differ from person to person. The relationship is managed and output to the viewpoint information management unit 1305, and the viewpoint information management unit 1305 identifies the viewpoint based on the relationship.

From the camera unit 1301 to the viewpoint information management unit 1305, fingers may be recognized instead of faces, and the positions and types of fingers may be specified. The input condition management unit 1309 may receive the finger position and type information from the viewpoint information management unit 1305 as an input, and transfer the input from the touch pad sense unit 1316 to the viewpoint information management unit 1305 . , the cursor may be displayed at the touch position.

FIG. 14 is a diagram showing an example of a second hardware configuration of the mobile terminal 810. As shown in FIG. Each part of the hardware described using FIG. 13 may be realized by software. The example of the hardware configuration shown in FIG. 14 is an example of the hardware configuration for its realization by software. Note that the hardware configuration of the mobile terminal 810 shown in FIG. 14 may be the hardware configuration of a general smart phone or tablet terminal.

A processor 1401 executes a program stored in a memory 1402, exchanges data with each unit shown in FIG. 14, and processes the data. A memory 1402 is, for example, a RAM (Random Access Memory), and stores programs and data.

A storage 1403 is, for example, a flash memory, and stores programs and data. Programs and data may be transferred between memory 1402 and storage 1403 . The communication unit 1404 communicates via a wired connection using the connection cord 930, a wireless telephone line, or a wireless LAN (Local Area Network).

The communication unit 1404 transmits and receives under the control of the processor 1401 that executes the application program stored in the memory 1402 . Also, the programs and data stored in memory 1402 or storage 1403 may be received by communication unit 1404 .

The input unit 1405 stores an input signal from the microphone 1407 or the like as data in the memory 1402 or the storage 1403 . A microphone 1407 is the microphone 150 shown in FIG. 1 or the microphone 490 shown in FIG. 4, and inputs an audio signal. Cameras 1408 and 1409 are the cameras 120 and 130 shown in FIG. 1 or the cameras 420 and 430 shown in FIG. 4, respectively, and receive captured video signals.

The output unit 1406 converts data stored in the memory 1402 or storage 1403 into a signal and outputs the signal to the speaker 1411 or the like. The touch pad 1410 is, for example, the touch pad 110 shown in FIG. 1 or the touch pad 410 shown in FIG. to display the screen.

FIG. 15 is a diagram showing an example of processing of the mobile terminal 810 shown in FIG. 14, and is an example of a flowchart of a program stored in the memory 1402 and executed by the processor 1401. FIG. When the mobile terminal 810 is powered on or wakes from sleep, the processor 140
1 starts processing.

First, processor 1401 determines whether calibration has not been completed (step 1501). For this determination, for example, a flag indicating whether or not calibration has not been completed may be stored in the storage 1403, and the processor 1401 may read the flag for determination. The flag is set to pending when the mobile terminal 810 is manufactured and reset.

If the processor 1401 determines that it has not been completed in step 1501 , it executes calibration (step 1502 ), and if it determines that it has not been completed yet, it skips step 1502 . The details of the calibration processing in step 1502 are as described with reference to FIGS. 7 and 8, and the cameras 1408 and 1409 of the mobile terminal 810 are enabled to capture an image of the user. Additionally, the flag is set to done.

Processor 1401 displays a home screen, also called a standby screen (step 1503). As shown in FIG. 2, the home screen includes icons corresponding to a plurality of application programs, and the application programs are activated by operating the icons. At this point, if the camera unit 1301 or cameras 1408 and 1409 of the mobile terminal 810 are not enabled, they are enabled.

Processor 1401 controls cameras 1408 and 1409 to capture images and recognize faces (step 1504). In this face recognition, processing for recognizing a face facing the cameras 1408 and 1409 is executed, and when the face is recognized, the position of the black eye and iris in the outline of the eye is recognized.

As a result of executing the processing for recognizing the face in step 1504, if the face can be recognized from the front, it is assumed that the face is facing the mobile terminal 810, and the processor 1401 proceeds to step 1506, and the face cannot be recognized. If so, or if it can be recognized that the face is facing sideways, it is assumed that the face is not facing the mobile terminal 810 and the processor 1401 proceeds to step 1512 .

At step 1512, processor 1401 outputs a message that calls for the user's attention or causes the user to turn his head. This message may be output from speaker 1411 .

At step 1506, the processor 1401 calculates the viewpoint from the relationship between the positions of the black eye and iris on the eye contour recognized at step 1504 and the positions of the black eye and iris on the eye contour recorded in the calibration at step 1502. . The distance to the face may be calculated from the parallax between the cameras 1408 and 1409 and used to calculate the viewpoint.

The processor 1401 applies a low-pass filter to prevent erroneous detection due to the calculated minute changes in viewpoint (step 1507). For example, viewpoints may be calculated from images captured at preset time intervals, and when a preset number of viewpoints are calculated, the positions of those viewpoints may be averaged.

The processor 1401 displays a cursor at the position corresponding to the low-pass filtered viewpoint (step 1508). If the position corresponding to the viewpoint matches the position of the icon on the home screen displayed in step 1503, the icon with the matching position may be highlighted.

Processor 1401 determines whether there is an input to select the icon specified by the viewpoint (step 1509). This selection input is, for example, input by the side button 140 in FIG. If it is determined that there is no input, the processor 1401 returns to step 1504 since the icon specified by the viewpoint may have just been seen by the user.

If it is determined that there is a selection input, processor 1401 determines whether the selection input satisfies the input conditions (step 1510). This input condition is as already described as an input condition with priority, and is set in advance. For example, if it is determined that the input condition is not satisfied even if the input is by the side button 140 , the processor 1401 proceeds to step 1512 .

At step 1512, the processor 1401 outputs a message that the input condition is not satisfied or that the input is invalid. This message may be output from speaker 1411 . It should be noted that step 1512 may be executed based on the determination result of step 1505 or based on the determination result of step 1510 .

If it is determined in step 1510 that the input condition is satisfied, processor 1401 executes the application program corresponding to the icon designated by the viewpoint (step 1511). A message as an execution result of the application program may be output from the speaker 1411 , or an execution error message may be output from the speaker 1411 .

After the application program is executed in step 1511 or the message is output in step 1512, the processor 1401 determines whether a preset time has elapsed without any operation. When determining that the preset time has passed, the processor 1401 puts the mobile terminal 810 into a sleep state, for example, and ends the processing of the flowchart.

The processor 1401 returns to step 1504 when determining that the operation has been performed before the preset time elapses. It should be noted that determination of non-operation in step 1513 is not limited to independent determination, and may be performed in cooperation with step 1506 or step 1509 .

In the above explanation using FIG. 15, an example using a viewpoint has been explained. It can be realized.

As described above, the mobile terminal operates by judging a plurality of input conditions in response to input by user's operation. Therefore, it is possible to reduce the possibility of erroneous operation due to user's erroneous operation.

It can be applied to portable terminals such as smart phones and tablet terminals, and it can also be applied to computers operated by users.

100 smartphone 110 touch pad 120 camera 130 camera 140 side button 150 microphone

Claims (1)

On your mobile device,
a touch panel that displays icons and detects touches;
The touch panel displays a first area that displays an icon for starting an application, and a second area that is different from the first area, and the first area and the second area are the touch panel. It is displayed at a predetermined position,
a camera that captures an image of the surroundings of the touch panel;
a processor;
The processor
detecting the viewpoint of the eye captured by the camera;
outputting the position of the detected viewpoint on the first region;
controlling to display a cursor over the first area;
determining whether or not a preset input condition is satisfied for both an input by touching the second area and an input of the viewpoint position on the first area;
A portable terminal characterized by activating an application program corresponding to the icon when a preset input condition is satisfied by the determination.

Images