JP7314196B2 - TERMINAL DEVICE, TERMINAL DEVICE CONTROL METHOD AND TERMINAL DEVICE CONTROL PROGRAM - Google Patents

Description

The present invention relates to a terminal device, a terminal device control method, and a terminal device control program.

Conventionally, there has been proposed a technique for estimating a gripping state of a terminal device owned by a user and providing various contents.

WO2018/105266

However, in the conventional technology, there is room for further improvement in terms of providing more appropriate content to users.

The present application has been made in view of the above, and an object thereof is to provide a terminal device, a terminal device control method, and a terminal device control program capable of providing appropriate content to a user.

A terminal device according to the present application includes an acquisition unit, an estimation unit, and a provision unit. The acquisition unit acquires a corneal image representing the user's cornea captured by a camera incorporated in a terminal device owned by the user. The estimation unit estimates the user's gripping state of the terminal device based on an image of a reflection area reflected by the cornea in the corneal image. The provision unit provides content according to the grip state estimated by the estimation unit.

According to one aspect of the embodiment, there is an effect that appropriate content can be provided to the user.

FIG. 1 is a diagram illustrating an example of control processing of a terminal device according to an embodiment. FIG. 2 is a diagram illustrating an example of a control system according to the embodiment; FIG. 3 is a block diagram illustrating a functional configuration example of a terminal device; FIG. 4 is a flowchart illustrating an example of processing executed by the terminal device according to the embodiment; FIG. 5 is a diagram illustrating an example of a hardware configuration;

Embodiments for implementing a terminal device, a terminal device control method, and a terminal device control program (hereinafter referred to as "embodiments") according to the present application will be described in detail below with reference to the drawings. The terminal device, the method for controlling the terminal device, and the control program for the terminal device according to the present application are not limited to this embodiment. Also, in each of the following embodiments, the same parts are denoted by the same reference numerals, and overlapping descriptions are omitted.

(embodiment)
[Overview of terminal device control processing]
First, an exemplary embodiment of the present disclosure will be described in detail with reference to FIG. FIG. 1 is a diagram illustrating an example of control processing of a terminal device according to an embodiment. The terminal device control process is a process performed by the terminal device 1 according to the embodiment executing a control method.

FIG. 1 shows a situation in which the user is holding (gripping) the terminal device 1 with the right hand and operating it. The terminal device 1 according to the embodiment first estimates the gripping state of the user on the terminal device 1 based on the image reflected by the user's cornea by control processing, and provides content according to the estimated gripping state.

Specifically, first, the terminal device 1 acquires a corneal image showing the user's cornea captured by the camera 5 (see FIG. 3) incorporated in the terminal device 1 (step S1). The corneal image may be acquired as a corneal image by extracting the corneal region from the face image of the entire face of the user captured by the camera 5 .

Although FIG. 1 shows the corneal image of the user's right eye, the corneal image of the left eye or the corneal images of both eyes may be used.

Subsequently, the terminal device 1 estimates the user's gripping state of the terminal device 1 based on the image of the reflection region R reflected by the cornea in the corneal image (step S2). As shown in FIG. 1, the reflection region R includes a display region R1, which is a bright region shaped by the display unit 8 of the terminal device 1, and a finger region R2, which is a dark region shaped by the user's finger (thumb). That is, the terminal device 1 estimates the grip state of the user based on the positional relationship between the display area R1 and the finger area R2.

A model generated by machine learning can be used for estimating the gripping state, and the details of this point will be described later.

In the example shown in FIG. 1, the terminal device 1 estimates a state in which the user is holding the terminal device 1 with the right hand and performing a touch operation on the display unit 8 with the thumb as the holding state. That is, the estimated grip state includes information about which finger the user is using to operate the display unit 8, in addition to information about which hand the user is gripping with, left or right. The gripping state may further include information about which position on the display unit 8 is being touched (touch position or touch coordinates).

In addition to the grasping state, the terminal device 1 can also estimate whether the display unit 8 is being operated with a pointing device such as a stylus, which will be described later in detail.

Then, the terminal device 1 provides content according to the estimated grip state (step S3). In the example shown in FIG. 1, the terminal device 1 moves the position of the icon image 8a of the application displayed on the display unit 8 to the lower right area, that is, the area close to the right thumb of the user.

This makes it easier for the user to access the icon image 8a with the thumb of the right hand, so that the user's operability of the terminal device 1 can be improved.

As described above, according to the terminal device 1 according to the embodiment, by estimating the user's grip state based on the image of the reflection area R reflected by the cornea, the grip state can be estimated with high accuracy. By estimating the gripping state with high accuracy, more appropriate content can be provided to the user.

In the example shown in FIG. 1, an example of providing content in which the position of the icon image 8a displayed on the display unit 8 is moved according to the gripping state is shown, but in addition to this, various content can be provided according to the estimated gripping state. The details of other provided content will be described later.

The terminal device 1 that performs such control processing will be described in detail below.

[Control system]
Next, a configuration example of a system including the terminal device 1 will be described with reference to FIG.

[Control system components]
FIG. 2 is a diagram showing an example of the control system S according to the embodiment. As shown in FIG. 2, the control system S according to the embodiment includes a terminal device 1 and a server device 100. As shown in FIG. In the control system S, a terminal device 1 and a server device 100 are connected via a network N. FIG.

The network N is a communication network such as a LAN (Local Area Network), a WAN (Wide Area Network), a telephone network (mobile telephone network, fixed telephone network, etc.), a local IP (Internet Protocol) network, or the Internet. The network N may include wired networks or wireless networks. Although one server device 100 is shown in FIG. 2 , the control system S may include a plurality of server devices 100 .

The terminal device 1 is, for example, a user terminal such as a smart device, as described above. The terminal device 1 can acquire predetermined content information from the server device 100 according to a user's operation.

In addition, the terminal device 1 can estimate the grip state of the user based on the corneal image, and provide content according to the estimated grip state. Note that detailed functions of the terminal device 1 will be described later.

The server device 100 provides various contents to the terminal device 1 . For example, the server device 100 is a device that distributes to the terminal device 1 information to be displayed in various applications installed in the terminal device 1 (for example, a portal application, a news application, an auction site, a weather forecast application, a shopping application, a finance (stock price) application, a route search application, a map providing application, a travel application, a restaurant introduction application, a blog browsing application, etc.). Alternatively, the server device 100 is a device that distributes to the terminal device 1 web pages in which various types of information related to portal sites, news sites, auction sites, weather forecast sites, shopping sites, finance (stock price) sites, route search sites, map providing sites, travel sites, restaurant introduction sites, web blogs, etc. are arranged in tiles. Further, the server device 100 may be a server that distributes application data itself, or may be a server that distributes web pages such as portal sites.

These contents are displayed on the display unit 8 (see FIG. 3) of the terminal device 1 based on the control information that the server device 100 transmits to the terminal device 1. FIG. The control information may be, for example, a program that controls an application installed in the terminal device 1, or a program embedded in content. For example, the control information may be a program written in a script language such as JavaScript (registered trademark). Also, the control information may be information written in a computer language other than the script language. For example, the control information may be information written in a style sheet language such as CSS3 (Cascading Style Sheets 3). Also, the control information may be a program written in a programming language such as Java (registered trademark), Swift, C language, C++, and compiled. It is also possible to regard application software itself distributed from a server such as a content distribution server as control information. At this time, the content information may be included in the control information. Note that the control information does not have to be a compiled program as long as the terminal device 1 can finally execute the program. Alternatively, the control information may be data (programs) provided in the form of an interpreter language (eg, script language), machine language, or intermediate language.

Note that the server device 100 may distribute all of the above-described contents by a single server device 100, or may distribute respective corresponding contents by a plurality of server devices 100. FIG.

[Configuration of terminal device]
Next, a configuration example of the terminal device 1 will be described with reference to FIG. FIG. 3 is a block diagram showing a functional configuration example of the terminal device 1. As shown in FIG. As shown in FIG. 3 , the terminal device 1 has a communication section 2 , a storage section 3 , a control section 4 , a camera 5 , an illuminance sensor 6 , a gyro sensor 7 and a display section 8 .

The camera 5 is a so-called in-camera built into the terminal device 1 . For example, the camera 5 is arranged on the same surface as the display unit 8 in the terminal device 1 . That is, the camera 5 is arranged at a position where the face of the user looking at the display unit 8 can be captured. The camera 5 outputs the captured face image of the user to the control unit 4 .

The illuminance sensor 6 is a sensor that detects illuminance indicating the brightness around the terminal device 1 . The illuminance sensor 6 outputs information on the detected illuminance to the control unit 4 .

The gyro sensor 7 is a sensor that detects the attitude of the terminal device 1 such as inclination and rotation. The gyro sensor 7 outputs information on the detected orientation of the terminal device 1 to the control unit 4 .

The display unit 8 is, for example, a liquid crystal display, and can display various information under the control of the control unit 4 . Note that the display unit 8 may have a touch panel function.

The communication unit 2 is realized by, for example, a NIC (Network Interface Card) or the like. The communication unit 2 is connected to a network by wire or wirelessly. The communication unit 2 is communicably connected to the server device 100 via the network N, and can transmit and receive information to and from the server device 100 .

The storage unit 3 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 3 , the storage unit 3 has model information 31 .

The model information 31 is information used by the estimation unit 42 to be described later, and is information including a learning model for estimating the user's gripping state of the terminal device 1 . By using the model in the model information 31, it is possible to calculate a score indicating one of a plurality of pre-classified gripping states.

Such a model can be generated, for example, by using the corneal image captured by the camera 5 in each grasping state as an explanatory variable and the corresponding grasping state as an objective variable. That is, the model is generated by learning the corneal image and the grasping state as a data set. As the learning algorithm, for example, a CNN (Convolutional Neural Network) can be used, but any learning algorithm limited to this can be used.

As for the corneal image used for the model, an image of a reflection region (details will be described later) reflected by the cornea may be extracted and used as an explanatory variable.

Note that the explanatory variables may further include information such as user hand feature information (finger length, hand size, etc.), camera 5 performance information (resolution, enlargement/reduction ratio, etc.), illuminance from the illuminance sensor 6, iris color, and the like.

The control unit 4 is a controller, for example, implemented by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing various programs (equivalent to an example of a control program) stored in a storage device inside the terminal device 1 using a RAM or the like as a work area. Also, the control unit 4 is a controller, and may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a GPGPU (General Purpose Graphic Processing Unit).

As shown in FIG. 3, the control unit 4 includes an acquisition unit 41, an estimation unit 42, and a provision unit 43, and implements or executes functions and effects of control processing described below. Further, the control unit 4 can implement the control processing described above with reference to FIG. One or more processors of terminal device 1 can realize the function of each control unit in control unit 4 by executing instructions stored in one or more memories of terminal device 1 . Note that the internal configuration of the control unit 4 is not limited to the configuration shown in FIG. 3, and may be another configuration as long as it is configured to perform the control processing described later. For example, the acquisition unit 41 may perform all or part of the control processing described later with respect to units other than the acquisition unit 41 .

The acquisition unit 41 acquires various types of information. For example, the acquisition unit 41 acquires a corneal image including the user's cornea based on the face image captured by the camera 5 . Specifically, the acquiring unit 41 acquires a corneal image by extracting a corneal region from the acquired face image by image analysis processing such as pattern matching. Note that the face image may be obtained as it is as the corneal image. A corneal image is an image of either the left or right eye of the user, or both eyes.

Further, the acquisition (imaging) timing of the corneal image by the camera 5 is, for example, the timing when the user touches the display unit 8 . Alternatively, such timing may be, for example, the timing at which the gyro sensor 7 detects the attitude change of the terminal device 1 . In other words, the corneal image is acquired at the timing when the posture of the user changes by holding the terminal device 1 .

Note that the acquisition unit 41 may acquire a corneal image captured while a single image (white image) having a specific luminance is displayed on the display unit 8 . As a result, the display area R1 reflected in the reflection area R can be made clearer, so that the estimation accuracy of the gripping state in the estimating section 42 in the subsequent stage can be improved. Also, it is preferable to display a single image with a number of frames that is difficult for the user to perceive. As a result, a corneal image can be obtained without giving the user a sense of discomfort.

The acquisition unit 41 also acquires the illuminance around the terminal device 1 detected by the illuminance sensor 6 . The acquisition unit 41 also acquires the orientation of the terminal device 1 detected by the gyro sensor 7 .

The estimation unit 42 estimates the user's gripping state of the terminal device 1 based on the image of the reflection area R reflected by the cornea in the corneal image acquired by the acquisition unit 41 . Specifically, the estimation unit 42 first extracts the reflection area R from the corneal image. For example, the estimation unit 42 extracts the pupil and iris regions as the reflection regions R by edge detection of the corneal image and image analysis such as pattern matching. Note that the reflective area R may be either one of the pupil and the iris.

Next, the estimation unit 42 determines whether or not the reflection area R includes at least the display area R1 out of the display area R1 and the finger area R2. When the display area R1 is included in the reflection area R, the estimation unit 42 inputs the image of the reflection area R to the model in the model information 31, and estimates the output result from the model as the gripping state. On the other hand, when the estimation unit 42 determines that the reflection area R does not include the display area R1, the image of the reflection area R is discarded without being input to the model. In other words, the grasping state is not estimated. As a result, unnecessary estimation processing can be reduced, and the load on the control unit 4 can be suppressed.

Note that the estimation unit 42 may input the image of the reflection area R to the model without determining whether the reflection area R includes the display area R1. In other words, the model may have a function of determining whether the reflection area R includes the display area R1.

The gripping state, which is the estimated result, can include information about which hand, left or right, or both hands the user is gripping the terminal device 1, information about which finger is operating the display unit 8, information about which position on the display unit 8 is touch-operated (or about to be touched), and information about whether the terminal device is placed on a stand such as a desk. That is, the estimation unit 42 estimates at least one of the hand holding the terminal device 1, the finger operating the display unit 8 of the terminal device 1, and the operation position on the display unit 8 as the holding state.

In addition to the grasping state, the estimation unit 42 can also estimate information as to whether the display unit 8 is being touched using a pointing device such as a touch pen. Such an estimation can be made by learning and generating a model in which the corneal image is used as an explanatory variable and information on whether or not the pointing device is used is used as an objective variable. That is, the estimating unit 42 estimates whether or not there is an operation with the pointing device as the gripping state.

Alternatively, the estimation unit 42 may adjust (correct) the luminance of the corneal image according to the illuminance detected by the illuminance sensor 6 before inputting it to the model. As a result, it is possible to improve the estimation accuracy of the gripping state, which is the output result from the model.

The providing unit 43 provides content according to the grip state estimated by the estimating unit 42 . For example, as shown in FIG. 1, the providing unit 43 moves the position of the icon image 8a displayed on the display unit 8 according to the holding state. Although FIG. 1 shows an example in which all the icon images 8a are moved closer to the hand holding the icon images, for example, only the icon images 8a designated in advance by the user or only a predetermined number of icon images 8a with the highest frequency of use may be moved. The position after movement is preferably within the reach of the fingers of the gripping hand. The reach range can be acquired in advance by input from the user.

Further, not only the icon image 8a, but also other interactive elements such as a home button, a back button, and advertising content may be moved closer to the hand holding the icon image. In other words, the interactive element such as the icon image 8a should not be placed outside the reach range.

Further, the providing unit 43 may change the setting mode of the terminal device 1 according to the estimated holding state. Specifically, the providing unit 43 may switch the mode of the web content or the like according to the holding state. Specifically, the providing unit 43 may activate the browser in secret mode when held with the right hand, and activate the browser in normal mode when held with the left hand. Such mode switching setting may be set in advance by the user or may be determined in advance.

Also, as an example of such mode switching, switching can be performed depending on whether or not the pointing device is used. For example, when it is estimated that a pointing device is used for operation, the providing unit 43 switches to the writing mode using the pointing device, and when it is estimated that the operation is performed with a finger (the pointing device is not used), it switches to the screen scrolling mode.

In addition, such mode switching may be performed depending on which hand, left or right, is used for gripping, rather than depending on the operating part. For example, the providing unit 43 may perform mode switching (or content provision) according to which part of the finger, nail, fist (knuckle), or finger joint the operation is performed.

Further, the providing unit 43 may provide content related to the user's degree of fatigue based on the estimated duration of the gripping state. For example, the providing unit 43 may provide warning information indicating that fatigue has accumulated or that tendonitis (De Quervain's disease) may develop if the same hand continues to hold the object for a predetermined time or longer. In addition, the providing unit 43 may suggest that the terminal device 1 be held with the other hand (or both hands) when the state of holding the terminal device 1 with one of the left and right hands continues for a predetermined time or longer.

Here, when the terminal device 1 is excessively tilted in one of the left and right directions when viewing the display unit 8, there is a possibility of eye strain due to the difference between the distance from the left eye to the display unit 8 and the distance from the right eye to the display unit 8. Therefore, the providing unit 43 provides content that guides how to view the display unit 8 correctly. Specifically, when the gyro sensor 7 tilts the terminal device 1 in the lateral direction by a predetermined amount or more and the display area R1 is not displayed in the corneal image, the providing unit 43 notifies the user of the possibility of eyestrain and guides the user to view the display unit 8 at a correct angle and distance.

More specifically, based on the tilt of the terminal device 1, the providing unit 43 notifies the user to change the tilt of the terminal device 1 until the display region R1 and the finger region R2 appear in the corneal image. For example, when the terminal device 1 is tilted rightward, the providing unit 43 notifies to tilt it leftward.

Further, the providing unit 43 instructs the terminal device 1 to move away from the user when the size (area) of the display region R1 reflected in the corneal image is equal to or larger than a predetermined value, and instructs the terminal device 1 to move closer to the user when the size (area) of the display region R1 is less than the predetermined value. That is, the providing unit 43 instructs to move the terminal device 1 until the distance between the terminal device 1 and the user becomes appropriate.

[Processing flow]
Next, a procedure of processing by the terminal device 1 according to the embodiment will be described with reference to FIG. FIG. 4 is a flowchart showing an example of processing executed by the terminal device 1 according to the embodiment.

As shown in FIG. 4, the acquiring unit 41 first acquires a corneal image based on the image captured by the camera 5 (step S101). Subsequently, the estimating unit 42 estimates the user's gripping state of the terminal device 1 based on the image of the reflection area reflected by the cornea in the corneal image (step S102). Subsequently, the providing unit 43 provides content according to the estimated gripping state (step S103), and ends the process.

〔others〕
Also, among the processes described in the above embodiments, some of the processes described as being automatically performed can also be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the illustrated one, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions.

For example, part or all of the storage unit 3 shown in FIG. 3 may be held in a storage server or the like instead of being held by the terminal device 1 . In this case, the terminal device 1 acquires various information by accessing the storage server.

[Hardware configuration]
Also, the terminal device 1 according to the above-described embodiments is implemented by a computer 1000 configured as shown in FIG. 5, for example. FIG. 5 is a diagram illustrating an example of a hardware configuration; A computer 1000 is connected to an output device 1010 and an input device 1020, and has a form in which an arithmetic device 1030, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 are connected by a bus 1090.

Arithmetic device 1030 operates based on programs stored in primary storage device 1040 and secondary storage device 1050, programs read from input device 1020, and the like, and executes various types of processing. The primary storage device 1040 is a memory device such as a RAM that temporarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device in which data used for various calculations by the arithmetic device 1030 and various databases are registered, and is realized by ROM (Read Only Memory), HDD (Hard Disk Drive), flash memory, or the like.

The output IF 1060 is an interface for transmitting information to be output to the output device 1010 that outputs various types of information, such as a monitor and a printer. Also, the input IF 1070 is an interface for receiving information from various input devices 1020 such as a mouse, keyboard, scanner, etc., and is realized by, for example, USB.

The input device 1020 may be, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase change rewritable disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a device that reads information from a semiconductor memory. Also, the input device 1020 may be an external storage medium such as a USB memory.

Network IF 1080 receives data from other devices via network N and sends the data to arithmetic device 1030, and also transmits data generated by arithmetic device 1030 via network N to other devices.

The arithmetic device 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070 . For example, arithmetic device 1030 loads a program from input device 1020 or secondary storage device 1050 onto primary storage device 1040 and executes the loaded program.

For example, when the computer 1000 functions as the terminal device 1 , the arithmetic device 1030 of the computer 1000 implements the functions of the control unit 4 by executing programs loaded on the primary storage device 1040 .

〔effect〕
As described above, the terminal device 1 according to the embodiment includes the acquisition unit 41 , the estimation unit 42 and the provision unit 43 . The acquisition unit 41 acquires a corneal image representing the user's cornea captured by the camera 5 built into the terminal device 1 carried by the user. The estimating unit 42 estimates the user's gripping state of the terminal device 1 based on the image of the reflection region R reflected by the cornea in the corneal image. The providing unit 43 provides content according to the grip state estimated by the estimating unit 42 . The estimating unit 42 estimates the gripping state using a model generated by using the corneal image captured in each gripping state as an explanatory variable and the corresponding gripping state as an objective variable. The estimating unit 42 extracts a region including at least one of the pupil and the iris as the reflection region R from the cornea. The estimation unit 42 estimates at least one of a hand holding the terminal device 1 , a finger operating the display unit 8 of the terminal device 1 , and an operation position on the display unit 8 as the holding state. The estimation unit 42 estimates the presence or absence of an operation with the pointing device as the grip state. The acquisition unit 41 acquires illuminance information from the illuminance sensor 6 that detects the illuminance around the terminal device 1 . The estimation unit 42 corrects the brightness of the corneal image based on the illuminance, and estimates the gripping state using the corrected corneal image. The acquiring unit 41 acquires a corneal image captured while a single image with a specific luminance is displayed on the display unit 8 of the terminal device 1 . The providing unit 43 moves the interactive element displayed on the display unit 8 to a position according to the grip state estimated by the estimating unit 42 . The providing unit 43 provides content related to the user's degree of fatigue based on the duration of the gripping state estimated by the estimating unit 42 . The providing unit 43 changes the setting mode of the terminal device 1 according to the grip state estimated by the estimating unit 42 . This makes it possible to provide appropriate content to the user.

As described above, some of the embodiments of the present application have been described in detail based on the drawings, but these are examples, and the present invention can be implemented in other forms with various modifications and improvements based on the knowledge of those skilled in the art, including the embodiments described in the disclosure of the invention.

〔others〕
Further, among the processes described in the above embodiments, all or part of the processes described as being automatically performed can be manually performed, or all or part of the processes described as being manually performed can be automatically performed by a known method. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the illustrated one, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions.

Further, each processing described in the above-described embodiments can be appropriately combined within a range that does not contradict the content of the processing.

Also, the "section, module, unit" described above can be read as "means" or "circuit". For example, the control unit 4 can be read as control means or a control circuit.

1 terminal device 2 communication unit 3 storage unit 4 control unit 5 camera 6 illuminance sensor 7 gyro sensor 8 display unit 31 model information 41 acquisition unit 42 estimation unit 43 provision unit 100 server device N network S control system

Claims (11)

an acquisition unit that acquires a corneal image showing the user's cornea captured by a camera incorporated in a terminal device owned by the user;
an estimating unit for estimating the user's gripping state of the terminal device based on an image of a reflection area reflected by the cornea in the corneal image;
a providing unit that provides content corresponding to the gripping state estimated by the estimating unit ;
The estimation unit
estimating presence/absence of operation with a pointing device as the grasping state;
A terminal device characterized by: The estimation unit
The terminal device according to claim 1, wherein the gripping state is estimated using a model generated by using the corneal image captured in each gripping state as an explanatory variable and the corresponding gripping state as an objective variable. The estimation unit
3. The terminal device according to claim 1, wherein an area including at least one of a pupil and an iris is extracted from the cornea as the reflection area. The estimation unit
3. The terminal device according to any one of claims 1 to 3, wherein at least one of a hand holding the terminal device, a finger operating the display unit of the terminal device, and an operation position on the display unit is estimated as the holding state. The acquisition unit
Acquiring the illuminance information from an illuminance sensor that detects the illuminance around the terminal device,
The estimation unit
The terminal device according to any one of claims 1 to 4 , wherein the luminance of the corneal image is corrected based on the illuminance, and the grip state is estimated using the corrected corneal image. The acquisition unit
The terminal device according to any one of claims 1 to 5, wherein the corneal image captured while a single image with a specific brightness is displayed on the display section of the terminal device is acquired. The providing unit
The terminal device according to any one of claims 1 to 6 , wherein an interactive element displayed on the display section of the terminal device is moved to a position according to the grip state estimated by the estimation section. The providing unit
The terminal device according to any one of claims 1 to 7 , wherein content regarding the degree of fatigue of the user is provided based on the duration of the gripping state estimated by the estimation unit. The providing unit
The terminal device according to any one of claims 1 to 8 , wherein a setting mode of the terminal device is changed according to the grip state estimated by the estimation unit. A control method executed by a terminal device,
an acquiring step of acquiring a corneal image showing the user's cornea captured by a camera incorporated in a terminal device owned by the user;
an estimating step of estimating the user's gripping state of the terminal device based on an image of a reflection area reflected by the cornea in the corneal image;
a providing step of providing content according to the gripping state estimated by the estimating step ;
The estimation step includes
estimating presence/absence of operation with a pointing device as the grasping state;
A terminal device control method characterized by: an acquisition procedure for acquiring a corneal image showing the user's cornea captured by a camera incorporated in a terminal device owned by the user;
an estimation step of estimating a gripping state of the user on the terminal device based on an image of a reflection area reflected by the cornea in the corneal image;
causing a computer to execute a provision step of providing content according to the gripping state estimated by the estimation step ;
The estimation procedure includes:
estimating presence/absence of operation with a pointing device as the grasping state;
A terminal device control program characterized by:

Images