JP6408045B2 - TERMINAL DEVICE, TERMINAL DEVICE CONTROL METHOD, AND COMPUTER PROGRAM - Google Patents

Description

  The present invention relates to a terminal device, a control method for the terminal device, and a computer program.

  A technique has been proposed in which the user's operation speed is measured and the display speed is higher than the conventional technique when the speed is higher than the average value (see Patent Document 1).

JP 2001-22488 A

  However, in the proposed technique, only the deviation from the average value is detected in the operation unit, and the display speed always changes if the operation deviates from the average value regardless of the state of the user who is performing the operation. turn into.

  Therefore, in the present invention, a condition for determining the state of the user who is operating the terminal is specified for each user, the state at the time of the user's terminal operation is determined according to the condition, and the terminal according to the state of the user A technique for controlling a process to be executed is provided.

A terminal device corresponding to one of the embodiments of the invention for solving the above-described problem is
Detecting means for detecting user input;
Storage control means for storing the user input detected by the detection means after the start of a predetermined operation in the terminal device in a storage unit in association with the predetermined operation;
Determining means for determining whether any user input stored in the storage unit in association with the predetermined operation has occurred when the predetermined operation of the terminal device is started;
Control means for controlling processing executed by the terminal device for the predetermined operation according to whether the determination means determines that any one of the user inputs has occurred.

The terminal device corresponding to the embodiment of the invention for solving the above-described problem is
Detecting means for detecting user input;
Storage control means for storing a plurality of user inputs as a user input set in a storage unit when the detection means detects a plurality of user inputs different from the average input of the user within a predetermined period;
Determination means for determining whether any user input set stored in the storage unit has occurred while the terminal device is performing a predetermined operation;
Control means for controlling processing executed by the terminal device for the predetermined operation according to whether the determination means determines that any one of the user input sets has occurred.

  According to the present invention, a condition for determining the state of a user who is operating a terminal is specified for each user, the state at the time of the user's terminal operation is determined according to the condition, and the terminal according to the user's state A technique for controlling a process to be executed can be provided.

The figure which shows the structural example of the whole client server system corresponding to embodiment of invention. The block diagram which shows an example of the hardware constitutions of the client 101 corresponding to embodiment of invention. The block diagram which shows an example of the hardware constitutions of the server 103 corresponding to embodiment of invention. The flowchart which shows an example of the flow of a user characteristic table production | generation process corresponding to Embodiment 1 of invention. The figure which shows an example of the user characteristic table corresponding to embodiment of invention. The flowchart which shows an example of the flow of a process in the client 101 corresponding to embodiment of invention. The flowchart which shows an example of the flow of the frustration elimination process in the client 101 corresponding to embodiment of invention. The flowchart which shows an example of the flow of an output process of the process result in the client 101 corresponding to embodiment of invention. The flowchart which shows an example of the flow of a user characteristic table production | generation process corresponding to Embodiment 2 of invention. The figure which shows an example of the average value table corresponding to Embodiment 3 of invention.

  Embodiments according to the present invention will be described below with reference to the drawings. In addition, embodiment shown below is an example and is not limited to this. Therefore, other configurations may be used as long as the characteristic configuration of the present invention can be applied.

[Embodiment 1]
Hereinafter, a first embodiment of the present invention will be described with reference to the accompanying drawings. However, the constituent elements described in this embodiment are merely examples, and are not intended to limit the scope of the present invention only to them.

<System configuration>
The configuration of the client server system corresponding to this embodiment will be described below. FIG. 1 is a block diagram showing an example of the overall system configuration corresponding to this embodiment when a system is constructed as an online shopping site. The system is configured by connecting user terminals and servers to a network. The server manages an online shopping site and manages store and customer user information in addition to image information of products sold on the site.

  Clients 101a, 101b, and 101c (hereinafter collectively referred to as “client 101”) are user terminals that can be serviced by the server after being operated by the user and receiving user authentication by the server. is there. In this embodiment, the service can include browsing of information provided on an online shopping site.

  The server 103 is a device that authenticates the user of the client 101 and provides the service to the client 101 used by the authenticated user. The client 101 and the server 103 are each connected to the network 102 and can communicate with each other. The network 102 can be constructed as, for example, the Internet, a local area network (LAN), or a wide area network (WAN). The Internet is a network in which networks all over the world are connected to each other. However, the network 102 may be a network that can be connected only within a specific organization, such as an intranet. A user information database 104 and a content database 105 are connected to the server 103.

  The client 101 can use a service provided by the server 103 by a user operation. The service includes, for example, browsing page information provided by the server 103, searching for information, narrowing down, sorting, acquiring other information, uploading information, and the like. The client 101 is a user terminal, a terminal device, an information processing device, or a communication device, and includes, for example, a smartphone, a tablet terminal, a notebook personal computer, a desktop personal computer, a portable information terminal, a mobile phone, and the like. The client 101 is assumed to be equipped with various sensors for detecting the user state. The sensor includes a sensor having a function of detecting video, touch, vibration, voice and the like. The client 101 is connected to the network 102 by wireless data communication means such as a wireless LAN or LTE. The network 102 may be further configured to be accessible by a LAN including a network cable such as Ethernet (registered trademark).

  The server 103 manages the user information database 104, holds registration information of each user of the client 101, and when each user wants to receive a specific service, the user receives the service provided. It can be used to determine whether or not it has authority. The server 103 can also manage data stored in the content database 105 and provide the data to the client 101 in response to a request from the client 101. The server 103 is connected to the user information database 104 and the content database 105 via, for example, a LAN. Each of the user information database 104 and the content database 105 is an information processing apparatus in which predetermined database software is installed, and manages various data.

  The user information database 104 manages registration information for each user. Specifically, a user identifier (user ID) for uniquely identifying a user, user registration information for determining whether or not the user is a registered user (for example, a set of a user name and a password, etc.) are stored in association with each other. To do.

  The content database 105 stores and manages content data provided from the server 103 to the client 101. In addition, content uploaded by the client 101 can be stored and managed. The content data includes data such as text, images, moving images, and sounds. These content data are assigned content IDs for uniquely identifying the content data.

  In the present specification, for convenience, the server 103, the user information database 104, and the content database 105 are described as being realized by physically independent information processing apparatuses, but the embodiment of the present invention is not limited thereto. It is not limited. For example, these may be realized by a single information processing apparatus. On the other hand, each device such as the server 103 may be configured redundantly or distributedly by a plurality of information processing devices. Although the user information database 104 is described as being connected to the server 103 via a LAN or the like, for example, the user information database 104 may be configured to be able to communicate with the server 103 via the network 102 or an intranet (not shown). The same applies to the relationship between the server 103 and the content database 105. The user information managed by the user information database 104 and the content-related data managed by the content database 105 may be managed integrally.

<Configuration of client 101>
FIG. 2 is a block diagram illustrating an example of the hardware configuration of the client 101. The client 101 corresponding to this embodiment can be configured as, for example, a mobile phone, a smartphone, or a tablet terminal. Hereinafter, the client 101 corresponding to the present embodiment will be described with reference to FIG. 2, but FIG. 2 illustrates an example in which the client 101 is configured as a smartphone.

  In FIG. 2, a CPU 200 includes an application program (including an application program for executing processing corresponding to the present embodiment) stored in a hard disk device (hereinafter referred to as HD) 205, an operating system (OS), and control. A program or the like is executed, and control for temporarily storing information, files, etc. necessary for executing the program in the RAM 202 is performed. Controls data transmission / reception with an external device via the interface 208, analyzes and processes data received from the external device, and generates data (including processing requests and data requests) to be transmitted to the external device. .

  The ROM 201 stores various data such as a basic I / O program and an application program for executing processing corresponding to the present embodiment. The RAM 202 temporarily stores various data and functions as a main memory, work area, and the like of the CPU 200.

  The external storage drive 203 is an external storage drive for realizing access to a recording medium, and can load a program or the like stored in the medium (recording medium) 204 into the computer system. The media 204 can use, for example, a flash (registered trademark) memory such as an SD card.

  In this embodiment, the external storage device 205 uses an HD (hard disk) that functions as a large-capacity memory. The HD 205 stores application programs, OS, control programs, related programs, and the like. Note that a nonvolatile storage device such as a flash memory may be used instead of the hard disk.

  The instruction input device 206 corresponds to a button, a keyboard, a mouse, a touch panel, or the like. The output device 207 outputs a command input from the instruction input device 206, a response output from the client 101 in response thereto, and the like. The output device 207 includes a display, a speaker, a headphone terminal, and the like. A system bus 209 manages the flow of data in the terminal device.

  An interface (hereinafter referred to as I / F) 208 serves as a transmission device, a reception device, or a communication device that mediates data exchange with an external device. Specifically, the I / F 208 can include a wireless communication module, which includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset. Well-known circuitry, including a subscriber identity module card, memory, and the like. In addition, a wired communication module for wired connection can be included. The wired communication module enables communication with other devices via one or more external ports. It can also include various software components that process the data. The external port is coupled to another device directly via Ethernet, USB, IEEE1394, or indirectly via a network. In addition, it can also comprise as a substitute of a hardware apparatus with the software which implement | achieves a function equivalent to each above apparatus.

  The imaging unit 210 can include an imaging optical system, an imaging element (such as a CMOS sensor), an A / D conversion unit, and the like. The image signal output from the imaging unit 210 may be stored in the RAM 202, converted to, for example, a JPEG image by compression / encoding processing by the CPU 200, and stored in the medium 204 or the HD 205. In addition, the CPU 200 may analyze the captured image in order to determine the user's state from the user's facial expression.

  The voice input unit 211 collects (collects) the voice around the client 101 using a built-in omnidirectional microphone, performs analog-digital conversion, and transmits the collected voice to the CPU 200. The CPU 200 may process the input digital audio signal and store it in the medium 204 or the HD 205 as necessary. In addition, the CPU 200 may analyze the voice generated by the user in order to determine the user state, or may perform voice analysis to determine the surrounding environment of the client 101.

  The vibration detection unit 212 can detect the vibration of the client 101 using an angular velocity sensor or an acceleration sensor and notify the CPU 200 of the vibration. The CPU 200 can analyze the vibration information in order to determine the installation environment and usage status of the client 101.

  The client 101 may load the program from the HD 205 in which the program is already installed into the RAM 202 every time the corresponding program is operated in order to execute processing corresponding to the present embodiment. It is also possible to record the program according to the present embodiment in the ROM 201, configure it as a part of the memory map, and execute it directly by the CPU 200. Furthermore, the corresponding program and related data can be directly loaded from the medium 204 into the RAM 202 and executed.

  Further, the user information database 104 and the content database 105 as the information processing apparatus described above may also be configured to have the same or equivalent hardware configuration from the CPU 200 to the system bus 209 in FIG.

<Configuration of Server 103>
FIG. 3 is a block diagram illustrating an example of a hardware configuration of the server 103. The functions and applications of the CPU 300, ROM 301, RAM 302, external storage drive 303, media 304, HD 305, instruction input device 306, output device 307, I / F 308, system bus 309, and the relationship between them are shown in FIG. It is the same as or equivalent to that described for the element having the same name.

  The server 103 can further include a database 310. The database 310 is connected to the system bus 309 and stores and manages predetermined data under the control of the CPU 300. The predetermined data can include content data provided to the client 101. The database 310 can include the user information database 104 and the content database 105 of FIG.

  Hereinafter, operation control of the client 101 based on the operation characteristics of the user executed in the client server system corresponding to the present embodiment will be described. The user who uses the client 101 may change the way of operation depending on the mental state / psychological state at that time. For example, when annoyed or feeling stressed, the number of operations increases or the operation speed increases. When the client 101 has a touch panel, the operation pressure on the touch panel may be increased. When the user is in such a state, it is assumed that the user is requesting that the terminal operated by the user responds quickly to the operation.

  Therefore, in the present embodiment, such a sign indicating the mental state / psychological state of the user is detected, and control for switching display processing is executed. At that time, since the manner of change of mental state and psychological state varies depending on the user, a parameter that easily changes the mental state of the user is identified from among a plurality of parameters, and the current state of the user is determined based on the change of the parameter. Determine the state. In this embodiment, first, the operation from the user is continuously monitored, the operation of the device is observed to see how the user reacts, and a change in the mental state of the user is output. Create easy-to-use parameters as a table and create a user characteristics table.

  The parameters that can be tabulated include any parameters that the client 101 can detect in the context of the user. For example, when the client 101 is a smartphone and has a touch panel, all input by various operations (tap pressure, swipe speed, tap count, tap position, pinch gesture count, etc.) accepted on the touch panel can be subject to table registration. . In addition, specific voice input (tongue, speech, etc.) by the user detected by the voice input unit 211, vibration of the client 101 detected by the vibration detection unit 212 (vibration generated when the user shakes the device, etc.) ) Can also be included.

  Hereinafter, with reference to the flowchart of FIG. 4, a user characteristic table generation process executed in the client 101 corresponding to the present embodiment will be described. The processing corresponding to the flowchart can be realized, for example, when the CPU 200 executes a corresponding program (stored in the ROM 102, the HD 105, or the like).

  First, in S401, the CPU 200 determines whether or not a predetermined operation instruction (or a predetermined operation input) has been detected. The detection target includes, for example, operation instructions such as information search, narrowing down, sorting, acquisition of other information, information upload, display of a new link destination, page switching of a web page, updating of a web page (or the like) Operation input for instructing execution of the operation). For example, when searching for information, it can be determined that a predetermined operation instruction has been detected when an operation on the search button is detected. If a new link destination is displayed, it can be determined that a predetermined operation instruction has been detected when a tap operation on the URL displayed in the page is detected. Similarly, when a tap operation corresponding to page switching or updating is detected, it can be determined that a predetermined operation instruction has been detected.

  When the CPU 200 detects the operation instruction in this way, the process proceeds to S402, and the CPU 200 delays execution of the instructed operation. For example, in the case of displaying a new link destination, the page information of the link destination may be acquired from an external web server or the like based on the URL, but even if the page information can be acquired, the client The operation is controlled so as not to be displayed on the display of the output device 107.

  The operation delay includes, for example, the following. Does not respond to user operations (for example, the screen does not move in response to a swipe), decreases the display switching speed of the website, decreases the scroll speed of the page, stops the display of the page, suddenly darkens the screen Such as closing a running application suddenly, or displaying a comment unrelated to the operation of the terminal.

  Next, the process proceeds to S403, and the CPU 200 determines whether or not an elapsed time Tp after detecting a predetermined operation instruction is equal to or greater than T1. If the elapsed time Tp is less than T1, the process proceeds to S404. If the elapsed time Tp is equal to or greater than T1, the process proceeds to S406. In S404, the CPU 200 determines whether an input is detected. The input includes touch panel input, voice input, vibration, and image detected while a process corresponding to the operation is not performed after the user performs an operation for performing a predetermined operation instruction in S401. It can also be included. These inputs indicate that the user feels frustrated. For example, input such as an operation input that strikes the screen, an input that strongly presses the screen, a quick swipe input, a tongue strike, or the utterance of a specific word. Expressions representing user dissatisfaction obtained by input and image analysis are included. In addition, it is possible to determine whether or not a user input different from the average input of the user has been detected by holding an average value of inputs that can be performed by the user and comparing the average values. For example, with respect to flick input, an average value of the moving speed of the finger may be held, and if it is faster than the average value, it may be determined that the input has been detected.

  If an input is detected in this way, the process proceeds to S405. On the other hand, if no input is detected, the process returns to S403 to monitor the elapsed time. In S405, the CPU determines the type of input detected in S404, temporarily registers it in the table in association with the operation detected in S401, and returns to S403.

  In S406, the CPU 200 executes the process delayed in S402. In subsequent S407, with reference to the registered contents of the table, whether or not there is a combination of the temporarily registered detection operation and detection input that is registered more than a predetermined number of registrations (for example, 5 times) in the same combination. Determine. If the same combination registered more than the predetermined number of registrations exists, the process proceeds to S408. If the same combination registered more than the predetermined number of registrations does not exist, the process returns to S401 and the above process is repeated.

  The CPU 200 performs the main registration in the user characteristic table for the same combination that is determined to have been registered a predetermined number of times or more in S408. Then, it returns to S401 and repeats the above process. At this time, the operation related to the combination registered in S408 may be excluded from the operation to be detected in S401. For example, when the main registration for the operation for displaying the new link destination is performed, in the subsequent processing, even when the operation for displaying the new link destination is detected, the processing delay in S402 is also delayed. If the link destination page can be received, the display may be performed immediately. Or you may make it include in operation | movement of a detection object after progress for a fixed period. Moreover, when not excluding from the operation | movement of search object, the combination of several user input may be registered about the same operation | movement.

  As described above, it is possible to register the input representing the frustration of the user and the operation of the apparatus in association with each other. The registered user characteristic table data may be stored in the HD 205 in the client 101 and may be stored in the server 103 or the user information database 104.

  In S402, the execution of the operation is intentionally delayed, but the same processing may be executed when a delay occurs accidentally in the normal operation. For example, after the operation instruction is detected in S401, S402 is not executed, and the process is executed when the process is ready in S406. At that time, the elapsed time from the detection of the operation instruction to the execution of the process is measured, and the input detected in S404 is provisionally registered in S405 when the process cannot be executed even after a certain elapsed time has passed. Thereby, it is possible to register an input representing the frustration of the user and the operation of the apparatus in association with the accidentally generated delay.

  FIG. 5 shows an example of the user characteristic table. In the user characteristic table 500, information including a registration type 501, a detection operation 502, a detection input 503, and a detection date 504 is registered. Although omitted in FIG. 5, the user characteristic table 500 can be associated with a user identifier for identifying the user.

  There are two types of registration types 501, temporary registration and main registration, and a detection operation 502 and a detection input 503 are registered for each. For temporary registration, the detection operation 502 registers the operation detected in S401 of FIG. In addition, the input detected in S404 is registered in the detection input 503, and the input detected until the elapsed time Tp reaches T1 can be registered. In the detection date and time 504, the date and time when the input is detected in S404 is registered, and this can be the date and time when the elapsed time reaches T1. When the detection date and time 504 has passed for a certain period of time compared to the current date and time, it may be deleted from the user characteristic table 500 as not being valid information.

  In the main registration, the detection operation 502, the detection input 503, and the detection date and time 504 are registered for the same combination of the detection operation and the detection input registered more than the predetermined number of registrations among the operations registered in the temporary registration. . For example, in the case of FIG. 5, the input 1 and the input 2 detected for the operation A are fully registered. These are the main registrations that have been registered, for example, five times or more in the temporary registration. As the detection date and time 504, the date and time of the main registration can be registered.

  The combination registered in the user characteristic table 500 is deleted from the temporary registration area, and thereafter, the same operation need not be temporarily registered. Alternatively, if a certain period of time has elapsed since the registration date and time, temporary registration is resumed for the same operation, and if the associated detection input combination is different, the content already registered at the time of the main registration is overwritten, You may register newly. This is to cope with the case where the user's reaction varies with time.

  When moving from temporary registration to main registration, there are a plurality of detection inputs 503 temporarily registered for the same detection operation, and all of them may not match. For example, paying attention to the operation B that has been temporarily registered in FIG. 5, since five temporary registrations have already been made, the main registration is possible. Of the detection inputs 503 for temporary registration contents, the input 3 and the input 1 are common to all, but the input 2 is detected only once. Therefore, in such a case, the input 2 can be ignored, and only the input 3 and the input 1 can be registered in association with the operation B. When the input 2 is more than a majority of the predetermined number of registrations, it may be the target of the main registration, or after only the input 3 and the input 1 that have exceeded the predetermined number of registrations are registered, the operation B continues. Temporary registration can be continued for the input 2, and after the predetermined number of registrations is registered for the input 2, it can be the target of the main registration.

  Here, by setting the predetermined number of registrations to a larger value, it is possible to statistically narrow down the operations that the user surely executes for a certain operation. As a result, for example, the input detected by chance under certain conditions is statistically excluded, so it is possible to specify the input that reflects the user's frustration without comparing with the average value. It becomes. In FIG. 5, a plurality of inputs are registered as the detection inputs 503. However, there is no need for a plurality of detection inputs, and a single input may be used. In that case, an input detected with the highest frequency for the associated detection operation 502 can be registered as a detection input.

  The input to be registered in the table is an input detected within a predetermined time T1, but it may be registered as a gesture input by collectively handling a plurality of detected same type (or different types) inputs. For example, while there are users who tap at a certain interval and tap the screen, there are users who repeat swipes in a certain direction. In this case, a single tap or swipe itself can be registered as an input, but multiple taps input within a certain period of time can be combined as a continuous tap gesture, and repeated swipes can be registered as a continuous swipe gesture. In this case, the state of the user can be determined by a gesture executed by the user. For example, if you are simply swiping in a single shot, you can determine that you are not feeling frustrated, but if you are repeatedly tapping continuously, you can determine that the user is feeling frustrated. it can.

  Next, a flow of processing corresponding to the present embodiment performed by the client 101 will be described with reference to FIG. FIG. 6 shows an example of a flowchart of processing executed by the client 101. The processing corresponding to the flowchart can be realized by, for example, executing a program corresponding to the CPU 200 of the client 101 and controlling the operation of each functional block.

  First, in step S601, the client 101 receives an operation from the user. The operations accepted here include, for example, information search, narrowing down, sorting, obtaining other information, uploading information, displaying pages, updating, presenting linked information, zooming in, zooming out, etc. Operations are included. When receiving an operation from the user, the CPU 200 starts a process corresponding to the operation received in S602. When the client 101 receives an operation for requesting acquisition of information from the server 103, the information acquisition request is transmitted using a Web API for acquiring the information (hereinafter simply referred to as API). . For example, in the case of information search, an API for requesting the server 103 to execute search is used to transmit a search request. At this time, if there are a plurality of usable APIs, the API designated as a default in the client 101 is used. For processing that can be executed on the client 101 side, the CPU 200 starts the processing. For example, when a zoom-in operation for the displayed page is accepted, a zoom-in operation corresponding to the operation amount received from the user can be executed.

  When the client 101 uses the API to request the server 103 to execute a predetermined process, when the server 103 receives the request from the client 101, the server 101 executes the process corresponding to the request. It is transmitted to the client 101 as a response to the request.

  While the processing in S602 is being executed, the user of the client 101 is placed in a waiting state waiting for the completion of the processing, but the waiting user feels frustration when the processing is not completed and a response to the user operation cannot be made. Become. Therefore, in subsequent S603, the CPU 200 determines whether the user feels frustrated. This determination can be made based on whether or not an input corresponding to the detection input 503 registered in the user characteristic table 500 is detected in association with the operation (predetermined process) received in S601. At this time, among the detected inputs, only the input in which the deviation from the average value given for each input is recognized may be determined as the input corresponding to the registered input 503. Further, when a plurality of inputs are registered in association with a predetermined process in the user characteristic table 500, it is determined that frustration is felt when any one or more inputs are detected in relation to the predetermined process. You may do it. Alternatively, when a plurality of inputs are registered, it may be determined that frustration is felt when at least two or all of the plurality of inputs are detected within a predetermined time.

  When the CPU 200 of the client 101 detects the frustration of the user, the process proceeds to S604, and the client 101 performs a process for eliminating the frustration. For example, if the user feels frustrated because of a long waiting time, the processing speed is increased. Details of the frustration elimination process will be described later with reference to FIG. In the subsequent S605, when the processing is completed, the processing result is output. For example, when the operation in S601 is an operation for requesting an information search, the search result received from the server 103 is displayed on the display. At this time, the output process can be executed so as to eliminate the frustration of the user. Details will be described later with reference to FIG.

  Next, the frustration elimination processing in the client 101 in S604 will be described in more detail with reference to FIG. Here, when using the API, processing that can be executed in cooperation with the server 103 is included.

  First, in S701, the CPU 200 determines whether there is an API that can respond faster on the server 103 side than the selected default API for the API for executing the process corresponding to the operation accepted in S601. Determine. The API that can respond at high speed may be, for example, an API that enables the server 103 to respond at high speed by increasing the processing speed on the server side. Further, for example, when a search is requested from the client 101, the server 103 responds at high speed by reducing the information included in the search result (for example, omitting the image, omitting the description information regarding the search target). It may be an API that makes it possible. If there is such an API that can respond at a higher speed, the request is transmitted again to the server 103 using the high-speed API in S702. At this time, information indicating which operation the user characteristic table 500 is based on may be included. If there is no API capable of a faster response, the process proceeds to S703.

  In step S703, the CPU 200 determines whether there is an API corresponding to the operation accepted in step S601. If there is a corresponding API, the process proceeds to S704, and if there is no corresponding API, the process proceeds to S705. For example, a certain operation can be executed using the function of the application on the client 101 side, but when the user feels frustration by executing the local function, the process can be switched to the processing on the server 103 side. At this time, if an API corresponding to the received user operation is prepared, a processing execution request may be transmitted to the server 103 using the API in S704. Thus, the frustration can be resolved by replacing the processing on the client 101 side, which the user felt frustrated with, the processing on the server 103 side.

  In addition, when the user feels frustration in the display form itself or in the process of changing the display form (for example, when the user feels frustration because the enlargement / reduction operation after image display cannot be performed well) In some cases, switching the CSS (Cascading Style Sheets) can change the layout of the display screen and eliminate frustration. Therefore, in S705, it is determined whether or not it is possible to change the CSS. If the change is possible, the CSS is changed in a direction to eliminate the user's frustration in S706. For example, when the user repeats zoom-in operations because the display is small, or when the user frequently scrolls because the display is large, it can be assumed that the user feels frustrated. In such a case, it is determined that the CSS can be changed, and the CSS is changed to a CSS layout in which the page display is enlarged. On the other hand, if it takes a long time to receive the search result, the CSS change is not directly related to eliminating the frustration.

  When the server 103 receives a request based on a new API by the process of FIG. 7, the server 103 executes a corresponding process. When the processing on the server 103 side is completed, the processing result is transmitted from the server 103 as a response to the request.

  Next, the output of the processing result in the client 101 in S605 will be described in more detail with reference to FIG. First, in step S <b> 801, it is determined whether an image is included in the transmission result from the server 103. The response result to the API is provided in, for example, JSON or XML format, and generates information to be displayed on the display based on the response result received by the client 101. For example, when a link (URL) for acquiring a display image is included in the response result, if the image size is large, the image quality (resolution) is high, and the data size is large. May be delayed and may not resolve user frustration. Therefore, if an image link is included, the process proceeds to S802, and if an image link is not included, the process proceeds to S803. In S802, the size of the image included in the image link is changed to a small value, and the process proceeds to S803. For example, when a URL for obtaining an image size of 128 × 128 is described in the response result, it may be changed to 64 × 64.

  Next, in S803, it is determined whether or not the number of display items is a predetermined number or more. If there are many items, it takes time to acquire and generate display information for all items, and the display on the client 101 is not completed. Therefore, there is a possibility that a delay occurs and the user's frustration cannot be solved. is there. For example, when the number of items and images included in the search result is a predetermined number or more, the display is not completed until each item and image are received. Further, when the page to be used for display is a page obtained by mashing up data acquired from a plurality of APIs, the display completion of the page depends on the API with the slowest response. In such a case, the item corresponding to the API with a slow response may be deleted, and only the item corresponding to the response result from the API with a quick response may be displayed.

  In the above description, the processing speed and the amount of information are reduced on the client 101 side to eliminate frustration without making an inquiry to the user. However, some users may feel frustrated and want to prioritize information volume and quality over speed. Therefore, when the user's frustration is detected in S603, the user may be inquired whether to execute the process of S604. In response to the inquiry, if the user wants to execute the process of S604, the process after S604 is executed. If the user does not want to execute the process, the process after S604 is not executed. The started processing can be continued as it is.

  As described above, in this embodiment, since an input generated in relation to a specific operation of the client 101 is monitored, whether or not the user feels frustration based on a predetermined user input generated in connection with the operation is determined. The display on the client 101 can be efficiently controlled.

  For example, a tap input for a delay at the timing at which the display is switched can be regarded as a cue for the user to speed up the display rather than an instruction operation on the terminal. Therefore, by associating such an operation with tap input, the display speed can be changed when a tap occurs at the timing of display switching. As described above, by storing the terminal operation and the parameter in association with each other, it is possible to prevent malfunction due to an operation change unrelated to the operation of the apparatus.

  Also, for example, if there is a user whose tongue is a habit, it is not known whether it is for the contents of the display content or for the operation of the device. In that case, if the display speed is switched every time a tongue hit occurs, there is a possibility that it is contrary to the user's intention. Further, if the API is changed each time and the request is retransmitted to the server 103, the load on the server 103 side increases. Therefore, by combining the tongue setting and other conditions (operation of the apparatus or other operation parameters), it is possible to accurately determine frustration for the apparatus and execute the processing.

  Furthermore, according to the present embodiment, it is possible to specify a gesture by handling not only individual inputs but also inputs detected within a certain period of time for input such as taps and swipes. As a result, when the user feels frustration, for the user who performs a gesture input that is a combination of specific inputs (for example, repeating a tap with a specific rhythm or repeating a swipe after several taps) It becomes possible to determine the state of the user depending on whether or not it has occurred.

  Furthermore, in the present embodiment, since the mental state of the user is determined based on a combination of a plurality of inputs and the relevance between the operation and the input, the display control is switched by accurately capturing a case where a quick display is necessary. be able to. Therefore, a high-speed API is frequently used according to simple input detection or input change detection, and as a result, it is possible to avoid a situation in which the load on the server becomes high. Can be improved.

[Embodiment 2]
In the first embodiment described above, user input performed in relation to a predetermined operation in the client 101 is detected and used as user characteristics. On the other hand, in the present embodiment, the user's frustration state can be determined by a combination of a plurality of inputs (user input set) without associating with a predetermined operation.

  In this embodiment, it is possible to monitor user input, associate changes in input detected at the same time, and distinguish them from the others. For example, when the user feels frustration because it takes time to switch the display, and a swipe operation and a tap operation are performed, a change is monitored for a combination of these parameters. If the parameter changes as a set, the display speed is changed assuming that the mental state of the user has changed. As a result, it is possible to avoid a situation in which the display resolution is always lowered simply because the swipe speed is accidentally high. In addition, if it is stored in relation to a specific device operation, it is necessary to associate it with each operation. However, if a set of parameters is stored, a change in operation that occurs in connection with a part of the operation will occur. It is not limited, and a wide range of correspondence is possible.

  Hereinafter, with reference to the flowchart of FIG. 9, a user characteristic table generation process executed in the client 101 corresponding to the present embodiment will be described. The processing corresponding to the flowchart can be realized, for example, when the CPU 200 executes a corresponding program (stored in the ROM 102, the HD 105, or the like).

  First, in S901, the CPU 200 determines whether a user input is detected. The user input can include any input that can be acquired by the instruction input device 208, the voice input unit 211, the vibration detection unit 212, and the imaging unit 210. If a user input is detected, the process proceeds to S802, and the CPU 200 determines whether the detected user input is an input deviating from the average value of the user input. For example, the client 101 can hold an average value for the number of operations per unit time, the pressing force at the time of tapping, and the like for tap input. If a tap input is detected as a user input in S901, it is determined whether or not a deviation from the average value of the number of detected taps or the pressing force is larger than a predetermined value. Can be determined that there is a deviation from the average. If it is determined that there is a deviation from the average, the process proceeds to S903, and the CPU 200 provisionally registers the detected user input in the user characteristic table. As the user characteristic table in the present embodiment, the user characteristic table 500 shown in FIG. 5 can be diverted. However, in this embodiment, the item of the detection operation 502 is blank, and the detection input 503 and the detection date 504 may be registered.

  Next, the process proceeds to S904, and the CPU 200 determines whether or not the elapsed time Tp ′ after detecting the user input determined to have a deviation from the average is equal to or greater than T2. Here, the value of T2 may be changed according to the content of the detected user input. If the elapsed time Tp ′ is less than T2, the process returns to S901 and the process is repeated. At this time, after the measurement of the elapsed time Tp ′ is started, the inputs detected within the elapsed time T2 and determined to deviate from the average are provisionally registered in association with each other in S903. If the elapsed time Tp ′ is equal to or greater than T2, the process proceeds to S905. In S905, referring to the registered contents of the table, among the combinations of detection inputs that are provisionally registered and associated with each other, whether or not there is a combination that is registered more than a predetermined number of registrations (for example, 5 times) with the same combination. Determine whether. If the same combination registered more than the predetermined number of registrations exists, the process proceeds to S906. If the same combination registered more than the predetermined number of registrations does not exist, the process returns to S901 and the above process is repeated.

  In S <b> 906, the CPU 200 performs main registration in the user characteristic table 500 for the same combination that is determined to have been registered a predetermined number of times or more. Then, it returns to S901 and repeats the above process. At this time, the operation related to the combination registered in S906 may be excluded from the operation to be detected in S901. For example, when the pinch input and the tongue input voice are registered as main, the same combination may not be temporarily registered. Alternatively, provisional registration may be performed after a certain period of time has elapsed since the main registration. In this embodiment, as in the first embodiment, a single input (tap, swipe, etc.) can be registered as an input. However, a plurality of inputs input within a predetermined time are collectively registered as a gesture input. May be.

  As described above, the user input set representing the frustration of the user can be registered in the user characteristic table 500. Also in the present embodiment, by executing the processing corresponding to FIGS. 6 to 8, the user's frustration is detected based on the user characteristics registered in the user characteristics table 500, and this is effectively eliminated. be able to. However, in the present embodiment, it is not necessary to determine whether or not the frustration state is associated with the operation received in S601, and can be performed based on user input after the process is started. If frustration is detected, the processing of S604 and S605 may be executed for the processing being executed at that time.

[Embodiment 3]
In the first embodiment and the second embodiment, the case where the detected user input is used in comparison with the average value of the user input is described. In these embodiments, although an average value corresponding to the type of user input is used, a uniform average value is used regardless of the usage status and usage period when the user is using the client 101. It was. However, even for the same user, the mode of user input may differ depending on the time zone used or the surrounding environment. In this embodiment, the usage time zone, the surrounding environment of the client 101, etc. The case where the determination is made in consideration of the external information is described.

  FIG. 10 shows an example of an average value table 1000 that manages the average value of each user input in associating the user input with conditions such as the surrounding environment and use time zone. In the average value table 1000, user inputs 1001 to be detected are registered, and a plurality of conditions are registered as conditions 1002 for each user input. The condition 1002 includes, for example, a condition regarding a time period in which the user uses the client 101 (for example, from 7 am to 9 am in the withdrawal time period, after 6 pm in the returning time period), Conditions related to the surrounding environment (such as riding on a train or being in a noisy environment) can be registered.

  Whether or not the use of the client 101 by the user matches any of the conditions is determined by, for example, whether the output from the CPU 200, the imaging unit 210, the audio input unit 211, or the vibration detection unit 212 satisfies any of the conditions. can do. For example, the CPU 200 can determine the time based on the clock inside the client 101 and determine whether the condition is met. Further, whether or not the user is on the train can be determined based on the voice detected by the voice input unit 211 and the vibration pattern detected by the vibration detection unit 212. Further, the level of noise in the surrounding environment of the client 101 can be determined based on the sound level detected by the sound input unit 211.

  In this way, when it is determined that any one of the conditions 1002 is met, the average value compared with the detected user input used in the determination of S603 is associated with the condition in the average value table 1000. Use registered values.

  This makes it possible to determine whether or not the user feels frustrated according to the user's usage environment. For example, when using during commuting, a user whose operating pressure is stronger than at other times is determined to be in the commuting time zone or determined to be on the train. In this case, it is possible to increase the average value so as not to impair the accuracy of determination. By appropriately setting the average value in this way, it is possible to limit unnecessary API requests from the client 101 to the server 103, so that it is possible to reduce the processing load on the server 103 side.

[Other Embodiments]
The present invention is not limited to the above-described embodiment, and various changes and modifications can be made without departing from the spirit and scope of the present invention. Therefore, in order to make the scope of the present invention public, the following claims are attached. The information processing apparatus according to the present invention can also be realized by a computer program that causes one or more computers to function as the information processing apparatus. The computer program can be provided / distributed by being recorded on a computer-readable recording medium or through a telecommunication line.

Claims (15)

A terminal device,
Detecting means for detecting user input;
Wherein a plurality of said user input is detected same or different by the detecting means within a predetermined period after the start of the predetermined operation in the terminal apparatus, the predetermined operation with associated and stored in the storage unit as the provisional registration, the temporary registered among the user input, those which are a predetermined number of times or more the temporary registration, a storage control unit that Ru is stored in the storage unit as a registration,
A determination unit that determines whether any user input stored in the storage unit as the main registration has occurred in association with the predetermined operation during the predetermined operation of the terminal device;
Control means for controlling processing executed by the terminal device for the predetermined operation according to whether the determination means determines that any one of the user inputs has occurred ;
A terminal device comprising: It said storage control means, a gesture input based on a combination of said detected plurality of said user input within the predetermined time period, in association with the predetermined operation as provisional registration is stored in the storage unit, which is the temporary registration Of the gesture input, the provisionally registered one or more times is stored in the storage unit as a main registration,
The terminal device according to claim 1 , wherein the determination unit determines whether or not the gesture input is performed. Said storage control means, said one of the user input is detected within a predetermined time period, different user input from the average input of the user input, the predetermined operation in association with the storage in the storage unit as the temporary registration is allowed, the temporary registered among the user input, what was said temporarily registered predetermined number of times or more, the terminal device according to claim 1 or 2, characterized in Rukoto is stored in the storage unit as a registration. The predetermined operation is started in response to an instruction from the user, and when the instruction is received, the terminal device transmits a request to execute a process corresponding to the predetermined operation to the server.
The control means, when it is determined that the determination means has occurred, performs control so that a request for speeding up the process of transmitting an execution request to the server is transmitted to the server. Item 4. The terminal device according to any one of Items 1 to 3 . A terminal device,
Detecting means for detecting user input;
Storage control means for storing a plurality of user inputs as a user input set in a storage unit when the detection means detects a plurality of user inputs different from the average input of the user within a predetermined period;
Determination means for determining whether any user input set stored in the storage unit has occurred while the terminal device is performing a predetermined operation;
Control means for controlling processing executed by the terminal device for the predetermined operation according to whether the determination means determines that any one of the user input sets has occurred,
The storage control means
A plurality of user inputs detected within the predetermined period are associated with each other as temporary registration and stored in the storage unit,
Among the plurality of provisionally registered user inputs, a plurality of provisionally registered user inputs are stored in the storage unit as the main registration of the user input set more than a predetermined number of times,
The determination means determines whether any of the user input sets registered in the main has occurred;
A terminal device characterized by that. The terminal device according to claim 5 , wherein the predetermined period is an elapsed period after the detection unit first detects a user input different from an average input of the user. The plurality of user inputs includes a plurality of user inputs of the same type or different types,
Said storage control means according to claim 5 or 6, characterized in that to store a gesture input based on a combination of the detected plurality of user input within the predetermined time period in the storage unit as the user input set Terminal device. The average input, the surrounding environment of the terminal device or, in accordance with the band when using the terminal device, any one of claims 3, characterized in that it is defined for each type of user input 7 of The terminal device according to item 1. The predetermined operation is started in response to an instruction from the user, and when the instruction is received, the terminal device executes processing corresponding to the predetermined operation in the terminal device.
Wherein, when said determination means determines to have occurred, one of claims 1 8, characterized in that control to transmit a request for execution of the process corresponding to the predetermined operation to the server The terminal device according to claim 1. The transmission to the execution of the processing request server, the terminal device according to any one of claims 5, characterized in that is based on the use of Web API 9 that the server provides. When the predetermined operation is an operation of manipulating information displayed on the display means,
The control means controls to edit a CSS description included in the information in order to change a display form of the information on the display means when it is determined that the determination means has occurred. The terminal device according to any one of claims 1 to 10 . When the display of a plurality of items is included in the result of executing the predetermined operation,
Wherein when said determination means determines to have occurred, to reduce the number of items to be displayed, or any one of claims 1 to 11, wherein the controller controls to lower the quality of the item The terminal device according to item 1. A control method for a terminal device comprising detection means for detecting user input,
The detected homologous or more of the user input disparate within a predetermined time period after the start of the predetermined operation in the terminal device, wherein the predetermined operation with associated and stored in the storage unit as the temporary registration is the temporary registration wherein among the user input, those which are more than a predetermined number of times the temporary registration, a storage control step of Ru is stored in the storage unit as the registration has,
A determination step of determining whether any user input stored in the storage unit as the main registration has occurred in association with the predetermined operation during the predetermined operation of the terminal device;
And a control step of controlling processing executed by the terminal device for the predetermined operation according to whether it is determined that any one of the user inputs has occurred in the determination step. Control method of the device. A control method for a terminal device comprising detection means for detecting user input,
A storage control step of storing the plurality of user inputs in a storage unit as a user input set when the detection unit detects a plurality of user inputs different from the average input of the user within a predetermined period;
A determination step of determining whether any user input set stored in the storage unit has occurred while the terminal device is performing a predetermined operation;
A control step of controlling processing executed by the terminal device for the predetermined operation according to whether it is determined that any one of the user input sets has occurred in the determination step, and
In the storage control step,
A plurality of user inputs detected within the predetermined period are associated with each other as temporary registration and stored in the storage unit,
Among the plurality of provisionally registered user inputs, a plurality of provisionally registered user inputs are stored in the storage unit as the main registration of the user input set more than a predetermined number of times,
In the determination step, it is determined whether any of the registered user input set has occurred,
A control method for a terminal device. The computer program for functioning a computer as each means of the terminal device of any one of Claim 1 to 12 .

Images