JP2015153325A - information processing apparatus, operation support method and operation support program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To shorten time until detection of a state where a user has trouble with an operation to increase reliability of detection.SOLUTION: A storage unit 3 stores troubled state information 3a indicating a state where it is assumed that a user has trouble with an operation. A detection signal analysis unit 2 analyzes a detection signal detecting the motion and look of the user to detect a user operation state at the time when the user performs an operation. An operation state determination unit 4 collates the user operation state with the troubled state information 3a to determine whether or not the user has trouble with the operation. When the operation state determination unit 4 determines that the user has trouble with the operation, an operation support unit 5 performs operation support for the user.

Description

  The present invention relates to an information processing apparatus, an operation support method, and an operation support program.

  In recent years, cellular phones, portable audio devices, digital cameras, in-vehicle navigations, and the like have been improved in functionality and multifunction, and can be positioned as information processing apparatuses equipped with a microprocessor.

  In such an information processing apparatus, the user needs to perform a predetermined operation when using a desired function. However, some users are familiar with the operation and others are unfamiliar with the operation. In addition, even if the same user uses different functions, there are cases where the user is unfamiliar with the operation, and it is not possible to simply determine whether the user is unfamiliar with the operation. When an unfamiliar operation is performed, the user feels uncomfortable and uncomfortable, such as repeating the same operation without knowing the operation method, or stopping the operation.

  Therefore, there is a monitoring support method for acquiring a user's operation history and collating it with an operation pattern that is likely to be confused with the operation, and determining whether the user is confused with the operation (for example, see Patent Document 1). ).

  In addition, there is a case where a help guidance screen is displayed when it is detected that a user of a copying machine is photographed with a camera and is looking confused (see, for example, Patent Document 2).

JP-A-10-222467 JP 2002-258682 A

  When a user who uses the information processing apparatus is confused about the operation, it is preferable that the operation support is performed as soon as possible. On the other hand, if the operation support is performed when the operation is not confused, the work up to that point is interrupted, which is not preferable.

  However, in the method of collating the operation history with the operation pattern, it is impossible to determine whether or not the operation is confused unless it is recognized that at least the user has performed the operation. For example, in the case of a touch panel operation, even if a user who is confused by the operation touches an area that does not correspond to a key, the operation may not be recognized. Moreover, even if it is recognized as an operation, it cannot be determined whether the operation is simple or unfamiliar with the operation. Therefore, a highly accurate determination can be made only after the user repeatedly performs this operation. For this reason, it takes time until it is detected that the user is troubled by the operation, and the operation support is delayed.

  Further, in the method of simply detecting a confused expression and performing operation support, it is not known whether the user is confused because of the unfamiliar operation. If the expression is confused due to other causes, unnecessary operation support is performed, which may cause discomfort to the user.

  In one aspect, the present invention provides an information processing apparatus capable of shortening the time until a user detects a state of being confused by an operation, increasing the accuracy of detection, and performing appropriate operation support. An object is to provide a support method and an operation support program.

  In order to achieve the above object, an information processing apparatus as shown below is provided. The information processing apparatus includes a storage unit, a detection signal analysis unit, an operation state determination unit, and an operation support unit. The storage unit stores confusion state information indicating a state in which the user is assumed to be confused with the operation. The detection signal analysis unit analyzes a detection signal that detects a user's movement and state, and detects a user operation state when the user is performing an operation. The operation state determination unit compares the user operation state with the puzzle state information and determines whether or not the user is puzzled by the operation. The operation support unit provides operation support to the user when the operation state determination unit determines that the user is confused about the operation.

  According to one aspect, it is possible to shorten the time until detection of a state in which the user is troubled by the operation, increase the accuracy of detection, and perform appropriate operation support.

It is a figure which shows an example of a structure of the information processing apparatus of 1st Embodiment. It is a figure which shows an example of a structure of the mobile telephone of 2nd Embodiment. It is a figure which shows an example of the hardware constitutions of the mobile telephone of 2nd Embodiment. It is a figure which shows an example of a function structure of the mobile telephone of 2nd Embodiment. It is a figure which shows an example of the operation history information of 2nd Embodiment. It is a figure which shows an example of the user facial expression information of 2nd Embodiment. It is a figure which shows an example of the audio | voice keyword information of 2nd Embodiment. It is a flowchart which shows the procedure of the operation unfamiliarization assistance process of 2nd Embodiment. It is a flowchart which shows the procedure of the operation unfamiliarity determination process of 2nd Embodiment. It is a flowchart which shows the procedure of the user support process of 2nd Embodiment. It is a figure which shows an example of a function structure of the mobile telephone of 3rd Embodiment. It is a figure which shows an example of the exclusion item information of 3rd Embodiment. It is a flowchart which shows the procedure of the exclusion operation determination process of 3rd Embodiment.

Hereinafter, the present embodiment will be described with reference to the drawings.
[First Embodiment]
First, the information processing apparatus according to the first embodiment will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of the configuration of the information processing apparatus according to the first embodiment.

The information processing apparatus illustrated in FIG. 1 includes a detection unit 1, a detection signal analysis unit 2, a storage unit 3, an operation state determination unit 4, an operation support unit 5, an operation determination unit 6, and an application control unit 7.
The detection unit 1 includes a touch detection unit 1a, a camera photographing unit 1b, an angular velocity detection unit 1c, and a voice detection unit 1d as means for detecting a user's movement and appearance. Each detection means outputs the detected movement and state of the user as a detection signal. The touch detection unit 1a constitutes a part of the touch panel, recognizes that a user's finger or the like has pressed the touch panel as a touch, the position coordinates of the pressed (hereinafter referred to as touch position), and the pressure of the pressed pressure. The size (hereinafter referred to as touch pressure) is detected. The camera photographing unit 1b photographs the user's face and generates an image signal. The angular velocity detection unit 1c detects a change in the direction of the information processing apparatus and an angular velocity. The voice detector 1d detects the user's voice. Note that the touch detection unit 1a, the camera photographing unit 1b, the angular velocity detection unit 1c, and the voice detection unit 1d in FIG. 1 are examples of the configuration of the detection unit 1, and the detection unit 1 includes a part of the configuration, or It is good also as a structure provided with another detection means.

  The detection signal analysis unit 2 analyzes the detection signal acquired from the detection unit 1 and detects a user operation state when the user is performing an operation. The detection signal analysis unit 2 sets the operation state of the information processing apparatus to “in operation” when the “in operation” condition is satisfied by the detection signal acquired from the detection unit 1. For example, it is possible to set the “operating” condition that a predetermined area is touched by the touch detection unit 1a. Further, it may be when there is a predetermined voice input. “In operation” is held for at least a certain period of time after the condition is satisfied, or while an effective operation continues. The user operation state is a user operation including an operation obtained from a detection signal during the operation by the user, and a facial expression or voice.

  The detection signal acquired from the touch detection part 1a is analyzed, and the touch position and the touch pressure which the user operated the touch panel are detected. Once acquired from the camera photographing unit 1b, the image signal of the user's face is analyzed to detect the type of facial expression of the user. For example, the facial features captured in the image signal are compared with features appearing in each facial expression such as a troubled facial expression and a sad facial expression, and the type of facial expression is specified based on the similarity to each facial expression. The detection signal acquired from the angular velocity detection unit 1c is analyzed, and the magnitude of impact applied to the information processing apparatus is detected. The voice signal acquired from the voice detection unit 1d is analyzed, and the words uttered by the user and the volume of the voice are detected. It is determined whether or not the user is operating by comparing these detected user states with the conditions for satisfying “in operation”. When “in operation”, the detected user state is output to the operation state determination unit 4 as a user operation state. Further, the detected touch position and touch pressure are also output to the operation determination unit 6 as operation information. When a voice operation is performed, the words uttered by the user are output to the operation determination unit 6 as operation information.

  The storage unit 3 stores in advance confusion state information 3 a that is referred to by the operation state determination unit 4. In the puzzle state information 3a, a user operation state that is assumed to be confused by the user and that can be detected by the detection signal analysis unit 2 is registered. The state in which the operation is confused means a state in which the operation is not successful or is in trouble, for example, the user does not know the operation at all, remembers the operation, or remembers the wrong operation. In such a case, for example, the person increases the touch pressure, increases the touch frequency, repeatedly touches a part other than the key, shakes the information processing device, gives an impact, makes the expression steep, sighs, and “don't know” It is known that the behavior and appearance are different from the normal state that is not confused, such as uttering the words such as, or having a loud voice. In the puzzle state information 3a, information indicating a characteristic user operation state at the time of such a puzzle is set. Note that features with large individual differences, such as increased touch pressure and louder voices, may be obtained by previously obtaining a threshold value from a value measured in a normal state and setting it in the trouble state information 3a.

  The operation state determination unit 4 acquires the user operation state detected by the detection signal analysis unit 2, and collates the acquired user operation state with the puzzle state information 3a. When the detected user operation state matches the state indicated in the puzzle state information 3a, it is determined that the user is troubled by the operation. If they do not match, it is determined that the user is not confused with the operation. The determination result is output to the operation support unit 5.

  The operation support unit 5 acquires a determination result as to whether or not the user is confused by the operation from the operation state determination unit 4. When it is detected that the user is confused with the operation, operation guidance is provided to assist the operation. Further, before performing the operation guidance, an operation guidance guidance screen for guiding the user to the operation guidance may be displayed to inquire whether the operation guidance is necessary. For example, a “help” key is displayed, and operation guidance is provided if the user operates it, and operation guidance is not performed if the user does not operate it. The operation guidance outputs an appropriate message based on the operation history so far held by the operation determination unit 6 and information on the currently operating application (hereinafter referred to as an application) held by the application control unit 7. Do.

The operation determination unit 6 holds the operation performed by the user as an operation history based on the operation information acquired from the detection signal analysis unit 2 and outputs an instruction corresponding to the operation to the application control unit 7.
Based on the instruction acquired from the operation determination unit 6, the application control unit 7 performs application control processing such as start and end processing of the instructed application and change of the operation state of the application. The information processing apparatus is equipped with a plurality of applications for realizing a predetermined function, and the user can activate and operate an arbitrary application. The application control unit 7 also holds application information related to the running application.

  As described above, in the information processing apparatus according to the first embodiment, the touch detection unit 1a, the camera photographing unit 1b, the angular velocity detection unit 1c, and the voice detection unit 1d that form the detection unit 1 detect the movement and state of the user. . For example, when the touch pressure or the touch interval of the user who is troubled by the operation is shortened, the touch detection unit 1a detects this. If the user has a confused expression, the camera photographing unit 1b catches this. If the information processing apparatus is shaken, the angular velocity detection unit 1c detects this. And if the word which expresses confusion or a voice becomes loud, the audio | voice detection part 1d will detect. As described above, the detection unit 1 detects an operation or a state assumed to be performed when the user is confused by the operation. The detection signal analysis unit 2 analyzes each detection signal acquired from the detection unit 1 to determine whether or not the user is operating, and if the user is operating, detects the user operation state at that time. . Touch position and pressure from the detection signal of the touch detection unit 1a, type of facial expression of the user from the detection signal of the camera photographing unit 1b, magnitude of impact applied to the information processing device from the detection signal of the angular velocity detection unit 1c, and voice detection unit The words and the volume produced by the user are detected as user operation states from the 1d detection signal. The detected user operation state is output to the operation state determination unit 4. The operation state determination unit 4 collates the detected user operation state with the state assumed to be confused by the user set in the confusion state information 3a stored in the storage unit 3, and the user performs the operation. Determine if you are confused. The operation support unit 5 provides operation support to the user when it is determined that the user is confused about the operation.

  As described above, in the information processing apparatus according to the first embodiment, in the method using the operation history, the determination must be performed based on the operation that occurs even in the normal state of “key operation”. It is determined by detecting a user operation state that is different from the normal state. Thereby, it is possible to quickly detect a state in which the user is troubled by the operation, and to increase the detection accuracy. Therefore, it is possible to perform appropriate operation support at an earlier timing without repeating the operation while the user is confused.

[Second Embodiment]
Next, a case where the information processing apparatus according to the first embodiment is applied to a mobile phone will be described as a second embodiment. FIG. 2 is a diagram illustrating an example of the configuration of the mobile phone according to the second embodiment.

  The mobile phone 100 includes a touch panel compatible display 101, a proximity sensor 102, an illuminance sensor 103, an earpiece 104, an in-camera 105, a one-segment FM transmitter antenna 106, a gyro / acceleration sensor 107, and a mouthpiece 108.

  The touch panel compatible display 101 is a display having a function as a touch panel. As a display, a moving image or a still image is displayed according to the activated application. Moreover, it is one form of the touch detection part 1a which detects the touch position and touch pressure which the user pressed while displaying the key for a touch in a predetermined area | region as a touch panel. The proximity sensor 102 detects that an object has approached, and the illuminance sensor 103 detects ambient brightness. When the earpiece 104 operates as a telephone, it reproduces the other party's voice. The in-camera 105 is a camera that performs shooting on the user side, and is a form of the camera shooting unit 1b. The one-segment FM transmitter antenna 106 is an antenna that receives broadcast signals. The gyro / acceleration sensor 107 is a sensor that detects acceleration in addition to angular velocity, and is a form of the angular velocity detector 1 c that detects the movement of the mobile phone 100, that is, the movement of the user to the mobile phone 100. The mouthpiece 108 is one form of a voice detection unit 1d that inputs a user's voice.

  As described above, the mobile phone 100 includes sensors corresponding to the touch detection unit 1a, the camera photographing unit 1b, the angular velocity detection unit 1c, and the voice detection unit 1d that constitute the detection unit 1 of the first embodiment. The in-camera 105 and the mouthpiece 108 are also positioned as sensors for detecting images and sounds.

  Next, the hardware configuration of the mobile phone 100 will be described with reference to FIG. FIG. 3 is a diagram illustrating an example of a hardware configuration of the mobile phone according to the second embodiment. The same components as those in FIG. 2 are denoted by the same reference numerals, and description thereof is omitted.

  The cellular phone 100 includes a control unit 110, a display 101a, a touch panel 101b, a proximity sensor 102, an illuminance sensor 103, an earpiece 104, an in-camera 105, a one-segment FM transmitter antenna 106, a gyro / acceleration sensor 107, and a mouthpiece 108. Prepare. In FIG. 3, the touch panel compatible display 101 shown in FIG. 2 is divided into a display 101a and a touch panel 101b for convenience.

  The control unit 110 comprehensively controls the mobile phone 100. The control unit 110 is entirely controlled by a CPU (Central Processing Unit) 111. A ROM (Read Only Memory) 112, a RAM (Random Access Memory) 113, a communication interface 114, and an input / output interface 115 are connected to the CPU 111 via a bus 116.

  The ROM 112 retains the stored contents even when the control unit 110 is powered off. The ROM 112 is, for example, a semiconductor storage device such as an EEPROM (Electrically Erasable Programmable Read-Only Memory) or a flash memory, an HDD (Hard Disk Drive), or the like. The ROM 112 is used as an auxiliary storage device for the control unit 110. The ROM 112 stores an OS (Operating System) program, firmware, application programs, and various data.

  The RAM 113 is used as a main storage device of the control unit 110. The RAM 113 temporarily stores at least a part of an OS program, firmware, and application programs to be executed by the CPU 111. The RAM 113 stores various data necessary for processing by the CPU 111. The RAM 113 may include a cache memory separately from a memory used for storing various data.

The communication interface 114 is connected to the network 200 and exchanges information with an external device via the network 200.
The input / output interface 115 is connected to the display 101a, the touch panel 101b, the proximity sensor 102, the illuminance sensor 103, the earpiece 104, the in-camera 105, the one-segment / FM transmitter antenna 106, the gyro / acceleration sensor 107, and the mouthpiece 108. The input signals from these units are taken into the control unit 110 via the bus 116. Moreover, according to CPU111, an output signal is output to each part.

With the hardware configuration described above, the processing functions of the mobile phone 100 of the second embodiment can be realized.
Next, functions of the mobile phone 100 will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of a functional configuration of the mobile phone according to the second embodiment. In FIG. 4, illustration of functional units not related to operation support is omitted.

  The mobile phone 100 includes a detection unit 10, a detection signal analysis unit 20, a storage unit 30, an operation unfamiliarity determination unit 40, and a user support implementation unit 50. The operation unfamiliarity determination unit 40 is one form of the operation state determination unit 4. In the following description, it is assumed that the operation is started when the user performs an effective touch on the touch panel 101b.

  The detection unit 10 includes a touch panel 101b, an in-camera 105, a gyro / acceleration sensor 107, and a mouthpiece 108 in order to detect a user's movement and appearance. The touch panel 101b generates a detection signal corresponding to the touch position and the touch pressure pressed by the user with a finger or the like. The in-camera 105 is a camera that performs shooting on the user side, and generates an image signal obtained by shooting the user's face. The gyro / acceleration sensor 107 generates a detection signal corresponding to the acceleration and angular velocity applied to the mobile phone 100. The mouthpiece 108 detects a user's voice and generates an audio signal. The detection signal of each sensor of the detection unit 10 is output to a corresponding analysis unit of the detection signal analysis unit 20. The detection signal of the touch panel 101b is output to the touch operation analysis unit 21. The image signal of the in-camera 105 is output to the user facial expression analysis unit 22. The detection signal of the gyro / acceleration sensor 107 is output to the impact analysis unit 23. The voice signal from the mouthpiece 108 is output to the voice analysis unit 24. Each unit of the detection unit 10 is activated at a predetermined cycle, for example, when the proximity sensor 102 detects the approach of the user or when the touch panel 101b detects a valid touch. And at least while the operation state of the mobile phone 100 is being operated, the user's movement and state can be detected. In addition, it can also be in the state which can always be detected.

  The detection signal analysis unit 20 analyzes a detection signal detected during operation by the user to detect the user operation state, and outputs the detected user operation state to the operation unfamiliarity determination unit 40. The touch operation analysis unit 21, the user facial expression analysis unit 22, the impact analysis unit 23, and the voice analysis unit 24 each detect a user operation state from a detection signal. The detection targets are different from each other, and at least one sensor detects a user's operation and state, and each unit of the corresponding detection signal analysis unit 20 analyzes the detection signal. The touch operation analysis unit 21 analyzes the detection signal acquired from the touch panel 101b, and detects the touch position and the touch pressure pressed by the user. When the touch position is included in a predetermined area and the touch pressure is a certain value or more, it is determined that the touch is valid. If the current state of the mobile phone 100 is in operation, the operation is continued. If the operation is not in progress, the operation state of the mobile phone 100 is set to the operation state because the operation is started. The detected touch position and touch pressure are output to the user operation determination unit 41 and the touch pressure determination unit 42. The user facial expression analysis unit 22 analyzes the image signal acquired from the in-camera 105 and detects the type of user facial expression. For example, a method based on coordinate changes of feature points of eyebrows, eyes, and mouth, which are facial components, is known for detection of the type of facial expression. In the embodiment, these known methods are used. Note that the user facial expression analysis unit 22 may determine only whether or not the facial expression is assumed based on the facial expression characteristic that is expected to appear in the confusion. The type of the detected facial expression or the determination result of whether or not it is a confused facial expression is output to the user facial expression determination unit 43. The impact analysis unit 23 analyzes the detection signal acquired from the gyro / acceleration sensor 107 and detects the magnitude of impact according to the acceleration or angular velocity applied to the mobile phone 100. The magnitude of the detected impact is output to the impact operation determination unit 44. The voice analysis unit 24 analyzes the voice signal acquired from the mouthpiece 108, and detects the words uttered by the user and the volume of the voice. The words uttered by the user are detected using a well-known voice recognition process. In addition, speech that cannot be recognized as a word such as sigh may be classified as an exclamation, for example. The detected word and volume are output to the voice keyword determination unit 45 and the volume determination unit 46.

  The storage unit 30 stores operation history information 31, touch pressure information 32, user facial expression information 33, impact operation information 34, voice keyword information 35, and volume information 36 in a storage area. Each of these pieces of information is an example of the puzzle state information 3a, and is prepared corresponding to each unit of the operation unfamiliarity determination unit 40. Each will be described.

  The operation history information 31 is information stored in the storage unit 30 corresponding to the user operation determination unit 41, and is referred to in order to detect confusion due to unfamiliar operation from the user operation history. FIG. 5 is a diagram illustrating an example of operation history information according to the second embodiment. In the operation history table 310, an operation that is often performed when the user is confused about the operation is set as an “operation for determining that the user is unfamiliar with the operation”. For example, there is “long-time leaving” in which an operation is interrupted because the next operation is unknown. In addition, operations such as “Repeat operation” that repeats the same operation because the operation is remembered or mistakenly remembered, and “Frequent call of setting screen” that returns frequently to the first setting screen because the operation is unknown are set. ing.

  The touch pressure information 32 is information stored in the storage unit 30 corresponding to the touch pressure determination unit 42, and is referred to in order to detect that the user is confused by the operation from the touch pressure. When the user is confused about the operation, the force to touch the touch panel 1b may become frustrated. In the touch pressure information 32, a threshold value for determining whether or not the user is confused about the operation is registered. In addition, although the magnitude | size of the touch pressure estimated that the user is confused with operation may be preset as a threshold value, it can also be set based on a measured value. For example, the normal touch pressure is measured at the initial setting or at a predetermined timing, and a threshold value for detecting that the user is troubled by the operation is obtained and set from the measured touch pressure. Although the detection is performed using the touch pressure here, not only the touch pressure but also a touch position when the user is confused about the operation, a touch interval, and the like may be set.

  The user facial expression information 33 is information stored in the storage unit 30 corresponding to the user facial expression determination unit 43, and is referred to in order to detect a user's confused facial expression. FIG. 6 is a diagram illustrating an example of user facial expression information according to the second embodiment. In the user facial expression table 330, the type of facial expression that is often used when the user is confused by the inexperienced operation is set as “an expression that is determined to be inexperienced in the operation”. In the example of FIG. 6, it is set that the user becomes an expression of “anger” or “sadness” in a state where the operation is confused.

  The impact operation information 34 is information stored in the storage unit 30 corresponding to the impact operation determination unit 44, and is setting information that is referred to in order to detect the magnitude of the impact applied to the mobile phone 100 by the user. . When the user is confused by the unfamiliar operation, the mobile phone 100 may be frustrated and hit or hit, and at that time, the mobile phone 100 moves more than the normal use state. In the impact operation information 34, when the mobile phone 100 is moved with an impact, a threshold value used for determining whether or not the impact is caused by the user being confused by the operation is set. Note that the magnitude of impact (angular velocity and acceleration) that is estimated to be confused by the user may be set in advance as a threshold value, but may be set based on measured values. For example, the angular velocity and acceleration detected at the time of initial setting or normal use at a predetermined timing are measured, and a threshold value for detecting that the user is in a confused state is obtained from the measured value and set.

  The voice keyword information 35 is information stored in the storage unit 30 corresponding to the voice keyword determination unit 45, and is referred to in order to detect words when the user is confused about the operation. FIG. 7 is a diagram illustrating an example of voice keyword information according to the second embodiment. In the speech keyword table 350, words often spoken when the user is confused about the operation are set as “speech keywords determined to be unfamiliar with the operation”. For example, when the operation is not known or when the mobile phone 100 does not operate as expected, it is set to detect a voice keyword such as “I don't understand”, “Unusable”, “Broken”.

  The volume information 36 is information stored in the storage unit 30 corresponding to the volume determination unit 46, and is referred to in order to detect that the user is confused and makes a loud voice. When the user is confused by the operation, the volume of the voice may be higher than the normal state by making a strange voice or yelling. In the volume information 36, a threshold value used for determining whether or not the user is confused about the operation is set. In addition to setting the threshold in advance, for example, the user's volume detected during normal use at the initial setting or at a predetermined timing is measured, and the user is in a confused state from the measured volume value. A threshold value for this may be obtained and set.

Note that threshold values such as touch pressure, angular velocity, acceleration, and volume may not be stored in the storage unit 30, but may be incorporated in the program in advance.
Returning to FIG.

  The operation unfamiliarity determination unit 40 includes a user operation determination unit 41, a touch pressure determination unit 42, a user facial expression determination unit 43, an impact operation determination unit 44, a voice keyword determination unit 45, and a volume determination unit 46. Based on the user operation state detected by the detection signal analysis unit 20 and the trouble state information 3a stored in the storage unit 30, it is determined whether or not the user is troubled by an unfamiliar operation. The determination result is output to the user support execution unit 50.

  The user operation determination unit 41 determines whether the operation is an effective operation based on the touch position and the touch pressure detected by the touch operation analysis unit 21. If the operation is valid, “in operation” is set, an operation corresponding to the touch position is specified, and the operation is stored in the operation history. The obtained operation history is collated with the operation history information 31 stored in the storage unit 30. If there is a match, it is determined that the user is confused by an unfamiliar operation. If there is no match, it is determined that the user is not confused. Further, when a series of operation inputs necessary for the operation of a predetermined function is completed, “in operation” is canceled, and an application that realizes the corresponding function is executed. In addition, in an apparatus in which operation buttons are arranged at locations other than the touch panel 101b, it is regarded as an effective operation when such an operation button is operated, and is “in operation”.

  Note that, unlike the other determination units of the operation unfamiliarity determination unit 40, the user operation determination unit 41 cannot detect that the user is confused by the operation until the user repeats the operation. However, there may be a case where the user does not perform an operation or a state that is detected by another determination unit that is troubled by the operation. In such a case, the mobile phone 100 can detect that the user is confused by the operation by the user operation determination unit 41.

  If it is “in operation”, the touch pressure determination unit 42 compares the touch position and the touch pressure detected by the touch operation analysis unit 21 with the threshold value of the touch pressure information 32 stored in the storage unit 30. If the detected touch pressure exceeds the threshold value, it is determined that the user is confused by an unfamiliar operation. If the threshold is not exceeded, it is determined that the user is not confused. In addition to the threshold value of the touch pressure, when the threshold value of the touch position or the touch interval is set in the touch pressure information 32 and the touch interval is obtained from the touch operation analysis unit 21, this can also be used for the determination. For example, when the touch position is not the operation key display area or when the touch interval is short, it is assumed that the user is frustrated, and this state is used for determination of confusion. As described above, when a touch operation different from the normal operation is performed, such as when the user is troubled by the operation and touches more strongly than the normal state, the touch pressure determination unit 42 can detect that the user is troubled by the operation.

  If it is “in operation”, the user facial expression determination unit 43 collates the user facial expression detected by the user facial expression analysis unit 22 with the setting of the user facial expression information 33 stored in the storage unit 30. If there is a matching facial expression, it is determined that the user is confused by an unfamiliar operation. If there is no matching facial expression, it is determined that the user is not confused. Note that when the user facial expression analysis unit 22 detects whether or not the facial expression is confused by the user, the result may be used without matching. In this way, when the user's facial expression becomes confused, such as when the user's facial expression becomes steep due to the operation, the user facial expression determination unit 43 can detect that the user is confused by the operation.

  If the impact operation determination unit 44 is “in operation”, the impact operation stored in the storage unit 30 is the acceleration and angular velocity when the mobile phone 100 detected by the impact analysis unit 23 is moved by the force applied from the outside. Compare with the threshold value of information 34. If the detected acceleration or angular velocity exceeds the threshold, it is determined that the user is confused by an unfamiliar operation. If the threshold is not exceeded, it is determined that the user is not confused. In this way, when a user who is troubled by the operation is frustrated and hits or shakes the mobile phone 100 and an impact that does not occur in the normal state is applied to the mobile phone 100, the impact operation determination unit 44 It is possible to detect that the user is confused with the operation.

  If the speech keyword determination unit 45 is “in operation”, the speech utterance detected by the speech analysis unit 24 is collated with the speech keyword set in the speech keyword information 35 stored in the storage unit 30. If the detected keyword of the user includes a voice keyword, it is determined that the user is confused by an unfamiliar operation. If the voice keyword is not included, it is determined that the user is not confused. As described above, when the user who is troubled by the operation utters a word indicating the trouble, the voice keyword determination unit 45 can detect that the user is troubled by the operation.

  If the volume determination unit 46 is “in operation”, the volume of the user's voice detected by the voice analysis unit 24 is compared with the threshold value of the volume information 36 stored in the storage unit 30. If the detected volume exceeds the threshold, it is determined that the user is confused by an unfamiliar operation. If the threshold is not exceeded, it is determined that the user is not confused. As described above, when the user who is troubled by the operation is frustrated and makes a louder voice than the normal state, the sound volume determination unit 46 can detect that the user is troubled by the operation.

  The user support execution unit 50 obtains a determination result as to whether or not the user is confused by an unfamiliar operation from the operation unfamiliarity determination unit 40, and supports the user's operation when it is determined that the user is confused. Implement user support processing. In the user support process, first, the user is inquired whether support is necessary. For example, a “help” key is displayed on a part of the screen, and a message is issued to operate “help” if support is necessary. The message may be displayed on the screen or output by voice. When the user operates “Help” and requests support, a support screen is displayed. In this way, by making an inquiry about the necessity of support before the start of the support process, it is possible to further reduce the user's discomfort when an unnecessary support screen is displayed.

  In the mobile phone 100 described above, each sensor of the detection unit 10 captures an operation and a state different from the normal time observed when the user who is operating is troubled by the operation. The detection signal analysis unit 20 analyzes the detection signal of the detection unit 10 and detects a user operation state including an operation and a state when the user is performing an operation. The operation unfamiliarity determination unit 40 collates the user operation state detected by the detection signal analysis unit 20 with the information set in the state in which the user unfamiliar with the operation stored in the storage unit 30 is confused, and the user is unfamiliar with it. Whether or not he / she is confused by the correct operation. The mobile phone 100 can detect a state in which the user is confused by the operation based on the user's touch pressure, facial expression, spoken words, loudness of voice, and the magnitude of impact applied to the device. When it is determined that the user is confused by an unfamiliar operation, the user support execution unit 50 presents a message inquiring whether support is necessary, and provides support when requested.

Next, the unfamiliar operation support process executed by the mobile phone 100 will be described with reference to FIG. FIG. 8 is a flowchart illustrating a procedure of operation unfamiliarity support processing according to the second embodiment.
The unfamiliar operation support process is started when the touch panel 101b, the gyro / acceleration sensor 107, and the mouthpiece 108 constituting the detection unit 10 detect the user's operation, or when the in-camera 105 captures the user's face. .

  [Step S11] The detection signal analysis unit 20 analyzes the detection signal detected by the detection unit 10 and detects a user operation state. The touch operation analysis unit 21 analyzes a detection signal of the touch panel 101b and detects at least a touch position and a touch pressure. The user facial expression analysis unit 22 analyzes the image signal of the in-camera 105 and detects the user's facial expression. The impact analysis unit 23 analyzes the detection signal of the gyro / acceleration sensor 107 and detects the magnitude of the impact applied to the mobile phone 100. The voice analysis unit 24 analyzes the voice signal of the mouthpiece 108 and detects the words uttered by the user and the volume of the voice. Analysis processing in each analysis unit is executed when a corresponding detection signal is obtained from the detection unit 10.

  [Step S12] The operation unfamiliarity determination unit 40 determines whether or not the user operation state analyzed by the detection signal analysis unit 20 is in a state of being confused with the operation. The operation unfamiliarity determination process will be described later with reference to FIG.

  [Step S13] The operation unfamiliarity determination unit 40 determines whether or not the user is confused by the operation due to the operation unfamiliarity. When a state in which the user is confused by an unfamiliar operation is detected, the process proceeds to step S14. When a state in which the user is confused by an unfamiliar operation is not detected, the process ends.

  [Step S14] When the user's unfamiliarity with the operation is detected, the user support execution unit 50 executes a process for supporting the user's operation. The user support process will be described later with reference to FIG.

  Next, the operation unfamiliarity determination process will be described with reference to FIG. FIG. 9 is a flowchart illustrating a procedure of an operation unfamiliarity determination process according to the second embodiment. In the unfamiliar operation support process, the detection signal is analyzed and activated after the user operation state is detected.

  [Step S121] The user operation determination unit 41 acquires a user's operation history, compares the obtained operation history with the operation history information 31, and detects an operation corresponding to a state in which the user is confused about the operation. It is determined whether or not. If the operation is detected, the process proceeds to step S127, and if not detected, the process proceeds to step S122.

  [Step S122] The touch pressure determination unit 42 compares the detected touch pressure of the user with the threshold value of the touch pressure information 32, and determines whether the touch pressure is a corresponding touch pressure exceeding the threshold value. If the touch pressure exceeds the threshold value, the process proceeds to step S127. If the touch pressure does not exceed the threshold value, the process proceeds to step S123.

  [Step S123] The user facial expression determination unit 43 compares the detected facial expression of the user with the user facial expression information 33 to determine whether or not a confused facial expression is detected. If a confused expression is detected, the process proceeds to step S127, and if not detected, the process proceeds to step S124.

  [Step S124] The impact operation determination unit 44 compares the detected magnitude (acceleration and angular velocity) of the impact received by the mobile phone 100 with the threshold value of the impact operation information 34, and the magnitude of the impact exceeds the threshold value. It is determined whether or not. If the magnitude of the impact exceeds the threshold value, the process proceeds to step S127, and if not, the process proceeds to step S125.

  [Step S125] The speech keyword determination unit 45 collates the detected words uttered by the user with the speech keyword information 35 and issues a speech keyword that is assumed to be spoken when the user is confused about the operation. It is determined whether or not. If the user has issued a voice keyword, the process proceeds to step S127, and if not, the process proceeds to step S126.

  [Step S126] The volume determination unit 46 compares the detected volume of the user's voice with the threshold value of the volume information 36, and determines whether or not the user has produced a loud voice. If the volume exceeds the threshold, the process proceeds to step S127. If the volume does not exceed the threshold, the process proceeds to step S128.

  [Step S127] Since at least one operation or state assumed to be confused by an unfamiliar operation is detected, it is determined that the user is confused by an unfamiliar operation.

  [Step S128] Since no operation or state assumed to be confused by an unfamiliar operation is detected, it is determined that the user is not confused by an unfamiliar operation.

  In the operation unfamiliarity determination process described above, the determination process is sequentially performed for each detection signal, and when an operation unfamiliarity is detected in any of the determination processes, the subsequent determination process is not performed. However, after all the determination processing for each detection signal is performed, the determination may be performed by calculating the logical sum of the results obtained by the respective determination processing.

  Next, user support processing will be described with reference to FIG. FIG. 10 is a flowchart illustrating a procedure of user support processing according to the second embodiment. In the operation unfamiliarity support process, it is activated when user's operation unfamiliarity is detected.

  [Step S141] The user support execution unit 50 performs a help guidance notification to the user and inquires whether the user needs to support the operation. For example, a start button for starting a help process that supports the operation and a message for inquiring whether support is necessary are displayed on the display screen.

  [Step S142] The user support execution unit 50 determines whether the user who has received the help guidance notification has requested help. For example, it is determined whether or not the start button has been pressed. If help is requested, the process proceeds to step S143. If help is not requested, the help guidance notification is deleted, and the process ends.

  [Step S143] The user support implementation unit 50 provides support information because the user requested help. For example, the display screen is switched to a support display screen and the support information is displayed.

  According to the operation support process of the mobile phone 100 as described above, the user support process is started when an operation or state assumed to be confused by the user is detected. Since these operations and states are different from the operations and states in the normal state, it is possible to detect the troubled state due to the user's unfamiliar operation with high accuracy. Moreover, since the user in a confused state can perform user support before repeatedly performing an operation, user support processing can be performed more quickly.

[Third Embodiment]
Next, as a third embodiment, a configuration in which a target operation state determination unit is provided in the configuration of the second embodiment, and the accuracy of detection of a state in which the user is troubled by the operation can be further increased. explain.

  The mobile phone 100 is remarkably multifunctional and highly functional, and has a camera function, a moving image playback function, a browser function, a navigation function, and the like in addition to the original call function. In such a multifunctional mobile phone 100, depending on the application being operated, the user's operation and state that the user is assumed to be confused with the operation are detected even if the operation is not confused. There are cases. In the third embodiment, by excluding such a case, the detection accuracy of a state in which the user is confused by the operation can be increased.

  The configuration of the third embodiment will be described with reference to FIG. FIG. 11 is a diagram illustrating an example of a functional configuration of the mobile phone according to the third embodiment. The same components as those in FIG. 4 are denoted by the same reference numerals, and description thereof is omitted.

  The mobile phone 300 includes a detection unit 10, a detection signal analysis unit 20, an operation unfamiliarity determination unit 40, a user support execution unit 50, and a target operation state determination unit 80. The detection unit 10, the detection signal analysis unit 20, the operation unfamiliarity determination unit 40, and the user support execution unit 50 are the same as the mobile phone 100 according to the second embodiment. In FIG. 11, the storage unit 30 shown in FIG. 4 is omitted.

  A mobile phone 300 according to the third embodiment has a configuration in which a target operation state determination unit 80 is added to the mobile phone 100 according to the second embodiment. Therefore, the target operation state determination unit 80 will be described below.

  The target operation state determination unit 80 includes an operation application determination unit 81, an excluded item information storage unit 82, and an exclusion operation state determination unit 83, and is disposed between the detection signal analysis unit 20 and the operation unfamiliarity determination unit 40. And it excludes what was specified from the user operation state which detection signal analysis part 20 detected, and outputs to operation unfamiliarity judgment part 40 which judges whether a user is perplexed by operation. The operation application determination unit 81 determines an application that is operating when the user is performing an operation. The excluded item information storage unit 82 is a storage unit that stores an excluded item table 82a.

  Here, the excluded item table 82a is information in which items to be excluded from the reference items when detecting a state in which the user is confused by the operation are set. FIG. 12 is a diagram illustrating an example of exclusion item information according to the third embodiment. In the excluded item table 82a, items to be excluded from the determination target whether or not the user is confused by the operation by the operation unfamiliarity determination unit 40 are associated with the action application as “items excluded from the operation unfamiliarity determination”. Is set. For example, when the operation application is a camera, if the detected operation is an operation on the finder screen, it is set to be excluded from the determination target. Further, when the operation application is a map, it is set to exclude the operation from the determination target when the operation is performed on the map screen. In addition to the operation, when the motion application is a video playback, it is set to exclude the user's facial expression from the determination target. This is because, depending on the content of the moving image to be viewed, it is assumed that the user's face becomes steep even though the operation is not confused.

  Returning to FIG. The exclusion operation state determination unit 83 acquires a user operation state (user's operation and state) from the detection signal analysis unit 20 and also acquires information on the operation application being operated from the operation application determination unit 81. Then, the acquired operation application and user operation state are checked against the excluded item table 82a stored in the excluded item information storage unit 82, and the presence or absence of a matching item is determined. When there is a matching item, the corresponding user operation state is deleted. To the operation unfamiliarity determination unit 40, the user operation state in which the excluded item is removed by the exclusion operation state determination unit 83 is output.

  Thus, in the mobile phone 300, the target operation state determination unit 80 excludes the exclusion items set in the exclusion item table 82a from the user operation states detected by the detection signal analysis unit 20, and only the remaining user information. Is output to the operation unfamiliarity determination unit 40. The operation unfamiliarity determination unit 40 determines whether the user is perplexed by the operation based on the user operation state excluding the items excluded by the target operation state determination unit 80. For example, even if a user who has performed an operation such as adjusting the volume while watching a video has a steep expression depending on the content of the video being viewed, the steep expression detected by the detection signal analysis unit 20 It is excluded from the reference items of the determination process by the state determination unit 80. Since the operation unfamiliarity determination unit 40 does not refer to the user's facial expression at the time of determination, the user support process is not performed in such a case. Thereby, it becomes possible to detect more accurately that the user is confused with the operation.

  Next, the exclusion operation determination process will be described with reference to FIG. The exclusion operation determination process is performed after the detection signal analysis process (step S11) is performed and before the operation unfamiliarity determination process (step S12) in the operation unfamiliarity support process illustrated in FIG. . FIG. 13 is a flowchart illustrating a procedure of exclusion operation determination processing according to the third embodiment. The exclusion operation determination process is started after the user operation state is detected by the detection signal analysis process.

  [Step S181] The operation application determination unit 81 acquires application information indicating which application is currently used, and determines whether or not an exclusion item is set for the operation application. The determination is performed by comparing the operation application with the “operation application” in the exclusion item table 82 a stored in the exclusion item information storage unit 82. If an exclusion item is set in the operation application, the process proceeds to step S182. If an exclusion item is not set, the process ends.

  [Step S182] The exclusion operation state determination unit 83 is set to the user operation state acquired from the detection signal analysis unit 20 and the corresponding “operation application” in the exclusion item table 82a stored in the exclusion item information storage unit 82. Compared with “items excluded from operation unfamiliarity determination”. If the user operation state detected by the detection signal analysis unit 20 matches the “item excluded from operation unfamiliarity determination”, the process proceeds to step S183, and if not, the process proceeds to step S184.

  [Step S183] Since the detected user operation state matches the “item excluded from the operation unfamiliarity determination”, the excluded operation state determination unit 83 determines the detected user operation state from the target of the operation unfamiliarity determination process. exclude.

  [Step S184] The excluded operation state determination unit 83 determines whether or not all detected “user operation states” are collated with “items excluded from the operation unfamiliarity determination”. If collation has not been completed for all detected “user operation states”, the process returns to step S182, and if collation has been completed, the process is terminated.

  In this manner, if the detected “user operation state” is an item excluded from the operation unfamiliarity determination target set for the application used by the excluded operation determination process, the “user operation” The “state” is excluded from the target of the operation unfamiliarity determination process. Thereby, the determination in the operation unfamiliarity determination process can be performed more accurately.

  The above processing functions can be realized by a computer. In that case, a program describing the processing contents of the functions that the information processing apparatus should have is provided. By executing the program on a computer, the above processing functions are realized on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of the computer-readable recording medium include a magnetic storage device, an optical disk, a magneto-optical recording medium, and a semiconductor memory. Examples of the magnetic storage device include a hard disk device (HDD), a flexible disk (FD), and a magnetic tape. Optical disks include DVD (Digital Versatile Disc), DVD-RAM, CD-ROM (Compact Disc Read Only Memory), CD-R (Recordable) / RW (ReWritable), and the like. Magneto-optical recording media include MO (Magneto-Optical disk).

  When distributing the program, for example, portable recording media such as a DVD and a CD-ROM in which the program is recorded are sold. It is also possible to store the program in a storage device of a server computer and transfer the program from the server computer to another computer via a network.

  The computer that executes the program stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in its own storage device. Then, the computer reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from the portable recording medium and execute processing according to the program. In addition, each time a program is transferred from a server computer connected via a network, the computer can sequentially execute processing according to the received program.

  In addition, at least a part of the above processing functions can be realized by an electronic circuit such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or a PLD (Programmable Logic Device).

DESCRIPTION OF SYMBOLS 1 Detection part 1a Touch detection part 1b Camera imaging part 1c Angular velocity detection part 1d Audio | voice detection part 2 Detection signal analysis part 3 Memory | storage part 3a Confusion state information 4 Operation state determination part 5 Operation support part 6 Operation determination part 7 Application control part 10 Detection Unit 20 detection signal analysis unit 30 storage unit 40 operation unfamiliarity determination unit 50 user support execution unit 80 target operation state determination unit

Claims (10)

A storage unit for storing confusion state information indicating a state assumed to be confused by the user;
A detection signal analysis unit that detects a user operation state when the user is performing an operation by analyzing a detection signal that detects the user's movement and appearance;
An operation state determination unit that compares the user operation state with the puzzle state information and determines whether or not the user is puzzled by the operation,
When the operation state determination unit determines that the user is confused by the operation, an operation support unit that provides operation support to the user;
An information processing apparatus. The detection signal is a signal detected by the user touching the touch panel, and the detection signal analysis unit detects the user operation state as the touch pressure of the user,
The operation state determination unit compares the detected touch pressure of the user with a threshold value of the touch pressure that is assumed to be confused by the user indicated by the confusion state information. When the touch pressure exceeds the threshold of the touch pressure indicated by the puzzle state information, it is determined that the user is troubled by the operation.
The information processing apparatus according to claim 1. The detection signal is a signal obtained by detecting at least one of an angular velocity and an acceleration when the information processing apparatus moves, and the detection signal analysis unit adds the user operation state to the information processing apparatus. Detected as the magnitude of the impact
The operation state determination unit compares the detected magnitude of the impact with a threshold value of the magnitude of the impact that is assumed that the user is perplexed by the operation indicated by the confusion state information, and detects the detected information. When the magnitude of impact applied to the processing device exceeds a threshold of magnitude of impact indicated by the perplexed state information, it is determined that the user is perplexed with the operation,
The information processing apparatus according to claim 1. The detection signal is a voice signal input from a user's voice, and the detection signal analysis unit detects the user operation state as a word uttered by the user,
The operation state determination unit collates the detected user's words with a speech keyword assumed to be issued when the user is confused in the operation indicated by the confusion state information, and detects the user's detected When the speech keyword indicated by the puzzle state information is included in a word, it is determined that the user is puzzled in operation.
The information processing apparatus according to claim 1. The detection signal is a voice signal inputted with a user's voice, and the detection signal analysis unit detects the user operation state as a volume of the user's voice,
The operation state determination unit compares the detected volume of the voice of the user with a threshold of the volume of the voice that is assumed to be confused by the user indicated by the puzzle state information. When the volume of the voice of the user exceeds the threshold of the volume of the voice indicated by the puzzle state information, it is determined that the user is confused with the operation.
The information processing apparatus according to claim 1. The detection signal is an image signal obtained by imaging a user's face, and the detection signal analysis unit detects the user operation state as a type of facial expression of the user,
The operation state determination unit collates and detects the detected facial expression type of the user and the assumed facial expression type that the user is confused about the operation indicated by the confusing state information. When the type of facial expression of the user is included in the type of facial expression indicated by the puzzle state information, it is determined that the user is confused with the operation,
The information processing apparatus according to claim 1. The user operation state detected by the detection signal analysis unit with reference to exclusion item information shown in association with an application using the user operation state item to be excluded from the determination processing target in the operation state determination unit Has a target operation state determination unit that excludes the user operation state from the target of the determination process when it matches the exclusion item corresponding to the running application,
The operation state determination unit compares the user operation state remaining as a target of the determination process in the target operation state determination unit with the confusion state information, and determines whether or not the user is troubled by the operation.
The information processing apparatus according to any one of claims 1 to 6. When it is determined that the user is confused by the operation, the operation support unit displays an operation guide guidance screen that guides the user to the operation guide, and the operation guide is displayed from the user on the operation guide guide screen. Start providing the operation guidance when instructed to provide
The information processing apparatus according to any one of claims 1 to 7. Computer
Analyzing the detection signal that detects the user's movement and appearance to detect the user operation state when the user is operating,
It is determined whether or not the user is confused with the operation by comparing the user operation state with the confused state information indicating the state assumed to be confused with the operation stored in the storage unit. And
When it is determined that the user is confused by the operation, operation support is provided to the user.
Operation support method. On the computer,
Analyzing the detection signal that detects the user's movement and appearance to detect the user operation state when the user is operating,
It is determined whether or not the user is confused with the operation by comparing the user operation state with the confused state information indicating the state assumed to be confused with the operation stored in the storage unit. And
When it is determined that the user is confused by the operation, operation support is provided to the user.
An operation support program that executes processing.

Images