JP5343876B2 - Electronic device, electronic device control method, and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an electronic apparatus whose operation method is switched according to the condition of a user, a method of controlling the electronic apparatus, and a computer program. <P>SOLUTION: An operation recognition module 3 recognizes an operation performed by the user and determines the continuity of the operation based on information input from either an input module 1 or an information collection module 2. A state recognition module 4 recognizes the behavior state of the user based on the information collected by the information collection module 2. A rule control module 5 controls the operation recognized by the operation recognition module 3 and the timing of application of operation rules corresponding to the behavior state of the user recognized by the state recognition module 4, based on the continuity of the operation determined by the operation recognition module 3. <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to an electronic device that switches an operation method according to a user's state, a method for controlling the electronic device, and a computer program.

  In recent years, there is a terminal device in which a motion sensor such as an acceleration sensor is mounted on a portable terminal device and an input operation by a gesture is possible. In such a terminal device, the recognition accuracy varies depending on the user's action state (for example, sitting in the office, walking toward the station, riding on a train, etc.). May decrease.

For example, when operating the terminal device while the user is sitting, the operation can be performed while holding the terminal device firmly, and since the influence of the shaking that the terminal device receives from the user is small, the gesture recognition accuracy is high.
On the other hand, when the terminal device is operated while the user is standing, the influence of shaking by the user is increased, and the gesture recognition accuracy is reduced. In addition, when the user performs an action such as moving or taking a seat while holding the terminal device, there is a possibility that the shaking generated during the action may be erroneously recognized as a gesture. In these cases, it is necessary to cancel the operation accepted due to misrecognition, and the operation is troublesome.

  As a technique for improving deterioration in operability due to a change in user behavior, a user's state is detected using a plurality of sensors, and when the user's state changes, the terminal according to the changed user's state A technique for controlling the apparatus is disclosed (for example, see Patent Document 1).

JP 2001-100870 A

  In the technique disclosed in Patent Literature 1, when it is determined that a change has occurred in the user's state, the control according to the state after the change is immediately applied, and thus the state of the device before the change is taken over. I can't. For this reason, in the case where a change in behavior is detected despite the operation being continued, such as when a user who is operating the terminal device while sitting is standing up, in Patent Document 1, after a state change (a standing state) Therefore, there is a problem that an operation performed before the state change (sitting state) may be discarded or stopped.

  In order to solve the above-described problem, the present application provides an electronic device, a control method for the electronic device, and a computer program that can take over the previous device state even when a change occurs in the user's behavior state. With the goal.

  The electronic device disclosed in the present application includes an accepting unit that accepts an operation by a user, an information collecting unit that collects information according to the user's state, and an operation content that is accepted by the accepting unit And a storage unit that stores an operation rule indicating a relationship between processing contents to be executed and a first determination unit that determines whether or not a user's state has changed based on information collected by the information collection unit, A determination result by the first and second determination units indicating the application timing of the second determination unit that determines the continuity of the operation received by the reception unit and the operation rule applied when the operation by the user is received by the reception unit. And when the user accepts an operation by the receiving unit, it should be executed based on the operation rule applied according to the application timing controlled by the control unit. A processing content determination unit that determines the physical contents, characterized by comprising an execution unit for executing processing according to the processing contents the processing content determination unit has determined.

  According to the present application, it is possible to estimate the continuation state of the operation from the user's operation, and to take over the previous operation rule when the operation is continued when the user's action state is changed. For this reason, even if a user's action state changes, operation by the same operation system is attained when operation by a user is continuing. In addition, by considering the application used by the user and the user's action state before and after the transition, it is possible to accurately estimate the continuation state of the user's operation and realize a better operating environment.

It is a schematic diagram which shows the internal structure of the terminal device which concerns on this Embodiment. It is a conceptual diagram which shows an example of operation rule DB. It is a flowchart explaining the procedure of the process which an operation recognition module performs. It is a flowchart explaining the procedure of the process which a state recognition module performs. It is a flowchart which shows the procedure of the process which a rule control module performs. It is a flowchart which shows the procedure of the process which a rule control module performs. It is a flowchart which shows the procedure of the process which an application control module performs. It is a schematic diagram which shows an example of operation history. 10 is a flowchart showing the contents of processing executed by the operation recognition module in the second embodiment. It is a conceptual diagram which shows an example of an operation space | interval table. 10 is a flowchart showing the contents of processing executed by an operation recognition module in the third embodiment. It is a conceptual diagram which shows an example of the operation kind mapping table. It is a conceptual diagram which shows an example of an operation space | interval table. 14 is a flowchart showing the contents of processing executed by an operation recognition module in the fourth embodiment. It is a conceptual diagram which shows an example of an action state memory | storage table. It is a conceptual diagram which shows an example of an operation space | interval table. 10 is a flowchart showing the contents of processing executed by an operation recognition module in the fifth embodiment. 10 is a flowchart showing the contents of processing executed by an operation recognition module in the fifth embodiment.

  Hereinafter, embodiments in which the electronic device of the present application is applied to a portable terminal device having functions such as a call function, a web browsing function, a music playback function, a calendar display function, and a document display function will be specifically described with reference to the drawings. To do.

Embodiment 1 FIG.
FIG. 1 is a schematic diagram showing an internal configuration of a terminal device according to the present embodiment. The terminal device according to the present embodiment includes an input module 1, an information collection module 2, an operation recognition module 3, a state recognition module 4, a rule control module 5, an operation rule database 6, an application control module 7, and an application group 8.
Further, the terminal device includes clock means (not shown), and each module can refer to the time output from the clock means.

The input module 1 includes at least one input unit such as a switch, a button, and a keyboard, and functions as a reception unit that receives an operation by a user. Information input by the input module 1 is sent to the operation recognition module 3.
The information collection module 2 includes detection means such as an acceleration sensor, and functions as an information collection unit for collecting information according to the user's behavior state. Information collected by the information collection module 2 is sent to the operation recognition module 3 and the state recognition module 4.

  The operation recognition module 3 recognizes an operation performed on the terminal device by the user based on information notified from the input module 1 or the information collection module 2. Specifically, based on information transmitted from the input module 1, it is recognized which button (switch, keyboard) has been pressed. In the following description, an operation recognized based on information notified from the input module 1 is referred to as a button operation.

  Further, the operation recognition module 3 performs an operation accompanied by the movement of the terminal device itself, such as an operation of tilting the terminal device, an operation of shaking the terminal device, or an operation of hitting the terminal device based on the information transmitted from the information collection module 2. recognize. For example, it is possible to recognize operations such as an operation of tilting the terminal device, an operation of shaking the terminal device, and an operation of hitting the terminal device by analyzing an acceleration pattern such as a time change of acceleration and an acceleration direction. A known method can be adopted as a method of analyzing an acceleration pattern and recognizing an operation. In the following description, an operation recognized based on information notified from the information collection module 2 is referred to as a gesture operation.

  The operation recognition module 3 notifies the rule control module 5 of the operation name of the recognized button operation or gesture operation and information on the operated time.

  The state recognition module 4 recognizes the user's action state based on the information sent from the information collection module 2. Specifically, whether the user is sitting or standing is recognized by analyzing the acceleration pattern. Similar to the recognition of the gesture operation, the user's action state can be recognized using a known method.

  The operation rule database 6 (hereinafter referred to as the operation rule DB 6) defines associations between operations recognized by the operation recognition module 3 (button operations and gesture operations) and processes to be executed by the terminal device when an operation is received. This is a database in which operation rules to be stored are accumulated. The operation rule is defined for each user action state.

  FIG. 2 is a conceptual diagram showing an example of the operation rule DB 6. FIG. 2A shows the operation rules when the user is sitting, and the association between each gesture operation and the processing content executed by each application is defined. For example, when the user is sitting and performing a gesture operation of “tilting” the terminal device, the browser performs scroll processing, the music app performs song feed, and the calendar app performs month feed. Is shown. The same applies to other gesture operations, and processing contents to be executed are defined for each application.

On the other hand, FIG. 2B shows an operation rule in a state where the user is standing, and the association between each gesture operation and the processing content executed by each application is defined. In the example of FIG. 2B, it is defined that the gesture operation itself is invalid when the user is standing.
Although FIG. 2 shows the operation rules for the gesture operation, the operation rule DB 6 prescribes the processing content to be executed for each application for the button operation. However, regarding the button operation, it is not necessary to prepare an operation rule for each user behavior state, and the same operation rule may be applied regardless of the user behavior state.

  In accordance with such an operation rule DB 6, the rule control module 5 determines the processing content to be executed. For this reason, the rule control module 5 acquires the operation rule according to an operation condition or a user's action state from operation rule DB6, and determines the processing content which should be performed by applying the acquired operation rule. When applying the operation rule, the rule control module 5 performs a search process based on the operation name notified from the operation recognition module 3 and the information of the currently running application, and acquires the processing content information as the search result. . The rule control module 5 notifies the application control module 7 of information on the determined processing content.

  Further, the rule control module 5 also performs processing for controlling the application rule application timing. As described above, the rule control module 5 is configured to acquire an operation rule corresponding to the user's behavior state, but when the user's behavior state changes, if any operation by the user is ongoing, Delay the user's operation until it is interrupted without immediately applying the newly acquired operation rule. On the other hand, when the user's action state changes, if the user's operation is not continuing, the newly acquired operation rule is immediately applied.

  The application control module 7 executes processing in the corresponding application based on the processing content information notified from the rule control module 5. The application group 8 stored in the terminal device includes a browser application, a music application, and a calendar application for realizing the functions of the terminal device, and a system application for realizing the overall function of the terminal device. included.

  Hereinafter, the process which a terminal device performs is demonstrated.

  FIG. 3 is a flowchart for explaining a procedure of processing executed by the operation recognition module 3. The operation recognition module 3 first determines whether or not the information sent from the input module 1 or the information collection module 2 has been acquired (step S11).

  When the operation recognition module 3 acquires information sent from the input module 1 (S11: YES), the operation recognition module 3 performs a process of extracting a button operation from the acquired information (step S12). In addition, when the operation recognition module 3 acquires information sent from the information collection module 2 (S11: YES), the operation recognition module 3 performs a process of extracting a gesture operation by analyzing an acceleration pattern included as the acquired information ( Step S13).

  The operation recognition module 3 determines whether or not it has been recognized as a specific operation in the extraction process in step S12 or step S13 (step S14). When the operation is recognized as an operation (S14: YES), the operation recognition module 3 stores information (operation information) in which the operation name and operation time of the recognized operation are paired (step S15), and the operation information is stored in the rule control module. 5 is notified (step S16).

  Next, the operation recognition module 3 determines the continuity of the operation. For example, the operation recognition module 3 compares the operation time of the operation stored as the latest operation with the current time (step S17), and the difference between the operation time and the current time is a predetermined time (for example, 30 seconds) or more. Whether or not (step S18).

  When the difference between the two times is less than the predetermined time (S18: NO), the operation recognition module 3 determines that the operation is continuous, and notifies the rule control module 5 that the operation is continuous. (Step S19). On the other hand, when the difference between the two operation times is equal to or longer than the predetermined time (S18: YES), the operation recognition module 3 determines that the operation is not continuous, and rules that the operation is discontinuous. The data is sent to the control module 5 (step S20).

Note that the processing after step S17 is also executed when it is determined that information has not been acquired at step S11 (S11: NO) and when it is not recognized as a specific operation at step S14 (S14: NO). The
Further, since the operation recognition module 3 needs to determine the continuity of the operation at any time, the processing of steps S11 to S20 is repeatedly performed as needed.

  FIG. 4 is a flowchart for explaining a procedure of processing executed by the state recognition module 4. The state recognition module 4 first determines whether or not the information sent from the information collection module 2 has been acquired (step S21).

  When it is determined that the information sent from the information collection module 2 has been acquired (S21: YES), the state recognition module 4 analyzes the acceleration pattern included in the acquired information to extract the user's behavior state Is performed (step S22).

  The state recognition module 4 determines whether or not a specific action state such as a user sitting state or a user standing state has been recognized by the process of extracting the action state in step S22 (step S23). When it is determined that a specific behavior state has been recognized (S23: YES), the recognized behavior state information is notified to the rule control module 5 (step S24).

  Note that the processing by the state recognition module 4 as described above is repeatedly executed as appropriate.

  5 and 6 are flowcharts showing a procedure of processing executed by the rule control module 5. The rule control module 5 determines whether or not the operation information notified from the operation recognition module 3 has been received (step S31). When it is determined that the operation information has been received (S31: YES), the operation details are notified to the application control module 7 (step S32).

  When it is determined that the operation information has not been received (S31: NO), the rule control module 5 determines whether or not the information related to the continuity of the operation has been received from the operation recognition module 3 (step S33). If it is determined that the information related to the continuity of the operation has been received (S33: YES), the rule control module 5 stores the received information as the current operation continuation state (step S34).

  When it is determined that the information related to the continuity of the operation has not been received (S33: NO), the rule control module 5 determines whether or not the action state information notified from the state recognition module 4 has been received ( Step S35). When it is determined that the behavior state information has been received (S35: YES), the rule control module 5 stores the received information as the latest user behavior state (step S36).

  Next, the rule control module 5 determines whether or not the previous user action state that has already been stored exists (step S37). If it is determined that there is no previous user action state that is already stored (S37: NO), the latest user action state information is stored as the previous action state for convenience (step S38).

  When it is determined that there is a previous action state that is already stored in step S37 (S37: YES), or when a past action state is stored in step S38 (S38), the rule control module 5 displays the latest action state. Are compared with the past action state (step S39), and it is determined whether or not the user action state matches (step S40).

  When a user's action state corresponds (S40: YES), the rule control module 5 complete | finishes the process by this flowchart. In this case, there is no change in the user's action state, and the operation rule currently in use can be taken over and applied, so there is no need to newly acquire an operation rule.

  If it is determined that the user's action state does not match (S40: NO), the rule control module 5 refers to the stored operation continuation state information to determine whether or not the current operation continuation state is continuous. Is determined (step S41). If the current operation continuation state is continuous (S41: YES), the rule control module 5 acquires an operation rule from the operation rule DB 6 using the previous action state as a search key (step S42), and the acquired operation rule. Is notified to the application control module 7 (step S44).

  On the other hand, when the current operation continuation state is not continuous (S41: NO), the rule control module 5 acquires an operation rule from the operation rule DB 6 using the latest action state as a search key (step S43), and the acquired operation The rule is notified to the application control module 7 (step S44).

  Note that the processing by the rule control module 5 as described above is repeatedly executed as appropriate.

  FIG. 7 is a flowchart showing a procedure of processing executed by the application control module 7. The application control module 7 first determines whether or not the operation content notified from the rule control module 5 has been received (step S51). When the operation content has not been received (S51: NO), it is determined whether or not the operation rule notified from the rule control module 5 has been received (step S55). When the operation rule is received (S55: YES), the application control module 7 stores the received operation rule as an operation rule to be applied (step S56).

  On the other hand, when the operation content is received in step S51 (S51: YES), it is determined whether or not there is an operation rule to be applied (step S52). That is, the application control module 7 determines whether or not the operation rule notified from the rule control module 5 has already been received and stored as the operation rule to be applied.

If there is an operation rule to be applied (S52: YES), the received operation rule is applied, and the operation name in the operation information is converted into a control command for the application being executed (step S53). Then, the application control module 7 notifies a control command obtained by converting the operation name to the executing application (step S54).
Based on the control command notified from the application control module, the application implements the process according to the user operation by executing the process specified by the control command.

  Note that the processing by the application control module 7 as described above is repeatedly executed as appropriate.

  As described above, in this embodiment, when the user's behavior changes, when the operation is continued, the immediately preceding operation rule is taken over. For this reason, even if a user's action state changes, if operation by a user is continuing, operation will be possible with the same operation system as before.

  In the present embodiment, the acceleration sensor is described as an example of the means for collecting information according to the user's action state in the information collection module 2, but the present invention is not limited to the acceleration sensor. For example, it is possible to use an appropriate sensor that can detect a user's surroundings such as an optical sensor, a geomagnetic sensor, a temperature sensor, a camera that acquires video data, a microphone that acquires audio data, and the like. In addition, in order to collect information according to the user's behavior state, it is not necessary for the terminal device to incorporate a sensor, a camera, and a microphone, and information obtained externally is obtained using communication means. It may be.

  In the present embodiment, the operations for the browser application, the music application, and the calendar application have been described as examples of operations according to the user's action state (see FIG. 2), but the operation target is limited to these applications. It is not something. Other applications may be included, and a task switching or setting changing function attached to a system application that directly operates on the terminal device may be used.

  Further, in this embodiment, a reference (threshold for the operation interval) for determining the continuity of the operation is stored in advance, but the user inputs an arbitrary threshold using the input module 1. The operation continuity may be determined based on a threshold set by the user.

Embodiment 2. FIG.
In the first embodiment, the continuity of the operation is determined based on whether or not the time interval between the latest operation and the previous operation is within a predetermined time. However, the continuity of the operation is determined based on the operation frequency. It is good also as a structure to determine.

  In the present embodiment, the operation recognition module 3 creates and stores an operation history 31 based on information input from the input module 1 and the information collection module 2. FIG. 8 is a schematic diagram showing an example of the operation history 31. As illustrated in FIG. 8, the operation history 31 stored in the operation recognition module 3 stores operation times and operation contents in association with each other.

  The operation recognition module 3 refers to the operation history 31 as shown in FIG. 8 and determines the continuity of the operation. FIG. 9 is a flowchart showing the contents of processing executed by the operation recognition module 3 in the second embodiment. The operation recognition module 3 recognizes the operation based on the information sent from the input module 1 or the information collection module 2 as in the first embodiment (steps S61 to S64).

When the operation recognition module 3 recognizes as an operation in the extraction process in step S62 or step S63, the operation recognition module 3 uses information (operation information) paired with the operation name and operation time of the recognized operation as the latest operation. (Step S65) and the operation information is stored in the operation history 31 (step S66).
Further, the operation recognition module 3 transmits operation information to the rule control module 5 (step S67).

  Next, the operation recognition module 3 calculates the operation frequency from the operation history 31 (step S68). The operation frequency is calculated, for example, by counting the number of operations recognized by the operation recognition module 3 within a predetermined time range starting from the current time.

  Next, the operation recognition module 3 compares the calculated operation frequency with a threshold value stored in advance (for example, 5 times) (step S69), and determines whether or not the current operation frequency is equal to or greater than the threshold value (step S70). ).

  When the operation frequency is equal to or higher than the threshold (S70: YES), it is determined that the operation is continuous, and the operation recognition module 3 sends a notification that the operation is continuous to the rule control module 5 ( Step S71). When the operation frequency is less than the threshold (S70: NO), it is determined that the operation is not continuous, and the operation recognition module 3 sends a notification that the operation is discontinuous to the rule control module 5 ( Step S72).

Note that the processing after step S69 is also executed when it is determined that information has not been acquired at step S61 (S61: NO) and when it is not recognized as a specific operation at step S64 (S64: NO). The
Moreover, since the operation recognition module 3 needs to determine the continuity of operation at any time, the process of steps S61-S72 is repeatedly performed as needed.

Embodiment 3 FIG.
In the first embodiment, the determination whether or not the time interval between the latest operation and the previous operation is within a predetermined time is performed without depending on the application. However, the determination is made according to the application in use. It is good also as a structure which changes a reference | standard.
In the third embodiment, a configuration for determining continuity of operation according to an application in use will be described.

  The terminal device according to Embodiment 3 includes an operation interval table 32 in which an operation interval threshold is defined for each application. FIG. 10 is a conceptual diagram showing an example of the operation interval table 32. In the operation interval table 32 shown in FIG. 10, the operation interval for the web browser is set to 10 seconds, and the operation interval for the document browsing application is set to 30 seconds. For example, for an application that is frequently operated such as a web browser, the operation interval is set to be relatively short, and for an application that is frequently used without frequent operation such as a document browsing application, the operation interval is relatively set. Set longer.

The operation recognition module 3 refers to the operation interval table 32 as shown in FIG. 10 and determines the continuity of the operation. FIG. 11 is a flowchart showing the contents of processing executed by the operation recognition module 3 in the third embodiment. The operation recognition module 3 extracts the button operation and the gesture operation based on the information sent from the input module 1 or the information collection module 2 as in the first embodiment (steps S91 to S93).
Further, the operation recognition module 3 extracts the application name of the application in use (step S94).

  Next, the operation recognition module 3 determines whether or not the operation has been recognized by the extraction processing in steps S92 and S93 (step S95). If it is determined that the operation has been recognized (S95: YES), the application name extracted in step S94 is stored (step S96).

  In addition, when the operation recognition module 3 recognizes as an operation by the extraction process in step S92 or step S93 (S95: YES), the operation recognition module 3 sets the information (the operation name and operation time of the recognized operation as a pair ( (Operation information) is stored as the latest operation (step S97), and the operation information is transmitted to the rule control module 5 (step S98).

  Next, the operation recognition module 3 refers to the operation interval table 32 and acquires an operation interval (threshold value) defined for the application in use (step S99).

  The operation recognition module 3 compares the operation time of the operation stored as the latest operation with the current time in the same manner as in the first embodiment (step S100), and the difference between the operation time and the current time is determined in step S99. It is determined whether or not the acquired threshold value is exceeded (step S101).

  When the difference between the operation time and the current time is less than the threshold (S101: NO), the operation recognition module 3 determines that the operation is continuous, and performs rule control to notify that the operation is continuous. The data is sent to the module 5 (step S102). On the other hand, when the difference between the operation time and the current time is equal to or greater than the threshold (S101: YES), the operation recognition module 3 determines that the operation is not continuous and notifies that the operation is discontinuous. It is sent to the rule control module 5 (step S103).

Note that the processing after step S100 is also executed when it is determined that information has not been acquired at step S91 (S91: NO) and when it is not recognized as a specific operation at step S95 (S95: NO). The
Further, since the operation recognition module 3 needs to determine the continuity of the operation at any time, the processing of steps S91 to S103 is repeatedly performed as needed.

  In the present embodiment, since the operation continuation state is determined in consideration of the application being used by the user, the possibility that the operation is determined to be interrupted despite the operation is reduced. For this reason, it becomes possible to estimate the exact continuation state of a user's operation, and a better operating environment can be realized.

In this embodiment, the operation interval table 32 is provided inside the terminal device. However, an operation interval table is prepared outside the terminal device, and the terminal device accesses the operation interval table at the time of determination. The continuity of operation may be determined.
Further, in the present embodiment, the configuration in which the threshold value for the operation time is set in the operation interval table 32 has been described. However, as in the second embodiment, the operation frequency may be set for each application.

Embodiment 4 FIG.
In the first embodiment, the determination as to whether or not the time interval between the latest operation and the previous operation is within a predetermined time is performed regardless of the type of operation. It is good also as a structure which changes a determination reference | standard according to.

  The terminal device according to Embodiment 4 includes an operation type mapping table 33 that defines the relationship between the operation content and the operation type, and an operation interval table 34 that defines a threshold value for the operation interval for each operation type.

  FIG. 12 is a conceptual diagram showing an example of the operation type mapping table 33. The operation type mapping table 33 defines the relationship between the operation content and the operation type. For example, an operation of selecting a menu and an operation of returning or moving forward on a tool bar indicate that they belong to an operation type related to a menu. The same applies to other operation contents, and the operation type mapping table 33 defines one or more operation contents for each operation type.

FIG. 13 is a conceptual diagram showing an example of the operation interval table 34. In the operation interval table 34, a threshold for the operation interval is defined for each operation type. For example, the operation interval is set to 15 seconds for the operation type of playlist / tag editing. The same applies to the operation types of music search and music operation, which are set to 30 seconds and 90 seconds, respectively.
In playlist / tag editing, it is expected that button operations are frequently performed for selecting a song name and inputting a tag. For such operation types that are frequently operated, the operation interval is set to be relatively short. On the other hand, for operation types that are not frequently operated, such as music operations, the operation interval is set to be relatively long.

  The operation recognition module 3 refers to the operation type mapping table 33 as shown in FIG. 12 and the operation interval table 34 as shown in FIG. 13 to determine the continuity of the operation. FIG. 14 is a flowchart showing the contents of processing executed by the operation recognition module 3 in the fourth embodiment. As in the first embodiment, the operation recognition module 3 extracts button operations and gesture operations based on information sent from the input module 1 or the information collection module 2, and recognizes the recognized operation as the latest operation. Store (steps S111 to S115).

  Next, the operation recognition module 3 refers to the operation type mapping table 33 and identifies the operation type indicated by the current operation (step S116). Further, the operation recognition module 3 transmits the operation information of the operation recognized in step S114 (that is, information in which the operation name and operation time of the recognized operation are paired) to the rule control module 5 (step S117).

  Next, the operation recognition module 3 refers to the operation interval table 34 and acquires an operation interval (threshold value) defined for the operation type of the recognized operation (step S118).

  Next, the operation recognition module 3 determines the continuity of the operation. The operation recognition module 3 compares the operation time of the operation stored as the latest operation with the current time (step S119), and whether or not the difference between the operation time and the current time is greater than or equal to the threshold acquired in step S118. Is determined (step S120).

  When the difference between the two times is less than the threshold (S120: NO), the operation recognition module 3 determines that the operation is continuous and notifies the rule control module 5 that the operation is continuous. It is sent out (step S121). On the other hand, when the difference between the two operation times is equal to or greater than the threshold (S120: YES), the operation recognition module 3 determines that the operation is not continuous, and performs rule control to notify that the operation is discontinuous. The data is sent to the module 5 (step S122).

Note that the processing after step S119 is also executed when it is determined that information has not been acquired at step S111 (S111: NO) and when it is not recognized as a specific operation at step S114 (S114: NO). The
Further, since the operation recognition module 3 needs to determine the continuity of the operation as needed, the processes of steps S111 to S122 are repeatedly performed as needed.

  As described above, in the present embodiment, since the operation interval according to the operation type is set, there is less possibility that the operation is determined to be interrupted even though the operation is being performed. For this reason, it becomes possible to estimate the exact continuation state of a user's operation, and a better operating environment can be realized.

  In the present embodiment, the operation interval table 34 is provided inside the terminal device. However, the operation continuity may be determined with reference to a table prepared in the terminal device.

  In the present embodiment, the threshold according to the operation type of the latest operation is acquired from the operation interval table 34. However, as in the second embodiment, the operation history is stored and the current time is set. The most frequent operation within a predetermined time range as a starting point may be specified, a threshold value may be acquired from the operation interval table 34 according to the operation type of the most frequent operation, and the continuity of the operation may be determined.

Embodiment 5 FIG.
In the first embodiment, it is determined whether or not the time interval between the latest operation and the previous operation is within a predetermined time, and the continuity of the operation is determined. It is good also as a structure which uses a different criterion according to the combination of the state of ().
In the fifth embodiment, a configuration for determining the continuity of an operation using a determination criterion set according to the transition of the behavior state will be described.

The terminal device according to the present embodiment includes a behavior state storage table 35 in which the latest behavior state and the previous behavior state are stored in order to identify the transition of the behavior state of the user. FIG. 15 is a conceptual diagram showing an example of the behavior state storage table 35. In the example illustrated in FIG. 15, the previous user behavior state is “standing”, and the latest user behavior state is “sitting”.
The behavior state storage table 35 is updated each time the behavior recognition state of the user is recognized by the state recognition module 4.

  In addition, the terminal device according to the present embodiment includes an operation interval table 36 that defines an operation interval threshold according to a transition of a user's behavior state (a set of behavior states). FIG. 16 is a conceptual diagram showing an example of the operation interval table 36. In the operation interval table 36 shown in FIG. 16, the threshold value of the operation interval is defined according to the set of action states. For example, it is stipulated that 20 seconds is used as an operation interval threshold for a transition from a standing state to a sitting state, and an operation interval is set for a transition from a sitting state to a standing state. It is specified that 40 seconds is used as the threshold value.

  The operation recognition module 3 refers to the operation interval table 36 as shown in FIG. 16 and determines the continuity of the operation. 17 and 18 are flowcharts showing the contents of processing executed by the operation recognition module 3 in the fifth embodiment. As in the first embodiment, the operation recognition module 3 extracts button operations and gesture operations based on information sent from the input module 1 or the information collection module 2, and recognizes the recognized operation as the latest operation. Store (steps S131 to S135).

  Next, the operation recognition module 3 determines whether a notification about the user's action state is received from the state recognition module 4 (step S136). When it is determined that the user behavior state notification has been received (S136: YES), it is determined whether or not the received behavior state matches the latest behavior state (step S137).

  When it is determined that the received behavior state does not match the latest behavior state (S137: NO), the operation recognition module 3 changes the latest behavior state stored in the behavior state storage table 35 to the previous behavior state. Overwriting (step S138), the behavior state notified from the state recognition module 4 is overwritten with the latest behavior state (step S139), and the behavior state storage table 35 is updated.

  Next, the operation recognition module 3 transmits the operation information of the operation recognized in step S134 (that is, information in which the operation name and operation time of the recognized operation are paired) to the rule control module 5 (step S140).

  Next, the operation recognition module 3 refers to the behavior state storage table 35 and obtains an operation interval (threshold value) set for the set of the previous behavior state and the latest behavior state from the operation interval table 36 (step) S141).

  Next, the operation recognition module 3 determines the continuity of the operation. The operation recognition module 3 compares the operation time of the operation stored as the latest operation with the current time (step S142), and whether or not the difference between the operation time and the current time is greater than or equal to the threshold acquired in step S141. Is determined (step S143).

  If the difference between the two times is less than the threshold (S143: NO), the operation recognition module 3 determines that the operation is continuous, and notifies the rule control module 5 that the operation is continuous. It is sent out (step S144). On the other hand, when the difference between the two operation times is greater than or equal to the threshold, that is, greater than or equal to the threshold (S143: YES), the operation recognition module 3 determines that the operation is not continuous, and the operation is discontinuous. A notification to that effect is sent to the rule control module 5 (step S145).

If it is determined in step S131 that information has not been acquired in step S131 (S131: NO), or if it is not recognized as a specific operation in step S134 (S134: NO), the process in step S136 is performed by the user. It is also executed when it is determined that there is no notification of the behavior state (S136: NO), and when it is determined that the behavior state received in step S137 matches the latest behavior state (S137: YES).
Further, in the operation recognition module 3, since it is necessary to determine the continuity of the operation at any time, the processes of steps S131 to S145 are repeatedly executed as needed.

  As described above, in the present embodiment, since the operation interval corresponding to the transition of the behavior state is set, the possibility that the operation is determined to be interrupted despite the operation is reduced. For this reason, it becomes possible to estimate the exact continuation state of a user's operation, and a better operating environment can be realized.

  In the present embodiment, the operation interval table 36 is provided inside the terminal device. However, the operation continuity may be determined by referring to a table prepared in the terminal device.

  Regarding the above embodiment, the following additional notes are disclosed.

(Appendix 1)
A reception unit that accepts user operations;
An information collection unit that collects information according to the user's condition;
A storage unit that stores the operation rules that are defined according to the user's state and that indicates the relationship between the operation content received by the reception unit and the processing content to be executed;
A first determination unit that determines whether or not a user's state has changed based on the information collected by the information collection unit;
A second determination unit that determines continuity of operations received by the reception unit;
A control unit that controls application timing of an operation rule to be applied when an operation by a user is received at the reception unit according to a determination result by the first and second determination units;
A process content determination unit for determining a process content to be executed based on an operation rule to be applied according to an application timing controlled by the control unit when an operation by a user is received by the reception unit;
An electronic device comprising: an execution unit that executes processing according to the processing content determined by the processing content determination unit.

(Appendix 2)
When the first determination unit determines that the user's state has changed and the second determination unit determines that the operation is continuous, the control unit determines an operation rule according to the user's state before the change. To keep
When the first determination unit determines that the user's state has changed and the second determination unit determines that the operation is not continuous, the control unit determines an operation rule according to the user's state before the change. The electronic device according to appendix 1, wherein the electronic device is switched to an operation rule according to a user state after the change.

(Appendix 3)
A measurement unit that measures an operation interval of operations received by the reception unit;
A threshold storage unit for storing the threshold,
The second determination unit is configured to determine the continuity of the operation based on the length of the operation interval measured by the measurement unit and the threshold value stored in the threshold value storage unit. The electronic device according to attachment 2.

(Appendix 4)
An application storage unit that stores a plurality of applications executed by an operation received by the reception unit;
The electronic device according to appendix 3, wherein the threshold storage unit stores a threshold for each application.

(Appendix 5)
The electronic device according to appendix 3, wherein the threshold storage unit stores a threshold according to an operation type including one or more operation contents.

(Appendix 6)
An operation history storage unit that stores a history of operations received by the reception unit;
The second determination unit is configured to determine operation continuity using a threshold value determined for an operation type including the most frequent operation within a predetermined time. The electronic device as described in.

(Appendix 7)
The electronic device according to appendix 3, wherein the threshold storage unit stores a threshold according to a combination of a plurality of states of the user.

(Appendix 8)
A setting unit for setting a threshold for the operation interval;
The electronic device according to appendix 3, wherein the threshold set by the setting unit is stored in the threshold storage unit.

(Appendix 9)
The electronic device according to any one of appendix 3 to appendix 8, wherein the measurement unit is configured to measure the number of times of operation within a predetermined time.

(Appendix 10)
Accept user operations,
Determine the continuity of accepted operations,
Collect information according to the user status,
Based on the collected information, determine whether the user's status has changed,
It is defined according to the user's state, and controls the application timing of the operation rule indicating the relationship between the operation content and the processing content to be executed according to the determination result of the user's operation continuity and the user's state change,
When the operation by the user is accepted, the processing content to be executed is determined based on the operation rule applied at the application timing controlled according to the determination result,
A control method for an electronic device, characterized in that processing is executed in accordance with the determined processing content.

(Appendix 11)
Causing the computer to determine the continuity of the accepted operations;
Causing the computer to determine whether or not the user's state has changed based on information acquired from outside;
Controls the application timing of operation rules that are defined in the computer according to the user's status and indicate the relationship between the operation content and the processing content to be executed according to the determination result of the continuity of the user's operation and the user's state change. Step to
When the computer accepts the operation by the user, the step of determining the processing content to be executed based on the operation rule applied at the application timing controlled according to the determination result;
And causing the computer to execute a process according to the determined processing content.

1 Input module (reception part)
2 Information collection module (information collection unit)
3 Operation recognition module (second determination unit)
4 State recognition module (first determination unit)
5 Rule control module (Processing content decision unit)
6 Operation rule database (storage)
7 Application control module (execution unit)
8 applications

Claims (5)

A reception unit that accepts user operations;
An information collection unit that collects information according to the user's condition;
A storage unit that stores the operation rules that are defined according to the user's state and that indicates the relationship between the operation content received by the reception unit and the processing content to be executed;
A first determination unit that determines whether or not a user's state has changed based on the information collected by the information collection unit;
A second determination unit that determines continuity of operations received by the reception unit;
A control unit that controls application timing of an operation rule to be applied when an operation by a user is received at the reception unit according to a determination result by the first and second determination units;
A process content determination unit for determining a process content to be executed based on an operation rule to be applied according to an application timing controlled by the control unit when an operation by a user is received by the reception unit;
An electronic device comprising: an execution unit that executes processing according to the processing content determined by the processing content determination unit. When the first determination unit determines that the user's state has changed and the second determination unit determines that the operation is continuous, the control unit determines an operation rule according to the user's state before the change. To keep
When the first determination unit determines that the user's state has changed and the second determination unit determines that the operation is not continuous, the control unit determines an operation rule according to the user's state before the change. The electronic device according to claim 1, wherein the electronic device is switched to an operation rule corresponding to a user state after the change. A measurement unit that measures an operation interval of operations received by the reception unit;
A threshold storage unit for storing the threshold,
2. The second determination unit is configured to determine continuity of an operation based on a length between an operation interval measured by the measurement unit and a threshold value stored in the threshold value storage unit. Or the electronic device of Claim 2. Accept user operations,
Determine the continuity of accepted operations,
Collect information according to the user status,
Based on the collected information, determine whether the user's status has changed,
It is defined according to the user's state, and controls the application timing of the operation rule indicating the relationship between the operation content and the processing content to be executed according to the determination result of the user's operation continuity and the user's state change,
When the operation by the user is accepted, the processing content to be executed is determined based on the operation rule applied at the application timing controlled according to the determination result,
A control method for an electronic device, characterized in that processing is executed in accordance with the determined processing content. Causing the computer to determine the continuity of the accepted operations;
Causing the computer to determine whether or not the user's state has changed based on information acquired from outside;
Controls the application timing of operation rules that are defined in the computer according to the user's status and indicate the relationship between the operation content and the processing content to be executed according to the determination result of the continuity of the user's operation and the user's state change. Step to
When the computer accepts the operation by the user, the step of determining the processing content to be executed based on the operation rule applied at the application timing controlled according to the determination result;
And causing the computer to execute a process according to the determined processing content.

Images