JP2017182571A - Information processing system, information processing method and information processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing system, an information processing method and an information processing program with which it is possible to easily switch the setting of whether or not to execute gesture-based operation at the discretion of a user.SOLUTION: The information processing system includes a wearable device 12 that detects the motion of a wrist around the longitudinal axis of an arm with a motion sensor unit 12a and outputs sensor data. An external device 14 is included that extracts the motion information of a user from the sensor data and executes a process that corresponds to gesture information assuming motion information that satisfies a specific condition as gesture operation information. The external device 14 has, as operation modes, an operation mode in which a process corresponding to the gesture operation information is executable and an ordinary mode in which the process corresponding to the gesture operation information is halted. The external device extracts around-axis motion information indicating the motion of a user around the wrist from the sensor data, determines that twist operation is detected when the around-axis motion information satisfies a specific condition, and switches the operation modes upon detecting the twist motion.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing system operable by a user's gesture, an information processing method executed by the information processing system, and an information processing program for causing the information processing system to execute the information processing method.

  Conventionally, a plurality of information processing systems have been proposed in which a user's gesture is detected by a wearable device, and an operation screen displayed on a personal computer or the like can be operated in a hands-free manner.

  For example, as disclosed in Patent Document 1, it is an apparatus that can be used even during work using both hands, and uses detection means such as a gyro sensor mounted on the user's head, shoulder, or arm. Thus, there has been an information input device that detects a gesture, and a control unit analyzes a detection signal of the detection unit to extract input information. This control means is set to consider the detection signal as noise when the magnitude of the detection signal is not within the predetermined range, and masks movements unrelated to the input operation, such as minute movements of the body and changes in posture. It is configured to be able to.

  Further, as disclosed in Patent Document 2, the movement of the hand including the wrist is detected using a state observation unit such as an acceleration / gyro sensor attached to the user's hand, and the processing unit includes a state observation unit. There has been a gesture input device that performs control in accordance with a detected gesture. This processing unit is configured to ignore the gesture detected by the state observation unit when the wrist is not dorsiflexed, and the user can select whether or not to perform an input operation by switching the wrist state It has become.

  Since the devices disclosed in Patent Documents 1 and 2 are configured to detect the movement of the user's body using an acceleration sensor or a gyro sensor, they can be detected with higher accuracy than using a geomagnetic sensor or the like. In addition, the configuration that detects the movement of the user's body using a camera or other imaging device is only used in places where conditions such as the direction of light and brightness are in place, such as when the camera is difficult to carry and install. Although there is an inconvenient aspect that cannot be performed, the configuration of detecting using an acceleration sensor or a gyro sensor has an advantage that such inconvenience does not occur.

JP-A-10-222293 Japanese Patent Laying-Open No. 2015-88161

  The information input device disclosed in Patent Document 1 determines whether the user's movement corresponds to an input operation or noise based on the magnitude of the detection signal, so that the user unexpectedly moves halfway. If this happens, there is a possibility of making an incorrect determination. Therefore, the user must always operate with care so as not to cause the device to malfunction, and in the event of malfunction, it is necessary to redo the processing operation, which is cumbersome and has poor operability. Met.

  On the other hand, in the case of the gesture input device of Patent Document 2, since the user can select whether or not to perform an input operation by switching his / her wrist state, the above malfunction is unlikely to occur. It is done. However, the user must perform a gesture while holding the wrist in a dorsiflexion state when performing an input operation. The posture with the wrist bent back is not an easy posture for people to relax, so it is painful for users to continue gesturing for a long time with this posture, and the operability is still not good. is there.

  The present invention has been made in view of the above-described background art. An information processing system, an information processing method, and an information processing method that can easily and accurately switch a setting as to whether or not to perform an operation based on a gesture. An object is to provide an information processing program.

  The present invention is an information processing system that can be operated by a user's gesture, and is mounted on a body part that the user can rotate by himself / herself, and the movement of the body part is detected by an acceleration sensor and a gyro sensor. A wearable device that detects and outputs sensor data detected by a motion sensor unit including at least one sensor element, acquires the sensor data output from the wearable device, and analyzes the sensor data to obtain a user's A device that extracts motion information, regards the motion information that satisfies a specific condition as gesture operation information, and performs various types of processing corresponding to the gesture operation information, and processing corresponding to the gesture operation information as an operation mode The operation mode that can be executed and the processing corresponding to the gesture operation information are stopped. An external device provided with a normal mode, and the external device, by analyzing the sensor data, to extract the pivoting movement information indicating the movement of the user pivoting the body part, Means is provided for determining that a twisting motion has been detected when this pivoting motion information satisfies a specific condition, and the external device, when detecting the twisting motion, is information for switching the current operation mode to the other operation mode. It is a processing system.

  The pivoting motion information is preferably information indicating a series of movements in which the user pivots the body part in the forward direction and the backward direction. The external device may be provided with a display unit that displays a GUI screen, and the wearable device may be used as a pointing device for operating the GUI screen.

  Further, the present invention is an information processing method executed by an information processing system operable with a user's gesture, wherein the information processing system is attached to a body part that the user can rotate on its own, A wearable device that detects the movement of the body part with a motion sensor unit including a sensor element including at least one of an acceleration sensor and a gyro sensor, and outputs sensor data, and the sensor data output by the wearable device A device that extracts user motion information by analyzing the sensor data, regards the motion information that satisfies a specific condition as gesture operation information, and performs various processes corresponding to the gesture operation information. As an operation mode, processing corresponding to the gesture operation information can be executed. An external device provided with an operation mode and a normal mode in which processing corresponding to the gesture operation information is stopped, and a movement in which the user pivots the body part by analyzing the sensor data Is an information processing method for extracting a twisting motion when the pivoting motion information satisfies a specific condition and determining that a twisting motion has been detected and switching the current motion mode to the other motion mode.

  A twist motion detection process for determining whether or not the twist motion has been detected is provided, and the twist motion detection process analyzes the sensor data and acquires the axis of the body part each time the sensor data is acquired. Each time the angular velocity is calculated, the step of calculating the angular velocity in the turning direction is compared with a predetermined angular velocity reference value, and the timing at which the angular velocity is equal to or higher than the angular velocity reference value is determined. The step of setting the twist detection start time for detecting the twist motion, and every time the angular velocity is calculated, using the angular velocity calculated last time and the angular velocity calculated this time, the amount of change in the twist angle of the body part And calculating the torsion angle change every time the torsion angle change is calculated and summing the torsion angle change calculated after the start time. A step of calculating a twist angle and comparing the twist angle with a predetermined twist angle reference value, and determining that the twist operation has been detected when the twist angle exceeds a predetermined twist angle reference value Can be. In this case, when the twist operation is not detected even after a lapse of a certain time from the start time, the setting of the start time is reset, and the calculated change in the twist angle and the information on the twist angle are discarded. It is preferable to do.

  The step of calculating the angular velocity in the axial direction of the body part is to calculate an angular velocity of an operation of rotating the wrist around the longitudinal axis of the user's arm.

  The pivoting movement information may be information indicating a series of movements in which the user pivots the body part in the forward direction and the backward direction. In this case, a twist motion detection process executed to detect the twist motion is provided, and the twist motion detection process analyzes the sensor data every time the sensor data is acquired, and rotates the axis of the body part. Each time the angular velocity is calculated, the angular velocity is compared with a predetermined angular velocity reference value, and the timing at which the angular velocity becomes equal to or higher than the angular velocity reference value is determined. The step of setting the twist detection start time for detecting motion, and every time the angular velocity is calculated, the change in the twist angle of the body part is calculated using the angular velocity calculated last time and the angular velocity calculated this time. And calculating the torsion angle change every time the torsion angle change is calculated and summing the torsion angle change calculated after the start time. A twist angle is calculated, and the twist angle is compared with a predetermined twist angle reference value.First, the twist motion detection process is performed for the forward axial motion, and the forward motion is detected. When the twist angle exceeds a predetermined twist angle reference value, it is determined that the forward twist has been detected, and after determining that the forward twist has been detected, within a certain time, The twist motion detection process is executed, and when the forward twist angle exceeds a predetermined twist angle reference value, it is determined that the reverse twist is detected and the twist motion is detected. It is preferable. Further, when a forward or backward twist is not detected even after a lapse of a certain time from the start time, the start time setting is reset and the calculated change in the twist angle and the information on the twist angle are discarded. It is preferable to adopt a configuration to

  The external device has a display unit for displaying a GUI screen, the wearable device is used as a pointing device for operating the GUI screen, and a cursor displayed on the GUI screen by a user A cursor stage for performing an operation of moving the cursor, the cursor stage displaying a cursor on the GUI screen when detecting specific gesture operation information of a user, and analyzing the sensor data The mobile motion information indicating the movement of the user moving the body part in a specific in-plane direction is extracted, the mobile motion information satisfying a specific condition is regarded as the gesture operation information, and the mobile motion information is included in the mobile motion information. And a step of moving the display position of the cursor correspondingly.

  The cursor stage determines that the specific position has been temporarily selected when the cursor has stopped at the specific position for a certain period of time and fixes the cursor to the specific position; and When it is determined that the twist operation is detected in a fixed state, it may be determined that the user has clicked at the specific position, and a process corresponding to the click is performed.

  Furthermore, the present invention is an information processing program comprising a program for each processing step for causing the information processing system to execute the information processing method.

  Since the information processing system of the present invention is configured so that a user wears a wearable device on a specific body part, it can be operated hands-free. Furthermore, since the movement of the user's body part is detected using an acceleration sensor or a gyro sensor, it can be detected with high accuracy in various environments and is easy to carry around.

  In addition, according to the information processing system, the information processing method, and the information processing program of the present invention, does the user perform an input operation by a gesture by performing a motion (twist motion) that pivots a specific body part? You can switch the setting of whether or not. Therefore, when the user wants to take a break, he / she can relax with ease and relax after switching to the normal mode. After the break, the user can switch to the operation mode and resume the input operation with a new feeling.

  Further, the twist operation is a simple operation that can be easily performed by a person and does not burden the user. Moreover, since the twist operation is a clear operation that can be easily distinguished from other operations, it is possible to easily prevent the external device from erroneously recognizing.

It is a system configuration figure showing one embodiment of an information processing system of the present invention. It is a figure which shows the state with which the wearable device used with the information processing system of this embodiment was mounted | worn on a user's wrist. It is a flowchart which shows the flow of the mode switching in one Embodiment of the information processing method of this invention. FIG. 4 is a flowchart illustrating a flow of a twist operation detection process part in FIG. 3. FIG. 6 is a flowchart showing a flow of a forward twist detection process in FIG. 4. FIG. 5 is a flowchart showing the flow of a backward twist detection process in FIG. 4. FIG. It is a figure which shows a GUI screen when performing the dialog stage in the information processing method of this embodiment. It is a flowchart which shows the flow of the dialog stage shown in FIG. It is a flowchart which shows the flow of the part of the swipe operation detection process in FIG. It is figure (a)-(d) which shows a GUI screen when performing the drawer stage in the information processing method of this embodiment. It is a flowchart which shows the flow of the dialog stage shown in FIG. It is a figure which shows a GUI screen when performing the cursor stage in the information processing method of this embodiment. It is a flowchart which shows the flow of the cursor stage shown in FIG. It is a flowchart which shows the modification of the flow of mode switching in the information processing method of this embodiment.

  Hereinafter, an embodiment of an information processing system, an information processing method, and an information processing program according to the present invention will be described with reference to the drawings. As shown in FIG. 1, the information processing system 10 according to this embodiment includes a wearable device 12 and an external device 14, and the wearable device 12 and the external device 14 are connected via a wired or wireless network 16. 12 is a system in which the external device 14 can be operated by a gesture of a user wearing the device 12. The information processing method of this embodiment is a processing method executed by the information processing system 10, and the information processing program of this embodiment includes programs for each processing step in order to cause the information processing system 10 to execute this processing method. It is a computer program.

  The wearable device 12 is attached to a body part where a user can perform an operation of rotating around a predetermined rotation axis of the body (hereinafter, referred to as “rotation”). , A device that detects and outputs sensor data by a motion sensor unit 12a including a sensor element including at least one of an acceleration sensor and a gyro sensor. The body parts that the user can rotate on their own are, for example, the wrist, ankle, fingertip, arm, head, etc. The wearable device 12 is structured to be worn on the wrist like a wristwatch. As shown in FIG. 2, the main body 18 including the motion sensor 12 a and the communication unit 12 b is disposed on the back side of the wrist, and is used by being fixed by the belt 20. That is, by moving the motion sensor unit 12a integrally with the wrist, the movement of the wrist moving or pivoting is detected. Hereinafter, in this embodiment, regarding the axial direction of the motion sensor unit 12a attached to the wrist or right hand of the right hand, a direction in which the right screw is tightened is referred to as a forward direction, and a direction in which the right screw is loosened is referred to as a backward direction. In addition, in the case where the motion sensor unit 12a is mounted on the wrist or left hand of the left hand, the reverse is preferable.

  The external device 14 is a computer device such as a personal computer, for example, and includes a communication unit 22, a sensor data analysis unit 24, a graphical user interface operation unit 26 (hereinafter referred to as a GUI operation unit 26), a display unit 28, and a control unit. 30.

  The communication unit 22 is a block that sequentially receives sensor data output from the wearable device 12, and the display unit 28 is a display that displays a graphical user interface screen 28a (hereinafter referred to as a GUI screen 28a). .

  The sensor data analysis unit 24 includes a data storage unit 24a that stores sensor data received by the communication unit 22, a gesture detection unit 24b that analyzes sensor data and determines whether or not a specific gesture has been detected, and gesture detection. A gesture determination condition database 24c in which a determination method (procedure, reference value, etc.) performed by the unit 24b is defined. The gesture detection unit 24b acquires necessary sensor data from the data storage unit 24a, analyzes the sensor data according to a procedure defined in the gesture determination condition database 24c, and extracts operation information indicating the movement of the user. Then, by comparing with a reference value or the like defined in the gesture determination condition database 24c, a specific gesture (for example, a twist operation for rotating the wrist around the longitudinal axis of the arm, a lateral direction) It is determined whether or not there is an item that corresponds to a swipe operation to move to. If there is corresponding motion information, it is determined that a specific gesture has been detected, and the motion information is output as gesture operation information.

  The GUI operation unit 26 includes a mode management unit 26 a that switches the operation mode of the GUI operation unit 26 and a GUI control command output unit 26 b that controls the GUI screen 28 a of the display unit 28. There are two operation modes: an operation mode in which processing corresponding to gesture operation information can be executed, and a normal mode in which processing corresponding to gesture operation information is stopped. The mode management unit 26a is described later with the sensor data analysis unit 24. When the twisting operation to be performed is detected, the current operation mode is switched to the other operation mode according to a command from the control unit 30. The GUI control command output unit 26b creates a GUI control command corresponding to the gesture operation information, and changes or switches the display of the GUI screen 28a.

  The control unit 30 is a block that comprehensively controls the order and timing of the operation of the communication unit 22, the sensor data analysis unit 24, the GUI operation unit 26, and the display unit 28, and further provides a GUI stage to be provided to the user. Also manage. The GUI stage is a generic name for a dialog stage, a drawer stage, a cursor stage, etc., which will be described later, and the control unit 30 displays a dialog on the GUI screen 28a, for example, at an arbitrary timing, or performs notification by a drawer. Or In addition, when creating a GUI control command based on gesture operation information, there are cases where the creation rule must be changed for each GUI stage, so the control unit 30 issues a command to the GUI control command output unit 26b, and the GUI Control is performed so that a GUI control instruction suitable for the stage is generated.

  Next, an embodiment of an information processing method used in the information processing system 10 described above will be described. Here, the wearable device 12 is already worn on the wrist of the user, detects the movement of the wrist with the motion sensor unit 12a, and sequentially outputs the sensor data through the communication unit 12b. It is assumed that the data is received by the communication unit 22 and sequentially stored in the data storage unit 24a.

  First, an example of a processing method for switching the operation mode of the GUI operation unit 26 when a twist operation of a user is detected in the information processing method of this embodiment will be described with reference to FIGS.

  Assuming that the operation mode at the start is the normal mode, first, as shown in FIG. 3, twist operation detection processing S11 is performed. The twist motion detection process S11 is a step of determining that the twist motion has been detected when a series of movements in which the wrist is rotated in the forward direction and the backward direction about the longitudinal axis of the user's arm satisfies a specific condition. It is. Generally speaking, first, forward movement information (movement information indicating the movement of the user pivoting the wrist in the forward direction) is extracted, and if this satisfies a specific condition, the forward twist is detected. After that, similarly, the backward axial movement information is extracted. If this information satisfies a specific condition, it is determined that the backward twist is detected, and it is determined that the twist movement is detected. Hereinafter, the flow of the twist operation detection process S1 will be described in detail.

  As shown in FIG. 4, the twist operation detection process S11 starts from start A and proceeds to step S11-1. At this time, since the forward twist has not been detected, the process proceeds to forward twist detection processing S11-2.

  The forward twist detection processing S11-2 starts from the start B as shown in FIG. First, in Step 11-2-1, the gesture detection unit 24b that receives a command from the control unit 30 acquires the latest sensor data from the data storage unit 24a (kth). In step 11-2-2, it is determined whether forward twist detection is being executed. At this point, since step S11-2-5, which will be described later, is not performed (because the start time Ta1 is not set), it is determined that it is not being executed, and the process proceeds to step S11-2-3.

  If it progresses to step S11-2-3, the gesture detection part 24b will analyze the acquired sensor data, will calculate the angular velocity of the axial direction (forward direction) of a wrist, and it calculated in step 11-2-4. It is determined whether the angular velocity is equal to or higher than the angular velocity reference value Va. If it is equal to or greater than the angular velocity reference value Va, it is determined that the operation of twisting the wrist in the forward direction has started, and this timing is set as the forward twist detection start time Ta1 (step S11-2-5). Thereafter, the process proceeds to end B and waits. When a command from the control unit 30 is received, the process returns to start B.

  On the other hand, if it is equal to or less than the angular velocity reference value Va, it is determined that the operation of twisting the wrist in the forward direction has not started, and the process proceeds to end B and waits. Thereafter, when a command from the control unit 30 is received, the process returns to start B.

  In step S11-2-5, the start time Ta1 is set. When the start time Ta1 is reached, when a command is received from the control unit 30, the process returns to start B and proceeds to step S11-2-1 to detect a gesture. The unit 24b obtains the latest sensor data again. Then, when proceeding to step S11-2-2, since it is already after the start time Ta1, it is determined that forward twist detection is being performed, and the process proceeds to step S11-2-6, where the gesture detection unit 24b acquires The sensor data is analyzed to calculate the angular velocity of the wrist in the axial direction (forward direction). In step S11-2-7, it is determined whether a predetermined time ta1 has elapsed from the start time Ta1.

  When the time ta1 has not elapsed from the start time Ta1, the process proceeds to step 11-2-8, and the angular velocity calculated in step S11-2-6 of this time (kth) and the previous (k-1th) step S11. A change ΔRa (k) of the wrist twist angle generated during this period is calculated using the angular velocity calculated in 2-6. And it progresses to step S11-2-9, each change part ΔRa (k) after start time Ta1 is totaled, twist angle Ra (k) is calculated, and it progresses to step 11-2-10.

  In Step 11-2-10, it is determined whether or not the calculated twist angle Ra (k) exceeds the twist angle reference value Ra. If the twist angle reference value Ra is exceeded, it is determined that a forward twist has been detected (step S11-2-11). This completes the forward twist detection process S11-2, and proceeds to the next step S11-3.

  On the other hand, if the twist angle reference value Ra is not exceeded, it is determined that the operation being detected (the movement of twisting the wrist in the forward direction) is not currently recognized as a twist, and the process proceeds to end B and waits. Thereafter, an instruction to continue the twist detection in the forward direction is received, and the process returns to start B again to proceed to step S11-2-1. Similar to the above, steps S11-2-2, S11-2-6, S11 are performed. -2-7, S11-2-8, S11-2-9 are performed. If the wrist is properly pivoted, the twist angle Ra (k) calculated in the current step S11-2-9 is the twist angle Ra (k) calculated in the previous step S11-2-9. Bigger than. When the current twist angle Ra (k) exceeds the twist angle reference value Ra, it is determined that the forward twist is detected (step S11-2-11), and the forward twist detection is performed in the same manner as described above. The process S11-2 is ended, and the process proceeds to the next step S11-3. On the other hand, if the twist angle reference value Ra is not exceeded, it is determined that the twist is not yet recognized at this moment, and the process proceeds to the end B and waits again. Thereafter, when there is an instruction from the control unit 30 to continue twist detection in the forward direction, the operation returns to start B and the above operation is repeated.

  If the torsional angle Ra (k) does not exceed the torsional angle reference Ra even if this repeated operation is continued for a certain period of time, it is determined that the motion being detected (the motion of twisting the wrist in the forward direction) does not correspond to the forward twist. And stop the twist detection in progress. That is, when the predetermined time ta1 has elapsed from the start time Ta1, the gesture detection unit 24b proceeds from step 11-2-7 to step S11-2-12, resets the setting of the start time Ta1, Information such as the twist angle change ΔRa (k) and twist angle Ra (k) calculated in the repetitive motion is discarded. Then, the process proceeds to end A and waits, and when there is a command from the control unit 30, the process returns to start A in FIG.

  After detecting the forward twist in the forward twist detection process S11-2, the process proceeds to step S11-4 through step S11-3 shown in FIG. 4, and sets the forward twist detection time Ta2 and ends. Go to A and wait.

  Thereafter, upon receiving a command from the control unit 30 promptly, the process returns to start A and proceeds to step S11-1. This time, since the twist in the forward direction has already been detected, the process proceeds to step S11-5, where it is determined whether or not a predetermined time ta2 has elapsed from the detection time Ta2 in the forward direction, and when the time ta2 has not elapsed, Reverse direction twist detection processing S11-6 is performed.

  As shown in FIG. 6, the contents of the backward twist detection process S11-6 are the same as those of the forward twist operation detection process S11-2 described above. That is, the start C and end C in the backward direction correspond to the start B and end B in the forward direction, respectively. Steps S11-6-1 to S11-6-12 in the backward direction correspond to steps S11-2-1 to S11-2-12 in the forward direction, respectively.

  Further, the backward twist detection start time Tb1 corresponds to the forward twist detection start time Ta1. The change ΔRb (k) and the twist angle Rb (k) in the backward twist angle calculated based on the sensor data are respectively changed to the change ΔRa (k) and the twist angle Ra (k) in the forward twist angle. Correspond. Further, the backward angular velocity reference value Vb, the time tb1, and the torsion angle reference value Rb, which are the reference values when making the determination, correspond to the forward angular velocity reference value Va, the time ta1, and the torsion angle reference value Ra, respectively. . These reference values (absolute values) and time may be set to the same value in the forward direction and the backward direction. If the turning operation is a little slow, a different value can be set accordingly.

  If the time ta2 has elapsed from the forward direction detection time Ta2 after the forward direction twist detection until the backward direction twist detection is started, the forward direction pivoting operation of the wrist is performed in the backward direction. It is determined that the operation is not a series of operations and does not correspond to the reciprocating twist operation, and the twist detection is stopped here. In other words, when the gesture detection unit 24b determines that the predetermined time ta2 has elapsed from the detection time Ta2 in several times of step S11-5, the process proceeds to step S11-7 and resets the settings of the times Ta1, Ta2, and Tb1. At the same time, the information on the twist angle changes ΔRa (k), ΔRb (k) and twist angles Ra (k), Rb (k) calculated in the repetitive motion is discarded, and the twist in the forward direction is detected. Reset the information that it has been completed. Then, the process proceeds to end A and waits, and then returns to start A when there is a command from the control unit 30.

  After the backward twist is detected in the backward twist detection process S11-6, the process proceeds to step S11-9 through step S11-8, and it is determined that the twist operation is detected. In step S11-10, the settings of the times Ta1, Ta2, and Tb1 are reset, and the twist angle changes ΔRa (k), ΔRb (k), the twist angle Ra (k), The information of Rb (k) is discarded, and the information that the forward twist has been detected is reset, and the twist operation detection process S11 is terminated. And as shown in FIG. 3, it progresses to the following step S12.

  Since the twist operation is detected in the twist operation detection process S11, the process proceeds to step S13 via step S12. The mode management unit 26a of the GUI operation unit 26 sets the operation mode of the GUI operation unit 26 according to the command of the control unit 30. The normal mode (operation mode in which processing according to gesture operation information is stopped) is switched to an operation mode (mode in which processing according to gesture operation information is executed). Thus, the user can operate the external device 14 with the gesture.

  The processing method for switching from the operation mode to the normal mode is the same as described above. That is, as shown in FIG. 3, in the operation mode, the twist operation detection process S11 and steps S12 and S13 similar to the above are performed. Thereby, the operation mode of the GUI operation unit 26 is switched from the operation mode to the normal mode. In the normal mode, since the external device 14 does not accept an input operation by a user's gesture, the user can relax and relax in a safe manner.

  Next, a dialog stage processing method in the GUI stage provided to the user by the information processing method of this embodiment will be described with reference to FIGS. As shown in FIG. 7, the dialog stage is a stage in which a dialog is displayed on the GUI screen 28a of the display unit 28, and the user selects a specific button by a swipe operation.

  Assuming that the operation mode at the start is the operation mode, as shown in FIG. 8, first, the GUI control command output unit 26b generates a predetermined GUI control command in response to a command from the control unit 30, and receives this command. The display unit 28 displays a dialog on the GUI screen 28a (step S21). Then, the sensor data analysis unit 24 performs a swipe operation detection process S22.

  The swipe operation detection process S22 starts from the start D as shown in FIG. First, in step S22-1, the gesture detection unit 24b that has received a command from the control unit 30 acquires the latest sensor data from the data storage unit 24a (kth). If the sensor data is acquired for the second time or later, the process proceeds to step S22-3. If the sensor data is acquired for the first time, the process proceeds to the end D as it is and waits.

  In step S22-3, the sensor data for a plurality of times are analyzed to determine whether or not the arm is stationary. If it is determined that the user is stationary, it is determined that the swipe operation has not been performed. In step S22-4, the start time Tc1 of the swipe operation detection is newly set, and in step S22-5, first. If there is calculated operation information (ΔLc (k), Lc (k) described later), it is discarded. And it progresses to the end D and waits, and when the command from the control part 30 is received after that, it will return to the start D.

  On the other hand, if it is determined that the arm is not stationary, it is determined in step 22-6 whether or not a predetermined time tc1 has elapsed from the previously set start time Tc1. If the time ta1 has not elapsed since the start time Tc1, the process proceeds to step 22-7, and the sensor data acquired in the current (k-th) step S22-1 and the previous (k-1th) step S22-1. Using the acquired sensor data, the change ΔLc (k) in the movement distance of the arm occurring during this time is calculated. Then, the process proceeds to step S22-8, where the total arm movement distance Lc (k) is calculated based on each change ΔLc (k) after the start time Tc1, and the process proceeds to step S22-9.

  In Step 22-9, it is determined whether or not the calculated total movement distance Lc (k) exceeds the movement distance reference value Lc. If the movement distance reference value Lc is exceeded, the process proceeds to step S22-10, and the movement direction of the arm is calculated. On the other hand, if the movement distance reference value Lc is not exceeded, it is determined that the motion (arm movement) being detected is not recognized as a swipe operation at the present time, and the process proceeds to the end D and waits. If there is a command to continue the swipe operation detection from 30, the process returns to start D.

  In step 22-10, when the movement direction of the arm cannot be specified as “up”, “down”, “right”, or “left”, it is determined that the detected motion (arm motion) is not a swipe operation. The process proceeds to step 22-4 via step S22-1. In step S22-4, a swipe operation detection start time Tc1 is newly set. In step S22-5, the previously calculated operation information (ΔLc (k), Lc (k)) is discarded. . And it progresses to the end D and waits, and when the command from the control part 30 is received after that, it will return to the start D.

  On the other hand, if the movement direction of the arm can be specified, the process proceeds to step S22-12 through step S22-11, and it is determined that the swipe operation in the specific direction is detected. Thereafter, in step S22-13, the previously calculated motion information (ΔLc (k), Lc (k)) is discarded, the swipe operation detection process S22 is terminated, and the process proceeds to the next step S23.

  If the time tc1 elapses from the swipe operation detection start time Tc1 while repeating the swipe operation detection, it is determined that the detected motion (arm movement) is not a swipe operation, and steps S22-6 to 22- Proceed to 4. In step S22-4, a swipe operation detection start time Tc1 is newly set. In step S22-5, the previously calculated operation information (ΔLc (k), Lc (k)) is discarded. . And it progresses to the end D and waits, and when the command from the control part 30 is received after that, it will return to the start D.

  After the swipe operation is detected in the swipe operation detection process S22, the process proceeds to step S23 as shown in FIG. Here, as shown in FIG. 7, since the dialog has two selection buttons arranged side by side, whether or not a right or left swipe operation has been detected in the previous swipe operation detection process S22 in step S23. Determine whether. For example, if a right swipe operation has been detected, the control unit 30 determines that the right “confirm” button has been selected (step S24), and the control unit 30 corresponds to the “confirm” button. Processing is performed (step S25). For example, when processing for changing the display of the GUI screen 28a is performed, the control unit 30 issues a command to the GUI control command output unit 26b, and the display unit 28 changes the display of the GUI screen 28a.

  On the other hand, if the right and left swipe operations are not detected, it is determined that the button is not properly selected, and the swipe operation detection process S22 is performed again. The above is a series of processing of the dialog stage.

  The user can switch the operation mode at an arbitrary timing by performing the above-described wrist twisting operation during the processing of the dialog stage. As shown in FIG. 8, the processing of the dialog stage is performed in the operation mode, and when a twist operation is detected during the processing, the mode management unit 26a operates according to the command of the control unit 30. The mode is switched to the normal mode, and the dialog stage processing is stopped and held in that state. Thereafter, when a twist operation is newly detected, the normal mode is switched to the operation mode in response to an instruction from the control unit 30, and the process is resumed from the held state. If the process stops during the execution of the swipe operation detection process S22, it is necessary to reset the start time Tc1, and the process is resumed from step S22-4.

  Next, a processing method of the drawer stage in the GUI stage provided to the user by the information processing method of this embodiment will be described based on FIGS. As shown in FIG. 10, the drawer stage is a stage in which a notification by the drawer is displayed on the GUI screen 28 a of the display unit 28, and the user selects what state the drawer should be in by a swipe operation.

  Assuming that the operation mode at the start time is the operation mode, as shown in FIG. 11, first, the GUI control command output unit 26b creates a predetermined GUI control command in response to a command from the control unit 30, and receives this command. The display unit 28 displays the notification by the drawer on the GUI screen 28a (step S31, FIG. 10A). Then, the sensor data analysis unit 24 performs the same swipe operation detection process S22 as described above, detects the swipe operation, and specifies the direction as “up”, “down”, “right”, or “left”.

  When the swipe operation detection process S22 ends, the process proceeds to step S32. In the drawer stage, as shown in FIGS. 10B, 10C, and 10D, the state of the drawer is selected by a swipe operation in the up, down, right, or left direction. Therefore, in step S32, it is determined whether or not a swipe operation in the up, down, right, or left direction is detected in the swipe operation detection process S22, and the process proceeds to step S33. When the swipe direction is up, the drawer is displayed (step S34, FIG. 10 (b)), and when it is down, the drawer is stored (step S35, FIG. 10 (c)). In some cases, the drawer is not displayed (step S36, FIG. 10D). The above is a series of processing of the drawer stage.

  The user can switch the operation mode at an arbitrary timing by performing the above-described wrist twisting operation during the drawer stage processing. As shown in FIG. 11, the drawer stage process is performed in the operation mode. When a twist operation is detected during the process, the mode management unit 26 a switches the operation mode according to a command from the control unit 30. Switching to the normal mode, the drawer stage processing is stopped and held in that state. Thereafter, when a twist operation is newly detected, the normal mode is switched to the operation mode in response to an instruction from the control unit 30, and the process is resumed from the held state. If the process stops during the execution of the swipe operation detection process S22, it is necessary to reset the start time Tc1, and the process is resumed from step S22-4.

  Next, the processing method of the cursor stage in the GUI stage provided to the user by the information processing method of this embodiment will be described with reference to FIGS. As shown in FIG. 12, the cursor stage is a stage where the cursor is displayed on the GUI screen 28a of the display unit 28, and the user moves the cursor to a specific position and clicks it.

  Assuming that the operation mode at the start is the operation mode, as shown in FIG. 13, first, the GUI control command output unit 26b creates a predetermined GUI control command in response to a command from the control unit 30, and receives this command. The display unit 28 displays a predetermined image and the cursor Cu on the GUI screen 28a (step S41). The predetermined image is, for example, a map A, B, C or the like. Although the control unit 30 may display the cursor Cu at an arbitrary timing, the sensor data analysis unit 24 detects specific operation information and performs the display at the timing sent to the GUI operation unit 26 as gesture operation information. Also good. For example, the logic may be such that the cursor Cu is displayed when a swipe operation in a predetermined direction (operation for switching maps A, B, and C to another map) and the twist operation described above are detected as a set. At this time, the display of the cursor Cu is in an unselected state (white arrow).

  Thereafter, the sensor data analysis unit 24 performs cursor movement detection processing S42. In the cursor movement detection process S42, first, sensor data is analyzed by the sensor data analysis unit 24, and movement operation information indicating movement in which the user moves his / her hand in the up / down / left / right direction (specific in-plane direction) is extracted. Here, since the wearable device 12 is attached to the wrist and rotates around the elbow, it is preferable to provide a gyro sensor in the motion sensor unit 12a to detect the angular velocity. For example, the sensor data analysis unit 24 shapes the time-series data of the acquired angular velocities using a technique such as a moving average method, and calculates the movement direction and movement amount (movement movement information) of the hand based on the shaped data. . When the movement operation information satisfying a specific condition is detected, the movement operation information is regarded as gesture operation information and sent to the GUI operation unit 26. Then, the GUI control command output unit 26b creates a GUI control command so that the cursor Cu moves corresponding to the movement operation information.

  After performing the cursor movement detection process S42, the control unit 30 determines whether or not the cursor Cu has stopped at a specific position for a predetermined time td (step S43). It is determined that a specific position has not been selected at the time, and the display of the cursor Cu is held in an unselected state (step S44).

  On the other hand, when it is stopped for the time td or longer, the control unit 30 determines that the stopped specific position has been temporarily selected, and as shown in FIG. 12, the display of the cursor Cu is temporarily selected (black arrow). ) And the cursor Cu is fixed at the specific position (step S45). After step S45, the same twist motion detection process S11 as described above is performed. When the twist motion is detected, the process proceeds to step S47 through step 46, and it is determined that the click is performed at the specific position, and the control unit 30 clicks. Then, the process returns to the cursor movement detection process S42. When the twist operation is not detected, the process returns to the twist operation detection process S11 again.

  Further, after the above-described step 44, when the twist motion detection process S11 is performed and the twist motion is detected, it is determined that the user desires to switch the operation mode to the normal mode, and the process goes to step S49 via step S48. Then, the cursor Cu is not displayed and the operation mode is switched to the normal mode. The above is a series of processing of the cursor stage.

  The processing of the cursor stage shown in FIG. 13 is characterized in that the movement of the wrist (twist operation) by the user is used for both the operation mode switching operation and the click operation. Since the twist operation is a simple and clear operation, it is difficult to be mistaken for another operation, and it is possible to accurately detect that the user has selected the specific position. Also, in step S45, when the temporary selection is made, the cursor Cu is fixed in position, so that even if there is a user's twisting operation (clicking operation), the cursor Cu is displaced due to the influence. There is no problem of end.

  The information processing system, the information processing method, and the information processing program of the present invention are not limited to the above embodiment. For example, in the flowchart shown in FIG. 3, the operation mode is switched when the forward and backward twists are detected by performing the twist motion detection process S11. However, the one-way twist motion is detected. In this case, the operation mode may be changed. For example, the steps related to the backward direction twist detection process can be omitted from the twist operation detection process S11 shown in FIG. 4, and only the steps related to the forward direction twist detection process can be included (FIG. 14). . Thus, if it is set as the structure which detects a one-way twist operation | movement, there exists an advantage that an information processing method and an information processing program can be made simpler. However, when it is desired to increase the detection accuracy of the twist operation, it is preferable to adopt a configuration in which the twist in the forward direction and the backward direction are detected.

  In the above embodiment, when determining whether the user's wrist movement corresponds to the twisting motion, the angular velocity and the twist angle are extracted as the pivoting motion information, and the angular velocity reference values Va and Vb and the twist angle reference value Ra are extracted. , Rb, and times ta1, ta2, tb1, so that it is preferable to mount a gyro sensor on the motion sensor unit 12a. However, even when the gyro sensor is not used, the same determination can be made by calculating the angular velocity and the twist angle from the sensor data of the acceleration sensor.

  Further, when determining whether or not the user's movement corresponds to various gesture operation information, it is free to use any parameter as the operation information indicating the user's movement. For example, in the above embodiment, when determining whether or not the user's hand movement is a swipe operation, the total movement distance of the arm is extracted as the operation information, and the determination is made using the total movement distance reference value Lc and the time tc1. Although it is configured, it may be changed to a parameter other than the total moving distance, or a parameter other than the total moving distance may be added in order to improve the determination accuracy.

  The system configuration inside the external device 14 shown in FIG. 1 is represented for convenience by using a plurality of functional blocks in order to easily understand the functions and operations of the external device 14. Therefore, the external device is not limited to the system configuration shown in FIG. 1 as long as the information processing method described above can be executed.

  Further, since the information processing system 10 is a system for realizing a graphical user interface, the display unit 28 for displaying the GUI screen 28a on the external device 14 and the GUI operation unit 26 for creating a GUI control command. Is provided. However, the configuration of detecting the twist operation and switching the operation mode can be applied to a system other than the graphical user interface. In this case, the display unit 28 and the GUI operation unit 26 may be omitted. it can.

DESCRIPTION OF SYMBOLS 10 Information processing system 12 Wearable device 12a Motion sensor part 16 External device 28 Display part 28a GUI screen S11 Twist action detection process S11-2 Forward direction twist detection process (twist action detection process)
S11-6 Reverse direction twist detection processing (twist motion detection processing)
Cu cursor

Claims (13)

In an information processing system that can be operated with user gestures,
The user is mounted on a body part that can be rotated by himself / herself, and the motion of the body part is detected by a motion sensor unit including a sensor element including at least one of an acceleration sensor and a gyro sensor, and sensor data A wearable device that outputs
The sensor data output by the wearable device is acquired, and the user's motion information is extracted by analyzing the sensor data. The motion information that satisfies a specific condition is regarded as gesture operation information. A device that performs various types of processing, and an operation mode capable of executing processing corresponding to the gesture operation information and a normal mode in which processing corresponding to the gesture operation information is stopped are provided as operation modes. With external devices,
By analyzing the sensor data, the external device extracts axial movement information indicating the movement of the user by rotating the body part. When the axial movement information satisfies a specific condition, the external device is twisted. Means for determining that the motion has been detected;
When the external device detects the twist operation, the external device switches the current operation mode to the other operation mode.   The information processing system according to claim 1, wherein the pivoting motion information is information indicating a series of movements in which a user pivots the body part in a forward direction and a backward direction.   The information processing system according to claim 1, wherein the external device is provided with a display unit that displays a GUI screen, and the wearable device is used as a pointing device for operating the GUI screen. In an information processing method executed by an information processing system operable with a user's gesture,
The information processing system includes:
The user is mounted on a body part that can be rotated by himself / herself, and the motion of the body part is detected by a motion sensor unit including a sensor element including at least one of an acceleration sensor and a gyro sensor, and sensor data A wearable device that outputs
The sensor data output by the wearable device is acquired, and the user's motion information is extracted by analyzing the sensor data. The motion information that satisfies a specific condition is regarded as gesture operation information. A device that performs various types of processing, and an operation mode capable of executing processing corresponding to the gesture operation information and a normal mode in which processing corresponding to the gesture operation information is stopped are provided as operation modes. Consisting of external devices,
By analyzing the sensor data, it is determined that the user has extracted the pivoting motion information indicating the motion of pivoting the body part, and the twisting motion is detected when the pivotal motion information satisfies a specific condition. And switching the current operation mode to the other operation mode. A twist motion detection process for determining whether or not the twist motion has been detected is provided,
The twist motion detection process includes
Each time the sensor data is acquired, the sensor data is analyzed to calculate an angular velocity in the axial direction of the body part; and
Each time the angular velocity is calculated, the angular velocity is compared with a predetermined angular velocity reference value, and a timing at which the angular velocity becomes equal to or higher than the angular velocity reference value is set as a twist detection start time of the body part;
Every time the angular velocity is calculated, using the angular velocity calculated last time and the angular velocity calculated this time, calculating a change in the twist angle of the body part;
Every time the change in the twist angle is calculated, the change in the twist angle calculated after the start time is summed to calculate the twist angle of the body part, and the twist angle and a predetermined twist angle reference value are calculated. A step of comparing
The information processing method according to claim 4, wherein when the twist angle exceeds a predetermined twist angle reference value, it is determined that the twist operation is detected.   6. The information processing method according to claim 5, wherein the step of calculating an angular velocity in the axial direction of the body part calculates an angular velocity of an operation of rotating a wrist around a longitudinal axis of the user's arm.   The information of the calculated change of the twist angle and the twist angle is discarded while the setting of the start time is reset when the twist operation is not detected even after a lapse of a certain time from the start time. Information processing method.   The information processing method according to claim 4, wherein the pivoting motion information is information indicating a series of movements in which a user pivots the body part in a forward direction and a backward direction. There is provided a twist motion detection process executed to detect the twist motion,
The twist motion detection process includes
Each time the sensor data is acquired, the sensor data is analyzed to calculate an angular velocity in the axial direction of the body part; and
Each time the angular velocity is calculated, the angular velocity is compared with a predetermined angular velocity reference value, and a timing at which the angular velocity becomes equal to or higher than the angular velocity reference value is set as a twist detection start time of the body part;
Every time the angular velocity is calculated, using the angular velocity calculated last time and the angular velocity calculated this time, calculating a change in the twist angle of the body part;
Each time the change in the twist angle is calculated, the change in the twist angle calculated after the start time is summed to calculate the twist angle of the body part, and the twist angle and a predetermined twist angle reference value are calculated. A step of comparing
First, the twist movement detection process is performed for the forward axial movement, and when the forward twist angle exceeds a predetermined twist angle reference value, it is determined that the forward twist is detected,
After determining that the forward twist is detected, the twist motion detection process is executed for the backward axial rotation within a predetermined time, and the forward twist angle exceeds a predetermined twist angle reference value. 9. The information processing method according to claim 8, wherein it is determined that the twist in the backward direction is detected and the twist operation is detected.   When a forward or backward twist is not detected even after a lapse of a certain time from the start time, the setting of the start time is reset, and the calculated change in the twist angle and the information on the twist angle are discarded. Item 10. The information processing method according to Item 9. The external device has a display unit for displaying a GUI screen, and the wearable device is used as a pointing device for operating the GUI screen.
A cursor stage is provided for the user to move the cursor displayed on the GUI screen.
The cursor stage is
Displaying a cursor on the GUI screen when detecting specific gesture operation information of the user;
By analyzing the sensor data, the movement operation information indicating the movement of the user moving the body part in a specific in-plane direction is extracted, and the movement operation information satisfying a specific condition is referred to as the gesture operation information. The information processing method according to claim 4, further comprising a step of moving the display position of the cursor corresponding to the movement operation information. The external device has a display unit for displaying a GUI screen, and the wearable device is used as a pointing device for operating the GUI screen.
A cursor stage is provided for the user to operate the cursor displayed on the GUI screen,
The cursor stage is
Displaying a cursor on the GUI screen when detecting specific gesture operation information of the user;
Analyzing the sensor data to determine that the specific position has been temporarily selected when the cursor has stopped at a specific position for a certain period of time, and fixing the cursor to the specific position;
12. A step comprising: determining that the twisting motion has been detected in a state where the cursor is fixed at a specific position, and determining that the user has clicked at the specific position and performing processing corresponding to the click. Any of the information processing methods of description.   An information processing program comprising a program for each processing step for causing an information processing system to execute the information processing method according to claim 4.

Images