TW202414164A - Information processing method, program, and information processing device - Google Patents

Abstract

This information processing device performs: sequential acquisition, from a sensor that is capable of measuring an action of a prescribed part of a user, of data relating to the action of the prescribed part at a plurality of time points; determination, based on the data, of whether or not the action of the prescribed part is a mode switching action, when in a non-movement mode; switching from the non-movement mode to a movement mode when it is determined that the action of the prescribed part is the mode switching action; outputting movement information relating to a pointer on the basis of the data when in the movement mode; and switching from the movement mode to the non-movement mode when a prescribed condition has been satisfied after the output of the movement information.

Description

Information processing method, program, and information processing device

The present invention relates to an information processing method, a program and an information processing device.

Previously, it is known that a sensor that detects the movement of the user's head is used to control input operations such as click operations on input mechanisms such as a mouse (for example, refer to patent documents 1 and 2). [Prior art document] [Patent document]

[Patent document 1] Japanese Patent Publication No. 4-309996 [Patent document 2] Japanese Patent Publication No. 9-258887

[The problem that the invention wants to solve]

However, in the prior art, when input operations are performed using the movement of the head, the pointer always moves, or unintentional operations are performed due to the natural movement of the head, which causes problems in usability.

Therefore, the purpose of the disclosed technology is to more appropriately implement the user's intended operation and improve usability when performing input operations by moving a specified part where a sensor is installed. [Technical means for solving the problem]

In an information processing method of one aspect of the disclosed technology, an information processing device performs: sequentially obtaining data related to the movement of the specified part at multiple time points from a sensor capable of measuring the movement of the specified part of the user; in a non-movement mode, determining whether the movement of the specified part is a mode switching movement based on the data; switching from the non-movement mode to the movement mode when it is determined that the movement of the specified part is a mode switching movement; in the movement mode, outputting movement information related to the pointer based on the data; and switching from the movement mode to the non-movement mode when a specified condition is satisfied after the movement information is output. [Effect of the invention]

According to the disclosed technology, when the movement of a specified part is measured by a sensor and an input operation is performed, the operation intended by the user can be more appropriately implemented, thereby improving usability.

Hereinafter, the embodiments of the present invention will be described with reference to the drawings. However, the embodiments described below are merely illustrative, and there is no intention to exclude the applicability of various modifications or technologies not expressly described below. That is, the present invention can be implemented with various modifications within the scope of its purpose. Furthermore, in the following drawings, the same or similar parts are represented by the same or similar symbols. The drawings are schematic drawings and may not necessarily be consistent with the actual dimensions or ratios. Sometimes, the drawings contain parts with different dimensional relationships or ratios.

[Example] In the example, glasses are taken as an example of a wearable terminal equipped with an acceleration sensor and/or an angular velocity sensor and, if necessary, a bioelectrode, but the present invention is not limited thereto. FIG. 1 is a diagram showing an example of an information processing system 1 of the example. The information processing system 1 shown in FIG. 1 includes an external information processing device (hereinafter also referred to as an "external device") 10 and glasses 30. The external device 10 and the glasses 30 are connected via a network and can perform data communication.

The glasses 30 are equipped with the information processing device 20, for example, at the nose bridge portion. The information processing device 20 may include a pair of nose pads and a nose bridge portion, each of which has a bioelectrode 32, 34, 36. The information processing device 20 includes a three-axis acceleration sensor and/or a three-axis angular velocity sensor (or a six-axis sensor). Furthermore, the bioelectrodes 32, 34, 36 are not necessarily required.

The information processing device 20 detects sensor signals or eye potential signals and transmits them to the external device 10. The information processing device 20 does not necessarily need to be placed at the nose bridge, as long as it is positioned at a position where the sensor signals or eye potential signals can be obtained when the glasses 30 are worn. In addition, the information processing device 20 can be detachably placed at the nose bridge.

The external device 10 is an information processing device with a communication function. For example, the external device 10 is a server device, a personal computer, a tablet terminal, a portable terminal such as a smart phone, etc. The external device 10 obtains data related to the movement of the user's head received from the information processing device 20, and performs a plurality of operation processes based on the data. The operation processes include, for example, pointer movement, clicking, dragging, scrolling, etc. In addition, the external device 10 can receive instructions for command operation processes or data indicating the movement amount of the pointer from the information processing device 20 corresponding to the movement of the head.

<Structure of external device 10> FIG. 2 is a block diagram showing an example of the structure of the external device 10 of the embodiment. The external device 10 includes: one or more processing devices (CPU) 110, one or more network communication interfaces 120, a memory 130, a user interface 150, a camera sensor 160, and one or more communication buses 170 for connecting these components to each other.

The network communication interface 120 is connected to the network via a mobile communication antenna or a wireless LAN communication antenna, and can perform data communication with the information processing device 20.

The user interface 150 may include a display device and an input device (keyboard and/or mouse, or any other pointing device, etc.). The user interface 150 may move a pointer displayed on the display device. The camera sensor 160 is an image sensor that receives a light signal, converts the light signal into an electrical signal, and generates an image. The image includes at least one of a still image and an animated image.

The memory 130 is, for example, a high-speed random access memory such as DRAM, SRAM, or other random access solid state memory devices. It can also be one or more magnetic disk memory devices, optical disk memory devices, flash memory devices, or other non-volatile solid state memory devices. It can also be a non-temporary recording medium that can be read by a computer.

The memory 130 stores data used by the information processing system 1. For example, the memory 130 stores data sent from the glasses 30 (information processing device 20).

As another example of the memory 130, one or more memory devices may be remotely located from the CPU 110. In one embodiment, the memory 130 stores programs, modules, and data structures executed by the CPU 110, or a subset thereof.

The CPU 110 configures the acquisition unit 112 and the processing control unit 114 by executing the program stored in the memory 130 .

The acquisition unit 112 acquires commands related to the operation or data indicating the movement amount of the pointer from the information processing device 20 of the glasses 30.

The processing control unit 113 executes the operation-related instructions acquired by the acquisition unit 112, or controls the movement of the pointer using the movement amount of the pointer. For example, the processing control unit 113 controls the operation of clicking or dragging using the instructions or data acquired from the glasses 30 having the same function as an input device such as a pointing device (e.g., a mouse).

Furthermore, the acquisition unit 112 of the CPU 110 can also sequentially acquire data related to the movement speed of the specified part from a sensor that can measure the movement of the specified part of the user (for example, a sensor installed at the specified part of the user). In this case, the processing control unit 113 can execute the mode control processing, determination processing, or output processing of the information processing device 20 described later, and control operations such as clicking or dragging. Furthermore, in addition to the mouse operations such as clicking, dragging, and scrolling, the processing of the processing control unit 113 can also establish a correspondence between the operation of the function key, the numeric key, or the setting change and the movement of the user's head.

<Configuration of information processing device 20> FIG. 3 is a block diagram showing an example of the configuration of the information processing device 20 of the embodiment. As shown in FIG. 3, the information processing device 20 has: a processing unit 202, a transmission unit 204, a six-axis sensor 206, a power supply unit 208, and each bioelectrode 32, 34, 36. In addition, each bioelectrode 32, 34, 36 is connected to the processing unit 202 using a wire, for example, via an amplifier. Furthermore, each bioelectrode 32, 34, 36 is not necessarily a necessary configuration. In addition, the information processing device 20 has a memory for storing processing data. The memory can be a non-temporary recording medium that can be read by a computer.

The sending unit 204 sends, for example, data related to the moving speed packetized by the processing unit 202, moving information, non-moving information, or a command to the external device 10. For example, the sending unit 204 sends the non-moving information, moving information, and a command to the external device 10 through wireless communication such as Bluetooth (registered trademark) and wireless LAN, or wired communication.

The six-axis sensor 206 is a three-axis acceleration sensor and a three-axis angular velocity sensor. In addition, each of these sensors can be provided individually, and any one of the sensors can be provided. The six-axis sensor 206 outputs the detected sensor signal to the processing unit 202. The power supply unit 208 supplies power to the processing unit 202, the transmission unit 204, the six-axis sensor 206, etc.

The processing unit 202 includes a processor, and processes the sensor signal obtained from the six-axis sensor 206 or the eye potential signal obtained from each bioelectrode 32, 34, 36 as needed, for example, the sensor signal or the eye potential signal can be packetized and the packet can be output to the sending unit 204. In addition, the processing unit 202 can also only amplify the sensor signal obtained from the six-axis sensor 206.

Furthermore, the processing unit 202 can generate the sensor signal from the six-axis sensor 206 as movement speed data based on information related to the movement of the head. The information related to the movement of the head is, for example, information related to the movement of the head up and down (forward and backward), left and right. The following describes an example in which the processing unit 202 has a processor and executes processing corresponding to the input device.

Fig. 4 is a block diagram showing an example of the structure of the processing unit 202 of the embodiment. In the example shown in Fig. 4, the processing unit 202 includes an acquisition unit 222, a mode control unit 224, and an output unit 228.

The acquisition unit 222 sequentially acquires data related to the moving speed of the specified part at multiple time points from a sensor that can measure the movement of the specified part of the user. For example, the acquisition unit 112 sequentially acquires angular velocity data sent from an angular velocity sensor that samples at a specified sampling frequency. Acquiring data from the sensor includes acquiring data directly or indirectly. The specified part of the embodiment is the head, but it can also be an arm or a foot, etc. The sensor is a sensor that can measure the moving speed of the specified part. In the embodiment, it is preferably an angular velocity sensor or an acceleration sensor that can be installed at the specified part of the user. For example, in the case where the sensor is a three-axis angular velocity sensor, the data related to the moving speed includes data of the angular velocity vector of the direction.

The mode control unit 224 controls the state transition between the movement mode for controlling the movement of the pointer of the external device 10 and the non-movement mode for not moving the pointer based on the data related to the movement speed. For example, the mode control unit 224 includes a determination unit 226 for performing various determinations.

For example, when the current state is the non-movement mode, the determination unit 226 of the mode control unit 224 determines whether the user's movement of the specified part is a mode switching movement based on the data related to the movement speed obtained in sequence. As a specific example, the mode switching movement is one or more preset movements of the specified part, and can also be a movement set by the user.

Furthermore, when the determination unit 226 determines that the movement of the specified part of the user is a mode switching movement, the mode control unit 224 switches from the non-movement mode to the movement mode. For example, when the determination unit 226 determines that the movement direction and/or movement amount of the specified part indicated by the data related to the movement speed obtained in sequence represents one of one or more mode switching movements, the mode control unit 224 switches from the non-movement mode to the movement mode. Regarding the determination of the mode switching movement, for example, it can be determined whether it is a movement with a threshold value or more in a specified direction within a specified time, or whether there are multiple combinations of arbitrary movements.

In addition, the threshold value may be preset to a predetermined threshold value capable of detecting sudden movement. For example, when the predetermined part is the head, the threshold value may be set to exceed the situation of rapidly shaking the head in the lateral direction or the longitudinal direction, or when the predetermined part is the arm or the foot, the threshold value may be set to exceed the situation of rapidly moving the arm or the foot. Furthermore, the threshold value corresponding to each direction may be set in a different manner in accordance with the movement of each.

The output unit 228 outputs the data processed by the processing unit 202 when the power of the information processing device 20 is turned on. The output data is sent from the sending unit 204 to the external device 10 via the network or Bluetooth (registered trademark).

For example, when the output unit 228 is in the non-movement mode, it outputs non-movement information for not moving the pointer on the external device 10. For example, the non-movement information indicates 0 (0, 0) for the movement amount of the X coordinate and the Y coordinate in order not to move the pointer regardless of the data related to the movement speed acquired sequentially.

Furthermore, when the current state is the moving mode, the output unit 228 outputs the moving information related to the pointer based on the data related to the moving speed obtained in sequence. For example, the output unit 228 outputs the information such as the moving amount (X1, Y1) of the X coordinate and the Y coordinate in order to move the pointer based on the data related to the moving speed obtained in sequence.

The mode control unit 224 switches from the moving mode to the non-moving mode when the moving mode is currently in the moving mode and a predetermined condition is satisfied after the moving information is output. The predetermined condition includes, for example, the moving amount indicated by the data related to the moving speed or a condition related to the action.

Through the above processing, when the sensor measures the movement of the specified part and performs input operation, the user's unintentional movement of the pointer can be prevented, and the user's intended operation can be more appropriately implemented, thereby improving usability. In addition, according to the above processing, when the user does not move the pointer, by introducing a non-movement mode that is unrelated to the movement of the specified part, when the user does not intend to perform an operation, the pointer can be prevented from moving in coordination with the movement of the specified part. In addition, the external device 10 only needs to move the pointer according to the non-movement information, movement information, instructions, etc. sent from the information processing device 20, just like when operating a mouse or a pointing device. Therefore, there is no need to add a special program on the external device 10 side, and it can be applied to the existing external device 10. In addition, as a result, the limited processing resources of the external device 10 can be effectively utilized.

Furthermore, the mode control unit 224 can set the non-movement mode as the starting mode when starting the control of the pointer movement using the data related to the movement speed. This prevents the pointer from moving immediately in coordination with the movement of the specified part after the power of the information processing device 20 is turned on, and can provide the user with the operating feeling that the pointer moves only when attempting to use it, such as a mouse.

Furthermore, the mode control unit 224 may switch from the non-movement mode to the movement mode, including switching from the non-movement mode to the first state of the movement mode when the movement of the specified part of the user is the first mode switching movement. For example, the mode control unit 224 switches from the non-movement mode to the first state of the plurality of movement modes when it is determined that the movement of the specified part is the first mode switching movement based on the movement direction and movement amount indicated by the data related to the movement speed obtained in sequence.

For example, when the first mode switching action is a left-right movement, the determination unit 226 determines whether there is a certain movement with a leftward movement amount exceeding a critical value and a rightward movement amount exceeding a critical value within a predetermined time based on the data related to the movement speed obtained in sequence. If the determination unit 226 detects such movement, it determines that the series of actions of the user's predetermined part is the first mode switching action of the processing from the pointer movement to the issuance of the command.

Furthermore, the output unit 228 may also include a case where, when the movement mode is in the first state, after the movement information is output, the pointer stops moving and then outputs a command corresponding to the first mode switching action. For example, the first state is the waiting state for clicking, and the command corresponding to the first mode switching action is left clicking.

Through the above processing, after the switching action from the non-moving mode to the moving mode, the pointer is only moved and stopped at the target position, and the left-click instruction can be output. That is, the user does not need to perform both the mode switching action and the instruction determination action. For the frequently used left click, a series of actions from the movement of the pointer to the left click can be performed through a series of actions from the mode switching action. In addition, by reducing the processing of the instruction determination, the processing load of the information processing device 20 can be reduced. Furthermore, the instruction corresponding to the first mode switching action is not limited to the left click, and can be changed to an instruction frequently used by the user.

Furthermore, the mode control unit 224 may switch from the non-movement mode to the movement mode, including switching from the non-movement mode to the second state of the movement mode when the movement of the specified part is the second mode switching movement. For example, the mode control unit 224 switches from the non-movement mode to the second state of the plurality of movement modes when it is determined that the movement of the specified part is the second mode switching movement based on the movement direction and movement amount indicated by the sequentially acquired data related to the movement speed.

For example, when the second mode switching action is right-left movement, the determination unit 226 determines whether there is a movement with a rightward movement amount greater than a critical value and a leftward movement amount greater than a critical value within a predetermined time based on the data related to the movement speed obtained in sequence. If the determination unit 226 detects the movement, it determines that the movement of the specified part of the user is the second mode switching action of the action determination processing determined by the instruction after the pointer moves.

Furthermore, when the movement mode is in the second state, the determination unit 226 determines whether the movement of the specified part is a commanded movement based on the data related to the movement speed after the movement of the pointer stops after the movement information is output. For example, when a plurality of commanded movements are set, the determination unit 226 determines whether it corresponds to one of the plurality of commanded movements based on the data related to the movement speed obtained in sequence.

Furthermore, the output unit 228 may output a command corresponding to the command action when the determination unit 226 determines that the command action is a command action. For example, when the command action is a left-move action, a left-click command is output; when the command action is a right-move action, a right-click command is output; when the command action is an upward-move action, a double-click command is output; and when the command action is a downward-move action, a drag start command is output.

Through the above processing, for example, by moving the pointer and temporarily stopping it on the target icon and then instantly moving the head to the right, a command to right-click on the icon is output, and a menu display of the icon can be opened. That is, commands can be set by combining multiple actions, and the variation of commands can be increased.

Furthermore, the prescribed condition for the transition from the moving mode to the non-moving mode may include the case where a command is outputted by the output unit 228. For example, when the mode control unit 224 detects that a command is outputted from the output unit 228, it controls the transition from the moving mode to the non-moving mode.

Through the above processing, it is possible to automatically switch to the non-movement mode after the command is output, thereby preventing the pointer from moving unnecessarily after the input operation is completed.

Furthermore, the output unit 228 may also include outputting movement information for causing the pointer to move corresponding to the mode switching action when switching from the non-movement mode to the movement mode. For example, the output unit 228 may replace the movement information with a predetermined time after the mode switching, output movement information of the pointer corresponding to the preset mode switching action, and then output movement information based on the movement speed data.

Through the above processing, the user can confirm whether the movement of the specified part is the intended movement by understanding the movement of the pointer.

Furthermore, the above-mentioned movement information may include movement determined independently of the movement of the user's specified part. For example, if the movement of the specified part is left-right-left-right, the movement information may be information that a circle is displayed three times.

Through the above processing, the user can master the simple movement of the pointer and confirm whether his or her complex movement can be input correctly.

Furthermore, when the current mode is the non-movement mode, the determination unit 226 can determine whether the user's movement of the specified part is a command movement not accompanied by the movement of the pointer based on the data related to the movement speed obtained in sequence.

In this case, when the output unit 228 determines that the movement of the specified part of the user is the above-mentioned command movement, it can output the command corresponding to the command movement.

Through the above processing, when in the non-movement mode, the user performs a direct command action, thereby releasing the non-movement mode, and outputting a direct command for a command that does not require the movement of the pointer (such as a rolling action, etc.).

<Data Example> FIG. 5 is a diagram showing an example of data related to the moving speed of the embodiment. The data shown in FIG. 5 represents data obtained by a predetermined sampling frequency (e.g., 20 Hz = 0.05 seconds). The data shown in FIG. 5 is, for example, stored in a memory in the information processing device 20. Furthermore, as data related to the moving speed, data related to the moving speed of the line of sight based on the eye potential signal can also be stored.

<Specific example> Figure 6 is a diagram showing an example of the relationship between data, movement, and output of an embodiment. In the example shown in Figure 6, the power of the information processing device 20 is turned on, and the external device 10 and the information processing device 20 are controlled to a non-moving mode after pairing by Bluetooth (registered trademark) or the like. That is, after pairing, the user's specified part moves, but the mode of the processing unit 202 is a non-moving mode, so non-moving information is output. In this way, the external device 10 obtains the non-moving information without moving the pointer.

Next, when the predetermined part moves left and right within the predetermined time of 0.85 seconds, for example, the determination unit 226 determines that it is the first mode switching action by sequentially acquiring data related to the movement speed indicating the movement left and right. The mode control unit 224 changes from the non-movement mode to the first state of the movement mode. At this time, the output unit 228 can output non-movement information, and in the case where there is a delay from the acquisition of data to the output, it can output movement information of a series of left and right actions. In addition, the output unit 228 can output movement information indicating the command.

Then, the prescribed part moves until 2.65 seconds, and the user moves the prescribed part so that the pointer is located at the target position set by the user. In the first state, the sensor detects the movement, the acquisition unit 222 acquires data related to the movement speed, and the output unit 228 outputs movement information based on the data.

Then, when the predetermined position stops for a predetermined time (e.g., between 2.70 and 2.80), the data related to the moving speed obtained in sequence indicates that the moving direction and the moving amount are substantially 0. Thus, the determination unit 226 determines that it stops, and the output unit 228 outputs a left click instruction corresponding to the first mode switching action.

FIG. 7 is a diagram showing an example of the relationship between the movement of the specified part and the mode and instruction of the embodiment. For example, in the state of the non-movement mode, the subsequent instructions for the movement of the specified part are as follows. Furthermore, after the instruction is issued, the state returns to the non-movement mode. Action of the specified part: subsequent commands Left-right (action to switch to move mode): pointer movement → stop → left click Right-left (action to switch to move mode): pointer movement → stop → click selection (left: left click, right: right click, up: double click, down: drag to start) Left-right-left or right-left-left (stop action): stop mode Vertical reciprocation 3 times (setting action): setting mode → mode selection (right: setting change mode, left: setting display mode) Up-down (direct command action): scroll up once Down-up (direct command action): scroll down once Up-down-up (direct command action): continuous scroll up → up or down → scroll stop Down-up-down (direct command action): continuous scroll down → up or down → scroll stop

As described above, it is possible to switch from the non-movement mode to the pointer movement mode (pointer movement) or the direct command issuing mode. For example, the scroll wheel of a mouse issues commands without accompanying the movement of the pointer. Since it is set to have the same function as the scroll wheel, if it is determined that there is a direct command action corresponding to the scroll command, the direct command is issued without accompanying the movement of the pointer.

Furthermore, in the stop mode, the pointer does not move and no command is issued unless the action to cancel the stop mode is set. For example, in order to prevent unintended operations from being performed by moving a specified part in sync with the other party during a meeting, a mode is set to not accept actions other than the action to cancel the stop mode. In this way, commands for actions intended by the user can be issued more appropriately.

FIG8 is a diagram showing an example of the relationship between the movement and instruction of the embodiment and the movement of the pointer. The movement of the pointer indicates the movement of the pointer set in accordance with the action or instruction of the specified part after the mode is switched. In the example shown in FIG8, the relationships are as follows. Movement shown by the pointer: Instruction: Movement of the pointer Left and right: Movement: Left and right Right and left: Movement: Right and left Left and right: Stop mode: Circle 3 times Longitudinal 3 times reciprocating: Set mode: Circle 2 times Up and down: Roll up 1 time: Up and down Down and up: Roll down 1 time: Down and up

In the case where the instruction is a movement, the movement shown in the data is the same as the movement of the pointer. In order to emphasize that the instruction is recognized, the movement of the pointer can be emphasized. The output unit 228 can output movement information having a faster movement or a larger movement than the movement shown in the actual data as an emphasis process. Since the movement shown in the data is the same as the movement of the pointer, the user can intuitively understand that the action is properly recognized.

Furthermore, when the command is a stop mode, etc., since there is no movement command, the pointer moves independently of the movement indicated by the data. In this case, by setting a simple movement that the specified part does not normally make, the user can understand whether the movement of the specified part is properly recognized.

<Action> Figures 9 and 10 are flowcharts showing an example of processing related to instruction issuance in the embodiment. The processing shown in Figures 9 and 10 is an example of processing performed by the information processing device 20.

In step S102 shown in Fig. 9, the acquisition unit 222 sequentially acquires data related to the moving speed of the specified part from the sensor that can measure the movement of the specified part of the user. The sensor is, for example, an angular velocity sensor installed on the glasses 30, the specified part is, for example, the user's head, and the data related to the moving speed is, for example, angular velocity data.

In step S104, the mode control unit 224 determines whether the current state is the non-mobile mode or the mobile mode. For example, the mode control unit 224 sets the non-mobile mode as a default, and sets the mobile mode after detecting the mode switching action until the specified conditions are met. Here, if the current state is the non-mobile mode (step S104-Yes), the processing proceeds to step S106, and if the current state is the mobile mode (step S104-No), it proceeds to step S122 shown in Figure 10.

In step S106, the determination unit 226 determines whether the sequentially acquired data related to the moving speed indicates one or more preset mode switching actions. If the data indicates a mode switching action (step S106-Yes), the process proceeds to step S114, and if the data does not indicate a mode switching action (step S106-No), the process proceeds to step S108.

In step S108, the determination unit 226 determines whether the data related to the movement speed obtained in step S106 is a command action not accompanied by pointer movement (direct command action). If the data indicates a direct command action (step S108-Yes), the processing proceeds to step S110, and if the data does not indicate a direct command action (step S108-No), the processing proceeds to step S112.

In step S110, the output unit 228 outputs a command corresponding to the direct command action.

In step S112, the output unit 228 outputs non-movement information for not moving the pointer of the external device 10, for example, information indicating that the movement amount is (0, 0).

In step S114, the mode control unit 224 switches from the non-moving mode to the moving mode.

In step S116, the determination unit 226 determines whether the data related to the moving speed obtained in step S106 indicates the first mode switching action. If the data indicates the first mode switching action (step S116-Yes), the processing proceeds to step S118, and if the data does not indicate the first mode switching action (step S118-No), the processing proceeds to step S120. In other words, if it is determined to be the first mode switching action, the processing proceeds to step S118, and if it is determined to be the second mode switching action, the processing proceeds to step S120.

In step S118, the mode control unit 224 is set to the first state of issuing a command after the pointer stops.

In step S120, the mode control unit 224 is set to the second state of waiting for an action corresponding to a prescribed command.

In step S122 shown in FIG10 , i.e., in the moving mode, the determination unit 226 determines whether the data related to the moving speed obtained in sequence indicates the stop action of the pointer. If the data indicates the stop action of the pointer (step S122-Yes), the processing proceeds to step S126, and if the data does not indicate the stop action of the pointer (step S122-No), the processing proceeds to step S124.

In step S124, the output unit 228 outputs movement information indicating the movement amount of the pointer based on the data related to the movement speed obtained sequentially.

In step S126, the mode control unit 224 determines whether the current state is the first state. If the current state is the first state (step S126-Yes), the process proceeds to step S128, and if the current state is the second state (step S126-No), the process proceeds to step S130.

In step S128, the output unit 228 outputs a command corresponding to the command action corresponding to the first mode switching action. In addition, the mode control unit 224 switches the movement mode to the non-movement mode after the command output.

In step S130, the output unit 228 outputs a command corresponding to a predetermined command action after the pointer stops. In addition, the mode control unit 224 switches the movement mode to the non-movement mode after the command is output.

Through the above processing, when the input operation is performed using the movement of the specified part where the sensor is installed, the operation intended by the user can be more appropriately implemented, thereby improving usability.

Furthermore, when the mode switching action or the command action is determined by the determination unit 226, the movement information of the pointer corresponding to the mode switching action or the command action can be output by the output unit 228. Through this processing, the user can confirm whether the action of the specified part is more accurately recognized by the information processing device 20.

Furthermore, the processing steps included in the processing flow illustrated in FIG. 9 can be arbitrarily changed in order or executed in parallel within the range that no contradiction occurs in the processing content, and other steps can be added between the processing steps. In addition, a step recorded as one step for the sake of ease of understanding can be divided into multiple steps and executed, and on the other hand, a step recorded as multiple steps for the sake of ease of understanding can be grasped as one step.

As described above, according to the embodiment, when the sensor measures the movement of the specified part and performs the input operation, the user's intended operation can be more appropriately implemented, thereby improving usability. In addition, according to the embodiment, when the user attempts to move the pointer or attempts to perform the specified operation using an input device such as a mouse, an interface can be provided for the user to recognize the intended movement or perform the input operation.

Furthermore, in the above-mentioned embodiment, the movement of the user's head, arm or foot is cited as an example of the movement of the predetermined part, but the movement of the line of sight of the eyeball may also be used. In this case, the movement of the eyeball is obtained based on the sensor signal detected from each bioelectrode (an example of a sensor), and the processing unit 202 can control the processing based on the data related to the movement.

Furthermore, in the embodiment, the sensor signal from the six-axis sensor mounted on the glasses 30 is used for explanation, but as mentioned above, the six-axis sensor is not only mounted on the head, but can also be mounted at any position on the human body.

Furthermore, in the embodiment, it is described that the data of the three-axis acceleration sensor can also be used for the movement speed data. In this case, after the acceleration is detected, it can be converted into speed through a prescribed operation process, and the converted speed data can be used.

The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. A person skilled in the art should clearly understand that various changes or improvements can be made to the above embodiments. Such changes or improvements are also included in the technical scope of the present invention, which is clearly stated in the scope of the patent application.

<Variation 1> In the above-mentioned embodiment, the following example is described: the information processing device 20 on the side of the glasses 30 functions as an interface such as a mouse, and the information processing device 20 outputs operation commands such as left click, right click, and scrolling to the external device 10. On the other hand, the information processing device 20 can also amplify and packetize the data related to the movement speed obtained in sequence and send it to the external device 10. In this case, the external device 10 can have the function of the above-mentioned processing unit 202, and output the determined command to the OS (Operating System) etc. to control the movement or operation of the pointer on the screen.

<Variation 2> In variation 2, the processing contents described above in the embodiment can be realized by an information processing device formed by integrating the information processing device 20 and a part of the external device 10. For example, the head mounted display may have a processing unit 202 of the information processing device 20 and a display 151 of the external device 10, and the image displayed on the display 151 is superimposed with a pointer corresponding to the movement of the user's head, so as to perform the input operation described above.

<Variation 3> In variation 3, the imaging sensor 160 is used as a sensor that can measure the movement of a specified part of the user. In the case of variation 3, the external device 10 is the above-mentioned information processing device, and the CPU 110 has a processing unit 202. As a specific example, the movement of the specified part (head, etc.) of the user is photographed by the live view mode of the imaging sensor 160, and the processing unit 202 obtains data related to the movement of the specified part identified by image recognition. The subsequent processing is the same as the processing of the embodiment. In this way, by using a personal computer with a camera, the above-mentioned non-moving mode can be introduced to more appropriately realize the operation intended by the user, thereby improving usability.

1: Information processing system 10: Information processing device (external device) 20: Information processing device 30: Glasses 32,34,36: Bioelectrodes 110: Processing device (CPU) 112: Acquisition unit 113: Processing control unit 120: Network communication interface 130: Memory 150: User interface 151: Display 160: Image sensor 170: Communication bus 202: Processing unit 204: Transmission unit 206: Six-axis sensor 208: Power supply unit 222: Acquisition unit 224: Mode control unit 226: Judgment unit 228: Output unit S102~S120,S122~S130: Steps

FIG. 1 is a diagram showing an example of an information processing system of an embodiment. FIG. 2 is a block diagram showing an example of the configuration of an external information processing device of an embodiment. FIG. 3 is a block diagram showing an example of the configuration of an information processing device of an embodiment. FIG. 4 is a block diagram showing an example of the configuration of a processing unit of an embodiment. FIG. 5 is a diagram showing an example of data related to movement speed of an embodiment. FIG. 6 is a diagram showing an example of the relationship between data, movement, and output of an embodiment. FIG. 7 is a diagram showing an example of the relationship between movement of a specified part and a mode and an instruction of an embodiment. FIG. 8 is a diagram showing an example of the relationship between movement and an instruction and movement of a pointer of an embodiment. FIG. 9 is a flow chart showing an example (part 1) of processing related to instruction issuance of an embodiment. FIG10 is a flow chart showing an example (part 2) of processing related to instruction issuance in an embodiment.

1: Information processing system

10: Information processing device (external device)

20: Information processing device

30: Glasses

32,34,36: Bioelectrodes

Claims (9)

An information processing method is performed by an information processing device: From a sensor capable of measuring the movement of a specified part of a user, sequentially obtain data related to the movement of the specified part at a plurality of time points; In a non-movement mode, determine whether the movement of the specified part is a mode switching movement based on the data; When the movement of the specified part is determined to be a mode switching movement, switch from the non-movement mode to the movement mode; In the movement mode, output movement information related to a pointer based on the data; When a specified condition is met after the movement information is output, switch from the movement mode to the non-movement mode. The information processing method of claim 1, wherein the switching from the non-moving mode to the moving mode comprises: When the action of the specified part is the first mode switching action, the switching from the non-moving mode to the first state of the moving mode; The information processing device further executes, When the moving mode is the first state, after the output of the movement information, after the movement of the pointer stops, the instruction corresponding to the first mode switching action is output. The information processing method of claim 1, wherein the switching from the non-moving mode to the moving mode comprises: When the movement of the specified part is the second mode switching movement, switching from the non-moving mode to the second state of the moving mode; The information processing device further executes, When the moving mode is the second state, after the output of the movement information, after the movement of the pointer stops, determining whether the movement of the specified part is a command movement based on the data, and when it is determined to be the command movement, outputting a command corresponding to the command movement. An information processing method as claimed in claim 2 or 3, wherein the aforementioned specified conditions include outputting the aforementioned instructions. The information processing method of claim 1 includes outputting movement information for causing the pointer to move corresponding to the mode switching action when switching from the non-movement mode to the movement mode. As in the information processing method of claim 5, the aforementioned movement information includes information representing a movement determined independently of the movement of the aforementioned specified part. The information processing method of claim 1, wherein the information processing device further executes, In the non-movement mode, based on the data, determining whether the movement of the specified part is a command movement not accompanied by the movement of the pointer; When it is determined that the movement of the specified part is a command movement not accompanied by the movement of the pointer, outputting a command corresponding to the command movement not accompanied by the movement of the pointer. A program that causes an information processing device to execute: From a sensor capable of measuring the movement of a specified part of a user, sequentially obtain data related to the movement of the specified part at a plurality of time points; In a non-moving mode, determine whether the movement of the specified part is a mode switching movement based on the data; When it is determined that the movement of the specified part is a mode switching movement, switch from the non-moving mode to the moving mode; In the moving mode, output movement information related to a pointer based on the data; When a specified condition is met after the movement information is output, switch from the moving mode to the non-moving mode. An information processing device includes a processor, and the processor executes: from a sensor capable of measuring the movement of a specified part of a user, sequentially obtaining data related to the movement of the specified part at a plurality of time points; when in a non-movement mode, determining whether the movement of the specified part is a mode switching movement based on the data; when it is determined that the movement of the specified part is a mode switching movement, switching from the non-movement mode to the movement mode; when in the movement mode, outputting movement information related to a pointer based on the data; when a specified condition is satisfied after the movement information is output, switching from the movement mode to the non-movement mode.