WO2023238568A1

WO2023238568A1 - Information processing method, program, and information processing device

Info

Publication number: WO2023238568A1
Application number: PCT/JP2023/017267
Authority: WO
Inventors: 泰生菰田
Original assignee: 株式会社ジンズホールディングス
Priority date: 2022-06-08
Filing date: 2023-05-08
Publication date: 2023-12-14
Also published as: JP2023179927A

Abstract

This information processing device performs: sequential acquisition, from a sensor that is capable of measuring an action of a prescribed part of a user, of data relating to the action of the prescribed part at a plurality of time points; determination, based on the data, of whether or not the action of the prescribed part is a mode switching action, when in a non-movement mode; switching from the non-movement mode to a movement mode when it is determined that the action of the prescribed part is the mode switching action; outputting movement information relating to a pointer on the basis of the data when in the movement mode; and switching from the movement mode to the non-movement mode when a prescribed condition has been satisfied after the output of the movement information.

Description

Information processing method, program, and information processing device

The present invention relates to an information processing method, a program, and an information processing device.

Conventionally, there is a known technique for controlling input operations such as click operations on input means such as a mouse using a sensor that detects the movement of a user's head (see, for example, Patent Documents 1 and 2).

Japanese Patent Application Publication No. 4-309996 Japanese Patent Application Publication No. 9-258887

However, in the conventional technology, when performing input operations using head movements, there were problems in usability, such as the pointer constantly moving and unintended operations being performed due to natural head movements.

Therefore, the disclosed technology aims to improve usability by making it possible to more appropriately realize the operation intended by the user when an input operation is performed using the movement of a predetermined part where a sensor is provided.

An information processing method in one aspect of the disclosed technology includes: an information processing apparatus sequentially acquiring data regarding the movement of a predetermined part of a user at a plurality of points in time from a sensor capable of measuring the movement of the predetermined part of the user; and a non-mobile mode. , determining whether the motion of the predetermined region is a mode switching motion based on the data; and when determining that the motion of the predetermined region is a mode switching motion, switching from the non-moving mode to the moving mode; when the pointer is in the moving mode, outputting movement information regarding the pointer based on the data; and switching from the moving mode to the non-moving mode if a predetermined condition is satisfied after outputting the movement information. Execute.

According to the disclosed technology, when an input operation is performed by measuring the motion of a predetermined part with a sensor, it is possible to more appropriately realize the operation intended by the user and improve usability.

FIG. 1 is a diagram showing an example of an information processing system in an embodiment. FIG. 2 is a block diagram illustrating an example of the configuration of an external information processing device in an embodiment. FIG. 1 is a block diagram showing an example of the configuration of an information processing device in an embodiment. It is a block diagram showing an example of composition of a processing part in an example. It is a figure which shows an example of the data regarding the moving speed in an Example. FIG. 3 is a diagram showing an example of the relationship between data, movement, and output in an example. FIG. 6 is a diagram showing an example of the relationship between the movement of a predetermined part, a mode, and a command in an embodiment. FIG. 6 is a diagram illustrating an example of the relationship between movements, commands, and pointer movements in the embodiment. 12 is a flowchart illustrating an example (part 1) of processing related to command issuance in the embodiment. 12 is a flowchart illustrating an example (part 2) of processing related to command issuance in the embodiment.

Embodiments of the present invention will be described below with reference to the drawings. However, the embodiments described below are merely examples, and there is no intention to exclude the application of various modifications and techniques not specified below. That is, the present invention can be implemented with various modifications without departing from the spirit thereof. In addition, in the description of the drawings below, the same or similar parts are denoted by the same or similar symbols. The drawings are schematic and do not necessarily correspond to actual dimensions or proportions. The drawings may also include portions that differ in dimensional relationships and ratios.

[Example]
In the embodiment, eyewear is taken as an example of a wearable terminal equipped with an acceleration sensor and/or an angular velocity sensor and, if necessary, a bioelectrode, but the present invention is not limited thereto. FIG. 1 is a diagram showing an example of an information processing system 1 in an embodiment. The information processing system 1 shown in FIG. 1 includes an external information processing device (hereinafter also referred to as "external device") 10 and eyewear 30, and the external device 10 and the eyewear 30 are connected via a network. is connected and data communication is possible.

The eyewear 30 has the information processing device 20 mounted on the bridge portion, for example. The information processing device 20 includes a pair of nose pads and a bridge portion, each of which may have

bioelectrodes

32, 34, and 36. The information processing device 20 includes a 3-axis acceleration sensor and/or a 3-axis angular velocity sensor (or a 6-axis sensor). Note that the

bioelectrodes

32, 34, and 36 are not necessarily required.

The information processing device 20 detects sensor signals, electro-oculography signals, etc. and transmits them to the external device 10. The installation position of the information processing device 20 does not necessarily have to be in the bridge portion, but it may be positioned at a position where the sensor signal and the electro-oculography signal can be obtained when the eyewear 30 is worn. Further, the information processing device 20 may be removably provided in the bridge portion.

The external device 10 is an information processing device that has a communication function. For example, the external device 10 is a server device, a personal computer, a tablet terminal, a mobile terminal such as a smartphone, or the like. The external device 10 acquires data regarding the movement of the user's head received from the information processing device 20, and executes a plurality of operation processes based on this data. The operation process includes, for example, pointer (cursor) movement, clicking, dragging, scrolling, and the like. Further, the external device 10 may receive a command for instructing an operation process or data indicating the amount of movement of the pointer from the information processing device 20 in accordance with the movement of the head.

<Configuration of external device 10>
FIG. 2 is a block diagram showing an example of the configuration of the external device 10 in the embodiment. External device 10 includes one or more processing units (CPUs) 110, one or more network communication interfaces 120, memory 130, user interface 150, image sensor 160, and one or more processors for interconnecting these components. one or more communication buses 170 .

The network communication interface 120 is connected to a network through a mobile communication antenna or a wireless LAN communication antenna, and is capable of data communication with the information processing device 20.

User interface 150 may include a display device and an input device (such as a keyboard and/or mouse, or some other pointing device). The user interface 150 is capable of moving a pointer displayed on a display device. The image sensor 160 is an image sensor that receives an optical signal, converts the optical signal into an electrical signal, and generates an image. The image includes at least one of a still image and a moving image.

Memory 130 can be, for example, a high speed random access memory such as DRAM, SRAM, other random access solid state storage, or one or more magnetic disk storage, optical disk storage, flash memory devices, or other non-volatile It may be a non-volatile memory such as a solid-state storage device, or it may be a computer-readable non-transitory recording medium.

The memory 130 stores data used by the information processing system 1. For example, the memory 130 stores data transmitted from the eyewear 30 (information processing device 20).

Further, as another example of the memory 130, one or more storage devices installed remotely from the CPU 110 may be used. In some embodiments, memory 130 stores programs, modules and data structures, or a subset thereof, that are executed by CPU 110.

The CPU 110 configures the acquisition unit 112 and the processing control unit 114 by executing a program stored in the memory 130.

The acquisition unit 112 acquires instructions (commands) regarding operations, data indicating the amount of movement of the pointer, etc. from the information processing device 20 of the eyewear 30.

The processing control unit 113 executes commands related to the operations acquired by the acquisition unit 112, and controls the movement of the pointer using the amount of movement of the pointer. For example, the processing control unit 113 controls operations such as clicking and dragging using commands and data obtained from the eyewear 30, which has a function similar to that of an input device such as a pointing device (eg, a mouse).

Note that the acquisition unit 112 of the CPU 110 may sequentially acquire data regarding the movement speed of a predetermined part of the user from a sensor capable of measuring the movement of a predetermined part of the user (for example, a sensor attached to a predetermined part of the user). good. In this case, the processing control unit 113 may execute mode control processing, determination processing, and output processing of the information processing device 20, which will be described later, and may control operations such as clicking and dragging. Note that in addition to mouse operations such as clicking, dragging, and scrolling, the processing of the processing control unit 113 may also include operations such as a function key, a numeric keypad, and a setting change, which may be associated with the movement of the user's head.

<Configuration of information processing device 20>
FIG. 3 is a block diagram showing an example of the configuration of the information processing device 20 in the embodiment. As shown in FIG. 3, the information processing device 20 includes a processing section 202, a transmitting section 204, a six-axis sensor 206, a power supply section 208, and each bioelectrode 32, 34, and 36. Further, each of the

bioelectrodes

32, 34, and 36 is connected to the processing unit 202 using an electric wire, for example, via an amplification unit. Note that each bioelectrode 32, 34, and 36 is not necessarily a necessary configuration. The information processing device 20 also includes a memory that stores processing data. This memory may be a computer readable non-transitory storage medium.

The transmitting unit 204 transmits, for example, data regarding the moving speed packetized by the processing unit 202, moving information, non-moving information, or a command, which will be described later, to the external device 10. For example, the transmitter 204 transmits non-movement information, movement information, commands, etc. to the external device 10 by wireless communication such as Bluetooth (registered trademark) and wireless LAN, or by wired communication.

The 6-axis sensor 206 is a 3-axis acceleration sensor and a 3-axis angular velocity sensor. Further, each of these sensors may be provided separately, or one of the sensors may be provided. The 6-axis sensor 206 outputs the detected sensor signal to the processing unit 202. The power supply unit 208 supplies power to the processing unit 202, the transmission unit 204, the 6-axis sensor 206, and the like.

The processing unit 202 includes a processor, and processes the sensor signal obtained from the 6-axis sensor 206 and the electro-oculography signal obtained from each bioelectrode 32, 34, and 36 as necessary, and processes, for example, the sensor signal and the electro-oculography signal. may be packetized and this packet may be output to the transmitter 204. Further, the processing unit 202 may simply amplify the sensor signal obtained from the 6-axis sensor 206.

Additionally, the processing unit 202 may generate the sensor signal from the 6-axis sensor 206 as movement speed data based on information regarding head movement. The information related to the movement of the head is, for example, information related to the movement of the head up and down (back and forth) and to the left and right. An example in which the processing unit 202 includes a processor and executes processing corresponding to an input device will be described below.

FIG. 4 is a block diagram showing an example of the configuration of the processing unit 202 in the embodiment. In the example shown in FIG. 4, the processing unit 202 includes an acquisition unit 222, a mode control unit 224, and an output unit 228.

The acquisition unit 222 sequentially acquires data regarding the movement speed of a predetermined region at multiple points in time from a sensor capable of measuring the motion of the predetermined region of the user. For example, the acquisition unit 112 sequentially acquires angular velocity data transmitted from angular velocity sensors that sample at a predetermined sampling rate. Obtaining data from a sensor includes obtaining data directly or indirectly. Although the predetermined region in the embodiment is the head, it may also be an arm, a leg, or the like. The sensor is a sensor capable of measuring the moving speed of a predetermined part, and in the embodiment, an angular velocity sensor or an acceleration sensor is suitable, and may be attached to a predetermined part of the user. For example, if the sensor is a three-axis angular velocity sensor, the data regarding the moving speed is data on an angular velocity vector including direction.

The mode control unit 224 controls state transition between a movement mode in which movement of the pointer of the external device 10 is controlled and a non-movement mode in which the pointer is not moved, based on data regarding movement speed. For example, the mode control unit 224 includes a determination unit 226 that performs various determinations.

For example, when the current state is the non-moving mode, the determination unit 226 of the mode control unit 224 determines whether the movement of a predetermined part of the user is a mode switching movement based on data related to movement speed that are sequentially acquired. Determine. As a specific example, the mode switching operation may be one or more operations of a predetermined part set in advance, or may be an operation set by the user.

Furthermore, if the determination unit 226 determines that the motion of the user's predetermined region is a mode switching motion, the determination unit 226 switches from the non-moving mode to the moving mode. For example, if the determination unit 226 determines that the moving direction and/or amount of movement of the predetermined part indicated by the sequentially acquired data regarding the moving speed indicates one of one or more mode switching operations, The mode control unit 224 switches from non-moving mode to moving mode. Regarding the determination of the mode switching operation, for example, it may be determined whether the operation involves a movement of more than a threshold value in a predetermined direction within a predetermined time, or whether there is a combination of a plurality of arbitrary operations.

Further, the threshold value may be a predetermined threshold value that can detect sudden movement. For example, if the predetermined part is the head, a threshold is set that is exceeded when the head is quickly shaken horizontally or vertically, and if the predetermined part is the arms or legs, a threshold is set that is exceeded when the head is quickly shaken horizontally or vertically. A threshold may be set that will be exceeded if the object is moved. Note that the threshold values for each direction may be set differently depending on each movement.

The output unit 228 outputs the data processed by the processing unit 202 when the information processing device 20 is powered on. The output data is transmitted from the transmitter 204 to the external device 10 via a network or Bluetooth (registered trademark).

For example, in the case of non-movement mode, the output unit 228 outputs non-movement information that does not move the pointer on the external device 10 side. For example, the non-movement information is information such as (0,0) indicating that the amount of movement of the X and Y coordinates is 0 in order to not move the pointer, regardless of the data regarding the moving speed that is sequentially acquired.

Furthermore, when the current state is in the movement mode, the output unit 228 outputs movement information regarding the pointer based on the data regarding the movement speed that is sequentially acquired. For example, the output unit 228 outputs information such as the amount of movement of the X coordinate and Y coordinate (X1, Y1) in order to move the pointer based on the data regarding the moving speed that is sequentially acquired.

If the current mode is the movement mode and a predetermined condition is met after the movement information is output, the mode control unit 224 switches from the movement mode to the non-movement mode. The predetermined conditions include, for example, conditions regarding the movement amount and motion indicated by the data regarding the movement speed.

Through the above processing, when the sensor measures the movement of a predetermined part and performs an input operation, the user can prevent the pointer from moving unintentionally, making it possible to more appropriately perform the user's intended operation, and improving usability. can be done. Furthermore, according to the above process, when the user does not move the pointer, by introducing a non-movement mode that is unrelated to the movement of a predetermined part, when the user does not intend to perform an operation, the pointer can be moved according to the movement of the predetermined part. It is possible to prevent this from moving. Further, the external device 10 only needs to move the pointer according to the non-movement information, movement information, commands, etc. transmitted from the information processing device 20 in the same way as when operating a mouse or pointing device. It can be applied to the existing external device 10 without the need to add a special program to the side. Moreover, as a result, it becomes possible to effectively utilize the limited processing resources of the external device 10.

Furthermore, when control of pointer movement is started using data related to movement speed, the mode control unit 224 may set a non-movement mode as the starting mode. This prevents the pointer from moving in accordance with the movement of a predetermined part immediately after the power of the information processing device 20 is turned on, and provides an operational feel that allows the pointer to move only when you want to use it, like with a mouse. can be provided to users.

Furthermore, the mode control unit 224 includes switching from the non-movement mode to the first state of the movement mode when the user's movement of a predetermined region is the first mode switching action. But that's fine. For example, if it is determined that the movement of the predetermined part is the first mode switching movement based on the movement direction and movement amount indicated by data related to movement speed that are sequentially acquired, the mode control unit 224 switches the movement mode from the non-movement mode. Switch to the first state of a plurality of movement modes.

For example, when the first mode switching operation is a left-right movement, the determination unit 226 determines, based on the sequentially acquired data regarding the movement speed, the amount of movement in the left direction is equal to or more than a threshold value and the amount of movement in the right direction is equal to or more than a threshold value within a predetermined time. It is determined whether there is a movement with a movement amount of . If the determination unit 226 detects this movement, it determines that the series of actions of the user's predetermined part is a first mode switching action that performs processing from pointer movement to command issuance.

Furthermore, when the movement mode is in the first state, the output unit 228 may include outputting a command corresponding to the first mode switching operation after the pointer stops moving, following output of the movement information. For example, the first state is a click wait state, and the command corresponding to the first mode switching operation is a left click.

With the above processing, it becomes possible to output a left-click command by simply moving the pointer and stopping at the desired position after switching from non-moving mode to moving mode. In other words, the user does not need to perform both the mode switching action and the command determination action, and for frequently used left clicks, the user can perform the series of actions from the pointer movement to the left click in the series of actions from the mode switching action. It becomes possible to do so. Furthermore, by reducing the command determination process, it is possible to reduce the processing load on the information processing device 20. Note that the command corresponding to the first mode switching operation is not limited to the left click, and may be changed to a command frequently used by the user.

Regarding switching from the non-movement mode to the movement mode, the mode control unit 224 may include switching from the non-movement mode to the second state of the movement mode when the movement of the predetermined region is a second mode switching operation. . For example, if it is determined that the movement of the predetermined part is the second mode switching movement based on the movement direction and movement amount indicated by data related to movement speed that are sequentially acquired, the mode control unit 224 switches the movement mode from the non-movement mode. Switch to the second state of the plurality of movement modes.

For example, when the second mode switching operation is a right-left movement, the determination unit 226 determines, based on the sequentially acquired data regarding the movement speed, the amount of movement in the right direction is equal to or more than a threshold value and the amount of movement in the left direction is equal to or more than a threshold value within a predetermined time. It is determined whether there is a movement with a movement amount of . If the determination unit 226 detects this movement, it determines that the user's movement of the predetermined part is a second mode switching movement in which a movement judgment process for determining a command is performed after the pointer is moved.

Further, when the movement mode is in the second state, following the output of the movement information, the determination unit 226 determines whether the movement of the predetermined part is a command movement based on the data regarding the movement speed after the pointer stops moving. judge. For example, if a plurality of command operations are set, the determination unit 226 determines whether one of the command operations corresponds to one of the plurality of command operations based on sequentially acquired data related to movement speed.

Further, the output unit 228 may output a command corresponding to the command operation when the determination unit 226 determines that the operation is a command operation. For example, if the command action is a momentary movement to the left, a left click command will be output, and if the command action is a momentary movement to the right, a right click command will be output, and if the command action is a momentary movement upward. , a double click is output, and if the command action is a momentary downward movement, a command to start dragging is output.

With the above process, for example, by moving the pointer, stopping once on the desired icon, and then momentarily moving the head to the right, a command to right-click on the icon will be output, and the menu display for the icon will be opened. be able to. That is, it is possible to set a command by combining a plurality of operations, and it is possible to increase the variations of commands.

Furthermore, the predetermined condition for transitioning from the moving mode to the non-moving mode may include that the output unit 228 outputs a command. For example, when the mode control unit 224 detects that a command has been output from the output unit 228, it controls the mode to transition from the moving mode to the non-moving mode.

With the above processing, it is possible to automatically transition to the non-movement mode after a command is output, and it is possible to prevent the pointer from moving unnecessarily after the input operation is completed.

Furthermore, when the non-moving mode is switched to the moving mode, the output unit 228 may include outputting motion information that causes a pointer to move corresponding to the mode switching operation. For example, after mode switching, the output unit 228 outputs pointer movement information corresponding to a preset mode switching operation instead of movement information for a predetermined time, and following the movement information, movement information based on movement speed data is output. may be output.

Through the above processing, the user can check whether the movement of the predetermined part is intended by understanding the movement of the pointer.

Furthermore, the above-mentioned movement information may include information indicating a predetermined movement regardless of the movement of a predetermined part of the user. For example, while the movement of the predetermined part is left, right, left, and right, the movement information may be information indicating three times in a circle.

Through the above processing, the user can check whether his or her complex movements have been input correctly by understanding the simple movements of the pointer.

Furthermore, when the current mode is the non-movement mode, the determination unit 226 determines whether the movement of the predetermined part of the user is a command movement that does not involve movement of the pointer, based on the data regarding the movement speed that is sequentially acquired. You may judge.

In this case, if the output unit 228 determines that the user's movement of the predetermined region is the above command movement, it may output a command corresponding to the command movement.

Through the above processing, the user can cancel the non-movement mode by directly performing a command operation while in the non-movement mode, and also issue direct commands for commands that do not require pointer movement (such as scrolling). It becomes possible to output.

<Data example>
FIG. 5 is a diagram showing an example of data regarding the moving speed in the example. The data shown in FIG. 5 represents data acquired at a predetermined sampling rate (for example, 20 Hz = 0.05 seconds). The data shown in FIG. 5 is stored in a memory within the information processing device 20, for example. Note that data regarding the movement speed of the line of sight based on the electro-oculogram signal may also be stored as the data regarding the movement speed.

<Specific example>
FIG. 6 is a diagram showing an example of the relationship between data, movement, and output in the embodiment. In the example shown in FIG. 6, after the power of the information processing device 20 is turned on and the external device 10 and the information processing device 20 are paired using Bluetooth (registered trademark) or the like, the information processing device 20 is controlled to be in a non-mobile mode. That is, after pairing, although the predetermined part of the user has moved, the mode of the processing unit 202 is the non-movement mode, so non-movement information is output. Thereby, the external device 10 acquires non-movement information and does not move the pointer.

Next, when the predetermined part moves left and right for a predetermined period of up to 0.85 seconds, for example, the determination unit 226 switches to the first mode when the sequentially acquired data regarding the movement speed indicates left and right movement. It is determined that it is a motion. The mode control unit 224 causes a transition from the non-mobile mode to the first state of the mobile mode. At this time, the output unit 228 may output non-movement information, or if there is a time lag between data acquisition and output, it may output movement information about a series of left and right movements. Further, the output unit 228 may output motion information indicating a command.

Next, the user moves the predetermined part until 2.65 seconds, and the pointer is located at the target position that the user is aiming for. In the first state, the sensor detects this movement, data regarding the movement speed is acquired by the acquisition unit 222, and movement information based on this data is output by the output unit 228.

Next, when the predetermined part stops for a predetermined period of time (for example, between 2.70 and 2.80), the sequentially acquired data related to the movement speed indicate data in which the movement direction and movement amount are approximately 0. As a result, the determining unit 226 determines to stop, and the output unit 228 outputs a left click command corresponding to the first mode switching operation.

FIG. 7 is a diagram showing an example of the relationship between the movement of a predetermined part, the mode, and the command in the embodiment. For example, in the non-movement mode, the subsequent commands for movement of a predetermined part are as follows. Note that after issuing the command, the state returns to non-mobile mode.
Movement of the specified part: Subsequent commands Left and right (switching to movement mode): Pointer movement → stop → left click Right and left (switching to movement mode): Pointer movement → stop → click selection (left: left click, right: Right click, top: double click, bottom: start dragging)
Left/right/left/right or right/left/left (stop action): stop mode vertically reciprocating 3 times (setting action): setting mode → mode selection (right: setting change mode, left: setting display mode)
Up/down (direct command action): Scroll one step up or down (direct command action): Scroll down one step up or down (direct command action): Continuous scroll up → up or down → scroll stop Down up/down (direct command action): Continuous scrolling Down → Up or Down → Stop scrolling

As described above, it is possible to transition from a non-movement mode to a mode in which the pointer moves (pointer movement), a mode in which a command is directly issued, and the like. For example, a mouse wheel is used to issue commands without moving the pointer. In order to perform the same function as this wheel function, if a direct command operation corresponding to a scroll command is determined, a direct command is issued without moving the pointer.

Furthermore, in the stop mode, the pointer does not move and no command is issued unless an action is taken to cancel the stop mode. For example, in order to prevent unintended operations from being performed by moving a predetermined part in sync with the other person during a meeting, etc., a mode is provided that does not accept any motion other than the motion to cancel the stop mode. This makes it possible to more appropriately issue commands based on the actions intended by the user.

FIG. 8 is a diagram illustrating an example of the relationship between movements, commands, and pointer movements in the embodiment. The movement of the pointer refers to the movement of the pointer that is set according to the movement of a predetermined part or a command after the mode is switched. In the example shown in FIG. 8, the relationships are as follows.
Movement indicated by the pointer: Command: Pointer movement Left/Right: Move: Left/Right/Right/Left: Move: Right/Left/Left/Left: Stop mode: Circle 3 times vertically 3 times back and forth: Setting mode: Circle 2 times up/down: Scroll one level up: Up/down/down/up :Scroll down one level:Down/Up

If the command is movement, etc., the movement indicated by the data and the movement of the pointer are the same, but the movement of the pointer may be emphasized to emphasize that the command has been recognized. As an emphasis process, the output unit 228 may output motion information having a motion that moves faster or moves more greatly than the motion indicated by the actual data, for example. Since the movement indicated by the data and the movement of the pointer are the same, the user can intuitively understand that the movement has been appropriately recognized.

Furthermore, if the command is in a stop mode or the like, the command does not move, so the pointer makes a predetermined movement regardless of the movement indicated by the data. At this time, by making a simple movement that the predetermined region does not normally perform, the user can understand whether the motion of the predetermined region has been appropriately recognized.

<Operation>
9 and 10 are flowcharts illustrating an example of processing related to command issuance in the embodiment. The processes shown in FIGS. 9 and 10 are examples of processes executed by the information processing device 20.

In step S102 shown in FIG. 9, the acquisition unit 222 sequentially acquires data regarding the movement speed of a predetermined region of the user from a sensor capable of measuring the motion of the predetermined region of the user. The sensor is, for example, an angular velocity sensor attached to the eyewear 30, the predetermined part is, for example, the user's head, and the data regarding the moving speed is, for example, angular velocity data.

In step S104, the mode control unit 224 determines whether the current state is a non-moving mode or a moving mode. For example, the mode control unit 224 sets the non-moving mode as a default, and sets the moving mode after a mode switching operation is detected until a predetermined condition is satisfied. Here, if the current state is the non-movement mode (step S104-YES), the process proceeds to step S106, and if the current state is the movement mode (step S104-NO), the process proceeds to step S122 shown in FIG. move on.

In step S106, the determining unit 226 determines whether the sequentially acquired data regarding the moving speed indicates one or more preset mode switching operations. If the data indicates a mode switching operation (step S106-YES), the process proceeds to step S114, and if the data does not indicate a mode switching operation (step S106-NO), the process proceeds to step S108.

In step S108, the determination unit 226 determines whether the data regarding the movement speed acquired in step S106 is a command operation that does not involve pointer movement (direct command operation). If the data indicates a direct command operation (step S108-YES), the process proceeds to step S110, and if the data does not indicate a direct command operation (step S108-NO), the process proceeds to step S112.

In step S110, the output unit 228 outputs a command corresponding to the direct command operation.

In step S112, the output unit 228 outputs non-movement information that does not move the pointer of the external device 10, for example, information indicating that the amount of movement is (0,0).

In step S114, the mode control unit 224 switches from the non-moving mode to the moving mode.

In step S116, the determination unit 226 determines whether the data regarding the moving speed acquired in step S106 indicates a first mode switching operation. If the data indicates a first mode switching operation (step S116-YES), the process proceeds to step S118, and if the data does not indicate a first mode switching operation (step S118-NO), the process proceeds to step S120. In other words, if it is determined that the operation is the first mode switching operation, the process proceeds to step S118, and if it is determined that the operation is the second mode switching operation, the process proceeds to step S120.

In step S118, the mode control unit 224 sets the first state in which a command is issued after the pointer stops.

In step S120, the mode control unit 224 sets the mode controller 224 to a second state in which it waits for an operation corresponding to a predetermined command.

In step S122 shown in FIG. 10, that is, in the movement mode, the determination unit 226 determines whether or not the sequentially acquired data regarding the movement speed indicates a stopping motion of the pointer. If the data indicates a stop operation of the pointer (step S122-YES), the process proceeds to step S126, and if the data does not indicate a stop operation of the pointer (step S122-NO), the process proceeds to step S124.

In step S124, the output unit 228 outputs movement information indicating the amount of movement of the pointer based on the data regarding the movement speed that is sequentially acquired.

In step S126, the mode control unit 224 determines whether the current state is the first state. If the current state is the first state (step S126-YES), the process proceeds to step S128, and if the current state is the second state (step S126-NO), the process proceeds to step S130.

In step S128, the output unit 228 outputs a command corresponding to the command operation corresponding to the first mode switching operation. Furthermore, after outputting the command, the mode control unit 224 switches the movement mode to the non-movement mode.

In step S130, the output unit 228 outputs a command according to a predetermined command operation after the pointer stops. Furthermore, after outputting the command, the mode control unit 224 switches the movement mode to the non-movement mode.

Through the above processing, when an input operation is performed using the movement of a predetermined part where a sensor is provided, it is possible to more appropriately realize the operation intended by the user and improve usability.

Note that when the determination unit 226 determines a mode switching operation or a command operation, the output unit 228 may output pointer movement information according to the mode switching operation or command operation. Through this process, the user can check whether the movement of the predetermined region is correctly recognized by the information processing device 20.

It should be noted that each processing step included in the processing flow explained in FIG. Other steps may be added. Furthermore, a step that is described as one step for convenience can be divided into multiple steps and executed, while a step that is described as multiple steps for convenience can be understood as one step.

As described above, according to the embodiment, when an input operation is performed by measuring the motion of a predetermined part with a sensor, it is possible to more appropriately realize the operation intended by the user and improve usability. Further, according to the embodiment, when performing an input operation using movement of a predetermined part, recognition of the movement intended by the user when the user wants to move the pointer or perform a predetermined operation, such as with an input device such as a mouse. It becomes possible to provide an interface that allows input operations.

Furthermore, in the above embodiments, movements of the user's head, arms, legs, etc. are taken as examples of movements of the predetermined parts, but movements of the line of sight of the eyeballs may also be used. In this case, the movement of the eyeball may be determined from the sensor signal detected from each bioelectrode (an example of a sensor), and the processing unit 202 may control the processing based on data regarding this movement.

In addition, in the embodiment, explanation was given using the sensor signal from the 6-axis sensor mounted on the eyewear 30, but as mentioned above, the 6-axis sensor can be attached not only to the head but also to any position on the human body. It would be fine if it had been done.

In addition, in the embodiment, it has been mentioned that the data of the 3-axis acceleration sensor may be used as the movement speed data, but in this case, after the acceleration is detected, it is converted to speed by predetermined calculation processing, and the converted Velocity data may also be used.

Although the present invention has been described above using Examples, the technical scope of the present invention is not limited to the scope described in the above Examples. It will be apparent to those skilled in the art that various changes or improvements can be made to the embodiments described above. It is clear from the claims that such modifications or improvements may be included within the technical scope of the present invention.

<Modification 1>
In the embodiment described above, the information processing device 20 on the eyewear 30 side functions as an interface that replaces a mouse or the like, and the information processing device 20 outputs operation commands such as left click, right click, and scroll to the external device 10. An example was explained. On the other hand, the information processing device 20 may amplify or packetize the sequentially acquired data regarding the moving speed and transmit the data to the external device 10. In this case, the external device 10 may be It may have a function and output the determined command to an OS (Operating System) or the like to control the movement and operation of a pointer on the screen.

<Modification 2>
In modification 2, the processing content described above in the embodiment may be realized by an information processing device in which the information processing device 20 and a part of the configuration of the external device 10 are integrally formed. For example, a head-mounted display includes the processing unit 202 of the information processing device 20 and the display 151 of the external device 10, and a pointer corresponding to the movement of the user's head is superimposed on an image displayed on the display 151. , input operations such as those described above may be performed.

<Modification 3>
In the third modification, an image sensor 160 is used as a sensor capable of measuring the motion of a predetermined region of the user. In the case of modification 3, the external device 10 is the information processing device described above, and the CPU 110 includes the processing section 202. As a specific example, the motion of a predetermined part (such as the head) of the user is photographed using the live view mode of the image sensor 160, and the processing unit 202 acquires data regarding the movement of the predetermined part recognized by image recognition. The subsequent processing is similar to that of the embodiment. Thereby, by introducing the above-described non-mobile mode using a personal computer having a camera, it is possible to more appropriately realize the operation intended by the user, and improve usability.

10 Information processing device (external device)
20 Information processing device 30 Eyewear 110 CPU
112 Acquisition unit 113 Processing control unit 120 Network communication interface 130 Memory 150 User interface 160 Image sensor 202 Processing unit 204 Transmission unit 206 6-axis sensor 208 Power supply unit 222 Acquisition unit 224 Mode control unit 226 Determination unit 228 Output unit

Claims

The information processing device
Sequentially acquiring data regarding the movement of the predetermined part at multiple points in time from a sensor capable of measuring the movement of the predetermined part of the user;
determining whether the motion of the predetermined part is a mode switching motion based on the data when the mode is a non-moving mode;
If it is determined that the motion of the predetermined part is a mode switching motion, switching from the non-moving mode to the moving mode;
When in the movement mode, outputting movement information regarding the pointer based on the data;
If a predetermined condition is satisfied after outputting the movement information, switching from the movement mode to the non-movement mode;
An information processing method that performs.
Switching from the non-mobile mode to the mobile mode includes:
When the operation of the predetermined part is a first mode switching operation, the method includes switching from the non-moving mode to the first state of the moving mode;
The information processing device further includes:
2. When the movement mode is the first state, following the output of the movement information, outputting a command corresponding to the first mode switching operation is performed after the pointer stops moving. Information processing method.
Switching from the non-mobile mode to the mobile mode includes:
When the operation of the predetermined part is a second mode switching operation, the method includes switching from the non-moving mode to the second state of the moving mode;
The information processing device further includes:
When the movement mode is the second state, following the output of the movement information, after the pointer stops moving, it is determined whether the movement of the predetermined part is a command movement based on the data, and the command movement is performed. 2. The information processing method according to claim 1, further comprising outputting a command corresponding to the command operation when it is determined that the command operation is the command operation.
The information processing method according to claim 2 or 3, wherein the predetermined condition includes outputting the command.
The information processing method according to claim 1, further comprising outputting movement information that causes the pointer to make a movement corresponding to the mode switching operation when switching from the non-movement mode to the movement mode.
The information processing method according to claim 5, wherein the movement information includes information indicating a predetermined movement regardless of the movement of the predetermined part.
The information processing device further includes:
If the mode is the non-movement mode, determining whether the movement of the predetermined part is a command movement that does not involve movement of a pointer based on the data;
If it is determined that the movement of the predetermined part is a command movement that does not involve movement of the pointer, outputting a command corresponding to the command movement that does not involve movement of the pointer;
2. The information processing method according to claim 1, wherein:
In the information processing device,
Sequentially acquiring data regarding the movement of the predetermined part at multiple points in time from a sensor capable of measuring the movement of the predetermined part of the user;
determining whether the motion of the predetermined part is a mode switching motion based on the data when the mode is a non-moving mode;
If it is determined that the motion of the predetermined part is a mode switching motion, switching from the non-moving mode to the moving mode;
When in the movement mode, outputting movement information regarding the pointer based on the data;
If a predetermined condition is satisfied after outputting the movement information, switching from the movement mode to the non-movement mode;
A program to run.
An information processing device including a processor,
The processor,
Sequentially acquiring data regarding the movement of the predetermined part at multiple points in time from a sensor capable of measuring the movement of the predetermined part of the user;
determining whether the motion of the predetermined part is a mode switching motion based on the data when the mode is a non-moving mode;
If it is determined that the motion of the predetermined part is a mode switching motion, switching from the non-moving mode to the moving mode;
When in the movement mode, outputting movement information regarding the pointer based on the data;
If a predetermined condition is satisfied after outputting the movement information, switching from the movement mode to the non-movement mode;
An information processing device that executes.