WO2023157121A1 - 情報処理装置および情報処理システム - Google Patents

情報処理装置および情報処理システム Download PDF

Info

Publication number
WO2023157121A1
WO2023157121A1 PCT/JP2022/006168 JP2022006168W WO2023157121A1 WO 2023157121 A1 WO2023157121 A1 WO 2023157121A1 JP 2022006168 W JP2022006168 W JP 2022006168W WO 2023157121 A1 WO2023157121 A1 WO 2023157121A1
Authority
WO
WIPO (PCT)
Prior art keywords
information processing
operation mode
processing device
action
motion
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/006168
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
仁 秋山
義憲 岡田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Maxell Ltd
Original Assignee
Maxell Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Maxell Ltd filed Critical Maxell Ltd
Priority to JP2024500764A priority Critical patent/JP7727084B2/ja
Priority to CN202280091777.6A priority patent/CN118696290A/zh
Priority to US18/837,708 priority patent/US12541246B2/en
Priority to EP22927024.4A priority patent/EP4481540A4/en
Priority to PCT/JP2022/006168 priority patent/WO2023157121A1/ja
Publication of WO2023157121A1 publication Critical patent/WO2023157121A1/ja
Anticipated expiration legal-status Critical
Priority to JP2025131942A priority patent/JP2025159051A/ja
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/014—Hand-worn input/output arrangements, e.g. data gloves
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/016—Input arrangements with force or tactile feedback as computer generated output to the user
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures

Definitions

  • the present invention relates to a technique for operating another information processing device by a user's action on one information processing device.
  • An example of an information processing device is a smartwatch worn on the wrist.
  • a smart watch is worn on the wrist and has various functions such as a clock and activity measurement.
  • There is also a technique of acquiring myoelectric information, detecting a user's motion (gesture), and operating a smart watch using the detected gesture see, for example, Patent Document 1).
  • HMD head-mounted display
  • An HMD is a type of information processing apparatus that mainly has a display function, and is an information processing apparatus that is worn on the user's head and that displays and visually recognizes information in a real space or a virtual space.
  • Patent Document 2 describes "an information processing device worn on a first part of a user, To a sensor that outputs a first signal in response to the user's motion, a first mode that uses the first signal to calculate information related to the motion, and a display device that is worn on a second part of the user a mode setting unit capable of setting the information processing apparatus to a second mode that handles the first signal as an input signal of the second mode, and a second signal generated based on the first signal in the second mode or the first and a transmitting unit that transmits a signal to the display device.
  • Patent Literature 2 describes a technique for switching operation modes to determine which of the information processing device and the display device is operated by the operation of the information processing device.
  • Patent Document 2 the switching of the operation mode, which is to operate the information processing device or the display device, is determined only by the operation on the side of the information processing device. . It is difficult to reliably switch the operation mode with higher accuracy, and there is a high possibility of misrecognition and malfunction.
  • the present invention has been made to solve the above problems, and is an information processing system having a plurality of information processing devices such as a smart watch and an HMD, wherein an information processing device to be operated by an operation on one information processing device It is an object of the present invention to provide a technique for reducing erroneous recognition and malfunction in switching an operation target with a simple operation in an information processing system capable of switching between .
  • the present invention is an information processing device capable of communicating with an external device, which is an external information processing device, and includes a storage unit for storing a predetermined cooperation start trigger operation, and a user wearing the information processing device.
  • a motion detection unit that detects a motion
  • a motion determination unit that determines whether the motion detected by the motion detection unit is the cooperation start trigger motion
  • a switching control unit that switches an operation mode of the information processing device.
  • the operation mode includes a cooperative operation mode in which an operation command for operating the external device is generated in the information processing device and transmitted to the external device, and the operation command for operating the information processing device in the information processing device. and operating the information processing apparatus according to the generated operation command, wherein the switching control unit is the cooperation start trigger operation by the action determination unit in the single operation mode. and reception of a detection notification from the external device, which means that a predetermined external cooperation start trigger action has been detected in the external device, both occur within a predetermined period of time. It is characterized by switching to the cooperative operation mode.
  • the present invention provides an information processing device capable of communicating with an external device, which is an external information processing device, and includes a storage unit for storing a predetermined trigger operation to start cooperation, and a use of the information processing device attached.
  • a motion detection unit that detects a motion of a person
  • a motion determination unit that determines whether the motion detected by the motion detection unit is the cooperation start trigger motion
  • a switching control unit that switches an operation mode of the information processing device; wherein the operation mode includes a cooperative operation mode in which an operation command for operating the information processing device is received from the external device and the information processing device is operated in accordance with the operation command, and an information processing device in the information processing device and a single operation mode in which the information processing apparatus is operated in accordance with the generated operation command, and the switching control unit is configured to operate the information processing apparatus according to the generated operation command in the single operation mode.
  • the present invention is an information processing system comprising a first information processing device and a second information processing device that are linked in communication with each other, wherein a predetermined first cooperation start trigger operation is performed in the first information processing device.
  • a predetermined second cooperation start trigger action is detected in the second information processing device within a predetermined time after the detection, the first information processing device and the second information processing device are operated in a cooperation operation mode
  • the cooperative operation mode includes a first cooperative operation mode in which the second information processing apparatus is operated in accordance with an action of the user of the first information processing apparatus detected by the first information processing apparatus, and a second cooperative operation mode in which the second information processing apparatus and a second cooperative operation mode in which the first information processing device is operated according to the user's motion of the second information processing device detected by the information processing device.
  • an information processing system having a plurality of information processing devices such as smart watches and HMDs, which can switch the information processing device to be operated by an operation on one information processing device.
  • erroneous recognition and erroneous operation in switching the operation target can be reduced with a simple operation. Further, problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.
  • FIG. 1 is an explanatory diagram for explaining an overview of an information processing system according to an embodiment of the present invention
  • FIG. It is an example of a figure explaining an outline of operation of an information processing system concerning an embodiment of the present invention.
  • 1A and 1B are a hardware configuration diagram and a functional block diagram, respectively, of a first information processing device (watch) according to an embodiment of the present invention
  • FIG. (a) and (b) are explanatory diagrams for explaining an example of a watch single operation database and a cooperative operation database, respectively, according to an embodiment of the present invention.
  • (a) and (b) are a hardware configuration diagram and a functional block diagram, respectively, of a second information processing device (HMD) according to an embodiment of the present invention.
  • HMD second information processing device
  • 4 is an explanatory diagram for explaining an example of the HMD single operation database according to the embodiment of the present invention
  • 4 is a flowchart of cooperative processing according to the embodiment of the present invention
  • (a) and (b) are explanatory diagrams for explaining display examples in the display device of the HMD according to the embodiment of the present invention. It is an explanatory view for explaining an outline of an information processing system concerning a modification of the present invention. It is an explanatory view for explaining two-party cooperation processing and three-party cooperation processing of the modification of the present invention.
  • An information processing system includes an information processing device (first information processing device) and an external device (second information processing device) that is communicable with the information processing device and is linked in advance for communication. Then, when the first information processing apparatus and the second information processing apparatus detect a predetermined trigger operation at substantially the same timing, both of them shift to the cooperative operation mode. In the cooperative operation mode, the second information processing device (operated device) is operated by the action detected by the first information processing device (operating device).
  • the first information processing device is a smart watch and the second information processing device (operated device) is a head mounted display will be described below as an example.
  • a smartwatch is an information processing device worn on the wrist in the same manner as a wristwatch.
  • a watch it will be called a watch.
  • a head-mounted display is an information processing device that is worn on the head of a user (user) and displays and visually recognizes information of a real space or a virtual space.
  • HMD head-mounted display
  • the watch In addition to its clock function, the watch also functions as an activity meter that measures the user's activity level (exercise amount), as well as functions such as receiving calls, receiving messages, and communicating with other devices.
  • the watch can acquire myoelectric information using a myoelectric sensor that detects biopotentials.
  • the watch can detect hand waving motions (gestures) of the user wearing the watch using myoelectric information obtained by the myoelectric sensor when the wrist moves.
  • the HMD not only has the function of displaying real space information and virtual space information, but also can detect the user's movement, posture state, and the user's line of sight using the various sensors it possesses.
  • FIG. 1 is a diagram for explaining an overview of an information processing system 900 according to this embodiment.
  • An information processing system 900 of this embodiment includes a watch 100 and an HMD 200 .
  • Watch 100 is worn on the wrist of user 800 .
  • the HMD 200 is worn on the head of the user 800 .
  • the watch 100 and the HMD 200 are linked in communication in advance.
  • the watch 100 includes a myoelectric sensor 131.
  • the HMD 200 also includes a left-eye line-of-sight sensor 235 and a right-eye line-of-sight sensor 236 that respectively detect the line of sight of the left and right eyes of the user 800, an acceleration sensor 232 that detects the posture state and movement of the user 800, a gyro sensor 233, and a geomagnetic sensor 234. and a camera 210 that captures an image of the outside world or the like.
  • the watch 100 of this embodiment has a watch-only operation mode and an HMD-linked operation mode as operation modes.
  • an operation command for operating the watch 100 is generated in the watch 100, and the watch 100 is operated according to the generated operation command. That is, in the watch single operation mode, the watch 100 detects an instruction by the user 800's action or input, and operates the watch 100 according to the detection result.
  • the watch 100 in the HMD linked operation mode, the watch 100 generates an operation command for operating the HMD 200 and transmits it to the HMD 200 . That is, in the HMD-linked operation mode, the watch 100 detects an instruction by an action or input by the user 800 , generates an operation command for the HMD 200 according to the detection result, and transmits it to the HMD 200 .
  • the HMD 200 of this embodiment has an HMD single operation mode and a watch cooperation operation mode as operation modes.
  • the HMD 200 detects an instruction by the user 800's motion or input, and operates the HMD 200 according to the detection result.
  • HMD 200 receives operation commands from watch 100 and operates HMD 200 according to the operation commands.
  • the watch 100 and the HMD 200 respectively operate in the watch single operation mode and the HMD single operation mode in the initial state (immediately after activation). Then, when the watch 100 side detects a predetermined trigger action and the HMD 200 side also detects a predetermined trigger action at the same time or within a predetermined period of time, the watch 100 and the HMD 200 enter the HMD linked operation mode and the watch mode, respectively. Switch to cooperative operation mode. In the HMD-linked operation mode and the watch-linked operation mode, the watch 100 transmits to the HMD 200, as operation information for operating the HMD 200, an operation command corresponding to a user action detected by the watch 100, as shown in FIG.
  • the watch single operation mode when there is no particular need to distinguish between the watch single operation mode and the HMD single operation mode, they will be referred to as the single operation mode. Also, when there is no particular need to distinguish between the HMD-linked operation mode and the watch-linked operation mode, they are referred to as linked operation modes.
  • actions and input instructions by the user 800 detected by the watch 100 and the HMD 200 are collectively referred to as actions.
  • actions an operation that triggers switching the operation mode from the single operation mode to the cooperative operation mode is called a cooperation start trigger operation, and an operation that triggers switching from the cooperative operation mode to the single operation mode is called a cooperation end trigger operation. Note that when there is no need to distinguish between the cooperation start trigger action and the cooperation end trigger action, they are simply referred to as trigger actions.
  • the cooperation start trigger action for the watch 100 and the cooperation start trigger action for the HMD 200 may be the same or different.
  • the cooperation start trigger action for the watch 100 will be referred to as the watch side cooperation start trigger action
  • the cooperation start trigger action for the HMD 200 will be referred to as the HMD side cooperation start trigger action (external trigger action). cooperation start trigger operation).
  • the cooperation end trigger action for the watch 100 and the cooperation end trigger action for the HMD 200 may be the same or different.
  • a case in which both are different referred to as a watch-side cooperation end operation and an HMD-side cooperation end operation
  • FIG. 1 as an example of the cooperation start trigger action on the watch 100 side, the action of the user 800 lifting and twisting the wrist on which the watch 100 is worn is shown. This is what the user 800 normally does when trying to look at the watch 100 .
  • the watch 100 performs a series of gesture actions from a state 802 in which the wrist is lowered while wearing the watch 100 to a state 803 in which the arm is lifted and the wrist is raised, and a state 804 in which the wrist is twisted.
  • the movement is detected by the myoelectric sensor 131 .
  • FIG. 1 shows an action of the user 800 closing his/her eyes for a moment as an example of the cooperation start trigger action on the HMD 200 side.
  • the HMD 200 uses the left-eye line-of-sight sensor 235 and the right-eye line-of-sight sensor 236 to capture a momentary change from the state 811 with both eyes open of the user 800 to the state 812 with the eyes closed, and detects this as a cooperation start trigger action.
  • the cooperation start trigger operation is not limited to these.
  • the action of lifting and twisting the wrist on which the watch 100 is worn which is the same as the cooperation start trigger action on the watch 100 side, may be used as the cooperation start trigger action on the HMD side.
  • the action of triggering the end of cooperation on the watch 100 side uses, for example, the action of the hand on the side not wearing the watch 100 . Specifically, it is an action such as touching the screen of the watch 100 or pressing the crown. This is because if the movement of the arm on which the watch 100 is worn is used to cancel (instruct to end cooperation), the possibility of erroneously recognizing that the movement is for operating the HMD 200 increases.
  • the cooperation end trigger action on the HMD 200 side uses, for example, the action of the hand on the side not wearing the watch 100 .
  • the user 800 who was looking at something else may start looking at the watch 100 again as the cooperation end trigger operation.
  • the sight line detection of the HMD 200 detects that the user 800 has started looking at the watch 100 again.
  • the reason why the user 800 is looking at the watch 100 is that the user 800's intention is assumed to be to return to the watch-only operation mode in order to cause the watch 100 to perform some action.
  • the cooperation end trigger operation is not limited to these.
  • FIG. 3(a) The hardware configuration of the watch 100 of this embodiment is shown in FIG. 3(a). 1 and 2 with the same reference numerals have the same operations as those already explained in these figures, so detailed explanation thereof will be partially omitted. .
  • the watch 100 of this embodiment includes a processor 101, a memory 102, a camera 111, a sensor 130, a display device 121, a second input interface (I/F) 123, and an audio input It comprises a device 124 , an audio output device 125 , a vibration generating device 126 and a communication device 127 .
  • a processor 101 a memory 102, a camera 111, a sensor 130, a display device 121, a second input interface (I/F) 123, and an audio input It comprises a device 124 , an audio output device 125 , a vibration generating device 126 and a communication device 127 .
  • the sensor 130 includes a myoelectric sensor 131, an acceleration sensor 132, a gyro sensor 133, and a geomagnetic sensor 134.
  • a first input I/F 122 is provided as the display device 121 .
  • Each of these configurations is interconnected via a bus 103.
  • the myoelectric sensor 131 is arranged on the housing of the watch 100, the back side of the wristband, or the like, and detects the bioelectric potential flowing near the wrist at the contact portion on the surface of the wrist.
  • the myoelectric sensor 131 captures the gesture motion of the wrist based on myoelectric information obtained by detecting the bioelectric potential that flows when the wrist performs a specific movement.
  • the myoelectric sensor 131 is a sensor for picking up and processing nerve pulse signals (biological signals) flowing through the motor nerves that move the muscles of the hand, such as the radial nerve, the median nerve, and the ulnar nerve, which pass through the wrist. With two sensors arranged side by side, it picks up nerve pulse signals that flow to the motor nerves when the hand is moved.
  • a biological signal for moving the thumb flows from the brain to the radial nerve and the median nerve, depending on the direction of movement, and the myoelectric sensor 131 picks up the signal to move the thumb. Detect the gesture motion.
  • the acceleration sensor 132 is a sensor that detects acceleration, which is a change in speed per unit time, and captures movement, vibration, impact, and the like.
  • the acceleration sensor 132 uses the gravity vector and its projection on the axis of the acceleration sensor 132 to determine the tilt angle and measure and detect how much it is tilted with respect to the ground when the only acceleration applied is gravity. .
  • the gyro sensor 133 is a sensor that detects the angular velocity in the rotational direction, captures the vertical, horizontal, and oblique postures, and measures and detects in which direction and how much it has moved.
  • the acceleration sensor 132 and the gyro sensor 133 it is possible to detect the orientation of the watch 100, such as its tilt and direction.
  • the geomagnetic sensor 134 is a sensor that detects the magnetic force of the earth, and detects the direction the watch 100 is facing.
  • the movement of the watch 100 can also be detected by using a three-axis type sensor that detects geomagnetism in the vertical direction as well as in the longitudinal and horizontal directions, and by capturing changes in the geomagnetism in response to the movement of the watch 100 .
  • the posture and motion of the user 800 wearing the watch 100 can be detected by the acceleration sensor 132, the gyro sensor 133, and the geomagnetic sensor 134.
  • it may be used in place of the myoelectric sensor 131 for motion detection in processing described later. Further, it may be used together with motion detection by the myoelectric sensor 131 . By using them together, even more accurate detection becomes possible.
  • the camera 111 is used to photograph the surroundings and recognize the actions of the user 800. For example, the action of looking at the watch 100, which has been described as the trigger action with reference to FIG.
  • the processor 101 is composed of a CPU (Central Processing Unit) or the like, and executes programs such as an operating system (OS) and motion control applications stored in the memory 102 to realize each function of the watch 100. . Functions to be implemented will be described later.
  • OS operating system
  • motion control applications stored in the memory 102 to realize each function of the watch 100.
  • the memory 102 is composed of a non-volatile storage device or the like, and functions as a storage unit that stores various programs and data handled by the processor 101 and the like.
  • the data to be stored includes data used by the watch 100 for processing, data generated during processing, and the like. Specifically, it includes motion data detected by the watch 100, various predetermined trigger motion data, start trigger information to be transmitted to the HMD 200 when the trigger motion is detected, an operation command, an operation command for the HMD 200, and the like.
  • the display device 121 displays the time, application images, notification information to the user 800, and the like.
  • Information to be notified to the user 800 includes information indicating that a trigger action has been detected, information indicating the start and end of finger tapping measurement on the display device 121 and head and neck region change measurement, and information indicating that a switching request has been received. , and information indicating the start and end of the smartwatch operation.
  • the first input I/F 122 accepts input information to the watch 100 by the user 800 .
  • the first input I/F 122 is provided on the display surface of the display device 121, detects touch operations with a finger or a touch pen, and receives inputs.
  • the second input I/F 123 is a crown, a switch, etc. provided on the side of the watch 100.
  • the user 800 can input input information to the watch 100 by operating these crowns and switches.
  • the voice input device 124 collects the voice of the user 800 himself/herself with a microphone and converts it into voice data.
  • the watch 100 can take instruction information by vocalization from the user 800 into the watch 100, and can conveniently execute operations in response to the instruction information. For example, the user 800 may utter a voice indicating an input operation, and the voice input device 124 may collect the voice to capture the input information.
  • the audio output device 125 outputs audio from the speaker based on the audio data. Notification information to the user 800 can be announced by voice.
  • the vibration generating device 126 generates vibration under the control of the processor 101 .
  • the processor 101 converts notification information to the user 800 into vibration, transmits vibration to the wrist on which the watch 100 is worn by the vibration generating device 126, and notifies the user 800 of the notification information. This makes it possible to improve usability.
  • the communication device 127 is a communication interface that performs wireless communication with the HMD 200 by short-range wireless communication or the like.
  • the communication device 127 includes a communication processing circuit, an antenna, etc. corresponding to various predetermined communication interfaces, and transmits and receives various information, control signals, and the like.
  • a telephone communication network may be included.
  • FIG. 3B is a functional block diagram of the watch 100. As shown in FIG.
  • the watch 100 of this embodiment includes a motion detection unit 151, a motion determination unit 152, a switching control unit 153, a watch motion control unit 154, an HMD operation command generation unit 155, and a single watch operation database (DB) 160. , and an HMD cooperative operation DB 170 .
  • the motion detection unit 151 detects the motion of the user 800 using the sensor 130 such as the myoelectric sensor 131 .
  • the detected actions of the user 800 include operation instructions input via the first input I/F 122, the second input I/F 123, and the voice input device 124, as described above.
  • the detected motion (hereinafter referred to as a detected motion) is output to motion determination section 152 and watch motion control section 154 or HMD operation command generation section 155 .
  • the detection operation is output to the operation determination unit 152 and the watch operation control unit 154, and when the operation mode is the HMD cooperation operation mode, the detection operation is output to It is output to the motion determination unit 152 and the HMD operation command generation unit 155 .
  • the action determination unit 152 determines whether or not the detected action matches a predetermined trigger action (a cooperation start trigger action or a cooperation end trigger action).
  • the trigger operation is stored in memory 102 in advance.
  • the motion determination unit 152 determines that the detected motion matches the trigger motion, it outputs a watch-side trigger motion detection instruction to the switching control unit 153 .
  • the motion determining section 152 outputs a mismatch signal to the motion detecting section 151 . Note that, upon receiving the mismatch signal, the motion detection unit 151 outputs the detection motion to the watch operation control unit 154 or the HMD operation command generation unit 155 as described above according to the operation mode.
  • the switching control unit 153 switches operation modes.
  • the switching control unit 153 causes the HMD 200 to perform an action within a predetermined period when the action detected by the action detecting unit 151 is determined by the action determining unit 152 to be the watch-side cooperation start trigger motion in the watch single operation mode.
  • the operation mode is switched to the cooperation operation mode.
  • the detection of the cooperation start trigger action may be performed first by the watch 100 or by the HMD 200 first.
  • the case where the HMD 200 also performs detection after detection by the watch 100 will be described below as an example. That is, when the watch 100 side first detects the cooperation start trigger action and the HMD 200 side detects the cooperation start trigger action within a predetermined time, an operation command for the HMD 200 is generated according to the action detected by the watch 100 and the watch is activated.
  • a case where the HMD 200 is operated by 100 will be described as an example.
  • the switching control unit 153 transmits start trigger information indicating that the cooperation start trigger operation has been detected to the HMD 200, and after the transmission, the switching request is sent from the HMD 200 within a predetermined period of time. is received, the operation mode is switched to the HMD cooperation operation mode. Whether or not the cooperation start trigger motion is detected is determined by whether or not the trigger motion detection instruction is received from the motion determination unit 152 . The switching control unit 153 also outputs the operation mode after switching to the motion detection unit 151 .
  • the switching control unit 153 switches from the HMD cooperation operation mode to the watch single operation mode when a cooperation end trigger operation is detected in the HMD cooperation operation mode.
  • the HMD 200 is sent end trigger information indicating that the cooperation end trigger operation has been detected.
  • end trigger information is received from the HMD 200 in the HMD linked operation mode, the HMD linked operation mode is switched to the watch single operation mode.
  • the watch operation control unit 154 controls the operation of the watch 100 according to the detected operation in the watch single operation mode.
  • a command for operating the watch 100 corresponding to the operation content (watch operation command; own device operation command) is obtained by referring to the watch single operation DB 160, which stores the operation content of the watch 100 in association with the detected operation. is generated and output to related functions.
  • the HMD operation command generation unit (external operation command generation unit) 155 generates a command for operating the HMD 200 (HMD operation command; external operation command) according to the detected operation, and outputs it to the HMD 200 . In this embodiment, it is transmitted to the HMD 200 via the communication device 127 .
  • the HMD operation command generation unit 155 refers to the HMD cooperative operation DB 170 that stores the operation content of the HMD 200 in association with the detected action, and generates an HMD operation command corresponding to the operation content.
  • FIG. 4A is a diagram for explaining an example of the watch single operation DB 160
  • FIG. 4B is a diagram for explaining an example of the HMD cooperative operation DB 170 according to this embodiment.
  • the single watch operation DB 160 registers watch operation details 162 in association with detection actions 161 .
  • the watch 100 performs an operation of "rejecting the incoming call”.
  • the watch operation command itself for realizing the operation may be registered.
  • HMD operation details 172 are registered in the HMD cooperative operation DB 170 in association with detected actions 171 .
  • the HMD 200 performs an operation of "displaying a menu”.
  • the HMD operation command itself for realizing the operation may be registered.
  • FIG. 5A is a hardware configuration diagram of the HMD 200 of this embodiment. 1 and 2 with the same reference numerals have the same operations as those already explained in these figures, so detailed explanation thereof will be partially omitted. .
  • the HMD 200 of this embodiment includes a processor 201, a memory 202, a camera 210, a sensor 230, a display device 221, an input I/F 223, an audio input device 224, and an audio output device. 225 , a vibration generating device 226 and a communication device 227 .
  • the camera 210 includes an out-camera 211 and an in-camera 212 .
  • the sensor 230 includes an acceleration sensor 232 , a gyro sensor 233 , a geomagnetic sensor 234 , a left-eye line-of-sight sensor 235 , a right-eye line-of-sight sensor 236 , and a distance sensor 237 .
  • Each of these configurations is interconnected via a bus 203.
  • the out-camera 211 and the in-camera 212 photograph the front of the HMD 200 and the user 800, respectively.
  • the out-camera 211 can also photograph the wrist of the user 800 and the watch 100. Thereby, the out-camera 211 can capture the gesture of the user 800 and detect the trigger operation of the watch 100 .
  • the in-camera 212 captures the eye of the user 800 and its surroundings as a subject, and measures and detects state changes such as movement of the eyeball and pupil. For example, a motion of closing one's eyes for a moment can also be detected as a trigger motion.
  • a method for measuring and detecting eye movement a method is known in which the eye is photographed with a visible light camera, the reference point is the inner corner of the eye, the iris is the moving point, and the line of sight is detected based on the position of the iris relative to the inner corner of the eye. This corresponds to a case where the eye of the user 800 is captured by the in-camera 212 and the movement of the eyeball or pupil is measured and detected from the captured image.
  • the left-eye line-of-sight sensor 235 and the right-eye line-of-sight sensor 236 are sensors that detect the movement and orientation of the left and right eyes, respectively, and capture eye movement and line of sight.
  • the process of detecting the movement of the eyeball may use a well-known technique that is generally used as eye tracking process. For example, as a method using corneal reflection, an infrared LED (Light Emitting Diode) is irradiated onto the face and photographed with an infrared camera. Techniques for detecting movement of the eyeball and line of sight based on the position of the pupil with respect to the position of the corneal reflex are known.
  • the acceleration sensor 232 is a sensor that detects acceleration, which is a change in speed per unit time, and captures movement, vibration, impact, and the like.
  • the acceleration sensor 232 can measure and detect how much it is tilted with respect to the ground by using the gravity vector and its projection on the axis of the acceleration sensor 232 to determine the tilt angle when the only acceleration applied is gravity. .
  • the tilt of the HMD 200 can be detected by the acceleration sensor 232 provided inside the HMD 200 .
  • the gyro sensor 233 is a sensor that detects the angular velocity in the rotational direction, captures the state of vertical, horizontal, and diagonal postures, and measures and detects in which direction and how much it has moved.
  • the acceleration sensor 232 and the gyro sensor 233 it is possible to detect the orientation of the HMD 200, such as its inclination and direction.
  • the geomagnetic sensor 234 is a sensor that detects the magnetic force of the earth, and detects the direction in which the HMD 200 is facing.
  • the motion of the HMD 200 can also be detected by using a three-axis type sensor that detects geomagnetism in the vertical direction as well as in the longitudinal and lateral directions, and by capturing changes in the geomagnetism with respect to the motion of the HMD 200 .
  • the posture of the user 800 wearing the HMD 200 can be detected and determined by the acceleration sensor 232, the gyro sensor 233, and the geomagnetic sensor 234. For example, it is possible to detect a state in which the user 800 is looking in the direction of the watch 100 from a state in which the user 800 is looking in a direction different from that of the watch 100 .
  • the distance measuring sensor 237 is a sensor that can measure the distance and angle to an object and can perceive the shape of an object such as an object as a three-dimensional object.
  • a LiDAR (Light Detection and Ranging) sensor, a TOF (Time Of Flight) sensor, a millimeter wave radar, or the like is used for the ranging sensor 237 .
  • LiDAR irradiates an object with laser light such as infrared rays, measures the reflected scattered light, and analyzes and detects the distance to a distant object and the state of the object.
  • the TOF sensor performs distance measurement by measuring the reflection time of pulsed light applied to a subject for each pixel.
  • the millimeter-wave sensor emits millimeter-wave radio waves, captures the reflected waves, and detects the distance to the reflecting object and the state of the opposite object.
  • the distance measurement sensor 237 measures the distance and angle to the index finger, thumb, etc., and based on the measurement information, it is possible to detect whether the hand is clenched or open, for example. Therefore, the ranging sensor 237 can detect an action such as a trigger of the watch 100 .
  • the processor 201 is composed of a CPU and the like, and executes programs such as an OS and operation control applications stored in the memory 202 to implement each function of the HMD 200 . Functions to be implemented will be described later.
  • the memory 202 is composed of a non-volatile storage device or the like, and functions as a storage unit that stores various programs and data handled by the processor 201 and the like.
  • the stored data includes data used by the HMD 200 for processing, data generated by the processing, and the like. Specifically, it includes motion data detected by the HMD 200, various kinds of predetermined trigger motion data, HMD operation commands, switching requests to be sent to the watch 100, and the like.
  • the display device 221 displays various information.
  • the display device 221 has a projection section and a transparent half mirror.
  • the projection unit projects, for example, an operation guide screen image for finger tap distance measurement, a virtual object such as an image that relaxes the user 800, notification information to the user 800, and the like.
  • a projected virtual object or the like is imaged and displayed on the half mirror in front of the eyes.
  • the user 800 can visually recognize both the physical object in the field of view in front of the user's eyes and the imaged virtual object as if they are floating.
  • the display device 221 When the HMD 200 is of the video see-through type, the display device 221 has a display such as a liquid crystal panel. On this display, the physical object in front of the eye captured by the out-camera 211, the virtual object, and the like are displayed together. As a result, the user 800 can visually recognize the physical object and the virtual object in the visual field image in front of the user 800 while superimposing them.
  • the input I/F 223 is input means such as key buttons, touch keys, etc., and accepts setting input of information desired to be input by the user 800 .
  • the input I/F 223 is provided in a position and form in which the user 800 can easily perform an input operation within the HMD 200 . It should be noted that it may be separated from the main body of the HMD 200 and connected by wire or wirelessly.
  • the input operation screen may be displayed within the display screen of the display device 221 . In this case, the input operation information may be acquired depending on the position on the input operation screen to which the line of sight is directed, or the input operation information may be acquired by displaying a pointer on the input operation screen and operating the pointer by the input I/F 223. .
  • the voice input device 224 uses a microphone to collect voices from the outside and voices of the user 800 and convert them into voice data. It is possible to take instruction information by vocalization from the user 800 into the HMD 200 and to execute an operation in response to the instruction information with ease of use. For example, the user 800 may utter a voice indicating the input operation, and the voice input device 224 may collect the sound to capture the input operation information.
  • the audio output device 225 outputs audio from the speaker based on the audio data. Notification information to the user 800 can be announced by voice.
  • the vibration generating device 226 generates vibration under the control of the processor 201 .
  • the processor 201 converts the notification instruction information transmitted by the HMD 200 to the user 800 into vibration, and transmits the vibration to the head of the user 800 wearing the HMD 200 by the vibration generation device 226 to inform the notification information. . This makes it possible to improve usability.
  • the communication device 227 is a communication interface that performs wireless communication with the watch 100 by short-range wireless communication.
  • the communication device 227 includes communication processing circuits, antennas, etc. corresponding to various predetermined communication interfaces, and transmits and receives various information, control signals, and the like.
  • a telephone communication network may be included.
  • FIG. 5B is a functional block diagram of the HMD 200. As shown in FIG.
  • the HMD 200 of this embodiment includes a motion detection section 251, a motion determination section 252, a switching control section 253, an HMD motion control section 254, a cooperation control section 255, and an HMD single operation DB 260.
  • the action detection unit 251 detects actions of the user 800 using the out-camera 211 , the in-camera 212 and the sensor 130 .
  • the sensor 130 for example, a left-eye line-of-sight sensor 235, a right-eye line-of-sight sensor 236, an acceleration sensor 232, a gyro sensor 233, a geomagnetic sensor 234, a distance measurement sensor 237, and the like are used.
  • the detected motion (hereinafter referred to as the detected motion) is output to the motion determination unit 252 and the HMD motion control unit 254 or the cooperation control unit 255 .
  • the detection operation is output to the HMD operation control unit 254, and when the operation mode is the watch cooperation operation mode, the detection operation is output to the cooperation control unit 255. do.
  • the action determination unit 252 determines whether or not the detected action matches a predetermined trigger action (a cooperation start trigger action or a cooperation end trigger action).
  • the trigger operation is stored in memory 202 in advance.
  • the motion determination unit 252 determines that the detected motion matches the trigger motion, it outputs a trigger motion detection instruction to the switching control unit 253 .
  • the motion determining section 252 outputs a mismatch signal to the motion detecting section 251 .
  • the motion detection unit 251 outputs the detection motion to the HMD motion control unit 254 if the operation mode is the HMD single operation mode. On the other hand, when the operation mode is the watch-linked operation mode, the detection operation is ignored.
  • the switching control unit 253 switches the operation mode.
  • the single operation mode when the action detected by the action detection section 251 is determined by the action determination section 252 to be an HMD side cooperation start trigger action, the switching control section 253 enables the watch side cooperation in the watch 100 within a predetermined period of time.
  • the operation mode is switched to the cooperative operation mode.
  • the detection of the cooperation start trigger operation may be performed first by the watch 100 or by the HMD 200 first.
  • the case where the HMD 200 also performs detection after detection by the watch 100 will be described below as an example.
  • the switching control unit 253 determines whether start trigger information has been received from the watch 100 at predetermined time intervals during the HMD single operation mode. Then, when a cooperation start trigger operation is detected within a predetermined period after receiving the start trigger information from the watch 100, an operation mode switching request is transmitted to the watch 100, and the operation mode of the HMD 200 is switched to the watch cooperation operation mode. .
  • the operation mode switching request is an instruction to switch the operation mode to the watch 100 of the transmission destination. Whether or not the cooperation start trigger motion has been detected is determined by whether or not a trigger motion detection instruction has been received from the motion determination unit 252 .
  • the switching control unit 253 also outputs the operation mode after switching to the motion detection unit 251 .
  • the switching control unit 253 switches from the watch-linked operation mode to the HMD single operation mode when a linkage end trigger operation is detected in the watch-linked operation mode. At this time, end trigger information is transmitted to the watch 100, which means that the cooperation end trigger operation has been detected. Further, when end trigger information is received from the watch 100 in the watch-linked operation mode, the watch-linked operation mode is switched to the HMD single operation mode.
  • the HMD operation control unit 254 controls the operation of the HMD 200 according to the detected operation in the HMD single operation mode.
  • the HMD single operation DB 260 which stores the operation details of the HMD 200 in association with the detected motion, is referred to, a command (HMD operation command) for operating the HMD 200 corresponding to the operation details is generated, and the related function unit output to
  • the cooperation control unit 255 When the cooperation control unit 255 receives an HMD operation command from the watch 100 via the communication device 227 in the watch cooperation operation mode, it outputs the HMD operation command to related function units. This realizes control of the operation of the HMD 200 according to the operation instruction from the watch 100 .
  • FIG. 6 is a diagram for explaining an example of the HMD single operation DB 260 of this embodiment.
  • the HMD single operation DB 260 registers HMD operation details 262 in association with detected actions 261 .
  • HMD operation details 262 For example, when the HMD 200 detects an action of “tapping the wrist with the opposite hand” by analyzing an image acquired by the out-camera 211 or the like, the HMD 200 performs an operation of “displaying a menu”. Note that instead of the HMD operation content 262, the HMD operation command itself for realizing the operation may be registered.
  • FIG. 7 is a processing flow of cooperation processing (operation mode switching processing) of the information processing system 900 of this embodiment. Note that in this figure, the case where both the watch 100 and the HMD 200 are both started in the single operation mode state as the initial state will be described as an example.
  • this processing ends when an action that signifies termination is detected in each action detection step.
  • the portions surrounded by dashed lines are portions to be processed in the single operation mode, and the portions surrounded by dashed-dotted lines are portions to be processed in the cooperative operation mode.
  • the motion detection unit 151 detects the motion of the user 800 at predetermined time intervals after activation (step S1101).
  • the motion determination unit 152 determines whether the detected motion matches a predetermined trigger motion (step S1102).
  • the current operation mode of the watch 100 is the single watch operation mode, it is determined whether or not there is a watch-side cooperation start trigger operation to shift to the cooperative operation mode.
  • step S1102 if it is not the watch-side cooperation start trigger action, the watch single operation mode continues, and the watch operation control unit 154 operates the watch 100 according to the detection action (step S1103), and returns to step S1101. As a result, when the trigger operation is not detected, the single operation mode continues.
  • step S1102 when it is determined that the watch-side cooperation start trigger motion has been detected, the motion determination unit 152 outputs a trigger motion detection instruction to the switching control unit 153. In response to this, the switching control unit 153 executes operation mode switching processing.
  • trigger information (start trigger information) indicating that the trigger operation has been performed is generated and transmitted to the HMD 200 (step S1201).
  • the motion detection unit 251 detects the motion of the user 800 at predetermined time intervals, similar to the watch 100 (step S2101). During this time, the switching control unit 253 determines whether trigger information (start trigger information) has been received from the watch 100 (step S2102). If the start trigger information has not been received, the HMD operation control section 254 controls the HMD 200 as it is according to the detected operation (step S2201), and returns to step S2101.
  • start trigger information start trigger information
  • the motion determination unit 252 determines whether or not the detected motion detected by the motion detection unit 251 is a predetermined trigger motion (step S2201).
  • the current operation mode of the HMD 200 is the HMD single operation mode, it is determined whether or not there is a trigger operation for starting the cooperation operation mode (HMD side cooperation start trigger operation). If it is not the HMD side cooperation start trigger operation, the process returns to step S2101 to continue the process.
  • the switching control unit 253 detects the HMD-side cooperation start trigger action on the HMD 200 side at the timing of receiving the start trigger information from the watch 100, and therefore generates an operation mode switching request. and transmits it to the watch 100 that sent the start trigger information (step S2202).
  • the switching control unit 153 determines whether or not there is a predetermined reply from the HMD 200 within a predetermined period of time from the generation and transmission of the start trigger information (step S1202). Here, it is determined whether or not an operation mode switching request has been received within a predetermined period (step S1202). Upon reception, the switching control unit 153 generates a result (so-called ACK) indicating that it has been received, returns it to the HMD 200 as the transmission source (step S1203), and switches the operation mode (step S1204). Here, the watch single operation mode is shifted to the HMD cooperative operation mode. It should be noted that, in step S1202, if not received, the process directly returns to step S1101.
  • step S1202 even if an operation mode switching request has not been received within a predetermined period of time, a result (ACK) indicating that the operation mode switching request has not been received may be generated and returned to the HMD 200 that is the transmission source. good.
  • the switching control unit 253 receives the result within a predetermined period after transmitting the operation mode switching request (step S2203), and switches the operation mode (step S2204).
  • the HMD single operation mode is shifted to the watch cooperation operation mode.
  • the process returns to step S2101 and continues.
  • the operation mode is switched only when both the watch 100 side and the HMD 200 side detect the cooperation start trigger action within a predetermined period.
  • the motion determination unit 152 determines that the detected motion is a predetermined trigger motion each time the motion detection unit 151 detects a motion. (step S1301).
  • the current operation mode of the watch 100 is the HMD cooperative operation mode, it is determined whether or not there is a trigger operation for ending the cooperative operation mode (watch-side cooperation end trigger operation).
  • the HMD operation command generation unit 155 generates an HMD operation command according to the detected action and transmits it to the HMD 200 (step S1303).
  • the action determination unit 152 determines whether or not the action is an action to end the operation on the watch 100 (step S1304). If it is an operation end instruction, the processing of the watch 100 is ended. At this time, the HMD 200 is notified that the operation of the watch 100 itself has ended. Upon receiving the notification, the HMD 200 shifts to the single operation mode.
  • step S1401 the switching control unit 153 performs operation mode return processing, which will be described later.
  • step S2301 when the cooperation control unit 255 receives the HMD operation command from the watch 100 (step S2301), the operation of the HMD 200 is controlled (operated) in accordance with the HMD operation command (step S2301). S2302).
  • the motion detection unit 251 continuously performs motion detection (step S2303).
  • the motion determination unit 252 determines whether the detected motion is a trigger motion (step S2304). Here, it is determined whether or not it is a cooperation end trigger operation. If it is not a trigger operation, the process returns to step S2301 to continue processing.
  • the action determination unit 252 first determines whether or not the action is an operation end instruction to the HMD 200 (step S2305). If it is an operation end instruction, the process ends. At this time, the watch 100 is notified that the operation of the HMD 200 itself has ended. Upon receiving the notification, the watch 100 shifts to the single operation mode.
  • step S2401 the switching control unit 253 executes operation mode return processing (step S2401).
  • the operation mode return processing executed in steps S1401 and S2401 will be described.
  • the cooperative operation mode returns to the single operation mode.
  • the switching control unit 153 communicates end trigger information indicating that the cooperation end trigger action has been detected to the HMD 200 , and the watch 100 Switches the operation mode of the to the watch single operation mode. Also, in the HMD 200 that has received the termination trigger information from the watch 100, the switching control unit 253 switches the operation mode of the HMD 200 to the HMD single operation mode.
  • the switching control unit 253 communicates end trigger information indicating that the cooperation end trigger action has been detected to the watch 100, and the operation of the HMD 200 is performed.
  • the mode is switched to the HMD single operation mode.
  • the switching control unit 153 switches the operation mode of the watch 100 to the watch single operation mode.
  • the watch 100 After completion of the operation mode return processing, the watch 100 returns to step S1101 and continues operating in the watch single operation mode. Similarly, in the HMD 200 as well, after the operation mode return processing ends, the process returns to step S2101 to continue the operation in the HMD single operation mode.
  • the HMD 200 may be configured to hold the trigger information received from the watch 100 for a period of time T1, and to determine in step S2201 whether or not there has been a cooperation start trigger operation during that period.
  • the HMD 200 detects the cooperation start trigger action within a predetermined period after the watch 100 detects the cooperation start trigger action, both of them enter the cooperation operation mode. transition to For example, if only one detects a cooperation start trigger action, or even if both detect the action, if it is not within a predetermined period of time, the transition to the cooperation operation mode is not made.
  • the cooperative operation mode when a cooperative end trigger operation is detected in either one of them, the operation mode is changed to the single operation mode.
  • both trigger operations are grasped in a simple manner to shift to the cooperative operation mode. , it is easy to use, reliable and highly accurate, and the possibility of erroneous recognition and malfunction in the operation mode switching selection can be eliminated or greatly reduced.
  • the HMD operation control unit 254 controls the display on the display device 221 (display unit) in the HMD single operation mode, and the cooperation control unit 255 in the watch cooperation operation mode.
  • the display device 221 of this embodiment includes a menu display area 270 and an operation mode display area 271.
  • FIG. 8(a) and 8(b) the display device 221 of this embodiment includes a menu display area 270 and an operation mode display area 271.
  • a menu is displayed as a virtual object in the menu display area 270 .
  • User 800 selects a desired operation from the menu to operate HMD 200 . Selection is made, for example, by sight.
  • the HMD 200 uses sensors such as the left-eye line-of-sight sensor 235 and the right-eye line-of-sight sensor 236 to detect the line-of-sight direction and the position of intersection with the display device 221 to determine which menu has been selected.
  • FIG. 8A is a display example when the HMD 200 is in the HMD single operation mode.
  • FIG. 8B is a display example when the HMD 200 is in the watch-linked operation mode.
  • an operation image display area 272 may be provided.
  • an operation image indicating what kind of operation the user 800 is performing is displayed.
  • the operation image is, for example, a simulated image of a hand that shows what kind of motion the user 800 has detected with the watch 100 .
  • An operation image is an image of an action corresponding to an HMD operation command transmitted from the watch 100 .
  • the watch 100 receives an HMD operation command corresponding to "clicking a selected item" in the HMD operation content 172 of the HMD cooperation operation DB 170 shown in FIG. Generate an image showing "lower".
  • the cooperation control unit 255 generates an action image corresponding to the received HMD operation command.
  • the HMD 200 retains in the memory 202 motion images associated with HMD operation commands.
  • motion information of the user 800 detected by the watch 100 may be transmitted to the HMD 200 together with the HMD operation command, and the cooperation control unit 255 may generate a motion image (simulated image) using the information.
  • the operation mode display may be an image display such as an icon instead of the character display.
  • the user 800 can intuitively understand how the motion is detected by the watch 100. As a result, it can be expected that the watch 100 will move its hand to improve the detection accuracy.
  • the watch 100 which is the first information processing device of the information processing system 900 of this embodiment, can communicate with the HMD 200, which is an external information processing device, and performs a predetermined cooperation start trigger operation.
  • a memory 202 that stores a motion detection unit 151 that detects the motion of the user 800 wearing the watch 100; and a motion determination unit 152 that determines whether the motion detected by the motion detection unit 151 is a cooperation start trigger motion.
  • a switching control unit 153 for switching the operation mode of the watch 100 .
  • the operation modes include a cooperative operation mode in which the watch 100 generates an operation command for operating the HMD 200 and transmits the operation command to the HMD 200, and a cooperative operation mode in which the watch 100 generates an operation command for operating the watch 100 and operates the watch 100 according to the generated operation command.
  • the HMD 200 which is the second information processing device, can communicate with the watch 100, which is an external information processing device.
  • a motion detection unit 251 that detects the motion of the motion detection unit 251
  • a motion determination unit 252 that determines whether or not the motion detected by the motion detection unit 251 is a cooperation start trigger motion
  • a switching control unit 253 that switches the operation mode of the HMD 200, Prepare.
  • the operation modes include a cooperative operation mode in which an operation command for operating the HMD 200 is received from the watch 100 and the HMD 200 is operated in accordance with the operation command; There is a single operation mode for operating the .
  • the switching control unit 253 determines that the action determination unit 252 is the HMD-side cooperation start trigger motion, and issues a detection notification indicating that the watch 100 has detected the watch-side cooperation start trigger motion. If both the reception from the watch 100 and the reception from the watch 100 occur within a predetermined period of time, the operation mode is switched to the cooperative operation mode.
  • the watch 100 operates in an operation mode for operating the watch 100 itself and an operation mode for operating the HMD 200. If the operation mode can be executed, the operation mode can be switched easily and with high accuracy. Specifically, when both the watch 100 and the HMD 200 detect a trigger action, the information processing device operated according to the user action detected by the watch 100 changes from the single operation mode, which is the watch 100, to It switches to the cooperative operation mode of the HMD 200 .
  • the operation mode switching is easy to use. It can be performed reliably and with high accuracy. Therefore, it is possible to eliminate or greatly reduce the possibility of misrecognition and malfunction in switching the operation mode.
  • myoelectric information acquired by the myoelectric sensor 131 is used to detect the trigger operation in the watch 100 .
  • the information processing devices of both the watch 100 and the HMD 200 can be freely operated.
  • the present embodiment in an information processing system having a plurality of information processing apparatuses capable of switching operation targets, when switching or selecting an operation mode to determine which one of the plurality of information processing apparatuses is to be operated, a simple method can be used.
  • the user-friendly operation mode can be switched and selected with high precision and accuracy. This is particularly effective in information processing devices such as the watch 100 and the HMD 200, which are difficult to operate and difficult to detect.
  • the watch 100 generates an HMD operation command based on the detected operation and transmits it to the HMD 200 in the cooperative operation mode, but the present invention is not limited to this.
  • information indicating the detected motion may be transmitted to the HMD 200, and the HMD 200 may generate an operation command corresponding to the motion.
  • the predetermined cooperation start trigger operation is not limited to the one described in the above embodiment.
  • FIG. 9 is a diagram explaining another example of the cooperation start trigger operation.
  • FIG. 9 shows a case in which an action of clenching a hand is predetermined as a cooperation start trigger action on the watch 100 side.
  • the myoelectric sensor 131 detects a gesture movement from the hand open state 821 to the hand closed and clenched state 822 .
  • the HMD 200 uses the left-eye line-of-sight sensor 235 and the right-eye line-of-sight sensor 236 to detect a state 831 in which the user 800 is looking in the direction of the watch 100 from a state 832 in which the user 800 is looking in a different direction from the watch 100, and Detected as a cooperation start trigger operation.
  • the acceleration sensor 232, the gyro sensor 233, and the geomagnetic sensor 234 that detect the posture and movement of the user 800 may be used to capture the state 831 in which the user 800 is looking toward the watch 100.
  • the watch 100 transmits to the HMD 200 an HMD operation command generated from the action detected by the watch 100 as operation information for operating the HMD 200 .
  • Actions on the watch 100 may be detected by other sensors 130 or means other than the myoelectric sensor 131 .
  • the sensors 130 such as the acceleration sensor 132, the gyro sensor 133, and the geomagnetic sensor 134 are used to detect the posture state and movement of the watch 100, and actions such as "lift and twist the hand” and “lift and stop the hand” are detected. may be obtained.
  • the line-of-sight sensor (the left-eye line-of-sight sensor 235 and/or the right-eye line-of-sight sensor 236) detects a change in the eye state of the user 800, the acceleration sensor 232, the gyro sensor, and the like.
  • the geomagnetic sensor 234 detects changes in the state of the user's 800 head, but other sensors 230 or other means may be used for detection.
  • the motions detected by the first information processing device and the second information processing device are not limited to the motions described above and may be any motions as long as they can be detected by some kind of sensor or means. , it is clear that the effect is obtained.
  • the cooperation start trigger action and the cooperation end trigger action are described as being different, but they may be the same.
  • the motion determination unit 152 and the motion determination unit 252 determine which trigger motion is in combination with the operation mode of the own device. That is, when the trigger action is detected in the single operation mode, it is determined as a cooperation start trigger action, and when the trigger action is detected in the cooperation operation mode, the cooperation end trigger Determined as an action.
  • the HMD 200 side may detect the trigger operation first and transmit it to the watch 100 .
  • the switching control unit 153 determines that the motion detected by the motion detection unit 151 within a predetermined period of time after receiving start trigger information from the HMD 200 indicating that the HMD-side cooperation start trigger motion has been detected is the cooperation start trigger motion. If so, an operation mode switching request is transmitted to the HMD 200 . Also, at this time, the operation mode of the watch 100 is switched to the HMD cooperation operation mode.
  • the switching control unit 253 transmits start trigger information to the watch 100 when detecting the HMD side cooperation start trigger operation. Then, if the operation mode switching request is received within a predetermined time after the transmission of the start trigger information, the switching control unit 253 switches the operation mode of the HMD 200 to the HMD cooperative operation mode.
  • the HMD 200 is operated by the action detected by the watch 100, but the reverse is also possible. That is, the watch 100 may be operated by actions detected by the HMD 200 .
  • the information processing system 900 of the present embodiment is an information processing system including the first information processing device and the second information processing device that are communicatively linked with each other. If the second information processing apparatus detects the second cooperation start trigger action within a predetermined time after detecting the start trigger action, the first information processing apparatus and the second information processing apparatus are operated in the cooperation operation mode.
  • This cooperative operation mode includes a first cooperative operation mode in which the second information processing device is operated in accordance with an action of the user of the first information processing device detected by the first information processing device, and a second information processing device. and a second cooperative operation mode in which the first information processing device is operated according to the user's motion of the second information processing device detected by the device.
  • the operation mode switching request is generated and transmitted by a device other than the device that first detected the cooperation start trigger action, but the present invention is not limited to this.
  • the operation mode switching request may be generated on the side of the device that first detects the cooperation start trigger operation.
  • the watch 100 first detects a cooperation start trigger action.
  • the switching control unit 153 When the cooperation start trigger operation is detected in step S ⁇ b>1102 , the switching control unit 153 generates start trigger information and transmits it to the HMD 200 .
  • the switching control unit 253 On the HMD 200 side, after receiving the start trigger information in step S2102, when the cooperation start trigger information is detected within the processing period in step S2201, the switching control unit 253 generates start trigger information (second start trigger information), Send to 100. That is, in the above embodiment, the HMD 200 generates an operation mode switching request and transmits it to the watch 100 as a reply. do.
  • the switching control unit 153 when the start trigger information is received from the HMD 200 within a predetermined period after the first detection of the cooperation start trigger operation or the generation and transmission of the start trigger information, the switching control unit 153 , generates an operation mode switching request and notifies it to the HMD 200 . Also, the operation mode of the watch 100 is switched. After receiving the operation mode switching request, the HMD 200 switches the operation mode.
  • the start trigger information is transmitted/received in reverse to the above.
  • the device switches operating modes.
  • the watch 100 is described as a specific example of the first information processing device
  • the HMD 200 is described as a specific example of the second information processing device, but the present invention is not limited to this.
  • the first information processing device and the second information processing device all devices with a function capable of detecting the operation of the device are targeted.
  • the first information processing device and the second information processing device are information processing devices other than smartwatches and HMDs, similar actions and effects can be obtained.
  • the first information processing device may be a smart phone that is held and used
  • the second information processing device may be a personal computer.
  • the posture change or movement is used as a trigger operation
  • the personal computer the posture, line of sight, or other movement of the user 800 is detected as the trigger operation.
  • Manipulate for example, in the smartphone, the posture change or movement is used as a trigger operation, and in the personal computer, the posture, line of sight, or other movement of the user 800 is detected as the trigger operation. Manipulate.
  • the information processing system 900 may further include a third information processing device.
  • FIG. 10 illustrates a case where the third information processing device is the smart phone 300 .
  • the watch 100, the HMD 200, and the smartphone 300 are linked in advance for communication.
  • the cooperation process may be two-party cooperation. Specifically, for example, when the watch 100 and the HMD 200 detect a predetermined first cooperation start trigger action at approximately the same timing, the watch 100 and the smartphone 300 start cooperation, and the watch 100 and the HMD 200 , and detects a predetermined second cooperation start trigger operation at substantially the same timing, the cooperation between the HMD 200 and the smartphone 300 is started.
  • the smartphone 300 transmits an operation command to the watch 100 and the HMD 200 respectively.
  • an operation command may be transmitted from watch 100 or HMD 200 to smartphone 300 .
  • the user 800 can operate the watch 100 or the HMD 200 to use the functions of the smartphone 300 .
  • the smartphone 300 can be operated while the smartphone 300 is in a pocket, bag, or the like.
  • it may be a three-party collaboration.
  • the watch 100 and the HMD 200 detect predetermined cooperation start trigger actions at approximately the same timing, the watch 100, the HMD 200, and the smartphone 300 start three-way cooperation.
  • three-party cooperation for example, as in the above-described embodiment or modification, cooperative processing is performed between the watch 100 and the HMD 200, and all the processing performed by each information processing device is transmitted to the smartphone 300 and processed. For example, all histories are accumulated in the smartphone 300 . Further, the three-party cooperation may be cooperation such that the screen of the smartphone 300 is transferred to the HMD 200 and displayed, and the operation of the smartphone 300 is performed on the watch 100 .
  • each processing example may be independent programs, or a plurality of programs may constitute one application program. Also, the order of performing each process may be changed.
  • the functions and the like of the present invention described above may be realized by hardware, for example, by designing them as integrated circuits.
  • the functions may be realized by software by having the microprocessor unit, CPU, etc. interpret and execute operation programs for realizing respective functions.
  • the implementation range of software is not limited, and hardware and software may be used together.
  • a part or all of each function may be realized by a server.
  • the server may be any form as long as it can cooperate with other components via communication to execute functions, and may be, for example, a local server, a cloud server, an edge server, a network service, or the like.
  • Information such as programs, tables, and files that implement each function may be stored in recording devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD cards, and DVDs. and may be stored in a device on a communication network.
  • recording devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD cards, and DVDs. and may be stored in a device on a communication network.
  • control lines and information lines shown in the diagram show what is considered necessary for explanation, and do not necessarily show all the control lines and information lines on the product. In practice, it may be considered that almost all configurations are interconnected.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Dermatology (AREA)
  • General Health & Medical Sciences (AREA)
  • Neurology (AREA)
  • Neurosurgery (AREA)
  • User Interface Of Digital Computer (AREA)
  • Position Input By Displaying (AREA)
PCT/JP2022/006168 2022-02-16 2022-02-16 情報処理装置および情報処理システム Ceased WO2023157121A1 (ja)

Priority Applications (6)

Application Number Priority Date Filing Date Title
JP2024500764A JP7727084B2 (ja) 2022-02-16 2022-02-16 情報処理装置および情報処理システム
CN202280091777.6A CN118696290A (zh) 2022-02-16 2022-02-16 信息处理装置和信息处理系统
US18/837,708 US12541246B2 (en) 2022-02-16 2022-02-16 Information processing device and information processing system
EP22927024.4A EP4481540A4 (en) 2022-02-16 2022-02-16 INFORMATION PROCESSING DEVICE AND INFORMATION PROCESSING SYSTEM
PCT/JP2022/006168 WO2023157121A1 (ja) 2022-02-16 2022-02-16 情報処理装置および情報処理システム
JP2025131942A JP2025159051A (ja) 2022-02-16 2025-08-07 情報処理装置

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2022/006168 WO2023157121A1 (ja) 2022-02-16 2022-02-16 情報処理装置および情報処理システム

Publications (1)

Publication Number Publication Date
WO2023157121A1 true WO2023157121A1 (ja) 2023-08-24

Family

ID=87577807

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/006168 Ceased WO2023157121A1 (ja) 2022-02-16 2022-02-16 情報処理装置および情報処理システム

Country Status (5)

Country Link
US (1) US12541246B2 (https=)
EP (1) EP4481540A4 (https=)
JP (2) JP7727084B2 (https=)
CN (1) CN118696290A (https=)
WO (1) WO2023157121A1 (https=)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20260099172A1 (en) * 2022-09-07 2026-04-09 Maxell, Ltd. Portable information terminal, terminal body, and attachment
US12379780B2 (en) * 2023-04-27 2025-08-05 Meta Platforms Technologies, Llc Manufacturing processes for biopotential-based wrist-wearable devices and resulting manufactured biopotential -based wrist-wearable devices

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016009503A (ja) * 2014-06-25 2016-01-18 トムソン ライセンシングＴｈｏｍｓｏｎ Ｌｉｃｅｎｓｉｎｇ デバイス同士をペアリングする方法と装置
US20160034887A1 (en) * 2014-07-31 2016-02-04 Lg Electronics Inc. Wearable device and method for controlling the same
JP2016085554A (ja) 2014-10-24 2016-05-19 セイコーエプソン株式会社 情報処理装置、表示装置、及び表示システム
US20160309286A1 (en) * 2015-04-17 2016-10-20 Samsung Electronics Co., Ltd. Electronic apparatus and method for identifying at least one pairing subject in electronic apparatus
US10671176B2 (en) 2014-09-30 2020-06-02 Apple Inc. Motion and gesture input from a wearable device
JP2021009619A (ja) * 2019-07-02 2021-01-28 富士ゼロックス株式会社 情報処理システム及びプログラム

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101698098B1 (ko) * 2014-07-23 2017-01-19 엘지전자 주식회사 이동 단말기 및 그것의 제어방법
US9978264B2 (en) 2016-07-28 2018-05-22 16Lab Inc Method for device interaction and identification

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016009503A (ja) * 2014-06-25 2016-01-18 トムソン ライセンシングＴｈｏｍｓｏｎ Ｌｉｃｅｎｓｉｎｇ デバイス同士をペアリングする方法と装置
US20160034887A1 (en) * 2014-07-31 2016-02-04 Lg Electronics Inc. Wearable device and method for controlling the same
US10671176B2 (en) 2014-09-30 2020-06-02 Apple Inc. Motion and gesture input from a wearable device
JP2016085554A (ja) 2014-10-24 2016-05-19 セイコーエプソン株式会社 情報処理装置、表示装置、及び表示システム
US20160309286A1 (en) * 2015-04-17 2016-10-20 Samsung Electronics Co., Ltd. Electronic apparatus and method for identifying at least one pairing subject in electronic apparatus
JP2021009619A (ja) * 2019-07-02 2021-01-28 富士ゼロックス株式会社 情報処理システム及びプログラム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4481540A4

Also Published As

Publication number Publication date
US20250138627A1 (en) 2025-05-01
JP7727084B2 (ja) 2025-08-20
EP4481540A4 (en) 2025-12-31
JPWO2023157121A1 (https=) 2023-08-24
EP4481540A1 (en) 2024-12-25
JP2025159051A (ja) 2025-10-17
CN118696290A (zh) 2024-09-24
US12541246B2 (en) 2026-02-03

Similar Documents

Publication Publication Date Title
WO2021073743A1 (en) Determining user input based on hand gestures and eye tracking
US20210263593A1 (en) Hand gesture input for wearable system
EP3090331B1 (en) Systems with techniques for user interface control
US9916006B2 (en) Eye-wearable device user interface and method
CN102779000B (zh) 一种用户交互系统和方法
EP4276591B1 (en) Interaction method, electronic device, and interaction system
JP2025159051A (ja) 情報処理装置
KR20150092165A (ko) Ｉｍｕ를 이용한 직접 홀로그램 조작
KR20160113139A (ko) 시선 스와이프 선택
US20160132189A1 (en) Method of controlling the display of images and electronic device adapted to the same
US20250208812A1 (en) Information processing apparatus, photographed image transfer and display system, and photographed image transfer and display method
US20250173000A1 (en) Systems for navigating user interfaces on multiple devices based on spatial orientations of a user's hand, and method of use thereof
JP2001154794A (ja) まばたきによるクリック機能を有するポインティング装置
US12197652B2 (en) Control device and control method with set priorities for input operations in competitive relationship
JP2024081664A (ja) 情報処理システム
US20240201502A1 (en) Wearable terminal device, program, and display method
WO2003003185A1 (en) System for establishing a user interface
CN103713387A (zh) 电子设备和采集方法
US12314470B2 (en) Information processing system used in cross reality
CN112445328A (zh) 映射控制方法及装置
JP7538886B2 (ja) 画像表示装置および画像表示方法
CN101002161A (zh) 将可选择的数据元受身体控制地传输给终端设备的方法、系统和设备
US20180143436A1 (en) Head-operated digital eyeglasses
US20260079580A1 (en) Information processing device, information processing method, and program
JP2026002360A (ja) 電子機器、電子機器の制御方法、およびプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22927024

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2024500764

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 18837708

Country of ref document: US

WWE Wipo information: entry into national phase

Ref document number: 202280091777.6

Country of ref document: CN

WWE Wipo information: entry into national phase

Ref document number: 2022927024

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2022927024

Country of ref document: EP

Effective date: 20240916

WWP Wipo information: published in national office

Ref document number: 18837708

Country of ref document: US

WWG Wipo information: grant in national office

Ref document number: 18837708

Country of ref document: US