WO2023286316A1 - 入力装置、システム、および制御方法 - Google Patents

入力装置、システム、および制御方法 Download PDF

Info

Publication number
WO2023286316A1
WO2023286316A1 PCT/JP2022/007274 JP2022007274W WO2023286316A1 WO 2023286316 A1 WO2023286316 A1 WO 2023286316A1 JP 2022007274 W JP2022007274 W JP 2022007274W WO 2023286316 A1 WO2023286316 A1 WO 2023286316A1
Authority
WO
WIPO (PCT)
Prior art keywords
input device
information
user
display
finger
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/007274
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
郁男 山野
邦仁 澤井
京二郎 永野
慧 高橋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Priority to CN202280048109.5A priority Critical patent/CN117616366A/zh
Priority to US18/576,165 priority patent/US20250093955A1/en
Priority to JP2023535096A priority patent/JPWO2023286316A1/ja
Publication of WO2023286316A1 publication Critical patent/WO2023286316A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/014—Hand-worn input/output arrangements, e.g. data gloves
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/016—Input arrangements with force or tactile feedback as computer generated output to the user
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/038—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
    • G06F3/0383—Signal control means within the pointing device
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/033—Indexing scheme relating to G06F3/033
    • G06F2203/0331—Finger worn pointing device
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/038—Indexing scheme relating to G06F3/038
    • G06F2203/0384—Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/02—Input arrangements using manually operated switches, e.g. using keyboards or dials
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0354—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of two-dimensional [2D] relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
    • G06F3/03547—Touch pads, in which fingers can move on a surface

Definitions

  • the present disclosure relates to input devices, systems, and control methods.
  • mice Conventionally, various input devices such as mice, keyboards, and touch pads are used for inputting operations to information processing devices.
  • Patent Literature 1 discloses a user operation sensing device that is worn in a ring shape on a user's finger.
  • the device described in Patent Document 1 uses two types of sensors to detect the movement of the finger (thumb) to which the device is attached, and determines a click operation or direction key operation.
  • a ring-shaped main body that is worn on a user's finger, one or more sensors provided on the main body that detect information of operation input by another finger, an external device and wireless
  • An input device comprising: a communication unit that performs communication; and a control unit that controls transmission of information of the operation input detected by the one or more sensors to the external device by the communication unit.
  • an operation input that acquires information on the operation input from one or more sensors that are provided in a ring-shaped main body that is worn on a user's finger and that detects information on the operation input by another finger.
  • a system comprising an information acquisition unit, a line-of-sight information acquisition unit that acquires line-of-sight information of the user, and a presentation control unit that controls information to be presented to the user based on the information of the operation input and the line-of-sight information. Suggest.
  • the processor detects information of an operation input by another finger using one or more sensors provided in a ring-shaped main body attached to a user's finger; and A control method is proposed, including controlling transmission of the operation input information to an external device by a communication unit that performs wireless communication.
  • the processor is provided in a ring-shaped main body that is worn on a user's finger, and acquires information on the operation input from one or more sensors that detect information on the operation input by another finger. acquiring line-of-sight information of the user; and controlling information presented to the user based on the information of the operation input and the line-of-sight information.
  • FIG. 1 is a diagram describing an overview of an information processing system according to an embodiment of the present disclosure
  • FIG. It is a block diagram showing an example of composition of a display and an input device which are included in an information processing system by this embodiment.
  • 1 is a schematic cross-sectional view showing an example of the structure of an input device according to this embodiment;
  • FIG. It is the figure which looked at the input device by this embodiment from the side.
  • 4 is a flow chart showing an example of the flow of operation processing of the information processing system according to the present embodiment; It is a figure explaining an example of the operation method with respect to the two-dimensional image by this embodiment. It is a figure explaining an example of the operation method with respect to the two-dimensional image by this embodiment.
  • FIG. 10 is a diagram illustrating UI display in an operation on a two-dimensional image according to this embodiment;
  • FIG. 10 is a diagram illustrating an example of an operation method (position adjustment mode) for a two-dimensional image according to this embodiment; It is a figure explaining an example of the relationship between the direction of touch operation in the position adjustment mode by this embodiment, and the moving direction of a pointer. It is a figure explaining an example of the relationship between the direction of touch operation in the position adjustment mode by this embodiment, and the moving direction of a pointer. It is a figure explaining the tap operation with the fingertip of the three-dimensional UI by this embodiment.
  • FIG. 11 is a diagram illustrating a gripping and releasing operation of a virtual object according to the embodiment; It is a figure explaining an example of detection of speaking operation by this embodiment.
  • FIG. 10 is a diagram for explaining determination of a releasing motion based on prediction of a center of rotation according to the present embodiment;
  • FIG. 10 is a diagram illustrating AR drawing of characters and pictures using a fingertip according to the present embodiment;
  • FIG. 10 is a diagram illustrating a gripping/releasing operation of a virtual object by a touch operation according to the embodiment; It is a figure explaining the remote operation of the virtual object by this embodiment.
  • FIG. 10 is a diagram illustrating movement of the pointer UI according to movement of the input device in the Yaw direction according to the embodiment; It is a figure explaining direction of pointer UI by this embodiment.
  • FIG. 10 is a diagram illustrating movement of the pointer UI according to movement of the input device in the Pitch direction according to the embodiment; It is a figure explaining the case where the input device by this embodiment is each mounted
  • FIG. 1 is a diagram illustrating an overview of an information processing system according to an embodiment of the present disclosure. As shown in FIG. 1, the information processing system according to this embodiment includes a display device 10 and an input device 20 .
  • the input device 20 is worn on the user's finger and has a function of detecting information of operation input to the display device 10 by the user.
  • the input device 20 is provided with a touch sensor 230 as an example of a sensor, and can detect touch operation information as operation input information.
  • the input device 20 transmits information on the detected operation input to the display device 10, which is an external device.
  • the display device 10 is an example of an information processing device that presents information to the user.
  • visual information is presented as an example of information presented to the user.
  • Other examples of information presented to the user include auditory information presented by an audio output device, vibration information presented by a vibration device, and the like.
  • the display device 10 may be an optically transmissive or non-transmissive HMD (Head Mounted Display) worn on the user's head.
  • HMD Head Mounted Display
  • the display device 10 can be realized by a spectacles-type device in which an optical transmission type display section 160 is provided in the lens section.
  • the display unit 160 superimposes an image (hereinafter referred to as a virtual object) as information of the virtual space on the real space so that the user can directly see the real space.
  • the display device 10 is an information processing device that provides a so-called AR (Augmented Reality) to the user.
  • An information processing apparatus that provides AR according to the present embodiment is not limited to an optically transmissive device, and may include a non-transmissive device using a video see-through display.
  • an image captured in the direction of the user's head (image in real space) is displayed in real time, and a virtual object is displayed superimposed on the image.
  • a video see-through display with a configuration that dynamically shields light in real space, it is possible to switch between a transmissive display (optical see-through display) and a non-transmissive display.
  • a virtual object may be a two-dimensional content, a three-dimensional object, or a UI (user interface).
  • Two-dimensional contents include, for example, websites, SNS (Social Networking Service) screens, photographs and videos, and electronic books.
  • the user can perform operation input from the input device 20 on the virtual object displayed on the display unit 160 of the display device 10 .
  • a touch operation such as a tap operation or a swipe operation with the thumb on the touch sensor 230 provided on the ring-shaped input device 20 fitted to the index finger
  • the virtual object can be selected or determined.
  • Manipulation and scrolling operations can be performed.
  • the ring-type input device 20 worn on the finger can be operated and input by another finger, the operation in daily life is more natural, less burdensome, and less troublesome. It is possible to perform operation input with reduced For example, a method of directly manipulating a virtual object by raising one's arm forward may cause conspicuous resistance in a city where there are people around. Also, the method of directly tapping the glasses-type device requires bringing the hand to the face, so the arm tends to get tired. In addition, the method using a grip-type remote controller (remote controller) is troublesome to carry around or take out.
  • the input device 20 that is worn on the finger and allows operation input to be performed at hand even when the hand is lowered, the above-described annoyance can be reduced, and the operation in daily life can be performed. , it is possible to realize more natural and low-load operation input.
  • FIG. 2 is a block diagram showing an example of the configuration of the display device 10 and the input device 20 included in the information processing system according to this embodiment. Each device will be described below.
  • the display device 10 (information processing device) includes, as shown in FIG. , and a storage unit 180 . Note that the configuration shown in FIG. 1 is an example, and the configuration of the display device 10 is not limited to this.
  • the communication unit 110 communicates with an external device by wire or wirelessly, and transmits and receives data.
  • the communication unit 110 can communicate with the input device 20 through wireless communication such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).
  • the communication unit 110 supports wired/wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), mobile communication network (LTE (Long Term Evolution), 3G (third generation mobile communication system), 4G (fourth generation mobile communication system), 5G (fifth generation mobile communication system)), etc., can be connected to the Internet to transmit and receive data to and from a server on the network.
  • the communication unit 110 also functions as an operation input information acquisition unit that acquires operation input information from the input device 20 .
  • control unit 120 functions as an arithmetic processing device and a control device, and controls overall operations within the display device 10 according to various programs.
  • the control unit 120 is realized by an electronic circuit such as a CPU (Central Processing Unit), a microprocessor, or the like.
  • the control unit 120 may also include a ROM (Read Only Memory) that stores programs to be used, calculation parameters, and the like, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.
  • ROM Read Only Memory
  • RAM Random Access Memory
  • the control unit 120 also functions as a display control unit 121.
  • the display control unit 121 controls display of the virtual object on the display unit 160 .
  • the display control unit 121 according to this embodiment is an example of a presentation control unit that controls the presentation of information to the user.
  • a virtual object is a two-dimensional or three-dimensional image. It should be noted that the display control unit 121, based on the analysis result of the captured image in the direction of the user's head acquired by the outward facing camera 140 (captured image including at least the user's visual field range), determines the virtual object to be superimposed on the real space. It is also possible to control the display position and orientation.
  • the display control unit 121 can also control the display position and orientation of the virtual object in consideration of the user's own position and orientation. Analysis of the captured image (object recognition, etc.) may be performed by the control unit 120 . Also, the user's own position and posture can be detected by various sensors provided in the display device 10 . Various sensors include, for example, the outward facing camera 140, a gyro sensor, an acceleration sensor, a geomagnetic sensor, a position measuring unit, and the like.
  • the position measurement unit may be a measurement unit that measures absolute position (for example, position measurement using GNSS (Global Navigation Satellite System)), or a measurement unit that measures relative position (for example, Wi-Fi or Bluetooth signal position measurement).
  • the data of the virtual object to be displayed may be data stored in the storage unit 180, data received from an external device via the communication unit 110, or data generated by the control unit 120. It may be data.
  • External devices include, for example, servers on the Internet and communication terminals owned by users (for example, smartphones, tablet terminals, wearable devices, etc.).
  • the display control unit 121 can perform display control of the virtual object based on the information of the operation input received from the input device 20 via the communication unit 110 .
  • the display control unit 121 performs control such as selection, determination, scrolling, or zooming with respect to the virtual object according to the received operation input information.
  • the operation input information is information detected by various sensors provided in the input device 20 .
  • touch operation information detected by the touch sensor 230 and hand movement information detected by the motion sensor 240 may be used.
  • the display control unit 121 may control the operation position on the display screen of the display unit 160 according to the user's line-of-sight information detected by the line-of-sight detection unit 130 . That is, the display control unit 121 can perform a pointing operation based on line-of-sight input.
  • the "line-of-sight information" is, for example, information indicating the direction of the user's line of sight.
  • the display control unit 121 recognizes the position of the user's line-of-sight center (gazing point) on the display screen of the display unit 160 as the operation position based on the user's line-of-sight direction.
  • the display control unit 121 recognizes the image displayed at the operation position as the operation target, and executes control on the image according to the operation input information input by the input device 20 .
  • the user can operate the input device 20 at hand while directing the line of sight to the virtual object to be operated, thereby performing an operation input (for example, a scroll operation, etc.) on any virtual object. becomes.
  • Such an operation method combining line-of-sight information and operation input information from the input device 20 may be used, for example, as an operation method mainly for two-dimensional images (including the above-described two-dimensional content). Specifically, it will be described later with reference to FIGS. 6 to 12.
  • FIG. Further, the display control unit 121 may perform vibration feedback using the vibrating unit 250 provided in the input device 20 as feedback of operation input by the user using the input device 20 .
  • methods of operation include a method of directly operating the virtual object and a method of remote operation.
  • the method of direct manipulation includes, for example, a method of controlling the display of a virtual object according to the user's movement of touching, grasping, and releasing the virtual object (see FIGS. 13 to 19).
  • the display control unit 121 appropriately uses the positions and movements of the user's hands and fingers obtained by analyzing the captured image acquired by the outward facing camera 140 and the information of the operation input received from the input device 20 .
  • the operation input information received from the input device 20 includes, for example, touch operation information detected by the touch sensor 230, motion information detected by the motion sensor 240, and information on the input device 20 and the thumb detected by the distance sensor.
  • the distance, the distance between the input device 20 and a predetermined knot portion, and the like can be mentioned.
  • the display control unit 121 may perform vibration feedback using the vibrating unit 250 provided in the input device 20 as feedback of operation input by the user using the input device 20 .
  • the remote control method includes, for example, a method of displaying a pointer UI 366 that moves according to the orientation of the input device 20 on the display unit 160 (see FIGS. 20 to 25).
  • the user moves the input device 20 to point the pointer UI 366 to an arbitrary virtual object (the virtual object can be recognized as an operation target by the display control unit 121), and then performs a touch operation or the like on the input device 20.
  • An operation input can be performed on the virtual object.
  • control unit 120 may perform power ON/OFF control, volume adjustment, display ON/OFF control, etc. according to an operation input from the operation input unit 150 provided in the display device 10 . Further, the control unit 120 may control audio output from the audio output unit 170 . For example, when the virtual object is a moving image, the reproduction of the sound is exemplified.
  • the line-of-sight detection unit 130 is a line-of-sight information acquisition unit that has a function of detecting (acquiring) information (line-of-sight information) regarding the user's line of sight.
  • the line-of-sight detection unit 130 may be an inward-facing camera that captures an image of the user's eyes when the display device 10 is worn, or may be a myoelectric sensor that detects an electrooculogram.
  • the line-of-sight detection unit 130 detects, for example, the line-of-sight direction as line-of-sight information based on information acquired by various sensors. There are no particular restrictions on the method of detecting line-of-sight information.
  • the outward facing camera 140 is an imaging unit provided in the display device 10 so as to capture an image in the direction of the user's head when the display device 10 is worn.
  • the angle of view of outward facing camera 140 includes at least a field of view range (also referred to as field of view).
  • the outward camera 140 includes a lens system including an imaging lens, an aperture, a zoom lens, a focus lens, and the like, a driving system for performing focusing and zooming operations on the lens system, and photoelectric conversion of imaging light obtained by the lens system.
  • Each has a solid-state imaging element array or the like that converts to generate an imaging signal.
  • the solid-state imaging device array may be implemented by, for example, a CCD (Charge Coupled Device) sensor array or a CMOS (Complementary Metal Oxide Semiconductor) sensor array.
  • Operation input unit 150 receives an operation from a user and outputs information on the received operation to control unit 120 .
  • the operation input unit 150 can be realized by, for example, push buttons, touch sensors, optical sensors, and the like.
  • the display unit 160 is implemented by, for example, a transmissive display.
  • a transmissive display is a display that can directly deliver light in the real space to the user's eyes. A user can directly view the real space through the transmissive display.
  • a transmissive display may be, for example, an optical see-through display.
  • the optical see-through display can employ known forms including a half-mirror system, a light guide plate system, a retinal direct drawing system, and the like.
  • the audio output unit 170 has a function of outputting audio.
  • the audio output unit 170 may be configured as headphones, earphones, or bone conduction speakers.
  • the storage unit 180 is implemented by a ROM (Read Only Memory) that stores programs, calculation parameters, and the like used in the processing of the control unit 120, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.
  • ROM Read Only Memory
  • RAM Random Access Memory
  • the configuration of the display device 10 according to the present embodiment has been specifically described above. Note that the configuration of the display device 10 is not limited to the example shown in FIG. For example, all the configurations shown in FIG. 2 may not be included, and other configurations may be included. Moreover, the display device 10 may be configured by a plurality of devices. Moreover, although not shown in FIG. 2 , the display device 10 has a power supply unit that supplies power to each component of the display device 10 .
  • the input device 20 has a communication section 210 , a control section 220 , a touch sensor 230 , a motion sensor 240 , a vibration section 250 and a storage section 260 .
  • the configuration shown in FIG. 1 is an example, and the configuration of the display device 10 is not limited to this.
  • the communication unit 210 communicates with an external device by wire or wirelessly, and transmits and receives data.
  • the communication unit 210 can communicate with the display device 10 through wireless communication such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).
  • control unit 220 functions as an arithmetic processing device and a control device, and controls overall operations within the input device 20 according to various programs.
  • the control unit 220 is implemented by an electronic circuit such as a CPU (Central Processing Unit), a microprocessor, or the like.
  • the control unit 220 may also include a ROM (Read Only Memory) that stores programs to be used, calculation parameters, and the like, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.
  • ROM Read Only Memory
  • RAM Random Access Memory
  • the control unit 220 controls transmission of information detected by the touch sensor 230 and information detected by the motion sensor 240 from the communication unit 210 to the display device 10 . These pieces of information (detection results) correspond to information of operation input to the display device 10 performed by the user using the input device 20 .
  • the control unit 220 also functions as a vibration control unit 221.
  • the vibration control section 221 performs vibration control of the vibration section 250 according to the control signal received from the display device 10 by the communication section 210 .
  • Such vibration control may be presented, for example, as feedback of operational input by the user.
  • the touch sensor 230 has a function of detecting touch operations by the user.
  • the touch sensor 230 is provided on the surface of the main body of the input device 20 worn on the user's finger, and at a position facing another finger other than the finger on which the touch sensor 230 is worn. For example, when the input device 20 is worn on the index finger as shown in FIG. 1, the touch sensor 230 is positioned opposite the thumb, so the user can perform a touch operation on the touch sensor 230 with the thumb.
  • the touch operation includes, for example, a tap operation, a swipe operation, a hold operation (long press), and the like.
  • the touch sensor system includes, for example, a resistive film system, a capacitive system, an ultrasonic system, an optical system, etc., and any of them may be used in this embodiment.
  • a single touch sensor 230 may be provided, or a plurality of touch sensors may be provided.
  • Motion sensor 240 has a function of detecting motion of the input device 20 itself.
  • motion sensor 240 may be a 3-axis acceleration sensor and a 3-axis gyro sensor.
  • the motion sensor 240 may further include a geomagnetic sensor.
  • the vibrating section 250 has a function of presenting tactile information (tactile stimulation to the skin). Note that the method of vibration is not particularly limited. A single vibrating unit 250 may be provided, or a plurality of vibrating units may be provided.
  • the storage unit 260 is implemented by a ROM (Read Only Memory) that stores programs, calculation parameters, and the like used in the processing of the control unit 220, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.
  • ROM Read Only Memory
  • RAM Random Access Memory
  • the configuration of the input device 20 has been specifically described above. Note that the configuration of the input device 20 is not limited to the example shown in FIG. For example, all the configurations shown in FIG. 2 may not be included, and other configurations may be included.
  • input device 20 may have one or more other sensors.
  • the input device 20 may further have a distance sensor.
  • the distance sensor may be an optical sensor (for example, ToF (Time of Flight) method).
  • the input device 20 has a power supply section that supplies power to each component of the input device 20 .
  • FIG. 3 is a schematic cross-sectional view showing an example of the structure of the input device 20 according to this embodiment.
  • the input device 20 according to this embodiment is formed in a shape that can be worn on a finger.
  • the input device 20 is formed with a ring-shaped shape.
  • the cross-sectional view shown in FIG. 3 is a view of the input device 20 viewed from the direction in which the finger is passed.
  • the ring-shaped shape may be a ring-shaped closed shape, or may be a shape (C-shaped ring) that is partially open and does not close ring-shaped as shown in FIG. It is also possible to deal with a slight difference in finger size by making the shape that does not close in a ring shape. Also, portions of the body may include deformable portions to accommodate different sizes. Details will be described later with reference to FIGS.
  • the input device 20 may be formed on the assumption that it is worn on, for example, the base or middle knuckle of the index finger.
  • a touch sensor 230 is provided on the surface of the main body of the input device 20 at a position facing a finger different from the finger on which the input device 20 is worn when worn. For example, when the input device 20 is worn on the index finger, it is assumed that touch operation is performed with the thumb of the same hand.
  • the touch sensor 230 can be formed vertically on one side of the surface of the main body.
  • FIG. 4 shows a side view of the input device 20 (the side where the touch sensor 230 is provided). In FIG. 4, the input device 20 is attached to the base of the index finger of the right hand.
  • the touch sensor 230 has a width that allows a swipe operation in the up, down, left, and right directions with the thumb.
  • a motion sensor 240 (for example, an acceleration sensor and a gyro sensor) is provided inside the main body of the input device 20 to detect the movement (orientation information) of the input device 20 itself.
  • the location of the motion sensor 240 is not particularly limited. For example, it may be arranged above the main body (at the dorsal side of the finger when attached to the finger), or may be arranged at the lower side (at the ventral side of the finger when attached to the finger). position), and may be provided on either the left or right side (side portion).
  • a push switch section 290 may be arranged on the back side of the touch sensor 230 .
  • a push operation (click operation) of the touch sensor 230 can be detected by the push switch section 290 .
  • a pressure sensor may be placed behind the touch sensor 230 . The pressure sensor detects the pressing pressure of the user's thumb, for example, on the touch sensor, and the ON operation is detected when the detected value exceeds a preset threshold value. At this time, control may be performed such that vibration feedback using the vibrating section 250 is returned to the user.
  • the touch sensor does not detect the user's touch
  • the ON operation is not detected even if the detected pressure exceeds the threshold value. You can avoid being detected.
  • analog value obtained by the pressure sensor for analog quantity operation such as zoom operation of the display area according to the pressure value.
  • a distance sensor 280 may be provided on the main body of the input device 20 .
  • the distance sensor 280 measures the distance to another finger (for example, thumb) and the distance to a predetermined joint (for example, middle joint) of the worn finger.
  • the distance sensor 280 may be, for example, a ToF (Time Of Flight) optical sensor. It is also possible to arrange a plurality of distance sensors 280 to widen the detection range.
  • a vibrating section 250 may be provided inside the main body section of the input device 20 .
  • the vibrating section 250 is provided above the main body section, but the present embodiment is not limited to this arrangement.
  • a unit section 300 is provided inside the main body section of the input device 20 .
  • Unit section 300 may be formed by one or more electronic circuits that implement control section 220 , communication section 210 , and storage section 260 .
  • the unit section 300 is provided above the main body section, but the present embodiment is not limited to this arrangement.
  • a power supply section 270 that supplies power to each component of the input device 20 is provided inside the main body of the input device 20 inside the main body of the input device 20, a power supply section 270 that supplies power to each component of the input device 20 is provided.
  • the power source section 270 is provided on the right side of the main body section, but this embodiment is not limited to this arrangement.
  • the power supply unit 270 may be arranged above the main body.
  • a ring for wearing on a finger in which parts are arranged in an L-shape on the dorsal side of the index finger and on the side facing the thumb, and the L-shaped part is fixed.
  • the ring-shaped part can be replaced according to the size of the user's finger.
  • an elastic member for the ring-shaped component it is possible to propose a stable input device that follows changes in thickness around the finger (proximal part) due to bending and stretching of the finger.
  • the main body of the input device 20 may be provided with a bent portion in which no substrate is arranged, and this portion may be configured to be plastically deformable so as to accommodate differences in finger size of each user.
  • the curved portion between the upper portion and the (right/left) side portion of the body portion of the input device 20 corresponds.
  • the user pushes the bent portion of the main body from the outside to plastically deform the finger so as to apply an appropriate pressure to adjust the shape and size.
  • the bent portion (or the main body portion including the bent portion) may be made of a plastically deformable material such as an aluminum plate. Also, the bent portion may be provided on both the left and right sides, or may be provided on only one side.
  • FIG. 5 is a flow chart showing an example of the flow of operation processing of the information processing system according to this embodiment.
  • the display control unit 121 of the display device 10 displays one or more virtual objects in AR on the display unit 160 (step S103).
  • AR display refers to a case in which a virtual object is superimposed and displayed on the real space in the optically transmissive display unit 160 . Also, here, as an example, it is assumed that a two-dimensional image virtual object is displayed.
  • the display device 10 uses the line-of-sight detection unit 130 to detect the user's line of sight (step S106).
  • the display control unit 121 of the display device 10 identifies the virtual object to be operated based on the line-of-sight information (step S109). Specifically, the display control unit 121 controls the virtual object (or part of the virtual object) is recognized as an operation target.
  • the display device 10 receives the detection result (touch operation information) of the touch sensor 230 from the input device 20 via the communication unit 110 (step S112).
  • the display control unit 121 of the display device 10 controls display of the virtual object to be operated according to the touch operation (step S115). For example, the display control unit 121 performs determination control according to a tap operation, or performs scroll control according to a swipe operation.
  • control unit 120 of the display device 10 controls the input device 20 to perform vibration feedback as necessary (step S118). That is, the control unit 120 transmits a vibration control signal to the input device 20 , and the vibration is presented in the vibration unit 250 of the input device 20 .
  • steps S106-S109 and step S112 may be continuously performed in parallel.
  • the virtual object is a two-dimensional image
  • an operation method that combines pointing by line-of-sight detection and operation instructions by touch operation has been explained.
  • an operation method that combines pointing based on line-of-sight detection and the detection result of the motion sensor 240 is also conceivable. For example, by twisting the wrist left and right while the input device 20 is worn on the finger, zoom control (magnification/reduction) of the operation target can be performed.
  • zoom control magnification/reduction
  • the analysis result of the captured image captured by the outward camera 140, the detection result of the motion sensor 240, the detection result of the distance sensor provided in the input device 20, the detection result of the touch sensor 230, etc. may be used. Details of each operation method will be described later with reference to FIGS.
  • FIG. 6 is a diagram illustrating an example of an operation method for a two-dimensional image according to this embodiment. For example, as shown in FIG. 6, two-dimensional image virtual objects 310 to 330 are displayed in the field of view of the user on the optical transmission type display unit 160 .
  • the virtual objects 310 to 330 are panels (two-dimensional images) that display two-dimensional content. For example, a panel that displays time-related information is displayed in the upper left area of the field of vision, a panel that displays video information such as sports broadcasts is displayed in the upper right area, and a panel that displays a web browser screen is displayed in the front area. is displayed.
  • a non-display area is provided in a part of the field of view (for example, the central area of the display unit 160) so as to avoid collisions with surrounding people and objects caused by walking.
  • the display positions of these virtual objects 310 to 330 (each panel displaying two-dimensional content) displayed on the display unit 160 are not localized in the real space, and relative display is possible even if the head moves.
  • the display is based on the coordinate axes of the display device 10 itself so that the position does not change. This enables the user to act like viewing a website while walking.
  • Decision control, scroll control, etc. for each of the virtual objects 310 to 330 can be performed by detecting the user's line of sight and operating the input device 20 .
  • the virtual object 310 is recognized as an operation target (no particular cursor is displayed).
  • the web browser displayed as the virtual object 310 can be controlled to scroll upward in FIG.
  • the touch sensor 230 can also perform horizontal swipe operations (see FIG. 4).
  • the operation target is a web browser
  • left and right scroll control may be performed. Whether analog screen scrolling control or command-like control is performed for the same swipe operation depends on the setting of the image to be operated.
  • FIG. 7 is a diagram illustrating an example of an operation method for a two-dimensional image according to this embodiment.
  • a virtual object 340 that is a panel displaying a web browser screen is displayed on the display unit 160 .
  • the display control unit 121 executes screen scrolling at a speed corresponding to the line-of-sight position V detected by the user's line of sight, for example.
  • the display control unit 121 scrolls the screen downward faster as the line-of-sight position V moves downward from the screen center line C, and scrolls upward more quickly as the line-of-sight position V moves upward from the screen center line C. to scroll the screen quickly.
  • the speed of screen scrolling is made faster as the distance from the screen center line C increases. This allows the user to scan the entire screen to find the place they want to read.
  • the display control unit 121 performs control to stop scrolling the screen while the touch sensor 230 is being touched by the input device 20 . This allows the user to stop scrolling the screen at any point while browsing the entire screen.
  • the display control unit 121 performs fine adjustment scrolling (slow scrolling) of the screen when an operation is performed to move the touch sensor 230 in the vertical direction while touching the touch sensor 230 with a finger.
  • scroll control in the horizontal direction can also be performed in the same manner.
  • up, down, left and right in the touch sensor 230 of the input device 20 are as shown in FIG. That is, when the input device 20 is fitted on the finger, the back side of the hand is the upward direction, and if the input device 20 is worn on the right hand, the base of the finger is the right direction.
  • the up, down, left, and right directions correspond to the up, down, left, and right directions on the web browser screen.
  • FIG. 8 is a diagram illustrating an example of an operation method for a two-dimensional image according to this embodiment.
  • the display control unit 121 can recognize the virtual object to be operated based on the line-of-sight detection by the line-of-sight detection unit 130 , and can perform determination control according to the tap operation on the input device 20 .
  • the display control unit 121 first recognizes the operation target from the line-of-sight position V (corresponding to the gaze point) by line-of-sight detection, as shown in the left part of FIG. 8 (in this case, the cursor may not be displayed).
  • the display area 342 of the virtual object 340 which is a panel displaying the web browser screen, can be recognized as an operation target.
  • the display control unit 121 performs determination control when a tap operation (or pressing of the push switch unit 290) is performed on the input device 20.
  • FIG. In the decision control, for example, screen transition to the link destination (web page of the article body) associated with the display area 342 is performed.
  • the UI (user interface)
  • the UI is not displayed at the tapped location, but the present embodiment is not limited to this, and the UI may be displayed temporarily at the moment of tapping. This makes it possible to explicitly feed back the operation position to the user.
  • FIG. 9 is a diagram illustrating UI display in operations on a two-dimensional image according to this embodiment.
  • the display control unit 121 displays a ring-shaped UI display 344a at the operation position (line-of-sight position V), and then displays the screen on the right side of FIG.
  • control is performed to make the circle smaller and disappear in a short time.
  • the UI display method is not limited to the example shown in FIG.
  • the display control unit 121 may highlight the link area selected by the tap operation (the display area where the link is pasted; for example, the display area 342 shown on the right side of FIG. 8) for a short time.
  • the operation position is the position of the operation target indicated by the line-of-sight position V (gazing position) obtained by line-of-sight detection.
  • V gaze position
  • FIG. 10 is a diagram for explaining an example of an operation method (position adjustment mode) for a two-dimensional image according to this embodiment.
  • the display control unit 121 recognizes the line-of-sight position V (operation position) by detecting the user's line of sight, as shown on the left side of FIG. 10 (at this time, the pointer does not appear).
  • the display control unit 121 displays a pointer 346 (an example of UI display) and shifts to the position adjustment mode. to go into.
  • the display control unit 121 moves the pointer 346 according to the movement of the finger on the touch sensor 230 (up, down, left, and right).
  • the display control unit 121 performs determination control by touch release by releasing the finger from the touch sensor 230, clicking operation by pressing the push switch unit 290 provided on the back of the touch sensor 230, or the like. I do.
  • FIG. 11 is a diagram illustrating an example of the relationship between the direction of the touch operation and the moving direction of the pointer in the position adjustment mode according to this embodiment.
  • the coordinate axes of the touch sensor 230 touch pad
  • the X-axis in touch operation with touch sensor 230 coincides with the Z-axis of input device 20 itself, which will be described later with reference to FIG. 21, and the Y-axis in touch operation coincides with the Y-axis of input device 20 itself.
  • the display control unit 121 aligns the coordinate axes of the touch operation with the coordinate axes on the screen of the virtual object 350, and moves the pointer 352a.
  • Such coordinate setting enables the user to intuitively perform operations.
  • the present embodiment is not limited to this, and as shown in FIG. A case in which the touch operation is performed by lifting is also assumed. In this case, by adjusting the coordinate axes of the touch operation, the orientations of the display system and the operation system may be brought closer to achieve intuitive operation.
  • FIG. 12 is a diagram illustrating an example of the relationship between the direction of the touch operation and the moving direction of the pointer in the position adjustment mode according to this embodiment.
  • the display control unit 121 rotationally offsets the coordinate axes of the touch sensor 230 (touch pad) of the input device 20 from the coordinate axes of the input device 20 itself.
  • the XY axes of the touch operation are rotated +45 degrees around the X axis direction of the input device 20 itself (see FIG. 21), for example.
  • the display control unit 121 aligns the coordinate axes of the rotationally offset touch operation with the coordinate axes on the screen of the virtual object 350, and moves the pointer 352b.
  • the user can ergonomically and naturally move the thumb in the left-right direction (X-axis direction of the touch operation shown in FIG. 12) (by only the movement of the thumb base joint), and the pointer can be moved. 352b also moves in the horizontal direction on the screen. Alternatively, it is also possible for the user to select whether to perform the operation shown in FIG. 11 or the operation shown in FIG. be.
  • the coordinate axes of the touch operation are similarly set. Basically, it is possible to set the coordinate axes as described with reference to FIG. 11, assuming the operation with the arm lowered. Further, when it is detected that the user has raised his or her arm, the coordinate axis setting described with reference to FIG. 12 may be used.
  • An example of visual information presented to the user using the display unit 160 is a virtual object generated from a three-dimensional image.
  • the virtual object of the three-dimensional image is superimposed and displayed on the real space on the display unit 160 .
  • the virtual object may be localized and displayed in the real space, or may be displayed without being localized.
  • FIG. 13 is a diagram for explaining a fingertip tap operation on the three-dimensional UI according to the present embodiment.
  • an example of a three-dimensional image is a three-dimensional UI image 360 displayed on the display unit 160 in an elevator, for example.
  • the control unit 120 analyzes the captured image of the user's field of view captured by the outward facing camera 140 (or analyzes the detection result of the depth sensor), and recognizes the fingertip position (or the position of the input device 20). Then, when the fingertip position touches the three-dimensional UI image 360 (depth information may be considered), the control unit 120 receives an operation input to the three-dimensional UI image 360 .
  • depth information may be considered
  • the control unit 120 determines that the 2nd floor is Information indicating the selection is sent to the system on the elevator side. Further, the control unit 120 may provide feedback of operation by sound, display, vibration, or the like. For example, the control unit 120 may control the display control unit 121 to change the color of the button image 360a. Further, the control unit 120 may control the vibrating unit 250 of the input device 20 to give the user tactile feedback (vibration such as clicking). With such tactile feedback, it is possible to create a sense of reality when operating a virtual object.
  • tactile feedback may be given when the button is turned on and when the button is turned off, and the vibration patterns of the two may be different.
  • ⁇ Robust detection of fingertip position Note that the “detection of the fingertip position” using the outward camera 140 and the depth sensor provided in the display device 10 may not be able to see the fingertip depending on the orientation of the hand and the angle of the finger, resulting in reduced accuracy. sometimes. Therefore, for example, a recognition marker may be added to the input device 20, the outward camera 140 may detect the marker, and the control unit 120 may estimate the fingertip position. Furthermore, the input device 20 is provided with a distance sensor to detect the distance to a predetermined knot (joint) of the finger worn. In this case, the control unit 120 can estimate the fingertip position based on the position of the input device 20 detected by a marker or the like and the distance from there to a predetermined knot (joint).
  • FIG. 14 is a diagram explaining robust detection of the fingertip position using the distance sensor according to this embodiment.
  • the distance from the input device 20 to the middle joint of the finger is measured by a distance sensor (one axis) provided in the input device 20 .
  • the distal joint and middle joint of the finger move together. For example, the longer the distance D from the input device 20 to the middle knuckle, the farther the fingertip, which is the tip of the distal knuckle of the finger.
  • the control unit 120 can estimate the fingertip position from the position of the input device 20 and the distance information to the middle joint.
  • the control unit 120 performs a grasp and release operation of the virtual object with the finger wearing the input device 20 based on the analysis of the image captured by the outward facing camera 140 and the detection result by the motion sensor 240 of the input device 20. can be detected.
  • FIG. 15A and 15B are diagrams for explaining the gripping and releasing operation of the virtual object according to the present embodiment.
  • a virtual object 362 three-dimensional image
  • a table 40 in real space can be directly grasped or released.
  • a grasping operation is easily detected by an acceleration sensor, which is an example of the motion sensor 240 provided in the input device 20, because the fingers (for example, index finger and thumb) collide with each other and vibration occurs.
  • an acceleration sensor which is an example of the motion sensor 240 provided in the input device 20
  • the fingers for example, index finger and thumb
  • the release operation it may be difficult to determine whether the finger is released or the entire hand is moved (is the finger moved from the wrist while the finger is being held) based only on the movement of the input device 20 alone? For example, the following methods are mentioned.
  • FIG. 16 is a diagram illustrating an example of detection of a speaking operation according to this embodiment.
  • the input device 20 is provided with a distance sensor 280 (single axis) to measure the distance to the thumb. This allows the control unit 120 to determine that the distance D10 is the grasped state and the distance D11 is the separated state. Note that calibration for each individual is performed first.
  • the value D of the uniaxial distance sensor is measured, the value at the timing when the "grasp" operation is detected by the acceleration sensor is recorded as Dt, and after that the value D is "released” at the timing when D>Dt+ ⁇ D ' Manipulation detection implementations are also contemplated.
  • ⁇ D is a value for ensuring robustness and stability of detection, and is set to 10% of Dt, for example.
  • a second solution is to make a judgment together with the results of hand recognition by the outward facing camera 140 of the display device 10 . For example, when the position of the back of the hand does not move and the motion sensor 240 of the input device 20 detects upward movement of the input device 20, the control unit 120 determines that the “release” action has been performed.
  • the control unit 120 predicts the center of rotation based on the detection result of the motion sensor 240 of the input device 20, and determines the "release" motion according to the position of the center of rotation. be done.
  • 17A and 17B are diagrams for explaining the determination of the releasing motion based on the prediction of the center of rotation according to the present embodiment. For example, as shown on the left side of FIG. 17, when the input device 20 itself moves upward, the center of rotation R predicted by the detection result of the motion sensor 240 is positioned around the base of the index finger. Therefore, in this case, the control unit 120 determines that it is a "release" action. On the other hand, as shown on the right side of FIG.
  • the control unit 120 determines that the motion is not a "release" action.
  • prediction of the center of rotation can be performed based on detection results of a gyro sensor and an acceleration sensor, which are examples of the motion sensor 240, for example. Further, by arranging a plurality of gyro sensors and acceleration sensors inside the input device 20 (for example, arranging them in the upper part and the lower part of the input device 20 respectively), the control unit 120 can make judgments with higher accuracy. .
  • the manner in which the input device 20 is attached may affect the detection accuracy.
  • the grab-and-release detection may use motion sensor 240 and distance sensor 280 (optical sensors).
  • the motion sensor 240 detects vibrations when, for example, the index finger and thumb touch each other.
  • the control unit 120 observes the x-, y-, and z-axis signals detected by the motion sensor 240, for example, and detects contact using the values themselves or values to which a bandpass filter or norm is applied. In the present embodiment, it has been described that contact between fingers is detected as a "grab" operation, but it may be detected as a tap operation.
  • the motion sensor 240 has a high degree of freedom with respect to the mounting angle. There is a risk of As for the mounting angle, contact can be detected without any problem by applying rotation processing to the sensor value or identifying using a feature amount that does not depend on the angle, even if the angle is arbitrary.
  • the control unit 120 instructs the user to re-wear the input device 20 so that it is more closely attached. Instructions may be given by displaying guidance on the display unit 160, by vibrating the input device 20, or by sound. The judgment that the detection of vibration is not sufficient may be made, for example, by looking at the amplitude of shaking of the terminal, or by previously learning and comparing vibration data when the terminal is loose. Alternatively, calibration may be performed by tapping the fingers once when the input device 20 is worn, and re-wearing may be instructed based on the amplitude of the calibration.
  • the distance sensor 280 which is mainly used to measure the distance between the fingers, does not cause any problem even if the input device 20 is not in close contact with the finger (there is no significant drop in detection accuracy). no), but there is an acceptable mounting angle.
  • the distance sensor 280 corresponds to, for example, a distance sensor (depth sensor) such as a uniaxial ToF, other ToF, a millimeter wave radar, a camera, or an IR sensor.
  • a distance sensor depth sensor
  • the recognizable range is solid-angle limited (for example, the plane angle is less than 60 degrees).
  • the control unit 120 instructs the user to re-attach the thumb when the distance from the thumb cannot be obtained.
  • the control unit 120 may display a guide to wear the input device 20 in the correct posture on the display unit 160 .
  • the input device 20 may be guided by display means (not shown) such as an LED display of the input device 20 .
  • display means not shown
  • the motion sensor 240 detects the contact (tapping motion) between the fingers but the distance sensor 280 does not detect that the thumb is approaching, the distance between the thumb and the thumb cannot be kept. It turns out that the measurement is not possible.
  • control unit 120 uses only the detection result of the motion sensor 240 to detect the gripping and releasing operation. You can do it.
  • the control unit 120 may assign, for example, contacting the fingers twice (tapping twice) as an alternative to the “release” action.
  • the control unit 120 uses only the distance sensor 280 to simulate holding and releasing. An operation may be detected.
  • control unit 120 can ensure operability by appropriately selecting the sensor used for recognition.
  • the control unit 120 of the display device 10 draws a line from the tip of the index finger on the display unit 160 . , etc. (images such as lines are displayed in AR), and control is performed to stop drawing in a non-touched state (touch OFF).
  • Images such as lines are virtual objects generated from three-dimensional information. Images such as lines may or may not be localized in real space.
  • the fingertip position can be determined by analyzing a captured image captured by the outward camera 140 of the display device 10 or by distance measurement by a distance sensor provided in the input device 20 (the distance from the input device 20 to the middle joint). ).
  • FIG. 18 is a diagram illustrating AR drawing of characters and pictures using fingertips according to the present embodiment. As shown in FIG. 18, when the input device 20 is touched ON, the control unit 120 AR-displays (AR drawing) the line image L1 following the position of the fingertip on the display unit 160, and when the touch is OFF. Stop drawing.
  • the line thickness, type, color, etc. can also be controlled by operations from the input device 20.
  • the control unit 120 may detect the force of pressing the input device 20 with the thumb and change the line thickness or the like according to the pressing force.
  • the input device 20 is provided with, for example, a touch sensor with a pressure detection function, other pressure sensors, and the like.
  • FIG. 19A and 19B are diagrams for explaining an operation of grabbing and releasing a virtual object by a touch operation according to the present embodiment. For example, as shown on the left side of FIG. ) is recognized by the outward facing camera 140 . A virtual object 364 is, for example, a three-dimensional CG image. Next, when a touch ON is detected on the input device 20, it is recognized that the virtual object 364 is grasped by the user, and the display of the virtual object 364 is controlled.
  • the control unit 120 controls the display position and orientation of the virtual object 364 to follow changes in the position of the fingertip. Also, the control unit 120 may provide feedback of the operation by changing the color of the virtual object 364 . Further, the control unit 120 may perform operation feedback by sound or vibration. Then, when the input device 20 detects that the touch is turned off (the thumb is released from the touch), the control unit 120 recognizes the released state, and performs control to release the virtual object 364 from the hand as shown in the right part of FIG. 19 . I do. In the example shown on the right side of FIG. 19 , the control unit 120 performs display control such that the virtual object 364 released from the hand falls onto the table 40 in the real space according to the parameters set for the virtual object 364 .
  • Which of the various operations described above is to be performed may be distinguished by the type of touch operation. For example, “touch”, “touch hold”, “tap and hold”, etc. may be used properly to distinguish which operation (AR drawing, grasping operation, etc.). Moreover, you may distinguish by the position and pressure which touch. For example, AR drawing may be performed when the lower portion of the touch sensor 230 is touched, and a grasping operation may be performed when the upper portion is touched.
  • FIG. 20 is a diagram explaining remote operation of a virtual object according to this embodiment.
  • a virtual object 368 which is a three-dimensional object
  • the user can draw a line like a laser that moves according to the direction of the input device 20.
  • a virtual object 368 can be moved using a pointer UI 366 that is a UI in the form of a pointer.
  • FIG. 20 it is assumed that the arrangement of a foliage plant virtual object 368 generated by three-dimensional CG data is examined through the display unit 160 of the display device 10 in order to examine the interior of the room. are doing.
  • the display of the pointer UI 366 is controlled so that it tilts left and right according to the posture of the input device 20, with the center of the lower side of the display unit 160 as a base point, for example.
  • the attitude of the input device 20 (the attitude in the Yaw and Pitch directions with respect to the ground) can be obtained based on the detection results of an acceleration sensor, a gyro sensor, and a geomagnetic sensor, which are examples of the motion sensor 240 .
  • the motion sensor 240 provided in the input device 20 is capable of detecting 3DOF (three degrees of freedom) (Roll direction in addition to Yaw and Pitch).
  • the base point of the pointer UI is fixed at one point in the center of the lower side of the display unit 160, as shown in FIG. 21, for example.
  • a two-dimensional marker or the like is provided on the input device 20, and the XYZ position coordinates of the input device 20 with respect to the display device 10 are detected by detecting this marker with a camera provided on the display device 10, and the detected position is It is also possible to display a pointer UI as a base point.
  • the user can grab the virtual object 368 that is out of direct reach and move it to an arbitrary location.
  • the length of the pointer UI 366 may be initialized to, for example, 5m. Also, if there is a virtual object or real object that can be remotely controlled in the length direction of the pointer UI 366, the control unit 120 may change the length to reach that position. Also, if a virtual object or a real object exists within the above 5 m, the display length of the pointer UI may be limited to that position.
  • the control unit 120 may display the pointer UI 366 only while the user is touching the touch sensor 230 of the input device 20 .
  • the touch sensor 230 is double-tapped, triple-tapped, long-touched (long-pressed), or pressed by a switch (provided on the back side of the touch sensor 230).
  • the pointer UI 366 may be displayed by performing a predetermined operation such as a click operation to press the push switch unit 290, a double click operation, or a long click operation.
  • the control unit 120 moves the touch sensor 230 of the input device 20 along the Y-axis (see FIG. 21) of the input device 20 in a state where there is no virtual object (such as a web browser screen) that can be scrolled in the line-of-sight direction of the user.
  • the pointer UI 366 may appear when a predetermined operation such as a swipe operation in the positive direction is performed. Further, the control unit 120 may hide the pointer UI 366 when a predetermined operation such as a swipe operation in the negative direction of the Y-axis of the input device 20 is performed.
  • control unit 120 may display the pointer UI 366 when a virtual object that can be remotely controlled is displayed in the approximate direction that the input device 20 points.
  • a pointer UI 366 may be displayed when a touch operation is performed.
  • control unit 120 may display the pointer UI 366.
  • the pointer UI 366 may be displayed when
  • control unit 120 may display the pointer UI 366 when the user lifts the arm while wearing the input device 20 (when the user lifts the arm in the pitch direction perpendicular to the ground).
  • the user controls the orientation of the pointer UI 366 by changing the posture of the input device 20, and performs pointing by aligning the pointer UI 366 with a virtual object 368 of a three-dimensional CG image such as a foliage plant.
  • the control unit 120 changes the display of the virtual object 368 (for example, changes the brightness value, casts a shadow, encloses it with a frame, or changes the display of the virtual object 368). blinking times, etc.) to let the user recognize that the focus is on.
  • the control unit 120 may further perform feedback by sound or vibration. For example, by causing the input device 20 to vibrate at the moment the virtual object 368 is focused, it is possible to create a sense of reality.
  • the control unit 120 determines whether the object is being grasped or not. It recognizes and controls display so that the virtual object 368 follows the pointer UI 366 (control similar to a drag operation using a mouse). In addition, when a touch release operation, an operation to press the switch again, or an operation to finish pressing (switch OFF) is performed, the control unit 120 recognizes that the target object is released, and releases the display position of the virtual object 368. Control to stay in place.
  • the pressing operation is not limited to the operation of the pressing switch section 290.
  • the pressing operation may be determined that the pressing operation has been performed when the value of the pressure sensor provided in the input device 20 exceeds a threshold value. Even when a pressure sensor is used, the feeling of operation can be improved by performing tactile feedback with the vibrating section 250 when it is determined whether the pressing operation is ON/OFF.
  • the control unit 120 controls the orientation of the pointer UI 366 to reflect the orientation of the input device 20 in the Pitch direction (vertical) and Yaw direction (horizontal) with respect to the ground.
  • Operation input using the input device 20 is two-axis (Yaw, Pitch). Movement in two directions (left-right and back-and-forth) can be achieved. That is, movement in the depth direction can be achieved by moving the virtual object back and forth (towards and behind the user) in response to the Pitch operation (vertical direction with respect to the ground).
  • FIG. 21 is a diagram illustrating movement of the pointer UI 366 according to movement of the input device 20 in the Yaw direction according to this embodiment.
  • the user moves the input device 20 in the yaw direction, which is a movement in the horizontal direction with respect to the ground, with the arm raised forward. is displayed and controlled to the left and right.
  • the base point (starting point) of the pointer UI 366 is fixed at the center of the lower side of the display section 160 .
  • the motion sensor 240 does not include a geomagnetic sensor capable of detecting an absolute posture angle in the Yaw direction, and only an acceleration sensor and a gyro sensor are used, the posture in the Yaw direction with respect to the ground is detected relative to the ground by the gyro sensor. A yaw attitude angle change is detected. Therefore, if control is intended such that the orientation of the pointer UI 366 and the orientation of the arm approximately match, the orientation of the arm and the orientation of the pointer UI 366 in the Yaw direction may not match. On the other hand, for example, when the user lifts the arm to a predetermined height in front of the body (for example, as shown in FIG.
  • the control unit 120 changes the pointer UI 366 to The display may be turned on, and the initial Yaw orientation of the pointer UI 366 at that time may be set to 0 degrees (facing forward as shown in FIG. 21).
  • control can be started from a state in which the orientation of the pointer UI 366 and the orientation of the hand are aligned with each other.
  • Subsequent posture changes in the Yaw direction are detected by the gyro sensor, so the orientation of the pointer UI 366 and the hand can continue to match.
  • the absolute attitude angle is obtained according to the direction of gravity (0 degrees in the pitch direction is the horizontal direction, as shown in FIG. 23).
  • the orientation (horizontal inclination) of the pointer UI 366 displayed on the display unit 160 is, as shown in FIG. intersecting directions).
  • the direction d shifted by about 45 degrees from the Y-axis of the input device 20 is the direction of the pointer UI 366 (horizontal inclination with the center of the lower side of the display unit 160 as the base point).
  • the orientation of the user's forearm and the orientation (horizontal tilt) of the pointer UI 366 can be approximately matched, enabling more intuitive operations.
  • the wrist can be flexed and extended from the state in the center of FIG. , it is possible to move the pointer UI 366 from the front to the left and right with a low physical load.
  • FIG. 23 is a diagram for explaining the movement of the pointer UI according to the movement of the input device in the pitch direction according to this embodiment.
  • the pointer UI 366 moves in two directions (left and right and front and back) on the display unit 160 in response to two-axis (Yaw, Pitch) input using the input device 20 .
  • the pointer UI 366 also extends diagonally upward from the horizontal direction by 45 degrees with respect to the real space and points toward the ceiling. It will be in a pointing state.
  • FIG. 24 is a diagram illustrating a case where the input device 20 according to this embodiment is worn on the left and right index fingers, respectively.
  • a touch sensor 230 is provided on one side of the main body of the input device 20, as shown in FIG. 3, for example.
  • a correct posture for wearing such an input device 20 on the index finger is a posture in which the touch sensor 230 faces the thumb. Therefore, as shown in FIG. 24, when the input device 20 is worn on the index finger in the correct posture, the left and right finger insertion directions are reversed (specifically, plus/minus directions of the Z axis).
  • the control unit 120 when the user wears the input device 20 on the right hand, the control unit 120 changes the orientation (direction d) of the pointer UI 366 around the Y axis of the input device 20 to the X axis. For example, the direction is rotated by about +45 degrees. 24, when the user wears the input device 20 on the left hand, the control unit 120 changes the direction (direction d) of the pointer UI 366 around the Y axis of the input device 20 to the X axis. For example, the orientation is rotated by -45 degrees. In this manner, the orientation of the pointer UI 366 can be changed according to information as to which hand the user wears the input device 20 on, left or right.
  • the user manually inputs using a screen UI displayed on the display unit 160 of the display device 10.
  • the user manually inputs using a screen UI displayed on the display unit 160 of the display device 10.
  • the user by recognizing the user's hand and the input device 20 with the outward facing camera 140 of the display device 10, it is possible to determine which hand the device is worn on. For example, as an initial operation when the input device 20 is worn, the user holds the hand wearing the input device 20 over the imaging area of the outward facing camera 140 to perform initial setting.
  • whether the input device 20 is attached or detached is determined by, for example, a proximity sensor installed inside the input device 20 .
  • the control unit 120 can also determine whether the input device 20 is worn on the right hand or the left hand. This is a method of performing estimation by statistically processing posture information because the direction of the index finger is mainly directed toward the ground when walking. Such estimation is performed each time the input device 20 is put on and taken off.
  • the input device 20 is attached to the proximal knuckle of the finger (root knuckle of the finger), but as another example of the attachment position, A case where it is installed is also assumed.
  • the input device 20 is attached to the middle knuckle portion in the state of the hand shown in FIG. 45 degrees, +45 degrees for the left hand), or 0 degrees. This makes it easier to operate the pointer UI 366 from the viewpoint of the joint range of motion of the wrist even when the input device 20 is attached to the middle joint.
  • Whether the input device 20 is attached to the proximal joint portion or the middle joint portion can be determined by, for example, capturing the hand and the input device 20 with the outward facing camera 140 in the same manner as the determination of the left and right hands described above. can be judged by
  • FIG. 25 is a diagram illustrating manipulation of a real object using the pointer UI according to the present embodiment.
  • the operation is not limited to the operation of a virtual object that is a two-dimensional or three-dimensional image, and the operation of a real object is also possible.
  • a temperature operation panel 370 is displayed.
  • the display device 10 can recognize the air conditioner 45 by the outward facing camera 140 .
  • the user can perform operations such as temperature adjustment and operation mode switching while the temperature operation panel 370 is displayed.
  • the display device 10 and the air conditioner 45 are paired in advance and can transmit and receive data.
  • the display device 10 can be wirelessly connected to the home network and control the air conditioner 45 via the home network.
  • the control unit 120 can also control ON/OFF of the display itself of the display unit 160 according to the operation of the input device 20 . By turning off the display of the display unit 160, power saving of the battery capacity of the display device 10 can be realized.
  • the display ON/OFF operation of the display unit 160 is, for example, a touch operation (tap, double tap, triple tap, long touch) on the touch sensor 230 of the input device 20, or a push operation (a click operation using the push switch unit 290). , double-click operation, triple-click operation, long-click operation) and the like.
  • a tap operation, a click operation, or the like may conflict with an operation used in remote operation using the pointer UI 366, so the control unit 120 prevents a predetermined tap operation or click operation from being performed, for example, when the pointer UI 366 is not displayed.
  • the display on the display unit 160 may be turned off when the information is received.
  • an operation to turn the display on/off of the display unit 160 includes, for example, a swipe operation on the touch sensor 230 of the input device 20 (a swipe operation in the upward direction turns the display on, and a swipe operation in the downward direction turns the display off, etc.). ).
  • a swipe operation on the touch sensor 230 of the input device 20 a swipe operation in the upward direction turns the display on
  • a swipe operation in the downward direction turns the display off, etc.
  • the edge which edge on the left or right is assigned to the display ON/OFF operation of the display unit 160 is based on the information on which hand the input device 20 is attached to, as described above. may be determined based on Alternatively, the operation of the entire surface of the touch sensor 230 instead of the edge may be used for the display ON/OFF operation of the display unit 160 .
  • the swipe operation for example, downward swipe operation
  • the control unit 120 reduces the sampling rate of the touch sensor 230 of the input device 20 or the motion sensor 240. It is also possible to reduce the power consumption of the input device 20 by, for example, turning off sensing.
  • the input device 20 may be worn on the proximal knuckle of the finger (root knuckle of the finger) as shown in FIG. middle node). Also, the user may appropriately change the mounting position according to the usage scene.
  • the structure of the input device 20 is such that a part of the C-shaped ring forming the main body of the input device 20 is plastically deformable so that fingers of different thicknesses can be accommodated. can also be installed. Note that the structure of the input device 20 according to this embodiment is not limited to the example shown in FIG. Another structural example of the input device 20 according to the present embodiment will be specifically described below with reference to FIGS. 27 to 29. FIG.
  • the input device 20 may have a ring-shaped closed shape, or may have a ring-shaped closed shape.
  • FIG. 27 is a diagram illustrating an example of the structure of the input device 20 according to this embodiment.
  • the cross-sectional view shown in FIG. 27 is a view of the input device 20 viewed from the direction in which the finger is passed.
  • the input device 20A shown on the left side of FIG. 27 has the shape of the C-shaped ring also shown in FIG.
  • An input device 20B shown in the center of FIG. 27 is an example of the shape of a notched ring.
  • the input device 20 can accommodate fingers of different thicknesses due to the ring shape having a non-closed portion.
  • the ring shape having a non-closed portion.
  • a wider range of adjustment is possible.
  • FIGS. 28 and 29 are diagrams explaining a structure in which a portion of the ring-shaped main body forming the input device 20 according to the present embodiment includes a deformable portion.
  • an input device 20D shown in the upper part of FIG. 28 includes deformation portions 21a and 21b formed of a flexible material by integral formation or the like in a part of a C-shaped ring. Since the input device 20D is attached to the finger in such a manner that the finger is inserted through the notched gap formed between the deformed portions 21a and 21b, it is highly convenient to remove. Also, even a finger 2 that is thicker than the finger 1 can be attached by the bending of the deformed portions 21a and 21b.
  • the input device 20E shown in the middle part of FIG. 28 has an O-shaped ring-shaped main body that is closed like a ring, and a deformable part 22 made of an elastic material that expands and contracts according to the thickness of the finger. Formation, etc. Since the input device 20E is attached to the finger by passing the finger through the hole, there is little risk of the device falling off.
  • the input device 20F shown in the lower part of FIG. 28 is configured by integrally forming a deformable portion 23 reversibly deformable according to the thickness of the finger on a part of the ring-shaped main body.
  • the input device 20 ⁇ /b>F is a belt-type ring that is worn on a finger by turning over the deformed portion 23 to insert the finger and winding the finger in the deformed portion 23 .
  • the belt-type ring input device 20F is easy to remove and less likely to come off and fall.
  • a method of locking the deformed portion 23 may be, for example, half force of deformation and attraction of a magnet or the like.
  • the input device 20G shown in the upper part of FIG. 29 is provided with deformable portions 24a and 24b that are movable according to the thickness of the finger on a part of the C-shaped ring-shaped main body.
  • the input device 20G is worn on a finger in such a manner that the finger is inserted through a notched gap formed between the deformed portions 24a and 24b.
  • the deformed portions 24a and 24b are configured, for example, in the form of presser springs whose strength is adjusted so that spring pressure is applied in the direction of pressing the finger and the latch mechanism does not return in the direction of opening.
  • An input device 20H shown in the lower part of FIG. 29 is formed with a structure in which attachments 25 (25a, 25b) that can be exchanged according to finger thickness are attached to a part of the main body.
  • attachments 25 25a, 25b
  • Each attachment 25 is formed in a shape along the pad of the finger.
  • one of the connecting portions provided at both ends of the attachment 25 may be removed and opened, and the finger may be inserted through the opened portion.
  • the deformed portions that are deformed according to the thickness of the finger described above are mainly provided at portions located on the pad side of the finger when the input device 20 is worn.
  • each configuration (unit) such as the communication unit 210, the control unit 220, the motion sensor 240, the vibration unit 250, and the storage unit 260 of the input device 20 is connected to the upper part of the input device 20 (mainly the finger when worn on the finger). portion located on the dorsal side).
  • the touch sensor 230 since the touch sensor 230 is provided on the side facing the thumb so that the thumb can be operated when the input device 20 is worn, the touch sensor 230 may be provided at a deformed portion of the input device 20 .
  • the input device 20 may be provided with an unvoiced voice input means in consideration of situations where it is difficult to speak, such as in a train.
  • non-speech refers to whispering at a very low volume or speaking without speaking at all.
  • the non-vocal voice input means includes, for example, throat vibration when uttering a whisper that cannot be heard by others, throat vibration when voicing with the mouth closed, and vocalization without uttering any voice ( It analyzes the information in the throat and oral cavity (for example, ultrasonic echo images) when performing so-called lip-syncing, and realizes voice text input and voice command recognition.
  • Throat vibrations and intraoral information can be acquired by various sensors (vibration sensors and ultrasonic sensors) provided in the input device 20 while the input device 20 is in contact under the chin. Analysis of these pieces of information may be performed by the control unit 120 of the display device 10 .
  • the input device 20 is not limited to the operation input for the display device 10 as described above, and may be used for the operation input for the audio output device, for example.
  • the audio output device may be various speakers worn by the user (a neck speaker, an earphone-type speaker, a headphone-type speaker), or may be an audio device installed in a room or a car.
  • the input device 20 may be used alone. For example, it may be used as a device for communicating with others.
  • the input device 20 may also be used as a remote controller (remote controller) for a TV or the like.
  • the operation input means in the input device 20 is not limited to a touch operation using the touch sensor 230, but a click operation using a push switch, motion input using the motion sensor 240, and pressing operation using a pressure sensor. etc. Moreover, these may be used in combination as appropriate.
  • the display device for displaying a virtual object
  • a glasses-type device worn on the user's head has been mentioned, but the display device according to this embodiment is not limited to this.
  • the display device may be, for example, a smart phone or a tablet terminal, or may be a display device (which can be worn on one ear) that presents an image to one eye of the user.
  • the object to be operated by the input device 20 is not limited to the virtual object superimposed on the real space. For example, it may be a virtual object in a virtual space displayed on a display device.
  • the virtual object included in the image projected by the projector may be the operation target.
  • one or more computer programs for causing hardware such as a CPU, ROM, and RAM incorporated in the display device 10 or the input device 20 described above to exhibit the functions of the display device 10 or the input device 20 are also created. It is possible. Also provided is a computer-readable storage medium storing the one or more computer programs.
  • the present technology can also take the following configuration.
  • An input device comprising: (2) The input device according to (1), wherein the one or more sensors are provided in the main body at a position facing the thumb when worn. (3) The input device according to (2), wherein the one or more sensors include at least one of a touch sensor that detects a touch operation and a switch sensor that detects a switch operation.
  • the input device is Further comprising a motion sensor for detecting movement provided in the main body, The input device according to any one of (1) to (7), wherein the control section controls transmission of the detection result by the motion sensor to the external device through the communication section.
  • the input device is further comprising a vibrating portion provided in the main body; The input device according to any one of (1) to (8), wherein the control section controls the vibration section according to information received from the external device by the communication section.
  • the input device is Further comprising a distance sensor provided in the main body for detecting the distance from the other finger or the distance to a predetermined knot of the finger worn,
  • the input device according to any one of (1) to (10), wherein the control section controls transmission of the detection result by the distance sensor to the external device through the communication section.
  • an operation input information acquisition unit that is provided in a ring-shaped main body that is worn on a user's finger and acquires information on the operation input from one or more sensors that detect information on the operation input by another finger; a line-of-sight information acquisition unit that acquires line-of-sight information of the user; a presentation control unit that controls information to be presented to the user based on the operation input information and the line-of-sight information;
  • a system comprising: (13) The system according to (12) above, wherein the information presented to the user includes at least one of tactile information provided by the vibrating section, visual information provided by the display section, and auditory information provided by the audio output section.
  • the system includes: the main body; the one or more sensors provided on the body; a communication unit that performs wireless communication with an information processing device used by the user; a control unit that performs control to transmit information of the operation input detected by the one or more sensors to the information processing device through the communication unit; an input device having the operation input information acquiring unit that acquires information of the operation input transmitted from the input device; the line-of-sight information acquisition unit; the presentation control unit; the information processing device having The system according to (12) or (13) above, comprising: (15) The information processing device is a display device worn on the user's head, The presentation control unit performing display control of a two-dimensional or three-dimensional image superimposed and displayed in real space on the display device; recognizing an image to be operated according to the line-of-sight information; The system according to (14) above, wherein the image of the recognized operation target is controlled based on the information of the operation input.
  • the presentation control unit recognizing an operation position on the display screen of the display device based on the line-of-sight information; Further, the system according to (15) above, wherein the operation position is adjusted according to information of a predetermined operation input.
  • the presentation control unit controls the image of the operation target based on the information of the operation input, and controls the presentation of tactile information by a vibrating unit provided in the input device.
  • the processor Detecting information of an operation input by another finger by one or more sensors provided in a ring-shaped main body attached to a user's finger; Controlling transmission of information of the detected operation input to an external device by a communication unit that performs wireless communication; control methods, including; (19) the processor Acquiring information of the operation input from one or more sensors provided in a ring-shaped main body attached to a user's finger and detecting information of the operation input by another finger; obtaining line-of-sight information of the user; controlling information to be presented to the user based on the operation input information and the line-of-sight information; control methods, including;

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Position Input By Displaying (AREA)
  • User Interface Of Digital Computer (AREA)
PCT/JP2022/007274 2021-07-14 2022-02-22 入力装置、システム、および制御方法 Ceased WO2023286316A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN202280048109.5A CN117616366A (zh) 2021-07-14 2022-02-22 输入装置、系统和控制方法
US18/576,165 US20250093955A1 (en) 2021-07-14 2022-02-22 Input device, system, and control method
JP2023535096A JPWO2023286316A1 (https=) 2021-07-14 2022-02-22

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021-116212 2021-07-14
JP2021116212 2021-07-14

Publications (1)

Publication Number Publication Date
WO2023286316A1 true WO2023286316A1 (ja) 2023-01-19

Family

ID=84918833

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/007274 Ceased WO2023286316A1 (ja) 2021-07-14 2022-02-22 入力装置、システム、および制御方法

Country Status (4)

Country Link
US (1) US20250093955A1 (https=)
JP (1) JPWO2023286316A1 (https=)
CN (1) CN117616366A (https=)
WO (1) WO2023286316A1 (https=)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2024137671A (ja) * 2023-03-21 2024-10-07 セリコ・インコーポレイテッド 網膜色素変性症および黄斑変性症患者のための拡張現実装置
WO2024252993A1 (ja) * 2023-06-09 2024-12-12 キヤノン株式会社 指輪型デバイス
WO2025018372A1 (ja) * 2023-07-18 2025-01-23 Tdk株式会社 リング型コントローラーおよび無線通信システム
WO2025018377A1 (ja) * 2023-07-18 2025-01-23 Tdk株式会社 リング型コントローラーおよび無線通信システム
WO2025033165A1 (ja) * 2023-08-07 2025-02-13 ソニーグループ株式会社 情報処理装置、入力装置、情報処理システム、及び、情報処理方法
WO2025169616A1 (ja) * 2024-02-09 2025-08-14 ソニーグループ株式会社 入力装置及び電子機器
WO2026004614A1 (ja) * 2024-06-25 2026-01-02 ソニーグループ株式会社 情報処理システムおよび情報処理方法
US12547246B2 (en) 2023-06-09 2026-02-10 Canon Kabushiki Kaisha Ring-type device
WO2026078754A1 (ja) * 2024-10-07 2026-04-16 株式会社ソニー・インタラクティブエンタテインメント 感覚提示装置及び感覚提示システム

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7563449B2 (ja) * 2020-05-11 2024-10-08 ソニーグループ株式会社 通信装置、方法、およびプログラム

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130135223A1 (en) * 2009-12-13 2013-05-30 Ringbow Ltd. Finger-worn input devices and methods of use
JP2016033815A (ja) * 2014-07-30 2016-03-10 エルジー エレクトロニクス インコーポレイティド 指輪型移動端末機
US20160077582A1 (en) * 2014-09-15 2016-03-17 Lg Electronics Inc. Ring-type mobile terminal
JP2017513126A (ja) * 2014-04-01 2017-05-25 アップル インコーポレイテッド リング形コンピューティングデバイスのための装置及び方法
JP2019526864A (ja) * 2017-06-29 2019-09-19 アップル インコーポレイテッドＡｐｐｌｅ Ｉｎｃ． センサ及び触覚を用いた指装着デバイス 本出願は、２０１８年６月２１日付出願の米国特許出願第１６／０１５，０４３号、及び２０１７年６月２９日付出願の米国仮特許出願第６２／５２６，７９２号に対する優先権を主張するものであり、それらの全体が参照により本明細書に組み込まれる。
JP2019536189A (ja) * 2016-11-30 2019-12-12 シューシングス インコーポレイテッドShoothings, Inc. 手指の関節間着用型の指輪タイプのユーザ操作センシング装置
US20200026352A1 (en) * 2018-07-17 2020-01-23 Apple Inc. Computer Systems With Finger Devices
JP2020174787A (ja) * 2019-04-16 2020-10-29 国立大学法人東京農工大学 手指運動推定システム
US20200356162A1 (en) * 2019-05-10 2020-11-12 Apple Inc. Electronic Device System With Controllers
US20210089131A1 (en) * 2019-09-23 2021-03-25 Apple Inc. Finger-Mounted Input Devices

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130135223A1 (en) * 2009-12-13 2013-05-30 Ringbow Ltd. Finger-worn input devices and methods of use
JP2017513126A (ja) * 2014-04-01 2017-05-25 アップル インコーポレイテッド リング形コンピューティングデバイスのための装置及び方法
JP2016033815A (ja) * 2014-07-30 2016-03-10 エルジー エレクトロニクス インコーポレイティド 指輪型移動端末機
US20160077582A1 (en) * 2014-09-15 2016-03-17 Lg Electronics Inc. Ring-type mobile terminal
JP2019536189A (ja) * 2016-11-30 2019-12-12 シューシングス インコーポレイテッドShoothings, Inc. 手指の関節間着用型の指輪タイプのユーザ操作センシング装置
JP2019526864A (ja) * 2017-06-29 2019-09-19 アップル インコーポレイテッドＡｐｐｌｅ Ｉｎｃ． センサ及び触覚を用いた指装着デバイス 本出願は、２０１８年６月２１日付出願の米国特許出願第１６／０１５，０４３号、及び２０１７年６月２９日付出願の米国仮特許出願第６２／５２６，７９２号に対する優先権を主張するものであり、それらの全体が参照により本明細書に組み込まれる。
US20200026352A1 (en) * 2018-07-17 2020-01-23 Apple Inc. Computer Systems With Finger Devices
JP2020174787A (ja) * 2019-04-16 2020-10-29 国立大学法人東京農工大学 手指運動推定システム
US20200356162A1 (en) * 2019-05-10 2020-11-12 Apple Inc. Electronic Device System With Controllers
US20210089131A1 (en) * 2019-09-23 2021-03-25 Apple Inc. Finger-Mounted Input Devices

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2024137671A (ja) * 2023-03-21 2024-10-07 セリコ・インコーポレイテッド 網膜色素変性症および黄斑変性症患者のための拡張現実装置
JP7614664B2 (ja) 2023-03-21 2025-01-16 セリコ・インコーポレイテッド 網膜色素変性症および黄斑変性症患者のための拡張現実装置
WO2024252993A1 (ja) * 2023-06-09 2024-12-12 キヤノン株式会社 指輪型デバイス
US12547246B2 (en) 2023-06-09 2026-02-10 Canon Kabushiki Kaisha Ring-type device
WO2025018372A1 (ja) * 2023-07-18 2025-01-23 Tdk株式会社 リング型コントローラーおよび無線通信システム
WO2025018377A1 (ja) * 2023-07-18 2025-01-23 Tdk株式会社 リング型コントローラーおよび無線通信システム
WO2025033165A1 (ja) * 2023-08-07 2025-02-13 ソニーグループ株式会社 情報処理装置、入力装置、情報処理システム、及び、情報処理方法
WO2025169616A1 (ja) * 2024-02-09 2025-08-14 ソニーグループ株式会社 入力装置及び電子機器
WO2026004614A1 (ja) * 2024-06-25 2026-01-02 ソニーグループ株式会社 情報処理システムおよび情報処理方法
WO2026078754A1 (ja) * 2024-10-07 2026-04-16 株式会社ソニー・インタラクティブエンタテインメント 感覚提示装置及び感覚提示システム

Also Published As

Publication number Publication date
US20250093955A1 (en) 2025-03-20
JPWO2023286316A1 (https=) 2023-01-19
CN117616366A (zh) 2024-02-27

Similar Documents

Publication Publication Date Title
WO2023286316A1 (ja) 入力装置、システム、および制御方法
JP7649396B2 (ja) 三次元環境と相互作用するためのデバイス、方法、及びグラフィカルユーザインタフェース
CN112424730B (zh) 具有手指设备的计算机系统
US12287921B2 (en) Methods for manipulating a virtual object
US12468393B2 (en) Systems for detecting in-air and surface gestures available for use in an artificial-reality environment using sensors at a wrist-wearable device, and methods of use thereof
JP2023515525A (ja) ウェアラブルシステムのための手のジェスチャ入力
US20190265802A1 (en) Gesture based user interfaces, apparatuses and control systems
JP5900393B2 (ja) 情報処理装置、操作制御方法及びプログラム
US12498792B2 (en) Systems for detecting gestures performed within activation-threshold distances of artificial-reality objects to cause operations at physical electronic devices, and methods of use thereof
CN102779000B (zh) 一种用户交互系统和方法
US20240094819A1 (en) Devices, methods, and user interfaces for gesture-based interactions
US20150002475A1 (en) Mobile device and method for controlling graphical user interface thereof
US20240272764A1 (en) User interface elements for facilitating direct-touch and indirect hand interactions with a user interface presented within an artificial-reality environment, and systems and methods of use thereof
US12204805B2 (en) Techniques for using in-air hand gestures detected via a wrist-wearable device to operate a camera of another device, and wearable devices and systems for performing those techniques
WO2020216106A1 (zh) 一种穿戴式计算设备和一种人机交互方法
US20250355535A1 (en) Devices, Methods, and Graphical User Interfaces for Interacting with System User Interfaces within Three-Dimensional Environments
CN108369451B (zh) 信息处理装置、信息处理方法及计算机可读存储介质
WO2024064016A1 (en) Devices, methods, and user interfaces for gesture-based interactions
JPWO2017110178A1 (ja) 情報処理装置、情報処理方法及びプログラム
US20250106498A1 (en) Techniques for using in-air wrist-roll gestures detected via a wrist-wearable device to operate another device, and wearable devices and systems for performing those techniques
WO2025033165A1 (ja) 情報処理装置、入力装置、情報処理システム、及び、情報処理方法
CN119343654A (zh) 使用腕部可穿戴设备处的传感器检测可用于人工现实环境的空中手势和表面手势的系统及其使用方法
WO2026004614A1 (ja) 情報処理システムおよび情報処理方法
WO2024168202A1 (en) User interface elements for facilitating direct-touch and indirect hand interactions with a user interface presented within an artificial-reality environment, and systems and methods of use thereof
CN117590936A (zh) 使用经由可穿戴设备的神经肌肉信号传感器检测的空中手势来导航用户界面，及其系统和使用方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22841666

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023535096

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 18576165

Country of ref document: US

WWE Wipo information: entry into national phase

Ref document number: 202280048109.5

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22841666

Country of ref document: EP

Kind code of ref document: A1

WWP Wipo information: published in national office

Ref document number: 18576165

Country of ref document: US