JP2022162702A5

JP2022162702A5 -

Info

Publication number: JP2022162702A5
Application number: JP2021067660A
Authority: JP
Filing date: 2021-04-13
Publication date: 2023-04-18
Anticipated expiration: 2041-04-13

Claims

A system that includes a sensor device, a controller that receives a user operation, and an information processing device that performs processing based on the user operation,
The sensor device is
a first image sensor that generates a first image signal by synchronously scanning all pixels at a predetermined timing;
a second image sensor that includes an event-driven vision sensor that asynchronously generates a second image signal upon detecting a change in intensity of light incident on each pixel;
Based on the first image signal and the second image signal, the state of the user, the controller held by the user, which part of the controller the user is holding, and the an estimating unit that estimates a posture of the controller including at least one of postures held by a user ;
an information output unit that outputs information indicating the state of the user and information indicating the attitude of the controller;
The information processing device is
a control value calculation unit that calculates a control value for feedback control to the controller based on at least one of information indicating the state of the user and information indicating the attitude of the controller;
The controller controls at least one of a force sense presentation device that presents a force sense based on the control value, a vibration device that vibrates based on the control value, or an audio output device that outputs sound based on the control value. have a system.

2. The system of claim 1, wherein the user's state includes at least one of the user's posture, the user's arm shape, or the user's finger shape.

The estimation unit estimates the first image signal based on a learned model constructed by learning a relationship between an image of a person having multiple joints and coordinate information indicating positions of the multiple joints. 3. The system according to claim 1 or 2, wherein coordinate information of at least one joint of said user included in a first image based thereon is calculated, and said user's state is estimated based on said coordinate information.

The information processing device further includes a first recognition unit that recognizes one or more users included in the object scene based on at least one of the first image signal and the second image signal,
4. The estimation unit according to any one of claims 1 to 3, wherein, for each user recognized by the first recognition unit, the estimation unit estimates the state of the user and the posture of the controller held by the user. The system described in .

the system includes a plurality of the controllers;
The information processing device further includes a second recognition unit that recognizes the controller held by the user for each of the users recognized by the first recognition unit,
5. The system according to claim 4, wherein said information output unit outputs information indicating a combination of said user recognized by said first recognition unit and said controller recognized by said second recognition unit.

An information processing method for outputting a control value for feedback control to a controller,
A first image signal is generated by a first image sensor that synchronously scans all pixels at a predetermined timing, and a second image signal is generated asynchronously when a change in intensity of light incident on each pixel is detected. Based on the second image signal generated by a second image sensor including an event-driven vision sensor, the state of the user and the controller held by the user, which part of the controller the user grips. and a pose of the controller including at least one of a pose in which the controller is held by the user ;
and an information output step of outputting information indicating the state of the user and information indicating the attitude of the controller.

A first image signal is generated by a first image sensor that synchronously scans all pixels at a predetermined timing, and a second image signal is generated asynchronously when a change in intensity of light incident on each pixel is detected. Based on the state of the user and the controller held by the user, based on the second image signal generated by a second image sensor including an event-driven vision sensor, what part of the controller the user holds. and a pose of the controller including at least one of a pose in which the controller is held by the user ;
An information processing program that causes a computer to implement a function of outputting information indicating the state of the user and information indicating the attitude of the controller.