WO2022220048A1

WO2022220048A1 - System, information processing method, and information processing program

Info

Publication number: WO2022220048A1
Application number: PCT/JP2022/013835
Authority: WO
Inventors: 直之宮田; 英樹柳澤; 麻美子石田; 英明岩木
Original assignee: 株式会社ソニー・インタラクティブエンタテインメント
Priority date: 2021-04-13
Filing date: 2022-03-24
Publication date: 2022-10-20
Also published as: JP2022162702A

Abstract

Provided is a system including a sensor device, a controller, and an information processing device. The sensor device comprises: a first image sensor for generating a first image signal by synchronously scanning all pixels at a predetermined timing; a second image sensor including an event-driven vision sensor for generating a second image signal; an estimation unit for estimating, on the basis of the first image signal and the second image signal, a state of a user, and a posture of the controller held by the user; and an information output unit for outputting the estimation result. The information processing device comprises a control value calculation unit for calculating a control value for feedback control to the controller, on the basis of at least one of information indicating the state of the user, and information indicating the posture of the controller. The controller has at least one of a force sense presentation device, a vibration device, and a sound output device.

Description

System, information processing method and information processing program

The present invention relates to a system, an information processing method, and an information processing program.

An event-driven vision sensor is known, in which pixels that detect changes in the intensity of incident light generate signals asynchronously with time. An event-driven vision sensor is advantageous in that it can operate at low power and at high speed compared to a frame-type vision sensor that scans all pixels at predetermined intervals, specifically image sensors such as CCD and CMOS. is. Techniques related to such an event-driven vision sensor are described in Patent Document 1 and Patent Document 2, for example.

Japanese Patent Publication No. 2014-535098 JP 2018-85725 A

However, although the above advantages of event-driven vision sensors are known, it is hard to say that enough proposals have been made for how to use them in combination with other devices.

Therefore, according to the present invention, based on an image signal generated by a sensor that synchronously scans all pixels at a predetermined timing and an image signal generated by an event-driven vision sensor, the user's state and the user's A system capable of realizing suitable feedback control to the controller while suppressing latency by estimating the attitude of the held controller and calculating a control value for feedback control to the controller based on the estimation result, An object is to provide an information processing method and an information processing program.

According to one aspect of the present invention, there is provided a system including a sensor device, a controller that receives user operations, and an information processing device that performs processing based on user operations, wherein the sensor device detects all pixels at a predetermined timing. A first image sensor that scans synchronously to generate a first image signal, and an event-driven vision sensor that asynchronously generates a second image signal when detecting a change in the intensity of light incident on each pixel. an estimation unit for estimating the state of the user and the orientation of the controller held by the user based on the first image signal and the second image signal; information indicating the state of the user; and an information output unit that outputs information indicating the posture of the controller, the information processing device controlling feedback control to the controller based on at least one of the information indicating the state of the user and the information indicating the posture of the controller. The controller includes a control value calculation unit that calculates a value, and the controller is a haptic presentation device that presents a haptic sensation based on the control value, a vibration device that vibrates based on the control value, or a voice that outputs sound based on the control value. A system is provided having at least one of the output devices.
According to another aspect of the present invention, there is provided an information processing method for outputting a control value for feedback control to a controller, the first image generated by a first image sensor synchronously scanning all pixels at a predetermined timing. Based on the image signal and a second image signal generated by a second image sensor including an event-driven vision sensor that asynchronously generates a second image signal upon detecting a change in the intensity of light incident on each pixel. an information processing method including an estimation step of estimating a user's state and the posture of a controller held by the user; and an information output step of outputting information indicating the user's state and information indicating the posture of the controller. provided.
According to still another aspect of the present invention, the first image signal is generated by a first image sensor that synchronously scans all pixels at a predetermined timing, and the intensity change of incident light for each pixel is detected. and a second image signal generated by a second image sensor including an event-driven vision sensor that asynchronously generates a second image signal with respect to the state of the user and the attitude of the controller held by the user. and a function of outputting information indicating the state of the user and information indicating the posture of the controller.

According to the above configuration, based on the image signal generated by the sensor that synchronously scans all pixels at a predetermined timing and the image signal generated by the event-driven vision sensor, the state of the user estimates the attitude of the controller held by , and calculates a control value for feedback control to the controller based on the estimation result. Therefore, it is possible to implement suitable feedback control to the controller while suppressing latency.

1 is a schematic diagram showing the entire system according to one embodiment of the present invention; FIG. 1 is a block diagram showing a schematic configuration of a system according to one embodiment of the present invention; FIG. It is a block diagram showing a schematic configuration of an estimating unit in the system according to one embodiment of the present invention. FIG. 4 is a diagram for explaining an example of estimation in one embodiment of the present invention; FIG. FIG. 4 is another diagram for explaining an example of estimation in one embodiment of the present invention; FIG. 4 is another diagram for explaining an example of estimation in one embodiment of the present invention; FIG. 4 is another diagram for explaining an example of estimation in one embodiment of the present invention; 1 is a flowchart illustrating an example of a processing method according to an embodiment of the invention; It is a figure explaining feedback control in one embodiment of the present invention. FIG. 4 is another diagram illustrating feedback control in one embodiment of the present invention; FIG. 4 is another diagram illustrating feedback control in one embodiment of the present invention; FIG. 4 is another diagram illustrating feedback control in one embodiment of the present invention; FIG. 4 is a block diagram showing a schematic configuration of a system according to another embodiment of the invention;

Several embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

FIG. 1 is a schematic diagram showing the entire system 1 according to this embodiment.
A system 1 according to the present embodiment is a game system including a camera unit 10, an information processing device 20, one or more controllers 30, and a display device 40, as shown in FIG. The information processing device 20 is connected to each of the camera unit 10, the controller 30, and the display device 40 via a wired or wireless network.
In the system 1, the information processing device 20 progresses the game in accordance with information transmitted from the camera unit 10 and the controller 30, and the display device 40 displays a screen during execution of the information processing device 20, such as a game screen.

In this embodiment, the camera unit 10 estimates the state of the user who is the game player and the orientation of the controller 30 held by the user, and transmits the estimation results to the information processing device 20 . Here, the state of the user includes at least one of the posture of the user, the shape of the user's arm, or the shape of the user's fingers.
The camera unit 10 functions as an operating device for accepting user operations, like the controller 30, by estimating and outputting the user's state and the orientation of the controller 30. FIG. In order to estimate the user's state and the orientation of the controller 30, such a camera unit 10 is arranged at a position where the user can fit in the field of view, for example, at a distance of about 1 meter from the user. In the example of FIG. 1, the camera unit 10 is arranged near the display device 40 . The optimum arrangement position of the camera unit 10 differs depending on the purpose. For example, depending on the content of the game to be played, it is desirable to arrange the camera unit 10 at a position where the target to be grasped, such as the user's whole body, upper body, hand, etc., fits within the field of view.
When arranging the camera unit 10, for example, the information processing device 20 may display a tutorial or the like on the display device 40 to guide the user to arrange the camera unit 10 at an appropriate position.

Each configuration of the system 1 will be described below.
FIG. 2 is a block diagram showing a schematic configuration of a system according to one embodiment of the invention.
Camera unit 10 includes RGB camera 11 , EDS (Event Driven Sensor) 12 , IMU (Inertial Measurement Unit) 13 , estimation section 14 , and information output section 15 .
The RGB camera 11 includes an image sensor 111 as a first image sensor and a processing circuit 112 connected to the image sensor 111 . The image sensor 111 generates RGB image signals 113 by synchronously scanning all pixels at predetermined intervals or at predetermined timings according to user operations, for example. Processing circuitry 112 converts, for example, RGB image signals 113 into a format suitable for storage and transmission. The processing circuit 112 also gives the RGB image signal 113 a time stamp.

EDS 12 includes sensor 121 , which is a second image sensor forming a sensor array, and processing circuitry 122 connected to sensor 121 . The sensor 121 includes a light receiving element, and is an event-driven vision that generates an event signal 123 when it detects a change in the intensity of light incident on each pixel, more specifically, a change in brightness exceeding a predetermined value. sensor. The event signal 123 output through the processing circuitry 122 includes the identity of the sensor 121 (eg, pixel location), the polarity of the luminance change (rising or falling), and a time stamp. Further, when detecting a luminance change, the EDS 12 can generate the event signal 123 at a significantly higher frequency than the RGB image signal 113 (frame rate of the RGB camera 11). In addition, the EDS 12 can generate the event signal 123 at a significantly higher frequency than the RGB image signal 113 (frame rate of the RGB camera 11).
In this specification, a signal from which an image can be constructed is called an image signal. Therefore, the RGB image signal 113 and the event signal 123 are examples of image signals.

In this embodiment, the time stamps given to the RGB image signal 113 and the event signal 123 are synchronized. Specifically, the time stamps applied to the RGB image signal 113 and the event signal 123 can be synchronized, for example, by providing the RGB camera 11 with time information that is used to generate the time stamps in the EDS 12 . Alternatively, if time information for generating time stamps is independent for each of the RGB camera 11 and the EDS 12, time stamps are generated based on the time when a specific event (for example, a subject change over the entire image) occurred. By calculating the offset amount, the time stamps given to the RGB image signal 113 and the event signal 123 can be synchronized afterwards.

Further, in this embodiment, the sensor 121 of the EDS 12 is associated with one or more pixels of the RGB image signal 113 by the calibration procedure of the RGB camera 11 and the EDS 12 performed in advance, and the event signal 123 is the RGB image. It is generated in response to light intensity variations in one or more pixels of signal 113 . More specifically, for example, a common calibration pattern is captured by the RGB camera 11 and the EDS 12, and corresponding parameters between the camera and the sensor are determined from the internal parameters and external parameters of the RGB camera 11 and the EDS 12, respectively. The calculation allows the sensor 121 to be associated with one or more pixels of the RGB image signal 113 .

The IMU 13 is a sensor that detects the orientation of the camera unit 10 itself. The IMU 13 acquires three-dimensional posture data of the camera unit 10 at a predetermined cycle or timing, and outputs the data to the estimation unit 14 .

Based on the event signal 123 generated by the EDS 12 and the RGB image signal 113 generated by the RGB camera 11, the estimation unit 14 estimates the state of the user and the orientation of the controller 30 held by the user. FIG. 3 is a block diagram showing a schematic configuration of the estimation unit 14. As shown in FIG. Estimating section 14 includes first recognizing section 141 , coordinate calculating section 142 , trained model 143 , state estimating section 144 , second recognizing section 145 , and posture estimating section 146 .
A first recognition unit 141 of the estimation unit 14 recognizes one or more users included in the field of view of the camera unit 10 based on at least one of the RGB image signal 113 and the event signal 123 . For example, the first recognition unit 141 detects an object existing in a continuous pixel region in which the event signal 123 indicates that an event of the same polarity has occurred, and detects the object based on the corresponding portion of the RGB image signal 113 . Recognize the user by performing recognition. When multiple users are included in the field of the camera unit 10, the first recognition unit 141 identifies each user.

The coordinate calculating unit 142 of the estimating unit 14 calculates, for each user recognized by the first recognizing unit 141, coordinate information indicating the positions of multiple joints of the user from the RGB image signal 113 based on the learned model 143. . The trained model 143 can be constructed in advance by executing supervised learning using, for example, an image of a person having multiple joints as input data and coordinate information indicating the positions of the multiple joints of the person as correct data. can. It should be noted that a detailed description of a specific method of machine learning is omitted because various known techniques can be used. Further, the estimating unit 14 includes a relationship learning unit that learns the relationship between the image based on the input RGB image signal 113 and the coordinate information indicating the position of the joint each time the RGB image signal 113 is input. A configuration in which the learned model 143 is updated may be used. Then, the state estimation unit 144 of the estimation unit 14 estimates the user's state based on the coordinate information calculated by the coordinate calculation unit 142 for each user recognized by the first recognition unit 141 .

The second recognition unit 145 of the estimation unit 14 recognizes the controller 30 held by each user recognized by the first recognition unit 141 based on at least one of the RGB image signal 113 and the event signal 123 . For example, the second recognition unit 145 performs object recognition on a portion of the RGB image signal 113 corresponding to the vicinity of the user's hand, based on the coordinate information indicating the positions of the multiple joints of the user calculated by the coordinate calculation unit 142. , the controller 30 is recognized. When multiple users are recognized by the first recognition unit 141, the second recognition unit 145 identifies the controller 30 for each user. The identification of the controllers 30, that is, the determination of which controller 30 is held by which user, may be performed in any manner. For example, an identification mark or the like may be attached to each of the plurality of controllers 30, and determination may be made based on the RGB image signal 113. Alternatively, each of the plurality of controllers 30 may output a predetermined identification signal, The determination may be made based on the identification signal received by the camera unit 10 or the information processing device 20 . Further, when the user does not hold the controller 30, the second recognition unit 145 recognizes that "the user does not hold the controller 30".

The posture estimation unit 146 of the estimation unit 14 estimates the posture of each controller 30 recognized by the second recognition unit 145 . For example, the posture estimation unit 146 estimates the shape of the controller 30 based on the result of subject recognition performed by the second recognition unit 145 in the RGB image signal 113, and estimates the posture of the controller 30 based on the estimated shape. do. Note that if the controller 30 has a sensor that detects the orientation of the controller 30 itself, the orientation of the controller 30 may be determined by taking into account the output of the sensor. When a plurality of controllers 30 are recognized by the second recognition unit 145 , the posture estimation unit 146 estimates the posture of each controller 30 . It should be noted that posture estimation section 146 may estimate the posture of controller 30 using a machine learning method using a learned model, similar to coordinate calculation section 142 described above.

4A to 4D are diagrams for explaining an example of estimation of the user's state by the estimation unit 14 and estimation of the posture of the controller 30. FIG. Figures 4A-4D illustrate a controller 30 having left and right grip portions.
The state of the user estimated by the estimation unit 14 includes at least one of the posture of the user, the shape of the user's arms, or the shape of the user's fingers, as described above. The posture of the user includes, for example, a state in which the user is seated on a chair, a state in which the user stands, a state in which the user faces the camera unit 10, and a state in which the user faces the side. including. Further, the shape of the user's arm includes, for example, a state in which the user raises the arm, a state in which the user moves the arm to take a predetermined pose, and the like. Further, the shape of the user's fingers includes, for example, a state in which the user is taking a predetermined pose such as a peace sign in which the user moves the fingers and raises two fingers, a state in which the user is gripping the controller 30, and the like. Also, the attitude of the controller 30 indicates which part of the controller 30 the user is holding and in what attitude the user is holding the controller 30 .

FIG. 4A illustrates a state in which the user takes a bow and arrow pose and holds the central portion of the controller 30 with one hand. The example of FIG. 4A shows an example of a game in which a virtual bow V1 and arrow V2 are manipulated based on the user's state and the posture of the controller 30. FIG.
FIG. 4B illustrates a state in which the user holds the grip portion of the controller 30 with both hands and rotates it. The example of FIG. B shows an example of a game in which a steering wheel of a motorcycle or the like is operated based on the state of the user and the posture of the controller 30 .
FIG. 4C illustrates a state in which the user takes a pose of holding a bat and grips the grip portion on one side of the controller 30 with one hand. The example of FIG. 4C shows an example of a game in which a virtual bat V3 is operated based on the state of the user and the posture of the controller 30. FIG.
FIG. 4D illustrates a state in which the user is in a shuriken throwing pose without holding the controller 30 . The example of FIG. 4D shows an example of a game in which a virtual shuriken V4 is manipulated based on the user's state without using the controller 30 .

The estimation processing by the estimation unit 14 described so far is merely an example, and how and when the RGB image signal 113 and the event signal 123 are used is not limited to this example. For example, based on the event signal 123, the estimation unit 14 may calculate the movement amount of part or all of the user included in the object scene and use the calculation result for the estimation process. Also, the estimation process may be performed using a known method such as a block matching method or a gradient method.
Further, for example, the three-dimensional posture data of the camera unit 10 detected by the IMU 13 may be used for the estimation processing by the estimation unit 14 .

The information output unit 15 outputs information indicating the state of the user estimated by the estimation unit 14 and information indicating the posture of the controller 30 to the information processing device 20 . The information indicating the state of the user may be coordinate information calculated by the coordinate calculation unit 142 or information indicating the state of the user estimated by the state estimation unit 144 .
Also, when a plurality of users are recognized by the first recognition unit 141 , the information output unit 15 outputs to the information processing device 20 information indicating combinations of users and controllers held by the users. When outputting such information, information such as a table showing a combination of information indicating the user's state and information indicating the attitude of the controller 30 may be output. The identification information of the controller 30 held by the user may be associated, or the information indicating the attitude of the controller 30 may be associated with the information indicating the state of the user holding the controller 30 .
Further, the information output unit 15 may output information indicating the user's operation to the information processing device 20 by predetermining the relationship between the user's state, the posture of the controller 30, and the user's operation in a table or the like. .

As described above, the camera unit 10 functions as an operating device for accepting user operations in the same way as the controller 30 by estimating the user's state and the orientation of the controller 30 . In other words, the user operation can be identified based on the state of the user estimated by the estimation unit 14 and the posture of the controller 30 . In addition, the camera unit 10 alone completes everything from generating the RGB image signal 113 and the event signal 123 to estimating the state of the user and estimating the posture of the controller 30, and transmits the RGB image signal 113 and the event signal 123 to the information processing device 20. The estimation result can be output without outputting the signal 123 . Note that the camera unit 10 preferably has an independent power source.

Referring to FIG. 2 again, the information processing device 20 is implemented by a computer having, for example, a communication interface, a processor, and a memory, and includes a communication section 21 and a control section 22. The control unit 22 includes functions of a control value calculation unit 221 and an image generation unit 222 implemented by the processor operating according to a program stored in memory or received via a communication interface. The function of each unit will be further described below.

The communication section 21 receives information output from the information output section 15 of the camera unit 10 . Further, the communication unit 21 can communicate with the controller 30 and outputs an image to be displayed on the display device 40 .
Based on at least one of the information received from the information output unit 15 of the camera unit 10 and the information received from the controller 30, the control value calculation unit 221 of the control unit 22 determines the output to the external device including the controller 30 and the display device 40. Calculate the control value for feedback control. As described above, the camera unit 10 and controller 30 function as an operation device for accepting user operations. Therefore, the control value calculation unit 221 calculates a control value for feedback control to an external device including the controller 30 and the display device 40 according to a user operation performed via at least one of the camera unit 10 and the controller 30. do. The calculated control value is output to the controller 30 and the display device 40 via the communication section 21 .
The image generator 222 of the controller 22 generates a display image to be displayed on the display device 40 according to the control value calculated by the control value calculator 221 . The generated display image is output to the display device 40 via the communication section 21 .
Details of calculation of the control values and generation of the display image will be described in connection with the description of the configurations of the controller 30 and the display device 40, which will be described later.

The controller 30 includes a communication section 31, an operation section 32, a force sense presentation section 33, a vibration section 34, and an audio output section 35, as shown in FIG. A user can perform various operations related to the game by operating the controller 30 .
The communication unit 31 receives the control values output from the communication unit 21 of the information processing device 20 and outputs the control values to the force sense presentation unit 33 , the vibration unit 34 , and the audio output unit 35 . The communication unit 31 also outputs information regarding user operations received by the operation unit 32 to the information processing device 20 .
The operation unit 32 includes a plurality of operators such as buttons and pads, and accepts user's operation input to the operators.
The haptic presentation unit 33 is provided in at least a part of the operation elements of the operation unit 32 , and presents the user with a force that resists or interlocks with the user's operation according to the control value supplied from the information processing device 20 . Specifically, the force sense presentation unit 33 can be configured by a motor, an actuator, or the like including a rotating rotor. A well-known device can be used as the haptic device that constitutes the haptic device 33, and detailed description thereof will be omitted here.

The vibrating section 34 generates vibration according to a control value supplied from the information processing device 20, and can be configured by a motor, for example. The vibrating unit 34 can notify the user that the user operation has been correctly performed and has been recognized by the information processing device 20 by generating vibration when the user operation is performed.
The audio output unit 35 outputs audio according to the control value supplied from the information processing device 20, and can be configured by, for example, a speaker. The audio output unit 35 can notify the user that the user operation has been correctly performed and has been recognized by the information processing apparatus 20 by outputting audio when the user operation is performed.
At least one of the vibration by the vibration unit 34 and the sound output by the sound output unit 35 is performed in conjunction with the presentation of the force sense by the force sense presentation unit 33 described above, thereby providing various feedback controls to the user. It is possible to improve

As described above, the control value calculation unit 221 of the information processing device 20 calculates a control value for feedback control to the controller 30. More specifically, the force sense presentation unit 33, vibration unit 34, and And a control value for feedback control to the audio output unit 35 is calculated. Regarding the force sense presentation unit 33, the control value calculation unit 221 calculates a control value indicating what kind of force sense is to be presented as feedback control according to the user's operation. Regarding the vibration unit 34, the control value calculation unit 221 calculates a control value indicating whether to present what kind of vibration is to be generated as feedback control according to the user's operation. Regarding the audio output unit 35, the control value calculation unit 221 calculates a control value indicating what kind of audio is to be output as feedback control according to the user's operation. The calculation of the control value by the control value calculator 221 can be performed according to a predetermined calculation formula, table, or the like.
When the first recognition unit 141 recognizes a plurality of users and the information output unit 15 outputs information indicating the combination of the user and the controller held by the user, the control value calculation unit 221 recognizes the user , the control value is calculated for each combination with the controller held by the user.

For example, as shown in the example of FIG. 4A described above, when the user performs the operation of shooting the arrow V2 with the virtual bow V1, the control value calculation unit 221 calculates the force based on the state of the user and the posture of the controller 30. For the sense presenting unit 33 and the vibrating unit 34, a control value is calculated which indicates the presenting of the sense of force corresponding to the recoil when the bow and arrow is actually shot, and the generation of the vibration. The control value calculator 221 also calculates a control value indicating the output of a sound corresponding to the sound when the bow and arrow are actually shot.
As shown in the example of FIG. 4B , when a user operation of operating a steering wheel of a motorcycle or the like is performed, the control value calculation unit 221 determines the force sense presentation unit 33 and the vibration of the controller 30 based on the state of the user and the posture of the controller 30 . With respect to the portion 34, a control value is calculated that indicates the presentation of the force sense corresponding to the reaction to the operation of the steering wheel and the generation of the vibration. Also, the control value calculator 221 calculates a control value indicating the output of the sound corresponding to the operation of the steering wheel.
As shown in the example of FIG. 4C , when a user operation to operate the virtual bat V3 is performed, the control value calculation unit 221 causes the force sense presentation unit 33 and vibration With respect to the section 34, a control value is calculated that indicates the presentation of a force sense corresponding to the recoil when the bat is actually operated and the generation of vibration. Also, the control value calculator 221 calculates a control value indicating the output of a sound corresponding to the sound generated when the bat is actually operated.
Note that, as shown in the example of FIG. 4D , when the user's operation is performed without using the controller 30 , the control value calculator 221 does not calculate the control value for feedback control to the controller 30 .

Various known configurations can be applied to the controller 30 described so far. For example, it may be configured by a pair of controllers that can be held with both hands, may be configured by a controller that allows character input such as a keyboard, or may be configured by an application such as a smartphone.
Further, the controller 30 may be provided with a contact sensor, and information indicating the contact state of the user to the contact sensor may be supplied to the information processing device 20 via the communication unit 31 . Such information can be used for calculation of the control value by the control value calculator 221 .
Also, the controller 30 may be provided with a voice input unit and a voice recognition technology may be applied. For example, the controller 30 may include a voice input unit such as a microphone and a voice recognition unit, and may supply commands uttered by the user and information indicating user calls to the information processing apparatus 20 via the communication unit 31. good.

The display device 40 includes a receiver 41 and a display 42, as shown in FIG.
The reception unit 41 receives information indicating the display image generated by the image generation unit 222 of the information processing device 20 via the communication unit 21 .
The display unit 42 has a monitor such as an LCD (Liquid Crystal Display) or an organic EL, and can present the information to the user by displaying a display image based on the information received by the reception unit 41 .

Various known configurations can be applied to the display device 40 described above. For example, it may be configured by the dedicated display device shown in FIG. 1, or may be configured by a display device such as an HMD mounted on the user's head. For example, the display unit of the HMD includes a display element such as LCD (Liquid Crystal Display) or organic EL, and an optical device such as a lens. It may be a transmissive display element. Furthermore, wearable devices such as AR (Augmented Reality) glasses and MR (Mixed Reality) glasses may be used as the HMD. Further, the display device 40 described above may be configured by a display device of a computer, or may be configured by a display device of a terminal device such as a smart phone. A touch panel for detecting contact may be provided on the surface of the display unit 42 .

As described above, the control value calculation unit 221 of the information processing device 20 calculates the control value for feedback control to the display image displayed on the display device 40. More specifically, the control value calculation unit 221 A control value is calculated that indicates how the display image is to be changed as feedback control according to the user's operation. The calculation of the control value by the control value calculator 221 can be performed according to a predetermined calculation formula, table, or the like.

The image generation unit 222 of the information processing device 20 generates a display image to be displayed on the display device 40 according to the control value calculated by the control value calculation unit 221, as described above. More specifically, the image generator 222 generates a new display image to be displayed on the display device 40 according to the control value for changing the display image. Note that the state of the user estimated by the camera unit 10 and the orientation of the controller 30 are reflected in the generation of the display image. Therefore, for example, when the user stands still and the posture of the controller 30 does not change, the change in the generated display image is small or there is no change. The image changes according to the user's operation. Also, when a plurality of users are included in the field of the camera unit 10, the generated display image is an image that changes according to the number of users.

FIG. 5 is a flow chart showing an example of processing of the camera unit 10 according to one embodiment of the present invention. In the illustrated example, the image sensor 111 of the RGB camera 11 generates the RGB image signal 113 (step S101) and the sensor 121 of the EDS 12 generates the event signal 123 (step S102).
Then, the first recognition unit 141 recognizes the user (step S103), and the coordinate calculation unit 142 and state estimation unit 144 estimate the state of the user (step S104). Next, the second recognition unit 145 estimates the state of the controller 30 (step S105), and the posture estimation unit 146 estimates the posture of the controller 30 (step S106).
Then, the information output unit 15 outputs information indicating the state of the user, information indicating the posture of the controller 30, and information indicating the combination of the user and the controller held by the user (step S107).
The estimating unit 14 continues outputting information by repeating steps S103 to S107 (the processing of steps S101 and S102 is also repeated, but the cycle may not necessarily be the same as the processing after step S103).

Through such processing, the camera unit 10 always outputs information indicating the latest state of the user, information indicating the posture of the controller 30, and information indicating the combination of the user and the controller 30 held by the user.
The information processing device 20 calculates a control value for feedback control to the controller 30 based on these pieces of information, thereby realizing suitable feedback control according to changes in the state of the user and the posture of the controller 30. .
For example, consider the case where the number of users changes, as illustrated in FIGS. 6A and 6B. As shown in FIG. 6A, when one user U1 exists in the field of the camera unit 10, the camera unit 10 receives information indicating the state of the user U1 and the attitude of the controller 30 _U1 held by the user U1. , and the information processing device 20 calculates a control value for feedback control to the controller _30U1 . After that, as shown in FIG. 6B, when another user U2 enters the field of view of the camera unit 10, the camera unit 10 displays the information indicating the states of the user U1 and the user U2 and the information held by the user U1. information indicating the attitude of controller 30 _U1 and controller 30 _U2 held by user U2, and information indicating the combination of the user and controller 30 held by the user, and information processing apparatus 20 outputs controller 30 _U1 to controller 30 U1. and the control value of the feedback control to the controller _30U2 .
In other words, even if the number of users dynamically changes, the information indicating the user's state, the information indicating the attitude of the controller 30, and the information indicating the combination of the user and the controller 30 held by the user are updated. , it is possible to realize feedback control that always matches the latest state.

Also, for example, as illustrated in FIGS. 7A and 7B, consider a case where the number of users changes and the users who hold the controller 30 change. As shown in FIG. 7A, when one user U1 exists in the field of view of the camera unit 10, the camera unit 10 receives information indicating the state of the user U1 and the attitude of the controller 30 _U1 held by the user U1. , and the information processing device 20 calculates a control value for feedback control to the controller _30U1 . After that, as shown in FIG. 7B, when another user U2 enters the field of view of the camera unit 10 and the user U2 holds the controller 30 _U1 held by the user U1, the camera unit 10 , information indicating the states of the user U1 and the user U2, information indicating the posture of the controller 30 _U1 held by the user U2, and information indicating the combination of the user and the controller 30 held by the user, and performing information processing. The device 20 calculates a control value for feedback control to the controller _30U1 .
In other words, even if the user holding a certain controller 30 changes dynamically, information indicating the state of the user, information indicating the posture of the controller 30, and information indicating the combination of the user and the controller 30 held by the user are Since it is updated, it is possible to realize feedback control that always matches the latest situation.
Although illustration is omitted, the same applies when the number of users does not change and the user holding the controller 30 changes. In either case, even if the number of users changes or the user holding the controller 30 changes, feedback control suitable for the situation can be realized without performing special setting processing or the like.

In one embodiment of the present invention as described above, the camera unit 10 is based on the RGB image signal 113 and the event signal 123 generated by the image sensor 111 and the sensor 121, respectively, based on the user's state and the state held by the user. The information processing device 20 performs feedback control to the controller 30 based on at least one of the information indicating the state of the user and the information indicating the orientation of the controller 30. Calculate the control value of
Therefore, when a user operation is performed via the camera unit 10, by calculating a control value for feedback control to the controller 30 based on information indicating the user's state and information indicating the attitude of the controller 30, the latency To realize suitable feedback control to a controller while suppressing

Further, the camera unit 10 performs from the generation of the RGB image signal 113 and the event signal 123 to the estimation of the state of the user and the posture of the controller 30, and the RGB image signal 113 and the event signal 123 are not output, and information indicating the estimation result can reduce the problem of communication load and communication delay. Furthermore, since it is not necessary to output the RGB image signal 113 and the event signal 123, it is also useful in terms of privacy protection.
In addition, since the camera unit 10 of one embodiment of the present invention can estimate the user's state and the attitude of the controller 30 and accept user operations, the cursor position can be maintained as in the conventional pointing device type operation device. It also does not cause physical fatigue for the user. In addition, the camera unit 10 does not require the user to wear a marker or an attachment to be recognized, unlike a conventional posture detection type operating device.

Also, in one embodiment of the present invention, both the EDS 12 and the RGB camera 11 are provided, and the user's state and the attitude of the controller 30 are estimated based on the event signal 123 and the RGB image signal 113 . Therefore, it is possible to realize suitable processing that takes advantage of the respective characteristics of the RGB image signal 113 and the event signal 123 .

In addition, in one embodiment of the present invention, the user's state estimated by the estimation unit 14 includes at least one of the user's posture, the shape of the user's arm, or the shape of the user's fingers. Therefore, it is possible to estimate the characteristic user state and accurately grasp the intention and content of the user's operation.

Further, in one embodiment of the present invention, the estimation unit 14 uses a learned model constructed by learning the relationship between an image of a person having multiple joints and coordinate information indicating the positions of the multiple joints. Based on this, the coordinate information of at least one joint of the user included in the image based on the RGB image signal 113 is calculated, and the state of the user is estimated based on the coordinate information. Therefore, the user's condition can be accurately and quickly estimated.

Further, in one embodiment of the present invention, the first recognition unit 141 recognizes one or more users included in the object scene based on at least one of the RGB image signal 113 and the event signal 123, and the estimation unit 14 estimates the user's state and the posture of the controller 30 held by the user for each user recognized by the first recognition unit 141 . Therefore, it is possible to grasp user operations for each of a plurality of users included in the field of the camera unit 10 .

Further, in one embodiment of the present invention, a second recognition unit 145 is provided for each user recognized by the first recognition unit 141, and the second recognition unit 145 recognizes the controller 30 held by the user. Information indicating the combination of the user recognized by 141 and the controller 30 recognized by the second recognition unit 145 is output. Therefore, even in a state where a plurality of controllers 30 are used by a plurality of users, user operations based on combinations of users and controllers 30 can be grasped and reflected in feedback control.

FIG. 8 is a block diagram showing a schematic configuration of a system according to another embodiment of the invention. 8 is a block diagram showing the configuration of a system 2 having a server 50 and a terminal device 60 instead of the information processing device 20 of FIG. Constituent elements having functional configurations are given the same reference numerals.

In the example of FIG. 8, the server 50 is a server (for example, a cloud server) communicably connected to the camera unit 10 and the terminal device 60 via the Internet communication network or wirelessly. The server 50 has the same configuration as the information processing apparatus 20 described with reference to FIG. 2, and performs various processes based on the information output by the camera unit 10. The terminal device 60 also includes a communication unit 61 , and the communication unit 61 receives information output from the server 50 . The communication unit 61 can communicate with the controller 30 and outputs an image to be displayed on the display device 40, like the communication unit 21 of the information processing apparatus 20 described with reference to FIG.
With such a configuration, the camera unit 10 performs everything from generating the RGB image signal 113 and the event signal 123 to estimating the state of the person, and outputting the estimated information to the server 50 enables the use of a server such as a cloud server. A similar effect can be obtained in a game system that has been used.

Also, in each of the above examples, the number of RGB cameras 11 and EDS 12 may be the same or may be different. Also, the number of RGB cameras 11 and EDS 12 may be one or more. For example, when a plurality of RGB cameras 11 are provided, it is possible to expand the range of the object field for generating the RGB image signals 113, or to estimate the state of a person in three dimensions from the plurality of RGB image signals 113. can. Further, for example, when a plurality of EDSs 12 are provided, the range of the object field for generating the event signal 123 is expanded, and the three-dimensional movement amount of the person is calculated based on the plurality of event signals 123. be able to.

In addition, the camera unit 10 described in each of the above examples may be implemented within a single device, or may be implemented distributed among a plurality of devices. For example, at least a portion of each sensor may be provided independently, and another configuration may be implemented as the camera unit 10 main body.

Although several embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field to which the present invention belongs can conceive of various modifications or modifications within the scope of the technical idea described in the claims. It is understood that these also belong to the technical scope of the present invention.

Reference Signs List 1 2 system 10 camera unit 11 RGB camera 12 EDS 13 IMU 14 estimation unit 14 information output unit 20 information processing device 21 31 61 communication unit 22 ... control section 32 ... operation section 33 ... force presentation section 34 ... vibration section 35 ... audio output section 40 ... display device 41 ... reception section 42 ... display section 50 ... server 60 ... terminal device , 111... Image sensor 112/122... Processing circuit 113... RGB image signal 121... Sensor 123... Event signal 141... First recognition unit 142... Coordinate calculation unit 143... Learned model 144... State Estimation unit 145 Second recognition unit 146 Attitude estimation unit 221 Control value calculation unit 222 Image generation unit.

Claims

A system that includes a sensor device, a controller that receives a user operation, and an information processing device that performs processing based on the user operation,
The sensor device is
a first image sensor that generates a first image signal by synchronously scanning all pixels at a predetermined timing;
a second image sensor that includes an event-driven vision sensor that asynchronously generates a second image signal upon detecting a change in intensity of light incident on each pixel;
an estimating unit that estimates a state of a user and an orientation of the controller held by the user based on the first image signal and the second image signal;
an information output unit that outputs information indicating the state of the user and information indicating the attitude of the controller;
The information processing device is
a control value calculation unit that calculates a control value for feedback control to the controller based on at least one of information indicating the state of the user and information indicating the attitude of the controller;
The controller operates at least one of a force sense presentation device that presents a force sense based on the control value, a vibration device that vibrates based on the control value, or an audio output device that outputs sound based on the control value. have a system.
The system according to claim 1, wherein the user's state includes at least one of the user's posture, the shape of the user's arm, or the shape of the user's fingers.
The estimation unit estimates the first image signal based on a learned model constructed by learning a relationship between an image of a person having multiple joints and coordinate information indicating positions of the multiple joints. 3. The system according to claim 1 or 2, wherein coordinate information of at least one joint of said user included in a first image based thereon is calculated, and said user's state is estimated based on said coordinate information.
The information processing device further includes a first recognition unit that recognizes one or more users included in the object scene based on at least one of the first image signal and the second image signal,
4. The estimation unit according to any one of claims 1 to 3, wherein, for each user recognized by the first recognition unit, the estimation unit estimates the state of the user and the posture of the controller held by the user. The system described in .
the system includes a plurality of the controllers;
The information processing device further includes a second recognition unit that recognizes the controller held by the user for each of the users recognized by the first recognition unit,
5. The system according to claim 4, wherein said information output unit outputs information indicating a combination of said user recognized by said first recognition unit and said controller recognized by said second recognition unit.
An information processing method for outputting a control value for feedback control to a controller,
A first image signal is generated by a first image sensor that synchronously scans all pixels at a predetermined timing, and a second image signal is generated asynchronously when a change in intensity of light incident on each pixel is detected. an estimating step of estimating the state of the user and the pose of the controller held by the user based on the second image signal produced by a second image sensor, including an event-driven vision sensor;
and an information output step of outputting information indicating the state of the user and information indicating the attitude of the controller.
A first image signal is generated by a first image sensor that synchronously scans all pixels at a predetermined timing, and a second image signal is generated asynchronously when a change in intensity of light incident on each pixel is detected. estimating the state of the user and the pose of a controller held by the user based on the second image signal generated by a second image sensor including an event-driven vision sensor;
An information processing program that causes a computer to implement a function of outputting information indicating the state of the user and information indicating the attitude of the controller.