WO2022220049A1 - システム、情報処理方法および情報処理プログラム - Google Patents
システム、情報処理方法および情報処理プログラム Download PDFInfo
- Publication number
- WO2022220049A1 WO2022220049A1 PCT/JP2022/013836 JP2022013836W WO2022220049A1 WO 2022220049 A1 WO2022220049 A1 WO 2022220049A1 JP 2022013836 W JP2022013836 W JP 2022013836W WO 2022220049 A1 WO2022220049 A1 WO 2022220049A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- information
- image
- user
- coordinate
- coordinate information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/20—Input arrangements for video game devices
- A63F13/21—Input arrangements for video game devices characterised by their sensors, purposes or types
- A63F13/211—Input arrangements for video game devices characterised by their sensors, purposes or types using inertial sensors, e.g. accelerometers or gyroscopes
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/20—Input arrangements for video game devices
- A63F13/21—Input arrangements for video game devices characterised by their sensors, purposes or types
- A63F13/213—Input arrangements for video game devices characterised by their sensors, purposes or types comprising photodetecting means, e.g. cameras, photodiodes or infrared cells
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/20—Input arrangements for video game devices
- A63F13/21—Input arrangements for video game devices characterised by their sensors, purposes or types
- A63F13/219—Input arrangements for video game devices characterised by their sensors, purposes or types for aiming at specific areas on the display, e.g. light-guns
Definitions
- the present invention relates to a system, an information processing method, and an information processing program.
- An event-driven vision sensor in which pixels that detect changes in the intensity of incident light generate signals asynchronously with time.
- An event-driven vision sensor is advantageous in that it can operate at low power and at high speed compared to a frame-type vision sensor that scans all pixels at predetermined intervals, specifically image sensors such as CCD and CMOS. is. Techniques related to such an event-driven vision sensor are described in Patent Document 1 and Patent Document 2, for example.
- the present invention calculates coordinate information about the user based on the image signal generated by the image sensor, and performs coordinate transformation on the coordinate information based on the orientation information of the image sensor. Then, by generating a display image showing the user's state based on the coordinate information after coordinate conversion, it is possible to generate a display image that more accurately corresponds to the user's state while suppressing latency.
- An object is to provide a processing method and an information processing program.
- a first image sensor generates a first image signal by synchronously scanning all pixels at a predetermined timing
- a a second image sensor including an event-driven vision sensor for asynchronously generating a second image signal; an inertial sensor for obtaining attitude information of the first image sensor and the second image sensor; a first arithmetic processing unit for recognizing a user by means of a second image signal and calculating coordinate information about the user based on at least the second image signal; and a second arithmetic processing unit for performing coordinate conversion processing on the coordinate information based on the posture information.
- an image generation device that generates a display image showing a user's state based on the coordinate information after coordinate conversion processing.
- an information processing method for generating a display image to be displayed on a display device wherein the first image is generated by a first image sensor that synchronously scans all pixels at a predetermined timing. and a second image signal generated by a second image sensor including an event-driven vision sensor that asynchronously generates a second image signal upon detecting a change in intensity of light incident on each pixel.
- the first image signal is generated by a first image sensor that synchronously scans all pixels at a predetermined timing, and the intensity change of incident light for each pixel is detected.
- a second image signal generated by a second image sensor including an event-driven vision sensor that asynchronously generates the second image signal; and recognizing a user based at least on the second image signal.
- An information processing program is provided that causes a computer to realize a function of generating a display image showing the state of the user based on subsequent coordinate information.
- coordinate information about the user is calculated based on the image signal generated by the image sensor, and coordinate conversion is performed on the coordinate information based on the posture information of the image sensor. Then, by generating a display image showing the user's state based on the coordinate information after coordinate conversion, it is possible to generate a display image that more accurately reflects the user's state while suppressing latency.
- FIG. 1 is a schematic diagram showing the entire system according to one embodiment of the present invention
- FIG. 1 is a block diagram showing a schematic configuration of a system according to one embodiment of the present invention
- FIG. It is a block diagram showing a schematic configuration of a calculation unit in the system according to one embodiment of the present invention.
- FIG. 4 is a diagram illustrating orientation information of a camera unit according to an embodiment of the present invention
- FIG. 11 is another diagram illustrating orientation information of the camera unit according to the embodiment of the present invention
- FIG. 4B is another diagram illustrating the orientation of the camera unit according to the embodiment of the present invention
- FIG. 4 is a diagram illustrating position information of a camera unit in one embodiment of the present invention
- FIG. 5 is another diagram illustrating position information of a camera unit in one embodiment of the present invention
- 1 is a flowchart illustrating an example of a processing method according to an embodiment of the invention
- 4 is another flow chart illustrating an example of a processing method according to an embodiment of the present invention
- FIG. 4 is a block diagram showing a schematic configuration of a system according to another embodiment of the invention
- FIG. 1 is a schematic diagram showing the entire system 1 according to this embodiment.
- a system 1 according to the present embodiment is a game system including a camera unit 10 which is a sensor device, an information processing device 20, a controller 30, and a display device 40, as shown in FIG.
- the information processing device 20 is connected to each of the camera unit 10, the controller 30, and the display device 40 via a wired or wireless network.
- the information processing device 20 progresses the game in accordance with information transmitted from the camera unit 10 and the controller 30, and the display device 40 displays a screen during execution of the information processing device 20, such as a game screen.
- the camera unit 10 calculates coordinate information regarding the user who is a game player, and transmits the information to the information processing device 20 .
- the coordinate information about the user means coordinate information about the posture of the user, coordinate information about the shape of the user's arm, coordinate information about the shape of the user's fingers, or an external device such as the controller 30 worn or held by the user. at least one coordinate information about the attitude of the
- the camera unit 10 functions as an operating device for accepting user operations, like the controller 30, by calculating and outputting coordinate information regarding the user.
- the camera unit 10 is arranged at a position where the user can be accommodated in the object field, for example, at a distance of about 1 meter from the user, and the elevation angle can be changed manually or electrically.
- the camera unit 10 is arranged near the display device 40 . Details of the arrangement of the camera unit 10 will be described later.
- FIG. 2 is a block diagram showing a schematic configuration of a system according to one embodiment of the invention.
- the camera unit 10 includes an RGB camera 11, an EDS (Event Driven Sensor) 12, an IMU (Inertial Measurement Unit) 13, a position information acquisition unit 14, a calculation unit 15 functioning as a first processing unit, an information and an output unit 16 .
- the RGB camera 11 includes an image sensor 111 as a first image sensor and a processing circuit 112 connected to the image sensor 111 .
- the image sensor 111 generates RGB image signals 113, which are first image signals, by synchronously scanning all pixels (all pixels) at predetermined intervals or at predetermined timings according to user operations.
- Processing circuitry 112 converts, for example, RGB image signals 113 into a format suitable for storage and transmission.
- the processing circuit 112 also gives the RGB image signal 113 a time stamp.
- EDS 12 includes sensor 121 , which is a second image sensor forming a sensor array, and processing circuitry 122 connected to sensor 121 .
- the sensor 121 includes a light-receiving element, and generates an event signal 123, which is a second image signal, when a change in intensity of light incident on each pixel, more specifically, a change in brightness exceeding a predetermined value is detected. It is an event-driven vision sensor that
- the event signal 123 output through the processing circuitry 122 includes the identity of the sensor 121 (eg, pixel location), the polarity of the luminance change (rising or falling), and a time stamp.
- the EDS 12 can generate the event signal 123 at a significantly higher frequency than the RGB image signal 113 (frame rate of the RGB camera 11).
- the EDS 12 can generate the event signal 123 at a significantly higher frequency than the RGB image signal 113 (frame rate of the RGB camera 11).
- a signal from which an image can be constructed is called an image signal. Therefore, the RGB image signal 113 and the event signal 123 are examples of image signals.
- the time stamps given to the RGB image signal 113 and the event signal 123 are synchronized.
- the time stamps applied to the RGB image signal 113 and the event signal 123 can be synchronized, for example, by providing the RGB camera 11 with time information that is used to generate the time stamps in the EDS 12 .
- time information for generating time stamps is independent for each of the RGB camera 11 and the EDS 12
- time stamps are generated based on the time when a specific event (for example, a subject change over the entire image) occurred.
- the time stamps given to the RGB image signal 113 and the event signal 123 can be synchronized after the fact.
- the sensor 121 of the EDS 12 is associated with one or more pixels of the RGB image signal 113 by the calibration procedure of the RGB camera 11 and the EDS 12 performed in advance, and the event signal 123 is the RGB image. It is generated in response to light intensity variations in one or more pixels of signal 113 . More specifically, for example, a common calibration pattern is captured by the RGB camera 11 and the EDS 12, and corresponding parameters between the camera and the sensor are determined from the internal parameters and external parameters of the RGB camera 11 and the EDS 12, respectively. The calculation allows the sensor 121 to be associated with one or more pixels of the RGB image signal 113 .
- the IMU 13 is an inertial sensor that detects the orientation of the camera unit 10 itself, more specifically, the image sensor 111 and the sensor 121 .
- the IMU 13 acquires three-dimensional posture information of the image sensor 111 and the sensor 121 at predetermined intervals or at predetermined timing. This posture information indicates the elevation angle of the camera unit 10 described above.
- the IMU 13 outputs the acquired posture information to the information output unit 16 .
- the location information acquisition unit 14 acquires location information from GPS satellites, for example, and outputs it to the information output unit 16 . Note that the location information acquisition unit 14 may acquire location information and the like from radio base stations instead of or in addition to GPS satellites.
- the calculator 15 calculates coordinate information about the user based on the event signal 123 generated by the EDS 12 and the RGB image signal 113 generated by the RGB camera 11 .
- FIG. 3 is a block diagram showing a schematic configuration of the calculator 15. As shown in FIG.
- the calculator 15 includes a coordinate calculator 151 and a trained model 152 .
- the coordinate calculation unit 151 of the calculation unit 15 detects an object existing in a continuous pixel region in which the event signal 123 indicates that an event of the same polarity has occurred, and calculates the corresponding part of the RGB image signal 113.
- the user is recognized by performing subject recognition based on this.
- the coordinate calculator 151 identifies each user.
- the coordinate calculation unit 151 calculates coordinate information indicating the positions of the joints of the user from the RGB image signal 113 based on the learned model 152 for each recognized user.
- the trained model 152 can be constructed in advance by executing supervised learning using, for example, an image of a person having multiple joints as input data and coordinate information indicating the positions of the multiple joints of the person as correct data. can. It should be noted that a detailed description of a specific method of machine learning is omitted because various known techniques can be used.
- the calculation unit 15 is provided with a relationship learning unit, and each time the RGB image signal 113 is input, the relationship between the image based on the input RGB image signal 113 and the coordinate information indicating the position of the joint is learned. A configuration in which the learned model 152 is updated may be used.
- the coordinate calculation unit 151 recognizes the controller 30 held by each recognized user. For example, the coordinate calculation unit 151 performs subject recognition on a portion of the RGB image signal 113 corresponding to the vicinity of the user's hand, based on the coordinate information indicating the positions of the joints of the user calculated by the coordinate calculation unit 151 . By doing so, the controller 30 is recognized. Further, when the user does not hold the controller 30, the coordinate calculation unit 151 calculates information indicating that "the user does not hold the controller 30" instead of the coordinate information. Then, the coordinate calculation unit 151 estimates the recognized orientation of the controller 30 .
- the coordinate calculation unit 151 estimates the shape of the controller 30 based on the result of subject recognition described above in the RGB image signal 113, and estimates the posture of the controller 30 based on the estimated shape. If the controller 30 has an inertial sensor or the like for detecting the orientation of the controller 30 itself, the orientation of the controller 30 may be estimated by taking into account the output of the sensor. As described above, when a plurality of users are included in the object scene, the coordinate calculation unit 151 estimates the orientation of the controller 30 held by each user. Note that the coordinate calculation unit 151 may estimate the orientation of the controller 30 using a machine learning method using the learned model described above.
- the information output unit 16 outputs the coordinate information about the user calculated by the calculation unit 15, the posture information acquired by the IMU 13, and the position information acquired by the position information acquisition unit 14 to the information processing device 20.
- the camera unit 10 alone completes the process from generating the RGB image signal 113 and the event signal 123 to calculating the coordinate information about the user. 123, the calculated coordinate information and orientation information can be output. Note that the camera unit 10 preferably has an independent power source.
- FIG. 4A is a side view of the positional relationship between the user U, the camera unit 10, and the display device 40
- FIG. 4B is a top view of the positional relationship between the user U, the camera unit 10, and the display device 40.
- FIG. It is a diagram. 4A and 4B, the user U is facing the display device 40, and the camera unit 10 is arranged facing the user U on the user U side of the display device 40, as shown in FIGS. 4A and 4B.
- the user U In a state where In the reference installation state, the user U, the imaging direction (arrow C) of the camera unit 10, and the display direction (arrow D) of the display device 40 are substantially parallel.
- a reference installation state is realized, for example, by displaying a tutorial or the like on the display device 40 by the information processing device 20 and prompting the user U to arrange the camera unit 10 correctly.
- the orientation information acquired by the IMU 13 and the position information generated by the position information acquisition unit 14 when the reference installation state is realized are the orientation information and the position information of the reference installation state.
- the orientation information and position information of the reference installation state are output by the information output unit 16 and used for information processing in the information processing device 20 .
- the calculator 15 may be difficult for the calculator 15 to calculate the coordinate information in the reference installation state.
- FIG. 4A consider an example of a game in which the user U assumes a bow and arrow pose, holds the central portion of the controller 30 with his left hand, and operates a virtual bow V1 and arrow V2 by moving his right hand. .
- both hands of the controller 30 and the user U may overlap when viewed from the camera unit 10 . Therefore, it is very difficult to separate both hands of the controller 30 and the user U in the generated RGB image signal 113 and the event signal 123 and calculate the coordinate information.
- FIG. 4A consider an example of a game in which the user U assumes a bow and arrow pose, holds the central portion of the controller 30 with his left hand, and operates a virtual bow V1 and arrow V2 by moving his right hand. .
- both hands of the controller 30 and the user U may overlap when viewed from the camera unit 10 . Therefore, it is very difficult to separate both hands of the controller 30 and the user U in the generated RGB image
- FIG. 5 is a side view of the positional relationship among the user U, the camera unit 10, and the display device 40, and shows an example in which the elevation angle is changed by an angle a1 without changing the position of the camera unit 10.
- the angle a1 can be obtained from posture information acquired by the IMU 13 .
- FIGS. 6A and 6B are top views of the positional relationship among the user U, the camera unit 10, and the display device 40.
- FIG. FIG. 6A shows an example in which the position of the camera unit is changed by a distance d1 in the horizontal direction when the user U sees the display device 40 without changing the elevation angle of the camera unit 10.
- the distance d1 can be obtained from the positional information acquired by the positional information acquisition unit 14 .
- the angle a2 which is the rotation angle about the user U, can be obtained from the posture information acquired by the IMU 13.
- the position of the camera unit is changed by a distance d2 in the horizontal direction when the user U sees the display device 40, and furthermore, in the front-rear direction, the position of the camera unit is changed.
- An example in which the position is changed by a distance d2 is shown.
- the distances d2 and d3 can be obtained from the position information acquired by the position information acquisition unit 14.
- FIG. Also, the angle a3, which is the rotation angle about the user U, can be obtained from the posture information acquired by the IMU 13 .
- the optimum installation state of the camera unit 10 with respect to the user U and the display device 40 differs depending on the purpose. For example, depending on the content of the game to be played, the scene, etc., the user's whole body, upper body, hands, etc., to be grasped differ, and the positions that can be grasped also differ. Therefore, as with the reference installation state described above, for the optimal installation state, for example, the information processing device 20 may display a tutorial or the like on the display device 40 to prompt the user U to correctly position the camera unit 10. . At this time, by performing test photography with the camera unit 10 and evaluating the generated RGB image signal 113 and event signal 123, it is possible to determine whether or not the optimal installation state is correctly realized.
- the information processing device 20 is implemented by a computer having a communication interface, a processor, and a memory, for example, and includes a communication section 21 and a control section 22 functioning as a second arithmetic processing device and an image generation device.
- the control unit 22 includes an estimation unit 221, a conversion processing unit 222, a control value calculation unit 223, an image generation unit 224, which are realized by the processor operating according to a program stored in a memory or received via a communication interface. , and the functions of the warning unit 225 . The function of each unit will be further described below.
- the communication section 21 receives each information output from the information output section 16 of the camera unit 10 . Further, the communication unit 21 can communicate with the controller 30 and outputs a display image to be displayed on the display device 40 .
- the estimation unit 221 of the control unit 22 estimates the relative positional relationship between the display device 40 and the camera unit 10 based on the posture information and position information acquired via the communication unit 21 . As described above, the positional relationship among the user U, the camera unit 10, and the display device 40 can be obtained from the posture information obtained by the IMU 13 and the positional information obtained by the positional information obtaining section 14.
- the transformation processing unit 222 Based on the positional relationship estimated by the estimation unit 221, the transformation processing unit 222 performs coordinate transformation processing on the coordinate information regarding the user. More specifically, the transformation processing unit 222 performs coordinate transformation processing for transforming the coordinate information about the user acquired via the communication unit 21 into the above-described coordinate system in the reference installation state. This conversion process can be realized by a known coordinate conversion process. Coordinate conversion processing converts coordinate information calculated based on the RGB image signal 113 and the event signal 123 generated when the camera unit 10 is displaced from the reference installation state into coordinate information generated in the reference installation state. be able to.
- the control value calculation unit 223 calculates a control value for feedback control to an external device including the controller 30 and the display device 40 based on at least one of coordinate information acquired via the communication unit 21 and information received from the controller 30. calculate. As described above, the camera unit 10 and controller 30 function as an operation device for accepting user operations. Therefore, the control value calculation unit 223 calculates a control value for feedback control to an external device including the controller 30 and the display device 40 according to a user operation performed via at least one of the camera unit 10 and the controller 30. do. The calculated control value is output to the controller 30 via the communication section 21 .
- the image generation unit 224 generates a display image indicating the state of the user based on the coordinate information after the coordinate conversion processing by the conversion processing unit 222. More specifically, the image generator 224 generates a display image to be displayed on the display device 40 according to the control value calculated by the control value calculator 223 . The generated display image is output to the display device 40 via the communication section 21 .
- the display image includes a so-called First Person View image that reproduces the user's point of view, a so-called Third Person View image that looks at the user from the outside, and the like.
- the type of display image to be used is determined according to the content of the game to be played and the purpose such as the scene. Details of calculation of the control value and generation of the display image will be described in connection with the description of the configurations of the controller 30 and the display device 40, which will be described later.
- the warning unit 225 performs a warning operation when the change in posture information or position information exceeds a predetermined threshold.
- the warning operation may be, for example, displaying a warning message on the display device 40, outputting a warning message by voice, or the like.
- a change in orientation information or position information exceeding a predetermined threshold means that the installation state of the camera unit 10 is not suitable for the calculation of coordinate information, or that the installation state of the camera unit 10 has changed and the calculation of coordinate information has failed.
- the installation state of the camera unit 10 is suitable for calculating coordinate information, or even if the installation state of the camera unit 10 has changed, the above-mentioned This is the case where it is not necessary to change the installation state of the camera unit 10 because it is within a range in which the change can be complemented by the processing performed by each unit.
- the threshold may be changed according to the purpose. For example, the threshold may be changed based on the content of the game to be played, the allowable range according to the scene, or the like.
- the controller 30 includes a communication section 31, an operation section 32, a force sense presentation section 33, a vibration section 34, and an audio output section 35, as shown in FIG.
- a user can perform various operations related to the game by operating the controller 30 .
- the communication unit 31 receives the control values output from the communication unit 21 of the information processing device 20 and outputs the control values to the force sense presentation unit 33 , the vibration unit 34 , and the audio output unit 35 .
- the communication unit 31 also outputs information regarding user operations received by the operation unit 32 to the information processing device 20 .
- the operation unit 32 includes a plurality of operators such as buttons and pads, and accepts user's operation input to the operators.
- the haptic presentation unit 33 is provided in at least a part of the operation elements of the operation unit 32 , and presents the user with a force that resists or interlocks with the user's operation according to the control value supplied from the information processing device 20 .
- the force sense presentation unit 33 can be configured by a motor, an actuator, or the like including a rotating rotor.
- a well-known device can be used as the haptic device that constitutes the haptic device 33, and detailed description thereof will be omitted here.
- the vibrating section 34 generates vibration according to a control value supplied from the information processing device 20, and can be configured by a motor, for example.
- the vibrating unit 34 can notify the user that the user operation has been correctly performed and has been recognized by the information processing device 20 by generating vibration when the user operation is performed.
- the audio output unit 35 outputs audio according to the control value supplied from the information processing device 20, and can be configured by, for example, a speaker.
- the audio output unit 35 can notify the user that the user operation has been correctly performed and has been recognized by the information processing apparatus 20 by outputting audio when the user operation is performed.
- At least one of the vibration by the vibration unit 34 and the sound output by the sound output unit 35 is performed in conjunction with the presentation of the force sense by the force sense presentation unit 33 described above, thereby providing various feedback controls to the user. It is possible to improve
- the control value calculation unit 223 of the information processing device 20 calculates the control value of the feedback control to the controller 30 as described above. And a control value for feedback control to the audio output unit 35 is calculated. At this time, the control value calculation unit 223 calculates a control value for feedback control to the controller 30 based on the coordinate information acquired via the communication unit 21 before conversion processing is performed by the conversion processing unit 222 .
- the coordinate information acquired via the communication unit 21 is coordinate information in an optimal installation state in which each element of the body of the controller 30 and the user U does not overlap. Therefore, by calculating a control value based on such coordinate information, it is possible to calculate a control value that can accurately grasp and reflect the state of the user. Note that coordinate information after coordinate conversion processing by the conversion processing unit 222 may be used as an auxiliary when the control value calculation unit 223 calculates the control value.
- the control value calculation unit 223 calculates a control value indicating what kind of force sense is to be presented as feedback control according to the user's operation.
- the control value calculation unit 223 calculates a control value indicating whether to present what kind of vibration is to be generated as feedback control according to the user's operation.
- the control value calculation unit 223 calculates a control value indicating what kind of audio is to be output as feedback control according to the user's operation. The calculation of the control value by the control value calculator 223 can be performed according to a predetermined formula, table, or the like.
- the controller 30 may be configured by a pair of controllers that can be held with both hands, may be configured by a controller that allows character input such as a keyboard, or may be configured by an application such as a smartphone.
- the controller 30 may be provided with a voice input unit and a voice recognition technology may be applied.
- the controller 30 may include a voice input unit such as a microphone and a voice recognition unit, and may supply commands uttered by the user and information indicating user calls to the information processing apparatus 20 via the communication unit 31. good.
- the display device 40 includes a receiver 41 and a display 42, as shown in FIG.
- the receiving unit 41 receives information indicating the display image generated by the image generating unit 224 of the information processing device 20 via the communication unit 21 .
- the display unit 42 has a monitor such as an LCD (Liquid Crystal Display) or an organic EL, and can present the information to the user by displaying a display image based on the information received by the reception unit 41 .
- the display device 40 described above may be configured by the dedicated display device shown in FIG. 1, or may be configured by a display device such as an HMD mounted on the user's head.
- the display unit of the HMD includes a display element such as LCD (Liquid Crystal Display) or organic EL, and an optical device such as a lens. It may be a transmissive display element.
- wearable devices such as AR (Augmented Reality) glasses and MR (Mixed Reality) glasses may be used as the HMD.
- the display device 40 described above may be configured by a display device of a computer, or may be configured by a display device of a terminal device such as a smart phone.
- a touch panel for detecting contact may be provided on the surface of the display unit 42 .
- the control value calculator 223 of the information processing device 20 calculates the control value for feedback control to the display image displayed on the display device 40 as described above. At this time, the control value calculation unit 223 calculates a control value for feedback control to the display image based on the coordinate information after conversion processing by the conversion processing unit 222 .
- the coordinate information after conversion processing is the coordinate information converted into the coordinate system in the reference installation state. Therefore, by calculating the control value based on such coordinate information, regardless of the installation state of the camera unit 10, it is possible to calculate the control value for feedback control for generating a display image from a fixed viewpoint. can.
- coordinate information acquired via the communication unit 21 before conversion processing is performed by the conversion processing unit 222 may be used as an auxiliary when calculating the control value by the control value calculation unit 223 .
- both the control value based on the coordinate information acquired via the communication unit 21 before the conversion processing is performed by the conversion processing unit 222 and the control value based on the coordinate information after the conversion processing may be calculated.
- the control value calculation unit 223 calculates a control value indicating how the display image is changed as feedback control according to the user's operation.
- the calculation of the control value by the control value calculator 223 can be performed according to a predetermined formula, table, or the like.
- the image generator 224 of the information processing device 20 generates a display image to be displayed on the display device 40 according to the control value calculated by the control value calculator 223, as described above. More specifically, the image generator 224 generates a new display image to be displayed on the display device 40 according to the control value for changing the display image.
- the control value for feedback to the display image is calculated based on the coordinate information converted into the coordinate system in the reference installation state. Therefore, even if the installation state of the camera unit 10 changes due to vibration or external force, the change can be compensated for and the display image can be continuously generated from a fixed viewpoint. With such a configuration, it is possible to suppress the occurrence of shaking and sudden changes in the display image, and stabilize the content of drawing on the display device 40 .
- control value calculation unit 223 calculates both the control value based on the coordinate information acquired via the communication unit 21 and the control value based on the coordinate information after conversion processing, detailed drawing is required. By using control values based on the coordinate information acquired via the communication unit 21, it is also possible to realize more delicate drawing for the part.
- FIG. 7 is a flow chart showing an example of processing of the camera unit 10 according to one embodiment of the present invention.
- the image sensor 111 of the RGB camera 11 generates the RGB image signal 113 (step S101) and the sensor 121 of the EDS 12 generates the event signal 123 (step S102).
- the calculation unit 15 calculates coordinate information regarding the user (step S103), and the information output unit 16 outputs coordinate information, posture information, and position information (step S104).
- the camera unit 10 repeats the processing from step S103 to step S104 (the processing from step S101 to step S102 is also repeated, but the cycle may not necessarily be the same as the processing from step S103 onward), thereby obtaining various information indicating the user's state. can be supplied to the information processing device 20 .
- FIG. 8 is a flow chart showing an example of processing of the information processing device 20 according to one embodiment of the present invention.
- the communication section 21 acquires various information supplied from the camera unit 10 (step S201).
- the warning unit 208 compares the amount of change in posture information with a predetermined threshold (step S202), and if the amount of change is equal to or greater than the predetermined threshold (NO in step S202), performs a warning operation (step S203), The user is prompted to change the setting state of the camera unit 10 .
- the process proceeds to step S204.
- the amount of change in position information may be compared with a predetermined threshold. By performing such a determination, it becomes possible to monitor the setting state of the camera unit 10 more strictly and perform a warning operation at an appropriate timing.
- the estimation unit 221 of the control unit 22 determines the relative relationship between the camera unit 10 and the display device 40.
- the positional relationship is estimated (step S204), and the conversion processing unit 222 performs coordinate conversion processing on the coordinate information (step S205).
- the control value calculation unit 223 calculates a control value for feedback control (step S206), the image generation unit 224 generates a display image based on the control value (step S207), and the display image is generated via the communication unit 21. is output (step S208).
- the camera unit 10 calculates coordinate information about the user based on the RGB image signal 113 and the event signal 123 respectively generated by the image sensor 111 and the sensor 121, and calculates the image. It is output to the information processing device 20 together with the posture information of the sensors 111 and 121 . Then, the information processing device 20 performs coordinate conversion processing on the coordinate information based on the posture information, and generates a display image indicating the state of the user based on the processed coordinate information. Therefore, regardless of changes in the installation state of the camera unit 10, it is possible to generate a display image that more accurately corresponds to the state of the user while suppressing latency. Also, the degree of freedom of the installation position of the camera unit 10 can be improved.
- the camera unit 10 performs from the generation of the RGB image signal 113 and the event signal 123 to the calculation of the coordinate information about the user, and does not output the RGB image signal 113 and the event signal 123, and the coordinate information, the image sensor 111 and the sensor By outputting H.121 attitude information, the problem of communication load and communication delay can be reduced. Furthermore, since it is not necessary to output the RGB image signal 113 and the event signal 123, it is also useful in terms of privacy protection. In addition, since the camera unit 10 according to one embodiment of the present invention can accept user operations by calculating coordinate information about the user, it is possible to maintain the cursor position like a conventional pointing device type operating device. It does not cause user physical fatigue. In addition, the camera unit 10 does not require the user to wear a marker or an attachment to be recognized, unlike a conventional posture detection type operating device.
- the camera unit 10 includes both the EDS 12 and the RGB camera 11, recognizes the user based on the event signal 123 with a relatively small time lag, and recognizes the RGB image with a relatively large amount of information. Coordinate information about the recognized user is calculated from the signal 113 . Therefore, it is possible to realize suitable processing that takes advantage of the respective characteristics of the RGB image signal 113 and the event signal 123 .
- the position information acquisition unit 14 of the camera unit 10 acquires the position information of the image sensor 111 and the sensor 121, and each unit of the control unit 22 of the information processing device 20 acquires the position information and the orientation information. Based on this, the relative positional relationship between the display device 40 and the camera unit 10 is estimated, and coordinate conversion processing is performed based on the estimated positional relationship. Therefore, the installation state of the camera unit 10 can be grasped and reflected in the generation of the display image.
- the coordinate information calculated by the calculation unit 15 of the camera unit 10 includes coordinate information regarding the user's posture, coordinate information regarding the shape of the user's arm, coordinate information regarding the shape of the user's fingers, Or at least one of coordinate information regarding the posture of the external device worn or held by the user. Therefore, it is possible to grasp characteristic user states and accurately grasp the intention and content of user operations.
- the calculator 15 of the camera unit 10 is constructed by learning the relationship between the image of a person having multiple joints and the coordinate information indicating the positions of the multiple joints. Based on the learned model, coordinate information of at least one joint of the person included in the first image (RGB image) based on the RGB image signal 113 is calculated. Therefore, it is possible to accurately and quickly calculate the coordinate information regarding the user.
- the warning unit 225 of the information processing device 20 performs a warning operation when the amount of change in posture information acquired by the IMU 13 of the camera unit 10 exceeds a predetermined threshold. Therefore, it is necessary to change the installation state of the camera unit 10, such as when the installation state of the camera unit 10 is not suitable for calculating the coordinate information, or when the installation state of the camera unit 10 has changed and it is difficult to calculate the coordinate information. If there is, the user can be prompted to change the setting state of the camera unit 10 .
- FIG. 9 is a block diagram showing a schematic configuration of a system according to another embodiment of the invention.
- 9 is a block diagram showing the configuration of a system 2 having a server 50 and a terminal device 60 instead of the information processing device 20 of FIG.
- Constituent elements having functional configurations are given the same reference numerals.
- the server 50 is a server (for example, a cloud server) communicably connected to the camera unit 10 and the terminal device 60 via the Internet communication network or wirelessly.
- the server 50 has the same configuration as the information processing apparatus 20 described with reference to FIG. 2, and performs various processes based on the information output by the camera unit 10.
- the terminal device 60 also includes a communication unit 61 , and the communication unit 61 receives information output from the server 50 .
- the communication unit 61 can communicate with the controller 30 and outputs an image to be displayed on the display device 40, like the communication unit 21 of the information processing apparatus 20 described with reference to FIG.
- the camera unit 10 performs from the generation of the RGB image signal 113 and the event signal 123 to the calculation of the coordinate information, and outputs the coordinate information and the orientation information of the camera unit 10 to the server 50.
- a server such as a cloud server.
- the camera unit 10 described in each of the above examples includes both the IMU 13 and the position information acquisition unit 14, it may be configured to include only one of them.
- the estimation section 221 may estimate the relative positional relationship between the display device 40 and the camera unit 10 based on either the orientation information or the position information.
- the conversion processing unit 222 may perform coordinate conversion processing of the coordinate information based on either the orientation information or the position information.
- a configuration may be adopted in which part of the processing performed by the information processing device 20 in each of the above examples is performed by the camera unit 10 .
- the functions of the estimation section 221 in the information processing device 20 may be provided in the camera unit 10 , or the functions of the estimation section 221 and the conversion processing section 222 in the information processing device 20 may be provided in the camera unit 10 .
- the camera unit 10 outputs coordinate information after coordinate conversion processing.
- the number of RGB cameras 11 and EDS 12 may be the same or may be different. Also, the number of RGB cameras 11 and EDS 12 may be one or more.
- the range of the object field for generating the RGB image signals 113 it is possible to expand the range of the object field for generating the RGB image signals 113, or to estimate the state of a person in three dimensions from the plurality of RGB image signals 113. can.
- the range of the object field for generating the event signal 123 is expanded, and the three-dimensional movement amount of the person is calculated based on the plurality of event signals 123. be able to.
- the camera unit 10 described in each of the above examples may be implemented within a single device, or may be implemented distributed among a plurality of devices.
- at least a portion of each sensor may be provided independently, and another configuration may be implemented as the camera unit 10 main body.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- Social Psychology (AREA)
- Psychiatry (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- User Interface Of Digital Computer (AREA)
- Image Input (AREA)
- Image Analysis (AREA)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US18/554,028 US12525062B2 (en) | 2021-04-13 | 2022-03-24 | System, information processing method, and information processing program |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2021-067661 | 2021-04-13 | ||
| JP2021067661A JP7434207B2 (ja) | 2021-04-13 | 2021-04-13 | システム、情報処理方法および情報処理プログラム |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2022220049A1 true WO2022220049A1 (ja) | 2022-10-20 |
Family
ID=83640590
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/JP2022/013836 Ceased WO2022220049A1 (ja) | 2021-04-13 | 2022-03-24 | システム、情報処理方法および情報処理プログラム |
Country Status (3)
| Country | Link |
|---|---|
| US (1) | US12525062B2 (cg-RX-API-DMAC7.html) |
| JP (1) | JP7434207B2 (cg-RX-API-DMAC7.html) |
| WO (1) | WO2022220049A1 (cg-RX-API-DMAC7.html) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2010045658A (ja) * | 2008-08-14 | 2010-02-25 | Sony Corp | 情報処理装置、情報処理方法および情報処理プログラム |
| JP2017191576A (ja) * | 2016-04-15 | 2017-10-19 | キヤノン株式会社 | 情報処理装置、情報処理装置の制御方法およびプログラム |
| JP2020086836A (ja) * | 2018-11-22 | 2020-06-04 | 株式会社デンソーアイティーラボラトリ | 情報処理装置、情報処理方法およびプログラム |
| JP2020181123A (ja) * | 2019-04-26 | 2020-11-05 | 塁 佐藤 | カラオケ演出システム |
| JP2021048456A (ja) * | 2019-09-17 | 2021-03-25 | 株式会社東芝 | 画像処理装置 |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5300777B2 (ja) * | 2010-03-31 | 2013-09-25 | 株式会社バンダイナムコゲームス | プログラム及び画像生成システム |
| KR101880998B1 (ko) | 2011-10-14 | 2018-07-24 | 삼성전자주식회사 | 이벤트 기반 비전 센서를 이용한 동작 인식 장치 및 방법 |
| US20180146149A1 (en) | 2016-11-21 | 2018-05-24 | Samsung Electronics Co., Ltd. | Event-based sensor, user device including the same, and operation method of the same |
| US10812731B2 (en) * | 2018-08-22 | 2020-10-20 | Qualcomm Incorporated | Adjustable receiver exposure times for active depth sensing systems |
| JP7394046B2 (ja) * | 2020-11-25 | 2023-12-07 | 株式会社ソニー・インタラクティブエンタテインメント | システム、撮像装置、情報処理装置、情報処理方法および情報処理プログラム |
-
2021
- 2021-04-13 JP JP2021067661A patent/JP7434207B2/ja active Active
-
2022
- 2022-03-24 WO PCT/JP2022/013836 patent/WO2022220049A1/ja not_active Ceased
- 2022-03-24 US US18/554,028 patent/US12525062B2/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2010045658A (ja) * | 2008-08-14 | 2010-02-25 | Sony Corp | 情報処理装置、情報処理方法および情報処理プログラム |
| JP2017191576A (ja) * | 2016-04-15 | 2017-10-19 | キヤノン株式会社 | 情報処理装置、情報処理装置の制御方法およびプログラム |
| JP2020086836A (ja) * | 2018-11-22 | 2020-06-04 | 株式会社デンソーアイティーラボラトリ | 情報処理装置、情報処理方法およびプログラム |
| JP2020181123A (ja) * | 2019-04-26 | 2020-11-05 | 塁 佐藤 | カラオケ演出システム |
| JP2021048456A (ja) * | 2019-09-17 | 2021-03-25 | 株式会社東芝 | 画像処理装置 |
Also Published As
| Publication number | Publication date |
|---|---|
| US20240127629A1 (en) | 2024-04-18 |
| JP7434207B2 (ja) | 2024-02-20 |
| JP2022162703A (ja) | 2022-10-25 |
| US12525062B2 (en) | 2026-01-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11443540B2 (en) | Information processing apparatus and information processing method | |
| JP6465672B2 (ja) | 情報処理装置および情報処理方法 | |
| JP6494305B2 (ja) | 情報処理装置、表示装置、および情報処理方法 | |
| WO2020110659A1 (ja) | 情報処理装置、情報処理方法、およびプログラム | |
| US12231781B2 (en) | Remote control system, and remote operation apparatus, video image processing apparatus, and computer-readable medium | |
| JP7247371B2 (ja) | 画像処理装置、システム、画像処理方法、および画像処理プログラム | |
| KR20220125362A (ko) | 각도 감지 검출기들을 포함하는 머리-착용 디스플레이 시스템들에 대한 위치 추적 시스템 | |
| JP2020154569A (ja) | 表示装置、表示制御方法、及び表示システム | |
| KR20190067523A (ko) | 글래스 타입 단말기 및 그것의 동작방법 | |
| US11275547B2 (en) | Display system, display method, and program | |
| US12315172B2 (en) | Information processing apparatus, system, information processing method, and information processing program | |
| JP2018085595A (ja) | 頭部装着型表示装置、及び、頭部装着型表示装置の制御方法 | |
| WO2021130986A1 (ja) | 映像表示装置及び映像表示方法 | |
| US11954269B2 (en) | Information processing apparatus, information processing method, and program for generating location data | |
| WO2017098999A1 (ja) | 情報処理装置、情報処理システム、情報処理装置の制御方法、及び、コンピュータープログラム | |
| JP7394046B2 (ja) | システム、撮像装置、情報処理装置、情報処理方法および情報処理プログラム | |
| JP7434207B2 (ja) | システム、情報処理方法および情報処理プログラム | |
| US10741028B2 (en) | Output control device, output controlling method and program | |
| KR20180068128A (ko) | 이동단말기 및 그 제어방법 | |
| CN111273885A (zh) | 一种ar图像显示方法及ar设备 | |
| JP7513564B2 (ja) | システム、情報処理方法および情報処理プログラム | |
| JPWO2018123022A1 (ja) | コンピュータプログラム、表示装置、頭部装着型表示装置、マーカ | |
| CN119998761A (zh) | 设置有360度图像视点位置识别单元的信息处理装置 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22787974 Country of ref document: EP Kind code of ref document: A1 |
|
| WWE | Wipo information: entry into national phase |
Ref document number: 18554028 Country of ref document: US |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 22787974 Country of ref document: EP Kind code of ref document: A1 |
|
| WWG | Wipo information: grant in national office |
Ref document number: 18554028 Country of ref document: US |