WO2023090163A1 - 情報処理装置、情報処理方法、およびプログラム - Google Patents

情報処理装置、情報処理方法、およびプログラム Download PDF

Info

Publication number
WO2023090163A1
WO2023090163A1 PCT/JP2022/041124 JP2022041124W WO2023090163A1 WO 2023090163 A1 WO2023090163 A1 WO 2023090163A1 JP 2022041124 W JP2022041124 W JP 2022041124W WO 2023090163 A1 WO2023090163 A1 WO 2023090163A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
space
presentation
information processing
interaction
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/041124
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
健司 徳武
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Priority to US18/709,035 priority Critical patent/US20250342663A1/en
Priority to JP2023561523A priority patent/JPWO2023090163A1/ja
Priority to EP22895445.9A priority patent/EP4425438A4/en
Publication of WO2023090163A1 publication Critical patent/WO2023090163A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/147—Digital output to display device ; Cooperation and interconnection of the display device with other functional units using display panels
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/131—Protocols for games, networked simulations or virtual reality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/52—Network services specially adapted for the location of the user terminal

Definitions

  • the present disclosure relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program that enable seamless service provision between different spaces.
  • Patent Document 1 discloses a technique for reflecting real-world objects in virtual space.
  • VR virtual reality
  • HMD Head Mounted Display
  • AR Augmented Reality
  • VR and AR are based on similar technologies, and in the future, it is expected that there will be technology that allows people to move back and forth between different spaces in social life, such as between VR and AR.
  • the present disclosure has been made in view of this situation, and is intended to enable seamless service provision between different spaces.
  • An information processing device includes a matching processing unit that matches coordinate systems set in each of a first space in which a first user exists and a second space in which a second user exists, and a presentation control unit that controls presentation of an interaction to the other user according to the action of one user based on position information and action information of each of the first user and the second user in the coordinate system. It is an information processing device provided.
  • the information processing device matches coordinate systems set in each of a first space in which a first user exists and a second space in which a second user exists, and performs matching.
  • the information processing device matches the coordinate systems set in each of the first space in which the first user exists and the second space in which the second user exists, and for executing a process of controlling presentation of an interaction to the other user according to the action of one user based on position information and action information of each of the first user and the second user in the coordinate system; It's a program.
  • the coordinate systems set in each of the first space in which the first user exists and the second space in which the second user exists are matched, and the first space in the matched coordinate system is matched. Based on the position information and motion information of each of the one user and the second user, presentation of an interaction to the other user according to the motion of one user is controlled.
  • FIG. 1 is a diagram showing an overview of a VR/AR synchronization service according to techniques of the present disclosure
  • FIG. 1 is a diagram illustrating a configuration example of a synchronization system according to an embodiment of the present disclosure
  • FIG. 3 is a block diagram showing a functional configuration example of a space recognition unit that executes SLAM
  • FIG. 4 is a flowchart for explaining the flow of operations of the VR device
  • 10 is a flowchart describing the flow of operations of the AR device
  • 4 is a flowchart for explaining the flow of operations of a synchronization service providing device
  • FIG. 10 is a diagram for explaining matching of coordinate systems of the VR space and the real space
  • FIG. 10 is a diagram showing an example of presentation of each of the VR space and the real space; It is a figure explaining the example of presentation of interaction. It is a figure explaining the example of presentation of interaction.
  • FIG. 10 is a diagram showing an example of presentation of each of the VR space and the real space;
  • FIG. 10 is a flowchart for explaining the flow of manipulation of a virtual object;
  • FIG. It is a figure explaining the example of presentation of interaction.
  • FIG. 4 is a diagram for explaining sharing of objects;
  • FIG. 10 is a diagram showing an example of presentation of each of the VR space and the real space; It is a figure which shows the other structural example of a synchronous system.
  • FIG. 10 is a diagram showing an example of presentation of each of the VR space and the real space;
  • FIG. 10 is a diagram showing an example of presentation of a VR space
  • FIG. 10 is a diagram showing an example of presentation of a VR space
  • FIG. 11 is a flowchart for explaining the flow of movement between metaverses
  • FIG. It is a block diagram which shows the structural example of the hardware of a computer.
  • FIG. 1 is a diagram showing an overview of the VR/AR synchronization service according to the technology according to the present disclosure.
  • AR space In the VR/AR synchronous service shown in FIG. 1, users who receive independent services in the VR space and the real space (hereinafter also referred to as AR space or AR display) can move back and forth between VR and AR. make it Specifically, between the users of the VR space and the real space, by presenting interactions to the other user according to the actions of one user, it is possible to provide seamless services between VR and AR. do.
  • users can log in to the VR space remotely and receive updates from the VR space in the real space. Activities can be shared.
  • Synchronization System Configuration and Operation The configuration and operation of a synchronization system that implements the VR/AR synchronization service shown in FIG. 1 will be described below.
  • FIG. 2 is a diagram showing a configuration example of a synchronization system according to an embodiment of the present disclosure.
  • the synchronization system 1 of FIG. 2 is configured to include a VR device 100, an AR device 200, a VR/AR synchronization service providing device 300 (hereinafter simply referred to as a synchronization service providing device 300), and a 3D map data server 350. be done.
  • the VR device 100 and AR device 200 are used by arbitrary users.
  • a user of the VR device 100 or AR device 200 can log into the VR space or AR space by obtaining an account for the VR service or AR service provided by the synchronization service providing device 300 .
  • the VR service and AR service here are integrally provided to the user as the same service. While enjoying one service, the user can communicate with other users who are enjoying the other service.
  • the VR device 100 is a device for providing a VR service to a user, and configured as an HMD, VR goggles, or the like worn by the user. Also, the VR device 100 may be configured as a wide-field display such as an omnidirectional, semi-spherical, or dome type connected to a computer.
  • the VR device 100 includes a GPS (Global Positioning System) sensor 111, a cell ID acquisition unit 112, a WiFi communication unit 113, a geomagnetic sensor 114, and a position detection unit 115.
  • GPS Global Positioning System
  • the GPS sensor 111 measures the position based on the radio waves from the GPS satellites, and supplies the position detection unit 115 with position information representing the position.
  • the cell ID acquisition unit 112 measures the position by acquiring the cell ID from the mobile phone base station, and supplies position information representing the position to the position detection unit 115 .
  • the WiFi communication unit 113 measures the position by detecting radio waves from surrounding access points through wireless communication such as Wi-Fi (registered trademark), and supplies position information representing the position to the position detection unit 115.
  • Wi-Fi registered trademark
  • the geomagnetic sensor 114 measures a direction by detecting geomagnetism, and supplies direction information representing the direction to the position detection unit 115 .
  • the position detection unit 115 detects the position and direction of the VR device 100 in real space based on the information from the GPS sensor 111, the cell ID acquisition unit 112, the WiFi communication unit 113, and the geomagnetic sensor 114.
  • the VR device 100 further comprises an IMU (Inertial Measurement Unit) 121, a camera 122, a ToF (Time of Flight) sensor 123, and a space recognition section 124.
  • IMU Inertial Measurement Unit
  • camera 122 a camera 122
  • ToF Time of Flight
  • space recognition section 124 a space recognition section 124.
  • the IMU 121 supplies motion information representing the motion of the VR device 100 to the space recognition unit 124 by detecting the angles and accelerations of the three axes.
  • the camera 122 acquires an RGB image by imaging the environment around the VR device 100 and supplies it to the space recognition unit 124 .
  • the ToF sensor 123 senses the surrounding environment of the VR device 100 to three-dimensionally recognize the surrounding environment and supplies the recognition result to the space recognition unit 124 .
  • the space recognition unit 124 performs space recognition processing in the VR space based on information from each sensor of the IMU 121, camera 122, and ToF sensor 123, and the position and direction detected by the position detection unit 115.
  • the space recognition unit 124 performs space recognition processing in 3DoF (Degree of Freedom) mode or space recognition processing in 6DoF mode in the VR space presented by the VR device 100 .
  • 3DoF Degree of Freedom
  • 6DoF space recognition processing in 6DoF mode
  • the 3DoF mode the user's position and motion in the VR space are recognized based on three movements around three axes, such as rotation and tilt of the user's head and neck.
  • the 6DoF mode in addition to the three movements in the 3DoF mode, the user's position and motion in the VR space are recognized based on the user's forward/backward, left/right, and up/down movements.
  • the user's position and motion in the VR space recognized by the space recognition unit 124 are supplied to the synchronization service application 130 .
  • the synchronization service application 130 is application software for providing VR services to the user wearing the VR device 100, and controls the presentation of the VR space to the user under the control of the synchronization service providing apparatus 300.
  • the synchronization service application 130 supplies the synchronization service providing device 300 with position information representing the position in the VR space of the user (own user) wearing the VR device 100 and motion information representing the motion.
  • the synchronization service application 130 acquires position information and motion information of the user (opposite user) wearing the AR device 200 from the synchronization service providing apparatus 300 .
  • the motion information may include, as the motion of the user, information representing a posture (pose), body direction (rotation), facial expressions, utterance content, and emotions such as emotions.
  • the synchronization service application 130 controls the display of the other user's avatar in the VR space presented to the user wearing the VR device 100 based on the other user's position information and motion information.
  • the VR presenting unit 140 is configured, for example, as a non-transparent display, and under the control of the synchronization service application 130, presents a VR space to the user wearing the VR device 100, and displays the other user in the VR space. display an avatar.
  • the GPS sensor 111 the cell ID acquisition unit 112, the WiFi communication unit 113, the geomagnetic sensor 114, and the position detection unit 115, which are surrounded by dashed lines in the figure, are not provided, but the VR device 100 can operate. It is possible to
  • the AR device 200 is a device for providing an AR service to a user, and is configured as a mobile terminal such as a smartphone held or worn by the user, AR glasses, or the like.
  • the AR device 200 includes a GPS sensor 211, a cell ID acquisition unit 212, a WiFi communication unit 213, a geomagnetic sensor 214, and a position detection unit 215.
  • the GPS sensor 211, the cell ID acquisition unit 212, the WiFi communication unit 213, the geomagnetic sensor 214, and the position detection unit 215 are provided in the VR device 100, respectively. , and position detection unit 115 .
  • the AR device 200 further comprises an IMU 221, a camera 222, a ToF sensor 223, and a space recognition section 224.
  • the IMU 221 , camera 222 and ToF sensor 223 may be configured similarly to the IMU 121 , camera 122 and ToF sensor 123 included in the VR device 100 .
  • the space recognition unit 224 performs space recognition processing in the real space based on information from each sensor of the IMU 221, camera 222, and ToF sensor 223, and the position and direction detected by the position detection unit 215.
  • the space recognition unit 224 uses SLAM (Simultaneous Localization and Mapping) and VPS (Visual Positioning Service) to perform space recognition processing in real space.
  • the space recognition unit 224 executes SLAM to estimate its own position and create a three-dimensional map. Then, the space recognition unit 224 recognizes the user's position and motion in the real space based on the three-dimensional map created by the VPS and the RGB image captured by the camera 222 .
  • SLAM Simultaneous Localization and Mapping
  • VPS Visual Positioning Service
  • FIG. 3 is a block diagram showing a functional configuration example of the space recognition unit 224 that executes SLAM.
  • the space recognition unit 224 that executes SLAM is composed of a front end unit 410, a back end unit 420, and a loop closure processing unit 430.
  • the front end unit 410 executes processing based on information from sensors.
  • the front end unit 410 has a feature extraction unit 411 and a data linking unit 412 .
  • a feature extraction unit 411 extracts feature points of surrounding objects based on IMU data and camera data.
  • the data linkage unit 412 associates the feature points extracted by the feature extraction unit 411 with each object. In this way, the positions of surrounding objects and the movement amount of the user are estimated.
  • the backend unit 420 executes processing that does not depend on information from sensors.
  • the backend unit 420 has a map estimation unit 421 and a map update unit 422 .
  • the map estimation unit 421 estimates a three-dimensional map based on the positions of surrounding objects and the movement amount of the user.
  • the map updating unit 422 updates the 3D map estimated by the map estimating unit 421 based on the positions of surrounding objects and the movement amount of the user.
  • the loop closure processing unit 430 reduces the cumulative error in self-position estimation by loop closure.
  • the spatial recognition unit 224 that executes SLAM is not limited to the configuration shown in FIG. 3, and can adopt other configurations.
  • the user's position and motion in the real space recognized by the space recognition unit 224 are supplied to the synchronization service application 230 .
  • the synchronization service application 230 is application software for providing AR services to the user wearing the AR device 200, and under the control of the synchronization service providing apparatus 300, presents AR information to the user and presents interactions based on the AR information. to control.
  • the synchronization service application 230 supplies the synchronization service providing device 300 with position information and motion information in real space of the user (self-user) holding or wearing the AR device 200 . Also, the synchronization service application 230 acquires position information and motion information of the user (opposite user) wearing the VR device 100 from the synchronization service providing apparatus 300 .
  • the synchronization service application 230 controls the display of the other user's avatar superimposed on the real space presented to the user via the AR device 200, based on the other user's position information and motion information.
  • the AR presentation unit 240 is configured, for example, as a transparent display, and under the control of the synchronization service application 230, displays AR information and the like on the real space imaged by the AR device 200 or on the display through which the real space can be seen. The other user's avatar is superimposed and displayed.
  • Synchronization service providing apparatus 300 is configured as a cloud server built on a so-called cloud.
  • the synchronization service providing device 300 includes a coordinate system matching processing unit 311, an interaction presentation control unit 312, and a user management unit 313.
  • the coordinate system matching processing unit 311 is set in each of the VR space in which the user of the VR device 100 (first user) exists and the real space (AR space) in which the user of the AR device 200 (second user) exists. match the coordinate system.
  • the interaction presentation control unit 312 based on the position information and the motion information of each of the first user and the second user in the coordinate system matched by the coordinate system matching processing unit 311, performs the motion of one user according to the motion of the other user. Control the presentation of interactions to the user. Further, the interaction presentation control unit 312 presents the avatar corresponding to one user to the other user based on the position information and motion information of each of the first user and the second user in the matched coordinate system. Control.
  • the user management unit 313 manages users who use the VR service or AR service provided by the synchronization service providing device 300. Specifically, the user management unit 313 assigns an account to each user who uses the VR service or the AR service, and based on the account, logs in, logs out, or logs into the VR space or the AR space. Manage information for all users.
  • the 3D map data server 350 can also be configured as a cloud server built on a so-called cloud.
  • the 3D map data server 350 stores 3D map data corresponding to the real space, and corresponds to position information from the VR device 100 based on instructions from the synchronization service providing device 300 (interaction presentation control unit 312). 3D map data is supplied to the VR device 100 .
  • the VR space based on the 3D map data from the 3D map data server 350 is presented under the control of the synchronization service application 130 (interaction presentation control unit 312).
  • step S11 the synchronization service application 130 activates the VR service according to the operation of the user wearing the VR device 100. This allows the user to log into the VR space.
  • step S12 the space recognition unit 124 performs space recognition processing in the VR space.
  • the synchronization service providing apparatus 300 performs matching between the coordinate systems set in the VR space presented by the VR device 100 and the real space.
  • step S13 the space recognition unit 124 determines whether the user wearing the VR device 100 can move sufficiently on the spot.
  • the IMU 121, camera 122, and ToF sensor 123 it is determined whether there are no obstacles around the user and the user can move forward, backward, leftward, rightward, and upward.
  • step S13 If it is determined in step S13 that the user can move sufficiently on the spot, the process proceeds to step S14, and the space recognition unit 124 starts space recognition processing in 6DoF mode.
  • step S15 the synchronization service application 130 sends the 3D map data corresponding to the real space to the 3D map data server 350 as 3D space data for presenting the VR space in accordance with the movement of the user in the 6DoF mode. read from.
  • step S13 if it is determined in step S13 that the user cannot move sufficiently on the spot, the process proceeds to step S16, and the space recognition unit 124 starts space recognition processing in 3DoF mode.
  • step S17 the synchronization service application 130 sends the 3D map data corresponding to the real space to the 3D map data server as 3D space data for presenting the VR space in accordance with the user's movement operation in the 3DoF mode. Read from 350.
  • step S15 or step S17 when the VR presentation unit 140 presents the VR space by reading the 3D space data, the process proceeds to step S18.
  • step S18 the synchronization service application 130 presents various interactions in the VR space under the control of the synchronization service providing device 300.
  • the user wearing the VR device 100 is presented with an interaction according to the action of the other user holding or wearing the AR device 200 .
  • step S19 the synchronization service application 130 determines whether or not to end the VR service according to the operation of the user wearing the VR device 100.
  • step S19 If it is determined in step S19 that the VR service is not to be terminated, the process returns to step S13, and presentation of the VR space and the interaction there is repeated to the user wearing the VR device 100.
  • step S19 if it is determined in step S19 to end the VR service, the presentation of the VR space and the interaction therein to the user wearing the VR device 100 ends, and the user logs out of the VR space.
  • step S21 the synchronization service application 230 activates the AR service according to the operation of the user holding the AR device 200. This allows the user to log in to the AR space.
  • step S22 the position detection unit 215 acquires position information of the AR device 200 in real space. Specifically, the position detection unit 215 detects the position and direction of the AR device 200 in real space based on the information from the GPS sensor 211, the cell ID acquisition unit 212, the WiFi communication unit 213, and the geomagnetic sensor 214. to detect
  • step S23 the space recognition unit 224 uses SLAM and VPS to perform space recognition processing in real space.
  • the synchronization service providing apparatus 300 performs matching between the coordinate systems set in the real space and the VR space presented by the VR device 100 .
  • step S24 the synchronization service application 230 presents various interactions in the real space under the control of the synchronization service providing device 300. For example, an interaction according to the action of the other user wearing the VR device 100 is presented to the user holding or wearing the AR device 200 .
  • step S25 the synchronization service application 230 determines whether or not to end the AR service according to the operation of the user holding or wearing the AR device 200.
  • step S25 If it is determined in step S25 that the AR service is not terminated, the process returns to step S24, and the real space (AR space) and the interaction there are presented to the user holding or wearing the AR device 200 repeatedly.
  • step S25 presentation of the real space (AR space) and the interaction there to the user holding the AR device 200 ends, and the user logs out of the AR space.
  • step S31 the coordinate system matching processing unit 311 acquires position information and motion information from the VR device 100.
  • step S ⁇ b>32 the coordinate system matching processing unit 311 acquires position information and motion information from the AR device 200 .
  • step S33 the coordinate system matching processing unit 311 matches the coordinate systems of the VR space and the real space based on the position information and motion information from the VR device 100 and the AR device 200, respectively.
  • step S34 the interaction presentation control unit 312 controls presentation of the interaction based on the position information and motion information of each user in the matched coordinate system.
  • step S35 the interaction presentation control unit 312 determines whether or not any user has finished using the service (logged out).
  • step S35 If it is determined in step S35 that none of the users has finished using the service (has not logged out), the process returns to step S31 to repeat the matching of the coordinate systems of the VR space and the real space and the presentation of the interaction. .
  • step S35 if it is determined in step S35 that any user has finished using the service (logged out), presentation of the interaction ends. Note that if additional users are logged into the service, the presentation of interactions between logged-in users continues.
  • FIG. 7 shows the flow of processing for the VR space where user A exists, and the lower part of FIG. 7 shows the flow of processing for the real space where user B exists.
  • position information (Xa, Ya, Za) and motion information (XA, YA, ZA) of user A at point a are acquired by performing VR space recognition processing based on the motion of user A. be.
  • the position information (Xb, Yb, Zb) and the motion information (XB, YB, ZB) are obtained.
  • the movement amount and movement speed of user A may be adjusted by a controller connected to the VR device 100 or other inputs.
  • user A's one-step movement may be the same scale as in real space, or may be adjusted in units of 1 km by the controller.
  • FIG. 8 is a diagram showing presentation examples of the VR space and the real space.
  • user A sees a VR image 100VS corresponding to the real world seen from the position where user B is in real space through the VR device 100 .
  • An avatar B corresponding to the user B is displayed in the VR image 100VS based on the user B's position information and motion information.
  • user B sees a real image 200RS of the real world taken from, for example, that position via the AR device 200.
  • An avatar A corresponding to the user A is superimposed on the real image 200RS based on the user A's position information and motion information.
  • the user's avatar existing in the VR space and the avatar existing in the real space can be distinguished, for example, even if the user's avatar existing in the real space is transparently displayed. good.
  • a digital human image may be displayed instead of the user's avatar so that the user existing in the VR space becomes familiar with the real space.
  • user A communicates with user B, who is enjoying the AR service, while enjoying the VR service
  • user B while enjoying the AR service
  • communicates with user A who is enjoying the VR service. It is possible to communicate with
  • users who receive independent services in the VR space and the real space (AR space) can move back and forth between VR and AR.
  • AR space real space
  • users who receive independent services in the VR space and the real space (AR space) can move back and forth between VR and AR.
  • by presenting an interaction between users in the VR space and the real space according to the actions of one user to the other user it is possible to realize seamless service provision between VR and AR. It becomes possible.
  • FIG. 9 is a diagram illustrating an example of real-time interaction presentation.
  • FIG. 9 shows the motions of user A in the VR space and user B in the real space (AR display) at each timing.
  • the VR service logged in by User A and the AR service logged in by User B are integrally provided as the same service, and User A and User B each have an account for the same service. .
  • user A goes (moves) to Shibuya in the VR space (hereinafter also referred to as the Metaverse).
  • user B goes (moves) to Shibuya in real space.
  • avatar B corresponding to user B exists near user A in the Metaverse
  • avatar A corresponding to user A exists near user B in the AR display.
  • user A finds avatar B corresponding to user B on the Metaverse.
  • user B receives a notification and interaction according to user A's action. Specifically, the AR device 200 held by the user B displays a notification that the user A tapped the shoulder, or the AR device 200 vibrates.
  • user B can notice the actions of user A in the VR space through real-time interaction according to user A's direct action on avatar B.
  • a real-time interaction for User A in the Metaverse may be presented in response to User B's direct action on Avatar A in the real space.
  • FIG. 10 is a diagram illustrating an example of presentation of interactions with a time delay.
  • user B finds avatar A that does not move on the AR display.
  • timing t36 user B receives an interaction according to user A's action.
  • a motionless avatar A and a letter A virtual object v11 simulating (envelope) is generated and displayed.
  • the message "Let's go to ⁇ first" left by User A is displayed so as to appear from virtual object v11. is displayed.
  • a hand gesture can be detected by bone estimation of user B's arm (hand) with respect to virtual object v11.
  • user A may set the deadline, time zone, and display content for displaying the message.
  • user B can confirm the message left by user A in the VR space by interacting with the time difference corresponding to the indirect action of user A on avatar B.
  • an interaction with user A in the metaverse may be presented with a time lag.
  • the interaction presented by the virtual object is not limited to the example in FIG.
  • a predetermined icon is displayed as a virtual object at a position corresponding to a store existing in real space, and the user makes a hand gesture with respect to the icon, a coupon that can be used at that store is presented. good.
  • the user's hand gesture may move the virtual object or place a new virtual object.
  • the process of FIG. 12 is executed by the interaction presentation control unit 312 and started when the user of the VR device 100 or AR device 200 on which the virtual object is displayed makes a hand gesture.
  • step S111 the interaction presentation control unit 312 detects the start of the user's hand gesture.
  • step S112 the interaction presentation control unit 312 determines whether or not the virtual object displayed on the device is being operated by a hand gesture.
  • step S112 If it is determined in step S112 that the virtual object is being operated, the process proceeds to step S113, and the interaction presentation control unit 312 changes the presentation mode of the virtual object according to the operation.
  • step S114 the interaction presentation control unit 312 determines whether or not the hand gesture has ended. If it is determined in step S114 that the hand gesture has not ended, the process returns to step S112 and the subsequent processes are repeated.
  • step S114 if it is determined in step S114 that the hand gesture has ended, the process ends.
  • step S112 is repeated until it is determined that the virtual object has been operated. You may do so.
  • the presentation mode of the avatar corresponding to the user in the other space may be changed according to the change in the user's situation in one space.
  • FIG. 13 is a diagram explaining an example of interaction presentation according to changes in the user's situation.
  • user A logs into the Metaverse (VR service).
  • user B logs into the AR display (AR service).
  • user B finds avatar A on the AR display.
  • Timing t54 User A purchases a costume at the virtual shop on the Metaverse and wears the costume.
  • the costume of avatar A found by user B on the AR display changes. Specifically, the costume of avatar A is switched to the costume purchased by user A at the virtual store on the Metaverse.
  • the presentation mode of avatar B displayed on the Metaverse may change according to changes in the situation of user B in the real space.
  • a change in the presentation form of the avatar according to the change in the user's situation is not limited to a change in the avatar's costume according to the user's costume.
  • the avatar may change to wear the item.
  • the presentation mode of the avatar may change according to changes in the user's physical condition, such as the user being infected with a specific virus or being vaccinated against a specific virus.
  • the recognized object may be presented in the other space, thereby sharing the object.
  • FIG. 14 is a diagram explaining sharing of objects.
  • user A logs into the Metaverse (VR service).
  • user B logs into the AR display (AR service).
  • user B designates an object displayed on the AR device 200.
  • the object specified here is an object in the real space, but it may be a virtual object on the AR display.
  • user B sets the specified object to be shared with user A.
  • image data of the designated object is transmitted from the AR device 200 to the synchronization service providing apparatus 300, and object recognition processing is performed on the object.
  • image data of a virtual object corresponding to the object is transmitted from the synchronization service providing device 300 to the VR device 100 and presented on the Metaverse where the user A is present.
  • user B designates a real object v21 such as a dog on a real image 200RS displayed on the AR device 200.
  • a real object v21 such as a dog
  • a real image 200RS displayed on the AR device 200.
  • the virtual object v21' corresponding to the real object v21 is displayed in the VR image 100VS displayed on the VR device 100.
  • an object specified by the user in the real space can be shared with the user in the VR space as a virtual object.
  • a virtual object specified by the user in the VR space may be shared with the user in the real space as a virtual object that is AR information.
  • the following describes the configuration and operation of a synchronization system that realizes a VR/AR synchronization service that can build a multi-metaverse.
  • FIG. 16 is a diagram showing a configuration example of a synchronization system that implements construction of a multi-metaverse.
  • the synchronization system 1 of FIG. 16 includes a coordinate system matching processing unit 311, an interaction presentation control unit 312, and a user management unit 313 that are configured to be connected to a plurality of metaverses (metaverses #1, #2, #3, . . . ). ) and in that a space management unit 510 is newly provided.
  • each of a plurality of Metaverses can be constructed for each community such as workplace, school, and hobby.
  • a coordinate system matching processing unit 311, an interaction presentation control unit 312, and a user management unit 313 control coordinate system matching and interaction presentation control for each corresponding metaverse (VR space) under the management of the space management unit 510. , to perform user management.
  • coordinate system matching processing unit 311, the interaction presentation control unit 312, and the user management unit 313 are provided in common to a plurality of Metaverses, and each process is executed across the plurality of Metaverses. can be
  • the space management unit 510 manages multiple coexisting Metaverses (VR spaces) by controlling the coordinate system matching processing unit 311, interaction presentation control unit 312, and user management unit 313 of each Metaverse. Specifically, the space management unit 510 provides a portal service that allows users to move between the Metaverses as a service that manages the multi-Metaverse.
  • the space management unit 510 accepts common operations (common operations) in each Metaverse, sets the foreground Metaverse for each user, and switches to the set Metaverse.
  • FIG. 17 is a diagram showing a presentation example of each of a plurality of Metaverses and real spaces.
  • user A can see any one of VR images 100VS-1 to 100VS-4 corresponding to the real world seen from the position where user B is in real space through the VR device 100 .
  • a notification regarding the Metaverse in which User A is present may be presented to User B present in the real space (AR space) according to the action of User A present in one of the Metaverses.
  • Metaverse #3 of Metaverses #1 to #4 is set to the foreground in the VR service that User A is logged into.
  • this state for example, when user A takes an action leading to interaction with user B, as shown on the right side of FIG. message n31 is displayed in a superimposed manner.
  • a notification regarding another Metaverse may be presented to User A existing in one Metaverse.
  • Metaverse #1 of Metaverses #1 to #4 is set to the foreground for User A.
  • the user A can see a VR image 100VS-1 in the metaverse #1 through the VR device 100.
  • FIG. In this state, for example, in the background Metaverse #3, when an interaction with User A occurs, a message n41 indicating that there is a notification in Metaverse #3 is superimposed on the VR image 100VS-1 that User A can see. make it visible.
  • Metaverse #3 is set to the foreground for User C
  • Metaverse #4 is set to the foreground for User D.
  • User C sees VR image 100VS-3 in Metaverse #3 through the VR device 100
  • User D sees VR image 100VS-4 in Metaverse #4 through the VR device 100.
  • FIG. 19 The processing of FIG. 19 is executed by the VR device 100 under the control of the synchronization service providing apparatus 300.
  • step S211 the synchronization service application 130 activates the VR service according to the operation of the user wearing the VR device 100. This allows the user to log into the VR space. It is assumed that a multi-metaverse is deployed in the VR space here.
  • step S212 the synchronization service application 130 determines whether or not an interaction for the user wearing the VR device 100 has been received. Step S212 is repeated until it is determined that an interaction has been received.
  • step S212 If it is determined in step S212 that an interaction has been received, the process proceeds to step S213, and the synchronization service application 130 determines whether the metaverse that received the interaction is not the foreground metaverse but another metaverse (background metaverse). determine whether the metaverse that received the interaction is not the foreground metaverse but another metaverse (background metaverse). determine whether the metaverse that received the interaction is not the foreground metaverse but another metaverse (background metaverse). determine whether
  • step S213 If it is determined in step S213 that the Metaverse that received the interaction is another Metaverse, the process proceeds to step S214, and the synchronization service application 130 determines whether to move to the Metaverse that received the interaction. The determination of whether to move to the Metaverse from which the interaction has been received is made based on the user's operation in response to the movement confirmation described with reference to FIG.
  • step S215 the synchronization service application 130 moves to the metaverse that received the interaction under the control of the space management unit 510 of the synchronization service providing apparatus 300.
  • step S ⁇ b>216 the synchronization service application 130 presents interactions in the moved Metaverse under the control of the synchronization service providing device 300 .
  • step S213 determines whether the metaverse that received the interaction is not another metaverse. If it is determined in step S213 that the metaverse that received the interaction is not another metaverse, steps S214 and S215 are skipped, and the interaction is presented in the foreground metaverse in step S216.
  • steps S215 and S216 are skipped and the interaction is not presented.
  • step S217 the synchronization service application 130 determines whether or not to end the VR service in accordance with the operation of the user wearing the VR device 100.
  • step S217 If it is determined in step S217 that the VR service is not to end, the process returns to step S212 and the subsequent processes are repeated.
  • step S217 if it is determined in step S217 to end the VR service, presentation of the Metaverse to the user wearing the VR device 100 ends, and the user logs out of the VR space.
  • coordinate system matching and interaction presentation control are performed by the synchronization service providing apparatus 300. It may be performed by the application 130 or the sync service application 230 of the AR device 200 .
  • FIG. 21 is a block diagram showing a hardware configuration example of a computer that executes the series of processes described above by a program.
  • the VR device 100, the AR device 200, and the synchronization service providing device 300 as information processing devices to which the technology according to the present disclosure can be applied are implemented by a computer 900 having the configuration shown in FIG.
  • a CPU 901 , a ROM (Read Only Memory) 902 and a RAM (Random Access Memory) 903 are interconnected by a bus 904 .
  • An input/output interface 905 is further connected to the bus 904 .
  • the input/output interface 905 is connected to an input unit 906 such as a keyboard and a mouse, and an output unit 907 such as a display and a speaker. Further, the input/output interface 905 is connected with a storage unit 908 including a hard disk and nonvolatile memory, a communication unit 909 including a network interface, and a drive 910 for driving a removable medium 911 .
  • the CPU 901 loads a program stored in the storage unit 908 into the RAM 903 via the input/output interface 905 and the bus 904 and executes the above-described series of processes. is done.
  • Programs executed by the CPU 901 are recorded on the removable media 911, or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and installed in the storage unit 908.
  • the program executed by the computer may be a program in which processing is performed in chronological order according to the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program in which processing is performed.
  • the present disclosure can be configured as follows.
  • a matching processing unit that matches the coordinate systems set in each of the first space in which the first user exists and the second space in which the second user exists;
  • a presentation control unit that controls presentation of an interaction to the other user according to the action of one user based on the matched position information and action information of the first user and the second user in the coordinate system.
  • An information processing device comprising: (2) one of the first space and the second space is a VR space; The information processing apparatus according to (1), wherein the other of the first space and the second space is a real space.
  • the presentation control unit controls presentation of the VR space based on 3D map data corresponding to the real space to the first user or the second user existing in the VR space. ).
  • the information processing apparatus controls presentation of the interaction based on AR information to the first user or the second user existing in the real space.
  • the presentation control unit controls presentation of the interaction when each of the first user and the second user has an account for the same service. information processing equipment.
  • the presentation control unit displays a first avatar corresponding to the first user in the second space based on the position information and the action information of each of the first user and the second user. and controlling display of a second avatar corresponding to the second user in the first space.
  • the presentation control unit presents the interaction in real time to the second user in the second space in response to a direct action of the first user on the second avatar in the first space.
  • the presentation control unit responds to an indirect action of the first user on the second avatar in the first space with a time delay of the interaction on the second user on the second space.
  • the information processing apparatus according to (6) which controls the presentation of the .
  • (9) wherein the presentation control unit generates a virtual object as the interaction with the second user, and changes a presentation mode of the virtual object in accordance with an operation of the virtual object by the second user;
  • the information processing device described. (10) The information processing device according to (6), wherein the presentation control unit changes a presentation mode of the first avatar in the second space according to a change in the situation of the first user in the first space. .
  • the presentation control unit instructs the second user present in the second space according to an action of the first user present in any of the first spaces to present the first user
  • the information processing apparatus according to (13) which controls presentation of a notification regarding the first space in which the is present.
  • (13) The information processing apparatus according to (13), wherein the presentation control unit controls presentation of a notification regarding another of the first spaces to the first user existing in one of the first spaces.
  • the presentation control unit controls the other of the first spaces.
  • the information processing device which controls presentation of confirmation of movement to space 1.
  • Synchronization system 100 VR device, 130 Synchronization service application, 140 VR presentation unit, 200 VR device, 230 Synchronization service application, 240 AR presentation unit, 300 VR/AR synchronization service providing device, 311 Coordinate system matching processing unit, 312 Interaction Presentation control unit, 313 user management unit, 350 3D map data server, 510 space management unit

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • User Interface Of Digital Computer (AREA)
  • Processing Or Creating Images (AREA)
PCT/JP2022/041124 2021-11-19 2022-11-04 情報処理装置、情報処理方法、およびプログラム Ceased WO2023090163A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US18/709,035 US20250342663A1 (en) 2021-11-19 2022-11-04 Information processing device, information processing method, and program
JP2023561523A JPWO2023090163A1 (https=) 2021-11-19 2022-11-04
EP22895445.9A EP4425438A4 (en) 2021-11-19 2022-11-04 Information processing device, information processing method, and program

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021-188349 2021-11-19
JP2021188349 2021-11-19

Publications (1)

Publication Number Publication Date
WO2023090163A1 true WO2023090163A1 (ja) 2023-05-25

Family

ID=86396810

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/041124 Ceased WO2023090163A1 (ja) 2021-11-19 2022-11-04 情報処理装置、情報処理方法、およびプログラム

Country Status (4)

Country Link
US (1) US20250342663A1 (https=)
EP (1) EP4425438A4 (https=)
JP (1) JPWO2023090163A1 (https=)
WO (1) WO2023090163A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4489390A1 (en) * 2023-07-06 2025-01-08 Mitel Networks Corporation Multitasking in a virtual reality environment

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010282497A (ja) 2009-06-05 2010-12-16 Ricoh Co Ltd 異世界状態反映装置
JP2020004388A (ja) * 2019-04-11 2020-01-09 株式会社コロプラ システム、プログラム、方法、および情報処理装置
JP2020035392A (ja) * 2018-08-31 2020-03-05 真一 福重 遠隔コミュニケーションシステム等

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010282497A (ja) 2009-06-05 2010-12-16 Ricoh Co Ltd 異世界状態反映装置
JP2020035392A (ja) * 2018-08-31 2020-03-05 真一 福重 遠隔コミュニケーションシステム等
JP2020004388A (ja) * 2019-04-11 2020-01-09 株式会社コロプラ システム、プログラム、方法、および情報処理装置

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4489390A1 (en) * 2023-07-06 2025-01-08 Mitel Networks Corporation Multitasking in a virtual reality environment

Also Published As

Publication number Publication date
US20250342663A1 (en) 2025-11-06
JPWO2023090163A1 (https=) 2023-05-25
EP4425438A4 (en) 2025-03-05
EP4425438A1 (en) 2024-09-04

Similar Documents

Publication Publication Date Title
KR102940079B1 (ko) 개인화된 아바타 실시간 모션 캡처
TWI900659B (zh) 用於產生虛擬內容的裝置及方法
KR102937748B1 (ko) 신체 애니메이션 공유 및 재혼합
EP3740849B1 (en) Hybrid placement of objects in an augmented reality environment
KR102911932B1 (ko) 웨어러블 장치를 이용한 제어가능한 장치의 위치 식별
WO2019142560A1 (ja) 視線を誘導する情報処理装置
US12472435B2 (en) External controller for an eyewear device
US12105283B2 (en) Conversation interface on an eyewear device
JP2017027206A (ja) 情報処理装置、仮想オブジェクトの操作方法、コンピュータプログラム、及び記憶媒体
KR102428438B1 (ko) 실시간 좌표 공유 기반의 다자 간 원격 협업 방법 및 시스템
US11907434B2 (en) Information processing apparatus, information processing system, and information processing method
WO2022140113A1 (en) Conversation interface on an eyewear device
US11553009B2 (en) Information processing device, information processing method, and computer program for switching between communications performed in real space and virtual space
WO2023090163A1 (ja) 情報処理装置、情報処理方法、およびプログラム
JP6223614B1 (ja) 情報処理方法、情報処理プログラム、情報処理システム及び情報処理装置
JP7625039B2 (ja) プログラム
EP4592809A1 (en) Electronic device for displaying change of virtual object, and method thereof
KR102748362B1 (ko) 다매체를 통해 원격 협업을 지원하는 웨어러블 디바이스
CN109062413A (zh) 一种ar交互系统及方法
JP2018163637A (ja) 情報処理方法、情報処理プログラム、情報処理システム及び情報処理装置
JP2019164614A (ja) アバター制御プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22895445

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2023561523

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 2022895445

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2022895445

Country of ref document: EP

Effective date: 20240529

NENP Non-entry into the national phase

Ref country code: DE

WWP Wipo information: published in national office

Ref document number: 18709035

Country of ref document: US