WO2023120217A1 - 情報処理装置、情報処理方法およびプログラム - Google Patents

情報処理装置、情報処理方法およびプログラム Download PDF

Info

Publication number
WO2023120217A1
WO2023120217A1 PCT/JP2022/045326 JP2022045326W WO2023120217A1 WO 2023120217 A1 WO2023120217 A1 WO 2023120217A1 JP 2022045326 W JP2022045326 W JP 2022045326W WO 2023120217 A1 WO2023120217 A1 WO 2023120217A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
viewpoint
virtual
image
information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/045326
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
貴文 渡辺
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Canon Inc
Original Assignee
Canon Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Canon Inc filed Critical Canon Inc
Publication of WO2023120217A1 publication Critical patent/WO2023120217A1/ja
Priority to US18/747,713 priority Critical patent/US20240338899A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—Three-dimensional [3D] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/003—Navigation within 3D models or images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/20—Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/62—Semi-transparency
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/024—Multi-user, collaborative environment
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/20—Indexing scheme for editing of 3D models
    • G06T2219/2012—Colour editing, changing, or manipulating; Use of colour codes

Definitions

  • the present invention relates to an information processing device, an information processing method, and a program.
  • VR virtual reality
  • MR mixed reality
  • AR augmented reality
  • a virtual viewpoint only the viewpoint for drawing a virtual 3D model (hereafter referred to as a virtual viewpoint) can be moved by user operation for a building or a huge building represented by a virtual 3D model.
  • a walkthrough function There is a function that can Such a function that allows the user to move the virtual viewpoint.
  • the walkthrough function allows the user to observe various locations without moving. In order to smoothly use the walk-through function, it is preferable to be able to grasp the positional relationship between the virtual three-dimensional model to be observed and the user himself.
  • Patent Literature 1 discloses a virtual reality presentation device that has a full-scale observation mode for immersed observation in a virtual three-dimensional space of the actual size and a reduced observation mode for observing a reduced virtual three-dimensional space CG. .
  • the virtual reality presentation device of Patent Document 1 presents a CG map to which the user's position and orientation information is added in a reduced observation mode, thereby allowing the user to grasp where he/she is in the virtual three-dimensional space.
  • the use of the walk-through function allows the virtual viewpoint to move, resulting in a positional shift between the real space and the virtual three-dimensional model. Inconsistency occurs.
  • the viewpoint of the actual head of the user during the walk-through as seen from other users hereinafter referred to as the real viewpoint
  • the three-dimensional model displayed to the user during the walk-through are displayed.
  • the real viewpoint the viewpoint of the actual head of the user during the walk-through as seen from other users
  • the three-dimensional model displayed to the user during the walk-through are displayed.
  • the present invention has been made in view of the problems described above, and aims to eliminate the inconsistency in the relationship between the virtual viewpoint and the real viewpoint of other users.
  • the present invention is an information processing apparatus for sharing a space in which a virtual three-dimensional model is presented with other users, comprising: acquisition means for acquiring information of the real viewpoint of the other user; and virtual viewpoint of the other user. is different from the real viewpoint of the other user, a generating means for generating a video effect based on the information of the real viewpoint of the other user.
  • FIG. 10 is a schematic diagram showing a case where a specific user uses the walkthrough function; It is a figure which shows the functional structure of the information processing system of 1st Embodiment. It is a figure which shows the hardware constitutions of an information processing apparatus.
  • FIG. 4 is a diagram schematically showing a mixed reality image viewed by the user; It is a figure which shows the functional structure of the information processing system of 2nd Embodiment. 4 is a flowchart showing processing of an information processing device;
  • FIG. 11 is a plan view schematically showing the situation of the third embodiment;
  • FIG. 4 is a diagram schematically showing a mixed reality image viewed by the user; 4 is a flowchart showing processing of an information processing device;
  • FIG. 1 is a plan view schematically showing the positional relationship when a specific user uses the walkthrough function when observing a virtual 3D model with a plurality of people.
  • user A101 and user B102 share a space in which a virtual three-dimensional model is presented.
  • User A101 is the own user, and user B102 is another user.
  • the real viewpoint 104 and the virtual viewpoint 105 of the user A 101 are at the same position.
  • the real viewpoint 106 and the virtual viewpoint 107 are different from each other because the user B 102 uses the walk-through function.
  • User B's avatar 103 is displayed at the position of user B 102's virtual viewpoint 107 .
  • the virtual viewpoint 107 of the user B 102 exists on the left side of the virtual model 108 even though the real viewpoint 106 of the user B 102 exists on the right side of the virtual model 108. become. This makes it difficult for the user A101 to grasp the position of the user B102.
  • the relationship between the other user's virtual viewpoint and the real viewpoint is generated by generating a video effect based on the information of the other user's real viewpoint. to eliminate inconsistencies.
  • a video effect is generated based on information about the real viewpoint of other users during the walkthrough. Specifically, an image indicating that the other user's virtual viewpoint does not exist at the position corresponding to the other user's real viewpoint is presented.
  • the user can recognize an image indicating that the other user's virtual viewpoint does not exist, so the own user's virtual viewpoint is not the position of the real viewpoint of the other user. can be easily grasped.
  • FIG. 4 is a diagram schematically showing a mixed reality image viewed from the real viewpoint of user A101. Note that the positional relationship of each object such as the avatar 103 and the virtual model 108 corresponds to FIG.
  • the user B102 is walking through, and the avatar 103 of the user B102 is displayed at the position and orientation of the virtual viewpoint of the user B102.
  • the position and orientation refer to the position and orientation (orientation).
  • the color of the area 401 surrounding the body of the user B102 is changed.
  • Character information 402 indicating that the walk-through is in progress is displayed near the area 401 .
  • FIG. 2 is a diagram showing an example of the functional configuration of the information processing system.
  • An information processing system is a mixed reality or augmented reality in which a plurality of people observe a virtual 3D model by cooperating with each other through a transmitting unit and a receiving unit that are installed in each device used by each user. feeling can be realized.
  • the information processing system shown in FIG. 2 assumes that there are two users, and has a configuration in which two devices, the device 201 and the device 221, cooperate with each other. When three or more persons observe the virtual three-dimensional model, three or more persons cooperate with each other. An embodiment in which there are two users will be described below.
  • the device 201 includes a terminal 202 , an operation input unit 205 and an information processing device 217 .
  • the terminal 202 captures images of the surrounding real environment and presents mixed reality images to the user.
  • the terminal 202 has an imaging unit 203 and a display unit 204 .
  • the imaging unit 203 captures images of the surrounding real environment.
  • a display unit 204 presents a mixed reality image to the user.
  • the terminal 202 of this embodiment uses a head-mounted display (hereinafter referred to as HMD) in which the imaging unit 203 and the display unit 204 are mounted on a head-mounted housing. By wearing the HMD on the head of the user, the HMD moves in conjunction with the movement of the user and the movement of the head.
  • the terminal 202 may be a tablet terminal, a smartphone terminal, or the like.
  • the terminal 202 of the present embodiment is a video see-through type in which a virtual three-dimensional model is superimposed on a real image captured by the imaging unit 203 and displayed on the display unit 204, but it may be an optical see-through type. .
  • the imaging unit 203 has a camera that captures an image to be presented to the user and a camera that acquires an image necessary for estimating the position and orientation of the terminal 202 .
  • the imaging unit 203 may have a function of capturing an image and a function of acquiring an image necessary for estimating the position and orientation with the same camera. Also, the imaging unit 203 can omit one or both of the two cameras depending on the method of acquiring the position and orientation of the terminal 202 and the method of superimposing the virtual three-dimensional model.
  • a camera that captures an image to be presented to the user in order to achieve stereoscopic vision and a display device that presents a mixed reality image are individually mounted for the right eye and the left eye.
  • the configuration of the camera and display device mounted on the terminal 202 can be changed according to the form of the terminal 202, the method of acquiring the position and orientation of the terminal 202, and the method of superimposing the virtual three-dimensional model.
  • the operation input unit 205 accepts an operation for enabling the user to use the walkthrough function, and an operation for moving the virtual viewpoint input for walkthrough. Specifically, the operation input unit 205 accepts one or more of button input from a controller device or the like, gesture input, voice input, and the like. The operation input unit 205 may be provided in the terminal 202 or configured as a separate device.
  • the information processing device 217 is a device for sharing the space in which the virtual three-dimensional model is presented with other users.
  • the information processing device 217 estimates the position and orientation of the terminal 202 and generates a mixed reality image.
  • the information processing device 217 includes an operation information acquisition unit 206, a captured image acquisition unit 207, a real viewpoint position/orientation acquisition unit 208, a virtual viewpoint position/orientation generation unit 209, a transmission unit 210, a reception unit 211, a virtual image generation unit 212, and a synthesis unit. 213.
  • the operation information acquisition unit 206 acquires operation information received by the operation input unit 205 .
  • the captured image acquisition unit 207 acquires a captured image of the real environment, which is transmitted from the terminal 202 to the information processing device 217 .
  • the real viewpoint position/posture acquisition unit 208 acquires information on the user's real viewpoint. Specifically, the real viewpoint position/orientation acquisition unit 208 acquires the real viewpoint information of the user by calculating the position and orientation of the user's real viewpoint from the captured image.
  • the virtual viewpoint position/orientation generation unit 209 generates information on the user's virtual viewpoint. Specifically, the virtual viewpoint position/posture generation unit 209 calculates the position/posture of the virtual viewpoint based on the information of the real viewpoint and the operation information acquired by the operation information acquisition unit 206, thereby determining the position and orientation of the user's virtual viewpoint. Generate information.
  • the transmitting unit 210 transmits to the device 221 information about the position and orientation of the user's real viewpoint and information about the position and orientation of the user's virtual viewpoint.
  • the receiving unit 211 receives, from the device 221, information on the position and orientation of the real viewpoint of the other user and information on the position and orientation of the virtual viewpoint of the other user.
  • the virtual image generation unit 212 generates information on the position and orientation of the virtual viewpoint of the user, information on the position and orientation of the real viewpoint of the user, information on the position and orientation of the virtual viewpoints of other users, and information on the position and orientation of the real viewpoints of the other users. generates a virtual three-dimensional model based on Also, when the other user is walking through, that is, when the other user's virtual viewpoint is different from the real viewpoint, the virtual image generation unit 212 generates a video effect based on the other user's real viewpoint information.
  • the synthesizing unit 213 synthesizes the virtual image and the image of the real environment to generate a mixed reality image.
  • the device 221 has the same configuration as the device 201. Therefore, the device 221 has an HMD as the terminal 202 worn on the head of another user. Further, the device 221 has an information processing device having the same configuration as the information processing device 217 of the device 201 . In FIG. 2, only the transmitting section 230 and the receiving section 231, which constitute the information processing device of the device 221, are illustrated.
  • FIG. 3 is a diagram showing an example of the hardware configuration of the information processing device 217. As shown in FIG.
  • the information processing device 217 has a CPU 301 , a ROM 302 , a RAM 303 , an input I/F 304 and an output I/F 305 . Each part of the hardware configuration of the information processing device 217 is connected by a bus 306 .
  • the CPU 301 controls the entire information processing device 217 . Further, the CPU 301 implements the functional configuration of the information processing device 217 described above by expanding the program stored in the ROM 302 into the RAM 303 and executing the program.
  • the ROM 302 stores programs executed by the CPU 301 and various data.
  • the RAM 303 temporarily stores the program when the CPU 301 executes the program stored in the ROM 302, and temporarily stores the execution result.
  • the input I/F 304 is an interface for inputting various data from the components within the device 201 or from the device 221 .
  • the information processing device 217 receives operation information from the operation input unit 205 , captured images captured by the imaging unit 203 , and information from the transmission unit 230 of the device 221 via the input I/F 304 .
  • the output I/F 305 is an interface for outputting various data to the components within the device 201 or the device 221 . Specifically, the information processing device 217 outputs images to the terminal 202 and outputs information to the receiving unit 231 of the device 221 via the output I/F 305 .
  • the information processing device 217 may be a stationary computer such as a personal computer or workstation, or a portable computing device. Also, the information processing device 217 may be integrated with the terminal 202 .
  • Each flowchart including FIG. 5 is implemented by the CPU 301 of the information processing device 217 developing a program stored in the ROM 302 in the RAM 303 and executing the program.
  • the own user and other users have set to use the walk-through function via the operation input unit 205 .
  • the real viewpoint position/posture acquisition unit 208 acquires information on the position/posture of the user's real viewpoint in world coordinates as the information on the user's real viewpoint.
  • the position and orientation of the real viewpoint can be estimated from the arrangement of the feature points of the markers in the image, based on the captured image of the markers arranged in space. There is a way.
  • SLAM Simultaneous Localization and Mapping
  • information on the position and orientation of the real viewpoint may be obtained by using an external measurement device such as motion capture.
  • the origin and orientation of the world coordinate system are determined in advance using markers or specific real objects. Also, the origin and direction of the world coordinate system are set so as to be common among a plurality of users.
  • the virtual viewpoint position/orientation generation unit 209 generates information on the position and orientation of the user's virtual viewpoint in the world coordinates as the information on the user's virtual viewpoint. Specifically, the virtual viewpoint position/orientation generation unit 209 calculates the viewpoint movement amount based on the operation information acquired by the operation information acquisition unit 206 . Next, the virtual viewpoint position/posture generation unit 209 adds the calculated viewpoint movement amount to the position/posture of the user's real viewpoint in the world coordinates, thereby calculating information about the position/posture of the user's virtual viewpoint in the world coordinates. .
  • the method of associating the amount of viewpoint movement from the operation information is not particularly limited.
  • the device that configures the operation input unit 205 or the device that configures the real viewpoint position/orientation acquisition unit 208 can acquire the positions and orientations of body parts such as the user's hands, the viewpoint can also be moved to the positions and orientations of the body parts. By adding the amount, it can be used for generating an avatar, which will be described later.
  • the transmission unit 210 transmits the real viewpoint information and the virtual viewpoint information of the user to the device 221 .
  • the transmitting unit 210 transmits the information on the position and orientation of the user's body part and the information obtained by adding the amount of viewpoint movement to the position and orientation of the body part. do.
  • the receiving unit 211 receives real viewpoint information and virtual viewpoint information of other users from the device 221 .
  • the receiving unit 211 adds the viewpoint movement amount to the information on the position and orientation of the other user's body parts and the position and orientation of the body parts. received information.
  • the device 201 and the device 221 can share information such as the user's real viewpoint with each other.
  • the virtual image generation unit 212 generates an avatar based on information on the position and orientation of the virtual viewpoint of the other user.
  • the virtual image generation unit 212 may generate an avatar of only the head, or may generate an avatar in which body parts such as the torso and legs are added to positions analogized from the position of the head. Further, when receiving information obtained by adding the amount of viewpoint movement to the position and orientation of the other user's body part, the virtual image generation unit 212 generates a virtual image of the other user who has moved through the walk-through based on the received information. An avatar with representations of body parts may be generated.
  • identification information may be added to the avatar in order to identify the user, in the vicinity of the avatar or superimposed on the avatar.
  • the identification information includes, for example, a user name, a user ID, a texture that indicates the characteristics of the user, and the like.
  • the virtual image generation unit 212 estimates the area where the other user's body exists based on the information on the position and orientation of the real viewpoint of the other user. Specifically, the virtual image generation unit 212 estimates the area including the head in the shape of an elliptical sphere based on the information about the position and orientation of the real viewpoint of the other user in the world coordinate system. In addition, for the whole body area other than the head, the feet are present at a predetermined distance vertically downward from the viewpoint position based on the vertical or floor surface information obtained by stipulating the vertical direction or estimating the floor surface. We estimate that Next, a cylinder existing with a predetermined radius around the vertical axis connecting the feet and the head is determined as the whole body area including the whole body of the other user.
  • the terminal 202 may be equipped with a posture sensor to estimate the vertical direction, or another method may be used.
  • the virtual image generation unit 212 appropriately corrects or adds areas where the other users' bodies exist, based on the received information. You may
  • the body region may be estimated by combining three-dimensional polyhedrons in the world coordinate system, or may be performed on a two-dimensional image presented to the user.
  • the position of the viewpoint on the world coordinates, the position of the other user's feet estimated by the above-described method, and the position and orientation of the other user's body parts have been received, based on the received information, An image to be presented to the user is projected onto the two-dimensional image.
  • the virtual image generator 212 may generate a polygon including the other user's body region on the projected two-dimensional coordinates.
  • the shape of the other user's area is not limited, and may be any shape.
  • the virtual image generation unit 212 generates an image showing that the virtual viewpoint of the other user is different from the real viewpoint of the other user at the position corresponding to the real viewpoint of the other user. In this embodiment, the virtual image generation unit 212 generates an image that presents that the other user's virtual viewpoint does not exist in the area where the other user's body exists.
  • the virtual image generation unit 212 generates an image in which the color of the region estimated in S505 is changed to a different color.
  • the color of the area can be changed by adding color to the estimated area and drawing it.
  • the virtual image generation unit 212 may be realized by drawing the three-dimensional polyhedron generated in S505 as it is, or may be realized by superimposing a specific texture image within the estimated area or within the area of the three-dimensional polyhedron. good.
  • the estimated area can be used to generate a mask, and the color of the area can be changed by changing the color of the corresponding real image within the mask area. This can be achieved by synthesizing the real image whose color has been changed by the virtual image generating unit 212 with the image of the real environment by the synthesizing unit 213 .
  • At least one of the saturation and brightness of the estimated area may be reduced to produce the absence of virtual viewpoints of other users.
  • a method for changing the color of the estimated area is not particularly limited, and any method may be used.
  • the virtual image generation unit 212 generates character information near the area where the other user's body exists or superimposed on the area to indicate that the other user does not exist, specifically that the user is walking through. .
  • the information indicating that the walkthrough is in progress is not limited to character information, and may be symbol information, an image such as an icon, or the like.
  • the synthesizing unit 213 synthesizes the virtual image generated by the virtual image generating unit 212 and the image of the real environment to generate a mixed reality image. Note that the synthesizing unit 213 also synthesizes the avatar image at the position of the virtual viewpoint of the other user during the walkthrough. The synthesizing unit 213 outputs the generated mixed reality image to the terminal 202 .
  • the terminal 202 presents the output mixed reality image to its own user.
  • the display unit 204 of the terminal 202 presents the mixed reality image to the user. Since the mixed reality image displayed on the display unit 204 includes an image showing that the virtual viewpoint of the other user is different from the real viewpoint of the other user at the position corresponding to the real viewpoint of the other user, the user can It is possible to easily grasp that the user's virtual viewpoint is not the position of the real viewpoint. Therefore, in mixed reality or augmented reality, it is possible to eliminate inconsistency in the relationship between the real and virtual viewpoints of the other users during the walkthrough.
  • the virtual image generation unit 212 when the other user's real viewpoint information and the other user's virtual viewpoint information are the same or substantially the same, the virtual image generation unit 212 generates a video effect based on the other user's real viewpoint information. processing can be omitted.
  • a so-called diminished reality is applied as a video effect based on the real viewpoint information of the other user, in which the area where the other user exists is changed to a background image.
  • the other user's body appears to have disappeared, so the own user can easily grasp that the virtual viewpoint of the other user is not the position of the real viewpoint.
  • FIG. 6 is a plan view schematically showing the situation of mixed reality or augmented reality to which this embodiment is applied. 1 are given the same reference numerals, and description thereof will be omitted as appropriate.
  • a background object 601 is placed behind the body of the user B102 who is walking through as viewed from the user A101.
  • a background object 601 is a real object placed in or around the mixed reality experience space.
  • FIG. 7 is a diagram schematically showing a mixed reality image viewed from the real viewpoint of user A101.
  • An image 701 of a background object 601 that can be observed when the body of the user B102 does not exist is displayed in the area where the body of the user B102 exists.
  • the avatar 103 of the user B102 is displayed at the position and orientation of the virtual viewpoint of the user B102.
  • FIG. 8 is a diagram showing an example of the functional configuration of the information processing system. It should be noted that the same reference numerals are given to the same configurations as in the first embodiment, and the description thereof will be omitted as appropriate. Also, the hardware configuration of the information processing device 217 is the same as that of the first embodiment, and description thereof will be omitted as appropriate.
  • a background information generation unit 801 is added to the information processing device 217 of the device 201 .
  • the background information generation unit 801 generates a captured image captured by the imaging unit 203, information on the position and orientation of the real viewpoint acquired by the real viewpoint position and orientation acquisition unit 208, and information on natural feature points when SLAM is used. Based on this, the background information when there is no other user is reconstructed.
  • the flowchart in FIG. 9 is, for example, a flowchart following the process of S504 in the flowchart in FIG.
  • the background information generation unit 801 generates background information by performing textured three-dimensional reconstruction of the real background space. For example, as a method of three-dimensional reconstruction, a method using structure from motion (SfM) based on information on the position and orientation of the real viewpoint and a real image, or using natural feature points obtained in the process of using SLAM There is a way. Also, if the imaging unit 203 is a stereo camera, the background information may be generated based on the information of the stereo camera. Specifically, the depth information in the image is generated from the stereo camera information, and the point cloud information generated from the depth information is combined by applying the ICP (Interactive Closest Point) algorithm between frames. Combine using the position and orientation.
  • ICP Interactive Closest Point
  • the background information generation unit 801 converts the three-dimensional reconstruction information into a mesh, and generates background information by associating the vertices of the mesh with the actual image captured during the three-dimensional reconstruction to form a texture.
  • the method of three-dimensional reconstruction is not limited, and any method may be used.
  • background information may be generated by a simplified method that does not perform three-dimensional reconstruction.
  • the background space as a polyhedron such as a sphere or a cube.
  • an image of the background space captured in advance using a fish-eye camera is attached as a texture to the inside of the polyhedron.
  • an image that can be obtained when the imaging unit 203 looks around is mapped inside the polyhedron according to the direction of the imaging unit 203 .
  • a simplified method that does not perform three-dimensional reconstruction lacks accuracy, but is useful when computational resources for performing three-dimensional reconstruction are insufficient.
  • the virtual image generation unit 212 estimates the area where the other user's body exists based on the information on the position and orientation of the real viewpoint of the other user.
  • the estimated area is calculated as the two-dimensional coordinates of the vertices of the contours of the other users on the image presented to the user. Specifically, first, if the position of the other user's real viewpoint during the walkthrough, the position of the other user's feet, and the position and orientation of the other user's body part have been acquired, the position and orientation of the other user's body part are obtained. A position information group in the added three-dimensional world coordinate system is acquired.
  • the method of estimating the presence of the foot described in S506 of the first embodiment can be used.
  • the acquired position information group is projected onto a two-dimensional image on the image presented to the user, and a region including the other user's body is generated from the projected position information group.
  • the distance information to the position of the real viewpoint of the other user is used to determine the margin amount on the two-dimensional image, and based on the determined margin amount, the upper right and upper left of the position of the real viewpoint, and the lower right of the foot position. Calculate the lower left four vertices.
  • the shape of the other user's area is not limited, and any shape may be used.
  • the calculation process and method are not limited as long as a result similar to the region estimation described above can be obtained.
  • the background information generation unit 801 generates a background image generated from the three-dimensional reconstruction information in the estimated area where the other user's body exists. Specifically, the background information generating unit 801 generates a background image by drawing the background information generated in S901 in the area estimated in S902 from the viewpoint of the user observing the background information.
  • the virtual image generation unit 212 generates a synthetic image of the real space by superimposing the generated background image on the real image. Therefore, a composite image is generated in which the estimated area where the other user's body exists is changed to a background image.
  • the body of the other user during the walkthrough is represented as if it does not exist. It is assumed that Therefore, the virtual image generator 212 changes the background image according to the positional relationship between the position of the user's real viewpoint and the position of the other user's real viewpoint during the walkthrough. Specifically, the virtual image generation unit 212 makes the background image transparent when the distance between the real viewpoints is equal to or less than a predetermined distance, or changes the background image continuously or stepwise according to the distance between the real viewpoints.
  • the synthesizing unit 213 may instruct the terminal 202 to sound a warning sound when the distance between the real viewpoints is equal to or less than a predetermined distance or when the distance between the real viewpoints is increased.
  • the composition unit 213 draws a virtual three-dimensional model on the composite image generated by the virtual image generation unit 212 to generate a mixed reality image.
  • the synthesizing unit 213 also synthesizes the avatar image at the position of the virtual viewpoint of the other user during the walkthrough.
  • the synthesizing unit 213 outputs the generated mixed reality image to the terminal 202 .
  • the terminal 202 presents the output mixed reality image to its own user. Specifically, the display unit 204 of the terminal 202 presents the mixed reality image to the user. In the mixed reality image displayed on the display unit 204, the background image is displayed at a position corresponding to the real viewpoint of the other user. can be grasped. Therefore, in mixed reality or augmented reality, it is possible to eliminate inconsistency in the relationship between the real and virtual viewpoints of the other users during the walkthrough.
  • a virtual space or a virtual three-dimensional model is deformed or processed as a visual effect based on information of another user's real viewpoint.
  • a part of the 3D model is displayed as a virtual 3D model viewed from the other user's virtual viewpoint. It can be easily grasped that the viewpoint is not the position of the actual viewpoint.
  • FIG. 10 is a plan view schematically showing the situation of mixed reality or augmented reality to which this embodiment is applied. 1 are given the same reference numerals, and description thereof will be omitted as appropriate.
  • FIG. 11 is a diagram schematically showing a mixed reality image viewed from the real viewpoint of user A101.
  • the virtual space itself or the virtual 3D model is transformed or processed. Specifically, a boundary surface 1003 is set between the position of the real viewpoint 104 of the user A 101 and the position of the real viewpoint 106 of the user B 102, and the virtual space is divided across this boundary. Then, in the virtual spaces of the user A 101 and the user B 102, respectively, coordinate systems are set in which the relative positions and orientations of the virtual viewpoints 105 and 1001 and the virtual model are maintained. As a result, the virtual model 1004 is divided and displayed in the virtual space on the side of the user B102, and the positions of the virtual viewpoint 1002 and the real viewpoint 106 of the user B102 match.
  • the flowchart in FIG. 12 is, for example, a flowchart following the process of S503 in the flowchart in FIG. Note that the functional configuration of the information processing system and the hardware configuration of the information processing device 217 are the same as those in the first embodiment, and description thereof will be omitted as appropriate.
  • the virtual image generation unit 212 obtains an inverse transformation matrix corresponding to the change from the position and orientation of the real viewpoint to the position and orientation of the virtual viewpoint, which change due to the walkthrough of other users.
  • the virtual image generator 212 applies the calculated inverse transformation matrix to the virtual space to generate a virtual space to which the inverse transformation is applied.
  • the virtual image generation unit 212 may generate a virtual three-dimensional model to which the inverse transformation is applied by applying the calculated inverse transformation matrix to the virtual three-dimensional model in the virtual space.
  • the virtual image generation unit 212 uses the information about the position and orientation of the real viewpoint of the user and the information about the position and orientation of the real viewpoint of the other user to set a boundary surface between the positions of the two real viewpoints.
  • the floor surface described in the first embodiment is estimated, the floor surface is set as the work surface. If there is no information about the floor surface, the average of the vectors in the upward direction of each user's field of vision, or the X, Y, or Z axis of the world coordinate system that forms the smallest angle with the average vector is taken as the normal, Let the plane passing through the appropriate coordinates be the working plane. Next, the foot of the perpendicular to the work plane of each user's real viewpoint is calculated. If there are two users, the boundary plane is a plane that passes through the perpendicular bisector of the foot of the perpendicular and is perpendicular to the work plane.
  • a Voronoi boundary for each user's perpendicular foot is generated on the work plane, and a plane passing through the Voronoi boundary and perpendicular to the work plane is defined as the boundary plane.
  • the user's real viewpoint and the other user's real viewpoint can be projected onto a work surface as a predetermined plane, and the Voronoi boundary can be calculated based on the projected points.
  • the virtual image generation unit 212 clips (cuts out) the virtual space based on the set boundary surface.
  • a case of clipping a virtual three-dimensional model included in the virtual space will be described with reference to FIG.
  • the virtual model 108 on the side of the user A101 clips the side of the user B102 at the boundary plane 1003.
  • the virtual model 1004 on the user B 102 side after inverse transformation is applied clips the user A 101 side at the boundary surface 1003 .
  • the virtual image generation unit 212 generates a virtual image drawn from the viewpoint of the user observing the clipped virtual space.
  • the generated virtual image is an image in which the position of the other user's virtual viewpoint matches the position of the other user's real viewpoint when the other user is viewed from the own user's real viewpoint.
  • a boundary surface is added to the virtual image, or the intersection of the boundary surface and the work surface is added.
  • a line may be added as a boundary line.
  • the virtual image generation unit 212 generates character information near the boundary surface or boundary line indicating that the other user's virtual viewpoint is different from the other user's real viewpoint, specifically, that the walk-through shown in FIG. 11 is in progress. 1101 is generated.
  • the information indicating that the walkthrough is in progress is not limited to character information, and may be symbol information, an image such as an icon, or the like.
  • the synthesizing unit 213 outputs the virtual image generated by the virtual image generating unit 212 to the terminal 202 . As necessary, the synthesizing unit 213 synthesizes the virtual image generated by the virtual image generating unit 212 and the image of the real environment to generate a mixed reality image, and outputs the generated mixed reality image to the terminal 202.
  • the terminal 202 presents the output virtual image or mixed reality image to its own user.
  • the display unit 204 of the terminal 202 presents the virtual image or the mixed reality image to the user. Since the image displayed on the display unit 204 displays a virtual three-dimensional model divided by the boundary plane, the own user can easily grasp that the virtual viewpoint of the other user is not the position of the real viewpoint. be able to. Therefore, in mixed reality or augmented reality, it is possible to eliminate inconsistency in the relationship between the real and virtual viewpoints of the other users during the walkthrough.
  • the case where the virtual space is clipped at the boundary surface has been described. may be used to generate a mixed reality image.
  • the boundary surface it is not limited to the case of clipping the virtual 3D model on the boundary surface, and an offset may be set for the boundary surface to provide a gap between the boundaries.
  • the virtual 3D model overlaps You may make it Alternatively, the side surface of a cylinder having a predetermined radius around the vertical axis passing through the real viewpoint of the user or another user may be used as the boundary surface.
  • transparency may be added to the boundary surface continuously or stepwise, or the color of the boundary surface may be changed.
  • the entire boundary surface may be blocked by the virtual 3D model, making it impossible to see what the user is doing.
  • the other user's body region is estimated by the method described in the first embodiment. Then, when the virtual three-dimensional model intersects with the region connecting the estimated region and the real viewpoint of the own user who observes other users, a virtual three-dimensional model including the intersecting portion or the periphery of the intersecting portion is generated. Regions of the dimensional model may be transparent or translucent.
  • the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a recording medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.
  • a circuit for example, ASIC
  • device 201 and the device 221 have the same configuration
  • the configuration is not limited to this and may be different.
  • device 201 may not have walkthrough functionality and device 221 may have walkthrough functionality.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Remote Sensing (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Architecture (AREA)
  • Computing Systems (AREA)
  • Geometry (AREA)
  • Human Computer Interaction (AREA)
  • Processing Or Creating Images (AREA)
  • User Interface Of Digital Computer (AREA)
PCT/JP2022/045326 2021-12-20 2022-12-08 情報処理装置、情報処理方法およびプログラム Ceased WO2023120217A1 (ja)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/747,713 US20240338899A1 (en) 2021-12-20 2024-06-19 Information processing apparatus information processing method and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021-206251 2021-12-20
JP2021206251A JP7760355B2 (ja) 2021-12-20 2021-12-20 情報処理装置、情報処理方法およびプログラム

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/747,713 Continuation US20240338899A1 (en) 2021-12-20 2024-06-19 Information processing apparatus information processing method and storage medium

Publications (1)

Publication Number Publication Date
WO2023120217A1 true WO2023120217A1 (ja) 2023-06-29

Family

ID=86902328

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/045326 Ceased WO2023120217A1 (ja) 2021-12-20 2022-12-08 情報処理装置、情報処理方法およびプログラム

Country Status (3)

Country Link
US (1) US20240338899A1 (https=)
JP (1) JP7760355B2 (https=)
WO (1) WO2023120217A1 (https=)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12519924B2 (en) * 2022-08-31 2026-01-06 Snap Inc. Multi-perspective augmented reality experience

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2018106298A (ja) * 2016-12-22 2018-07-05 キヤノンマーケティングジャパン株式会社 画像処理装置、画像処理方法、コンピュータプログラム
JP2018147377A (ja) * 2017-03-08 2018-09-20 株式会社コロプラ 情報処理方法、コンピュータ、および当該情報処理方法をコンピュータに実行させるためのプログラム

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10395435B2 (en) * 2016-04-04 2019-08-27 Occipital, Inc. System for multimedia spatial annotation, visualization, and recommendation
US12211151B1 (en) * 2019-07-30 2025-01-28 Onpoint Medical, Inc. Systems for optimizing augmented reality displays for surgical procedures
US11769301B2 (en) * 2020-08-03 2023-09-26 Urban District Realty, LLC Virtual reality simulations of three-dimensional structures
JP7543959B2 (ja) * 2021-03-19 2024-09-03 株式会社Ｊｖｃケンウッド 画像処理装置、画像処理方法及びプログラム

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2018106298A (ja) * 2016-12-22 2018-07-05 キヤノンマーケティングジャパン株式会社 画像処理装置、画像処理方法、コンピュータプログラム
JP2018147377A (ja) * 2017-03-08 2018-09-20 株式会社コロプラ 情報処理方法、コンピュータ、および当該情報処理方法をコンピュータに実行させるためのプログラム

Also Published As

Publication number Publication date
JP7760355B2 (ja) 2025-10-27
US20240338899A1 (en) 2024-10-10
JP2023091481A (ja) 2023-06-30

Similar Documents

Publication Publication Date Title
AU2021480445B2 (en) Avatar display device, avatar generation device, and program
EP4058870B1 (en) Co-located pose estimation in a shared artificial reality environment
JP5818773B2 (ja) 画像処理装置、画像処理方法及びプログラム
JP7262973B2 (ja) 情報処理装置、情報処理方法及びプログラム
KR101687017B1 (ko) 머리 착용형 컬러 깊이 카메라를 활용한 손 위치 추정 장치 및 방법, 이를 이용한 맨 손 상호작용 시스템
CN109598798B (zh) 虚拟物拟合方法及虚拟物拟合服务系统
JP5871345B2 (ja) ３次元ユーザインタフェース装置及び３次元操作方法
JP5709440B2 (ja) 情報処理装置、情報処理方法
JP2008210276A (ja) 三次元モデル情報の生成方法及び装置
CA2550512A1 (en) 3d videogame system
JP2023532285A (ja) アモーダル中心予測のためのオブジェクト認識ニューラルネットワーク
KR20200138349A (ko) 화상 처리 방법 및 장치, 전자 디바이스, 및 저장 매체
JP2019008623A (ja) 情報処理装置、及び、情報処理装置の制御方法、コンピュータプログラム、記憶媒体
JP2020052790A (ja) 情報処理装置、情報処理方法、およびプログラム
WO2020036114A1 (ja) 画像処理装置、画像処理方法、およびプログラム
JP2009169622A (ja) 画像処理装置、画像処理方法
JP6723743B2 (ja) 情報処理装置、情報処理方法、およびプログラム
JP7175715B2 (ja) 情報処理装置、情報処理方法及びプログラム
JP7760355B2 (ja) 情報処理装置、情報処理方法およびプログラム
JP2019133310A (ja) 画像処理装置、画像処理プログラム及び画像処理方法
WO2024004321A1 (ja) 画像処理装置、頭部装着型表示装置、画像処理装置の制御方法、頭部装着型表示装置の制御方法、およびプログラム
US10798360B2 (en) Information processing system, method for controlling same, and program
JP7321029B2 (ja) キャリブレーション装置及びその制御方法及びプログラム及び記憶媒体
US20240233269A1 (en) Information processing apparatus, information processing method, and program
US20260039781A1 (en) Display image generation apparatus and image display method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22910928

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22910928

Country of ref document: EP

Kind code of ref document: A1