WO2018039071A1 - Method and system for showing remote meeting sites from user-dependent viewpoints - Google Patents
Method and system for showing remote meeting sites from user-dependent viewpoints Download PDFInfo
- Publication number
- WO2018039071A1 WO2018039071A1 PCT/US2017/047605 US2017047605W WO2018039071A1 WO 2018039071 A1 WO2018039071 A1 WO 2018039071A1 US 2017047605 W US2017047605 W US 2017047605W WO 2018039071 A1 WO2018039071 A1 WO 2018039071A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- remote
- user
- local
- telepresence
- site
- Prior art date
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/15—Conference systems
- H04N7/157—Conference systems defining a virtual conference space and using avatars or agents
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/141—Systems for two-way working between two video terminals, e.g. videophone
- H04N7/142—Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
- H04N7/144—Constructional details of the terminal equipment, e.g. arrangements of the camera and the display camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/141—Systems for two-way working between two video terminals, e.g. videophone
- H04N7/147—Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals
Definitions
- This disclosure relates to systems and methods for telepresence systems. More specifically, this disclosure relates to systems and methods for formation of user-centric views of local and remote users within a consistent virtual geometry.
- Telepresence (remote interaction) systems give users the feeling of more personal interaction. In basic telepresence systems, this is accomplished by one camera in each local meeting room shared by all remote participants. In more advanced systems supporting directions and spatiality, multiple physical or virtual cameras are used in each meeting space.
- a wide video display may be used for showing all of the participants at each meeting site.
- multiple physical or virtual cameras are used in each meeting space, with different participants being shown in different displays. Showing each participant or group of participants in separate video windows, however, is unsatisfactory due to not supporting enough the feeling of continuity and immersion.
- a panorama from the correct viewpoint is provided individually for all participants of the telepresence or interaction session.
- a panorama is formed locally, and derived views are sent to each receiver as a panorama video.
- a front-end system may omit transmitting excessive amounts of real-time 3D data for deriving panoramas at each receiving site.
- a method for conducting a teleconference showing remote meeting sites from a user-dependent viewpoint.
- An exemplary teleconference system operates to determine a multi-party, multi -location telepresence session geometry.
- Systems and methods operate to determine and track the location of a far-end (remote) user.
- the location of the far-end user is provided to the local conferencing system.
- the local system captures video feeds of local participants.
- the local system further operates to construct a panorama video corresponding to the viewpoint of a first far-end user using the location of the far-end user, the session geometry, and video feeds of local participants.
- the local system communicates the constructed panorama video to the remote user.
- properties of the display pose and the geometry of a far end user are signaled to the local conferencing system.
- details of the display pose, geometry, position of a first far end user, and meeting geometry are used to produce a virtual panorama image of the local scene for the first far end user.
- FIG. 1 is a schematic plan view illustrating use of camera-centric viewpoints in panorama generation.
- FIG. 2 is a schematic plan view illustrating formation of a user-centric panorama in a three-party meeting, where user positions are tracked and combined in a unified geometry (coordinate system).
- FIG. 3 is a schematic functional block diagram of a system architecture employed in some embodiments.
- FIG. 4 is a schematic plan view illustrating an example of a shared geometry for a meeting between five participants at three local sites.
- FIG. 5 is a schematic plan view illustrating an example of a linear sensor arrangement to support each remote user centric view.
- FIG. 6 is a schematic plan view illustrating an embodiment in which restrictions to the visibility of users are caused either by the span of the camera array or the display.
- FIG. 7 is a schematic plan view illustrating the use of separate user centric camera views in accordance with an embodiment.
- FIG. 8A is a schematic plan view of camera views illustrating equal distribution of cameras around a circular constellation.
- FIG. 8B is a schematic plan view of camera views illustrating equal angular distribution of local cameras.
- FIG. 8C is a schematic plan view of camera views illustrating equal angular distribution of cameras that capture the view seen by each user's eyes.
- FIG. 8D is a table of camera and position angles for an example embodiment with six users and telepresence sites.
- FIG. 9A is a schematic plan view of camera views, where each camera represents a view seen by a user's eyes.
- FIG. 9B is a schematic perspective view of camera views with one camera on the circular geometric setup replaced by a user.
- FIG. 10 is a schematic plan view of camera views where each physical camera is replaced with a virtual camera.
- FIG. 11 is a flow chart illustrating a method performed in an exemplary embodiment.
- FIG. 12 is a flow chart illustrating a method performed in an exemplary embodiment.
- FIG. 13 is a message flow diagram illustrating information exchanged in an exemplary embodiment.
- FIG. 14 is a schematic plan view illustrating an exemplary virtual geometry.
- FIGS. 15A-15D are perspective views illustrating images captured from respective cameras in the virtual geometry of FIG. 14.
- FIG. 16 is a schematic illustration of a user-dependent panorama view constructed from the captured images illustrated in FIGS. 15B-15D.
- FIG. 17A is a system diagram illustrating an example communications system in which one or more disclosed embodiments may be implemented.
- FIG. 17B is a system diagram illustrating an example wireless transmit/receive unit (WTRU) that may be used within the communications system illustrated in FIG. 17A according to an embodiment.
- WTRU wireless transmit/receive unit
- FIG. 1 illustrates two issues that are addressed in some embodiments by the generation of a user-centric panorama video stream.
- the first issue is the use of camera-centric viewpoints (e.g., cameras on the wall 103 capture views 111 along lines of sight 109 from a set point 107) instead of user-centric viewpoints (e.g., a user 105 looking at a display with field of view 106).
- the second issue is the formation of the panorama from an arbitrary or even ambiguous eye-point (e.g., point 107).
- an environment with two users 125a, 125b
- an array of cameras on the wall 103 dashed line
- panorama appearance For simplicity, showing the panorama is seen from one participant's (105) viewpoint only. Furthermore, in current systems, due to using separate camera viewpoints, stitching sub-views into a panorama induces visible distortions on sub-view edges.
- One technique that may be used to improve panorama appearance is to employ a normalized, evenly colored background at each meeting room, which reduces discontinuity if bringing them side by side on a large screen. This works even if views from different meeting rooms are taken side by side.
- the views may be stitched into a large panorama.
- Generation of a panorama is more complicated if images of one physical space are collected using multiple spatially dispersed cameras, but various techniques have been developed for generating such a panorama.
- such a panorama may be generated using approximation, using the views from the spatially dispersed cameras.
- neighboring camera views are stitched into a panorama so that the distortions on image borders stay small enough. Discontinuities on sub-view edges may be reduced by various filtering and interpolation operations.
- the viewpoint of the resulting panorama may be an ambiguous average viewpoint chosen on a viewing line perpendicular to the panorama image's plane.
- forming a 3D reconstruction using different camera views is used to obtain the combined panorama view from a correct view-point, for example as described in T. Moons et al., 3D Reconstruction from Multiple Images Part 1: Principles, FOUNDATIONS AND TRENDS IN COMPUTER GRAPHICS AND VISION, Vol. 4, No. 4, 287-398 (2008).
- accuracy may be improved by hybrid embodiments using depth sensors in addition to visual cameras, as described in Tola et al., Virtual View Generation with a Hybrid Camera Array, CVLAB-REPORT-2009-001, EPFL (2009).
- a far enough viewpoint is chosen on a viewing line perpendicular to the panorama image's plane.
- techniques are used to compile a consistent panorama for a remote viewer in a specific location. Such embodiments are particularly useful in telepresence systems that provide various viewpoints to remote sites, e.g., for supporting spatial orientation.
- positions of the users are tracked using one or more known position-tracking techniques, and the positions of the users are brought into a common coordinate system, referred to as a common geometry.
- the formation of a 3D reconstruction from multiple camera and sensor views is one available technique to support formation of panoramas from desired viewpoints.
- Known panorama formation embodiments generally provide a (one) perspective approximation to each local site and its participants.
- the panorama is formed from a viewpoint which is typically further away than one or more system camera(s). Even though true eye- directions or eye-contact are not supported, there is less horizontal parallax distortion, and the result is more acceptable if viewed on a large display.
- panorama formation may be made either in a system front-end component, e.g., in the transmitting terminals or in the receiving devices, provided that the information for forming the panorama is received for this purpose.
- the chosen embodiment may have considerable effects to the required bitrate and other system properties.
- a user-centered perspective may be used to view an environment through a display with an attached camera and/or an attached 3D sensor, e.g., a device such as a tablet computer.
- an attached camera and/or an attached 3D sensor e.g., a device such as a tablet computer.
- a 3D sensor e.g., a device such as a tablet computer.
- One system for providing a user-centered perspective is described in Domagoj Bancevic, Cha Lee, Matthew Turk, Tobias Hollerer, Doug A. Bowman, A Hand-Held AR Magic Lens with User-Perspective Rendering, ISMAR 10 (2012).
- methods and systems described in Moons et al. may be used for forming a correct user's perspective.
- a depth sensor and user tracking may be used to generate the video from the user's viewpoint.
- a panorama of several camera views is created by, e.g., positioning tablets showing the video with corrected viewpoints side by side.
- Analogous embodiments may generate panoramas in remote settings where the viewer and camera are not physically in the same environment.
- a technique for panorama generation that may be employed in embodiments herein is described in Dou et al., Room-Sized Informal Telepresence System, IEEE VIRTUAL REALITY 4 (2012).
- a frontal image of a person is stitched into a consistent background panorama (for example, the person may be turned to appear more front-facing in the central panorama).
- the results may still be approximate. For example, if a user is looking towards the camera, all viewers may feel looked at (e.g., the Mona Lisa effect).
- Analogous technologies are described in US Patent Application 2012/0050458 (filed Aug. 31, 2010).
- a 3D reconstruction of each local site and its participants is formed, and the reconstruction is used for forming a virtual panorama perspective from an arbitrary viewpoint.
- Systems and methods that may be used for generating such 3D reconstructions are described, for example, in Zhang et al., Viewport: A Distributed, Immersive Teleconferencing System with Infrared Dot Pattern, ⁇ MULTIMEDIA, Vol. 20, No. 1, 17-27 (2013).
- all 3D information is captured and transmitted to a receiving terminal for deriving user-centered views or panoramas.
- an arbitrary viewpoint of a scene depicted by an array of cameras may be generated by forming a high-quality depth image of the scene using the array of cameras.
- LFR light field rendering
- One technique for the generation of a panorama view may be a variant of the light field rendering method, as described in Yang, et al., Creating Adaptive Views for Group Video Teleconferencing- An Image-Based Approach, INTERNATIONAL WORKSHOP ON IMMERSIVE TELEPRESENCE (2002).
- the panorama is collated from a set of vertical image slices received from an array of cameras.
- the panorama is generated using a collection of pixel columns from various camera views on each user centric line-of-sight.
- the generation of a view may be performed using a crossed slits projection technique as described in the Zomet et al., Mosaicing New Views: The Crossed-Slits Projection, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, Vol. 25, No. 6, 741-754 (2003).
- a horizontal camera array may provide only one fixed vertical viewpoint (and focal length), but in the horizontal direction a number of viewpoints may be provided (using a modified focal length).
- Some embodiments may make use of a hybrid sensor, such as by complementing video cameras with depth sensors. Such systems may improve results for producing 3D surfaces or light field renderings, and may also be used to improve the view-point calculation using techniques such as those described in Tola et al.
- the positions of a camera and a viewer are determined in order to calculate an accurate user perspective.
- the viewer and camera are placed in a virtual geometry that is shared by all the local and remote sites. Generation of such a shared virtual geometry is described in International Application No. PCT/US 16/46848, filed Aug. 12, 2016, entitled “System and Method for Augmented Reality Multi-View Telepresence," and U.S. Provisional Patent Application No. 62/357,060, filed June 30, 2016, entitled “System and Method for Spatial Interaction Using Automatically Positioned Cameras,” both of which are incorporated herein by reference in their entirety.
- Methods for representing 3D information may be adapted from 3DTV technologies and from game development technologies.
- Techniques for presenting 3D content with one monocular video and the respective depth map are available, although such techniques have problems, for example, in edge regions revealing content behind occluding objects.
- This representation is supported, for example, by the Kinect sensor.
- 3D formats may be used, including, for example, Dolby 3D, XpanD 3D, Panavision 3D, Masterlmage 3D, and ⁇ 3D. Better quality with more free view-point selection generally calls for more complex methods.
- H.264 MVC multiple view video coding
- the standard is used for coding stereoscopic 3D content.
- Exemplary methods that may be used include those described in Chen and Vetro, Next-Generation 3d Formats with Depth Map Support, IEEE MULTIMEDIA, Vol. 21, No. 2, 90-94 (2014).
- Exemplary embodiments described herein operate to form consistent panoramas with accurate individual viewpoints to one or more remote viewers. Disclosed embodiments further operate to tracking the users' positions and place them in a shared virtual geometry to provide for consistent tracking of user viewpoints.
- FIG. 2 illustrates an embodiment of generating a user-centric panorama in a meeting taking place in three locations (201, 202, 203) (although participants in location 203 are not illustrated), where user positions (such as those of user 225a and 225b at location 202) are tracked and combined in a unified geometry (coordinate system).
- the panorama e.g., views 207) from only one remote participant's (user 205) viewpoint is shown.
- the herein disclosed systems may be implemented so that the panorama (207) is formed locally (at the location of the cameras), and the derived views are sent to each remote receiver as a panorama video.
- a front-end system may omit transmitting excessive amounts of real-time 3D data to be used for deriving panoramas at each remote site.
- FIG. 3 An exemplary system architecture using components described herein is shown in FIG. 3.
- a system' s architecture may however be implemented in various ways, for example using client- side or server-based components.
- a user 303 at a first site is using a user terminal 335 to view video of a second user 305 at a second site.
- Communications between sites may occur over the internet and/or through servers 301.
- a telepresence system may include, but is not limited to, the following components.
- a sensor arrangement 325 is provided for supporting remote viewpoint selection.
- the sensor arrangement 325 may include, for example, a set of depth sensors and a set of 2D video cameras.
- the video cameras may be arranged in a horizontal array (e.g., a linear array).
- Software and hardware may be provided to track position of each participant, such as tracking systems 310 and 312.
- the software and hardware may operate to use the sensor arrangement 325 for remote view point selection. Determining the position of each participant may be performed, for example, based on tracking features in the videos filmed by the cameras in augmented reality (AR) glasses used by local users.
- AR augmented reality
- Software may be provided to form a shared virtual geometry 315 containing the participants with respect to the sensor arrangement.
- the software may further be operative to form a virtual panorama 330 from a remote user's viewpoint in a virtual geometry (which may be retrieved from a database for virtual geometry 320).
- Software is further provided that is capable of manipulating the panorama video 340, e.g., augmenting virtual objects to the video or changing background of the video (such as from a database of objects 345).
- Software and appropriate communication interface hardware is provided to transmit panorama video between the participant sites.
- Aterminal 335 may be provided for displaying the user-centric panorama.
- the terminal 335 may be, for example, augmented reality glasses (or other augmented reality display system, such as other head-mounted displays), used by the collaboration participants.
- the terminal 335 may be a multiview display, showing different view to each local participant.
- An audio system may be provided to produce and capture spatial audio so that remote participant's sound seems to be coming from direction of the corresponding video stream.
- a session manager may be provided for organization of sessions, sites and users.
- Exemplary embodiments of the disclosed systems and methods operate to support formation of a consistent panorama of each local meeting site from a freely selected viewpoint and to support meeting the natural focal length of a human observer. Exemplary embodiments avoid typical panorama distortions caused by stitching multiple uncalibrated camera views from arbitrary or ambiguous viewpoints.
- Exemplary systems and methods employ detection and tracking of user (e.g., head) positions during system use, such as by tracking modules 310 and 312. Any suitable method of position tracking may be used.
- user position is tracked by an embedded camera in AR glasses capturing graphical markers or other known features in the environment.
- the telepresence session virtual geometry may be defined for a telepresence session involving a local site having a local telepresence system and at least a first remote site having a first remote telepresence system.
- the viewpoint position of a first remote user at the first remote site may be determined at the remote telepresence system, with respect to the telepresence session virtual geometry, and communicated to the local telepresence system.
- the local telepresence system may capture video or image feeds of at least a first local user at the local site using a plurality of cameras or other image detection means.
- the local telepresence system may generate, from the plurality of captured video or image feeds, a video (such as a panorama video, but not necessarily) corresponding to the viewpoint position of the first remote user with respect to the telepresence session virtual geometry.
- the local telepresence system may then send or otherwise communicate the generated panorama video to the first remote telepresence system at the first remote site, for viewing by the first remote user (such as with a set of AR glasses, or other display system).
- the local telepresence system may communicate the captured video or image data to the remote telepresence system, which may then generate the appropriate panorama video.
- the local telepresence system may determine the viewpoint position of at least a first local user at the local site, and communicate this viewpoint position to one or more remote telepresence systems at one or more remote sites.
- These remote telepresence systems may use the local user's viewpoint position to generate videos or images (such as panorama videos, but not necessarily) which are then communicated back to the local telepresence system for display to the local user (such as with AR glasses, or some other display system).
- each meeting room participating in the session users select their respective positions (e.g., particular seats), which - after user positions are being detected - leaves a system with a mutual, virtual orientation of meeting sites.
- the local geometries are configured into a desired constellation, with the number of local geometries being the number of meeting sites.
- the number of sites (and people's locations in them) suggests a type of virtual geometry to be used, e.g., three sites with participants in a row suggest a triangular type of geometry. If the local participants are sitting more in an arc form, a circular type of virtual geometry may be selected.
- FIG. 4 an example of a shared geometry for a meeting between five participants 401a-e at three sites 403, 405, 407 is shown.
- an exemplary geometry optimization results in a virtual meeting geometry
- the participants 401a-e are in a circle form 410 (dashed circle).
- the room walls 415 with camera arrays are shown by dashed lines.
- the shaded triangle 420 in the middle is a virtual space between real physical spaces.
- view-points such as views 430 from each user to the users at different sites.
- the shared geometry is recalculated during an ordinary session management process, e.g., if more sites with user(s) enter or leave the meeting. This may be compared to a physical meeting, where people may relocate after changes in participation.
- the policy in triggering geometry changes may vary, as it may not be desirable to change the geometry for every move of individual participants.
- consistent high-quality panorama viewpoints are generated for a plurality of viewpoints based on the positions of remote viewers in the defined meeting geometry.
- FIG. 5 illustrates an example of a linear sensor arrangement to support each remote user centric view.
- a user 501 at a first site 505 may observe a view 503 that is captured by a plurality of sensors 510 at a second site 507.
- a linear array of cameras such as those described in PCT/US 16/46848, may be used in some embodiments.
- Another linear array of cameras that may be used in some embodiments is described in R. Yang, et al.
- Such linear camera arrays may be used to forming remote user centered panoramas based on either camera based 3D reconstruction or light-field-rendering. Note that a horizontal camera array provides only one fixed vertical viewpoint (and focal length), but in horizontal direction, a number of viewpoints is supported (using a modified focal length).
- a linear camera array may be described as an application of the crossed-slits projection described in Zomet et al.
- a two-dimensional light-field capture setup may be employed, such as an 8x8 array of sixty-four cameras, such as that described in J. Yang et al., A Real-Time Distributed Light Field Camera, EUROGRAPHICS WORKSHOP ON RENDERING 1-10 (2002).
- a two- dimensional light-field-camera may naturally provide symmetrical perspectives in horizontal and vertical directions.
- a linear camera array may be employed in some embodiments for light-field capture.
- the cameras may have a sufficiently high density, as known to those of skill in the art.
- rendering the remote user centered panorama may be done either at the local site (if the user position is made available) or optionally at the receiving site (if a complete 3D reconstruction or light-field is transmitted for this purpose, or all video views required for their construction). Rendering at the local site may result in significant savings in communication bandwidth, but may also result in additional latency caused by collection and transmission of information regarding remote user positions.
- a viewpoint from light-field Providing that the light-field camera captures the scene without disturbing occlusions, a panorama from the desired remote viewpoint may be formed using the known principles of light-field-rendering.
- Various coding methods and standards described herein may be used for coding and transmission of the information required for remote user-centered panorama.
- the chosen architecture affects to the choice, as the information may be sent as separate videos, multi-view video (such as H.264 MVC), depth plus video, mesh plus texture, light-field, and so on.
- multiple remote user centric panoramas are formed in a local front end and are transmitted using H.264 MVC type of coding method exploiting their remaining redundancies (due to views showing the same environment from slightly different viewpoints).
- an individualized panorama is displayed to each remote user.
- the display may be provided using Augmented Reality glasses or HMDs, among other options.
- the supported field-of-view of existing AR glasses is typically rather narrow.
- the glasses may display a correct sub-view of a wider panorama, according to the user's viewing direction.
- tracking of both user's position and viewing direction may be performed. This tracking may be implemented either by the HMD camera (e.g., by tracking markers or features) or by some other suitable tracking arrangement. Technologies such as those used by the Oculus Rift may be employed to support sub-view formation and position tracking.
- multi-view displays may be used, with the information provided to each viewing segment being changed according to users' movements in the space, including the users' distance from the display.
- Embodiments disclosed herein thus provide a more realistic viewpoint than a viewpoint that would result from merely rendering a constant 3D or stereoscopic information to each viewing segment.
- Multiscopic displays may serve multiple users, showing different content to different display directions. However, they do not typically support naturally changing parallax for a user moving further or closer to the display, inside his/her viewing segment. Embodiments disclosed herein control the multi-view content according to a user's distance as part of the user tracking process.
- Systems set forth herein may use an array of cameras (and/or other sensors) for capturing data for panoramas.
- the array may be positioned on the eye-level of the viewing participants. If using a hardware display, positioning of the array without blocking the display, or vice versa, may require additional effort.
- Imaging through a display has been used to solve the camera positioning problem (and the implied camera-display parallax problem).
- this has more recently been addressed by time-multiplexing an LCD screen between display and see-through modes and/or by integrating a matrix of cameras inside the display.
- Techniques that may be used in embodiments disclosed herein include techniques described in US Patent Application 2014/0146127 (Filed Nov. 29, 2012) and US Patent 7,808,540 (Filed Jan. 11, 2007).
- an analogous imaging system is implemented through an auto- multiscopic display. Particularly, this may be done using a display based on parallax barriers instead of lenticular lenses.
- BiDi for bi-directional
- image capturing display as described in Hirsch et al., BiDi Screen: A Thin, Depth-Sensing LCD for 3D Interaction Using Light Fields, ACM TRANSACTIONS ON GRAPHICS (TOG), Vol. 28, No. 5, 9 (2009).
- a dense matrix of camera elements is integrated into the display for the light field capture.
- the size of the display may be selected in order to support the defined (wide) geometry; the display orientation with respect to remote user(s) is determined (e.g., for showing content in the correct viewing segment).
- the view may be restricted if the hardware display is too narrow to support the full geometry.
- the properties (size and type) and pose of each display in the unified meeting geometry are determined.
- the system may then serve users with correct content and lines-of-sight.
- FIG. 6 illustrates an embodiment in which restrictions to the visibility of users (605, 606, 607) are caused either by the span of the camera array 610 or the display 615.
- the situation is shown in meeting in three locations 601, 602, 603 (although participants at location 603 are not illustrated). Users 605 and 606 do not see each other without changing their positions.
- a respective restriction may happen if a camera array 610 is too short for the used geometry. This situation may be compared to a physical window between two rooms, where seeing users in the other room depends on the viewer's position and the window size. Differing from a physical window, viewing restrictions in the disclosed system may also be asymmetrical.
- a system may also make use of information regarding the span of the camera array 610 in use. In cases where a camera array 610 cannot support lines-of-sight to all remote participants (analogous to a case where a window is too small), panorama views may be cropped respectively.
- the properties and pose of each display are included in "user terminal parameters" transmitted to a system front-end for panorama formation 330.
- the properties and pose of each camera/sensor array 325 are included in "sensor positions and parameters" used by a system in forming a unified meeting geometry 315 and in a system front- end for panorama formation 330.
- the span of both the display and camera setups may be taken into consideration whether the panorama display is based on AR glasses or flat panels.
- panorama formation has the basic benefit of more complete and consistent perception of the received far-end view.
- panorama formation operates to reduce redundancy and bitrate for transmitting remote user-centric views. Redundancy is reduced whether the panorama is formed by making 3D reconstruction from multi-camera views, or by using light-field-rendering techniques. In the latter, however, the amount of redundancy is considerably large and so is the benefit of forming the panorama at a system front-end.
- FIG. 7 instead of a combined panorama, separate user-centric camera views may be, in effect, brought to a remote user's eye-point.
- this may be implemented by the same components and procedures as any of the above discussed embodiments using panorama formation.
- the sub-views 705 may be taken along the line-of-sight of a remote viewer 701, and be made correct from the viewer's perspective, rather than being camera centric (as in FIG. 1).
- This variation may be preferred by a user in some circumstances.
- the remote user may not want to cover his/her view with large panoramas (e.g., due to using a small display, or wanting to save display space for showing some other information).
- the local user may want to protect his/her privacy by restricting the view.
- a system may support users viewing either sub-views or panoramas according to their preferences.
- FIGS. 8-10 Additional embodiments are discussed in relation to FIGS. 8-10.
- One embodiment uses a circular meeting table with participants placed uniformly around the table.
- the table is a virtual, circular table.
- users may see each other in a spatially faithful way, where users and cameras are placed in a unified meeting setting with a global coordinate system.
- Cameras are remote users' proxies (eyes) in each local user's environment. Each view is rendered respectively in these proxy eye positions, resulting in natural eye-contact and directions between users.
- FIGS. 8 A to 8C show one embodiment with a spatially-oriented telepresence between multiple persons in a circular constellation.
- the person on the lower left of FIG. 8 A may be the local participant 801 and may see five remote participants 805a-e distributed spatially around him/her.
- FIG. 8 A shows remote users 805a-e seen by a local participant 801 in a circular constellation 810 (dashed circle).
- a local participant 801 is shown at the lower left of FIG. 8A.
- remote users 805a-e are distributed around a virtual circle 810, where each of the remote users 805a-e is displayed locally looking at the local participant 801.
- the solid, straight lines 815 that go from a remote user (as rendered locally) to the local participant 801 show line-of-sight lines reflecting the view that remote users 805a-e would have if those remote users 805a-e were present locally.
- FIG. 8B depicts an exemplary scenario in which remote users occupy equally-sized segments of 30 degrees. The example of FIG.
- FIG. 8B shows six users distributed 30° apart.
- FIG. 8B is similar to FIG. 8A, with the addition of the example 30° angular distribution.
- the equal angular distributions for each wedge segment are distributed evenly on either side of a line-of-sight between the local participant 801 in the lower left of FIG. 8B and each remote user 805a-e as rendered locally.
- two smaller wedges, each 15° wide are shown between a line 820 perpendicular to the local participant 801 and an edge of one of the outermost 30° wedges.
- FIG. 8C shows an example respective camera capture setup.
- Each of the six users may be captured by a capture setup as depicted in FIG. 8C.
- video data is captured for each user from five different viewpoints.
- FIG. 8C depicting a single user 801 of the six users, each user is captured from 30°, 60°, 90°, 120°, and 150°.
- the field-of-view for each camera/user 806a-e may depend on the maximum number of remote participants.
- the focal length for each camera 806a-e is set the same as a human eye, which means cameras with an "normal" objective (e.g., 50 mm for a full-frame, sensor-equipped DSLR camera) are used.
- the diameter of the capture setup affects the distances of users at the virtual table in the global coordinate system.
- the diameter of the capture circle 810 may be selected based on personal and cultural preferences. In order for the users to have the same perception of distances, the diameter should be the same for all user environments. For example, if the main purpose of a system is personal communication, the diameter may be relatively small (for example, 1.5 meters).
- the diameter of the circle 810 may be used to create a virtual table.
- FIG. 8D shows a table of user positions and camera angles for an example set of six positions.
- a local user is in position 1 and remote users are in positions 2 to 6.
- user A is in position 1 and user B is in position 2.
- This pattern is repeated for the rest of the remote users, with user F in position 6 if user A is a local user.
- this paradigm is repeated if user B is a local user, with user B in position 1 and user C in position 2. Again, this patter is repeated for the rest of the remote users, with user F in position 5 and user A in position 6.
- these patterns are repeated if any of the other users is a local user, as shown in the table.
- FIG. 9A shows an embodiment of camera capture locations for a circular table geometry 900.
- the example capture set-up uses a set of cameras 905 in a circular arrangement, with equal distances in between each camera 905.
- a local user 901 replaces one of the cameras in the circle 900.
- each user is distributed an equal angle from each other, measured from the center of the circle.
- each user is separated by 60° as measured from the center of the circle. In other embodiments, any number of users may be evenly distributed around a circle.
- FIG. 9B shows an embodiment where each camera 905 represents a user's eyes.
- a number of cameras are chosen to support a predefined number of users, but the number of users supported may reach a maximum number, above which, the field-of-view for each camera becomes too narrow.
- the embodiment of FIG. 9B shows a camera setup that captures video data of a user 901 from five different locations.
- the number of cameras used is N-l, where N is the number of total users.
- FIG. 10 shows a camera setup using virtual cameras.
- a virtual rendering threshold physical cameras may be replaced with virtual cameras.
- a three-dimensional (3D) reconstruction model may be captured for a user's environment 1000 (or a local telepresence site), and virtual viewpoints 1008 (or virtual camera video streams) may be calculated for locations of virtual cameras 1010.
- depth sensors are calibrated and used to make 3D depth measurements of a user's environment.
- four depth sensors 1005 are shown to create a 3D model of the environment 1000.
- depth sensors 1005 are shown, with two depth sensors 1005 on the top side wall 1020 and two depth sensors 1005 on the right side wall 1022. Other embodiments may have more or fewer depth sensors 1005.
- virtual camera viewpoints 1008 may be calculated from the 3D model for each of the plurality of virtual cameras 1010.
- Physical cameras may be replaced with virtual cameras (such as by using a calibrated set of depth sensors). Even though physical cameras may be small, such cameras (along with their supporting frames) occupy and occlude part of the meeting space. Virtual camera views allow the number and positioning of remote users (or cameras) in each local space to be flexible.
- FIG. 11 is a flow chart illustrating steps in an exemplary embodiment for defining the virtual meeting geometry in embodiments used herein.
- FIG. 12 is a flow chart (continuing from FIG. 11) illustrating steps in an exemplary embodiment for providing each remote user with a user-centered panorama including augmented objects (if any).
- a user's position may be tracked relative to their camera setup, and an AR object's position (for example, given by a user) may be read or otherwise obtained. These positions may be stored until all sites, users, and AR objects have been analyzed. Using the stored positions, a geometry for each meeting site may be formed. Such geometry may consider display and sensor setup parameters at each site.
- a user centered panorama may be formed. This may comprise capturing videos of each user, and using the geometry of each meeting site to form the remote user centered panorama. Also, the user centered panorama may be augmented with any AR objects (if any) that were added by a user. The created (and possibly augmented) user centered panorama may then be transmitted to other users and displayed, in some cases in consideration of the display setup parameters (and in some cases sensor setup parameters) for each receiving user (e.g., type, size, etc.). Panoramas may be created and transmitted until all sites and all users have been processed. The panoramas may be maintained until a change in the meeting setup is triggered (e.g., a user leaves the meeting, etc.). If a change is triggered, if the meeting has not ended, the meeting site geometry may be accordingly updated, and new user centered panoramas may be formed.
- FIG. 13 is a message flow diagram illustrating a process performed in an exemplary embodiment.
- a local and remote conferencing system may communicate to calibrate a conference makeup, e.g., determining a virtual geometry and locations of the users in the virtual geometry (such as based at least in part on the display and/or sensor setup parameters at each site or system).
- the remote conferencing system sends to the local conferencing system information on its properties (e.g., display/sensor type, size, and position).
- the remote conferencing system also determines the location of the remote user (e.g., the position of the remote user's head or eyes) and conveys that location information to the local conferencing system. The location may be determined by, for example, a camera at the remote conference site.
- the location information may be conveyed in a variety of formats such as, for example, as coordinates in a common coordinate system of the virtual geometry.
- An array of local cameras at the local conference site collect video feeds of a local participant. Based on these video feeds, the local conferencing system constructs a panorama view representing the viewpoint of the remote user. The local conferencing system supplies a video feed of the constructed panorama view to the remote conferencing system for display on the remote display.
- FIGS. 14-16 One exemplary method for constructing a panorama video based on a viewpoint of a remote user is described with respect to FIGS. 14-16.
- FIG. 14 involves only two sites— a single local site 1401 and a single remote site 1403— although the principles disclosed herein may be implemented in circumstances with different numbers of sites.
- components are shown only for displaying a local site 1401 to a remote user 1425 (e.g., cameras A-E of camera array 1420 at the local site 1401 and a display 1430 at the remote site 1403).
- Components for displaying a remote site 1403 to a local user may be employed in embodiments but are not illustrated in FIG. 14.
- the display 1430 at the remote site 1403 is depicted as a wall-mounted display screen, although other display types as described herein may alternatively be used.
- the local site 1401 and remote site 1403 are arranged in a virtual geometry that simulates a face-to-face meeting. For ease of illustration of parallax and foreshortening effects (discussed with respect to FIGS. 15 and 16), an empty chair is depicted in the position of the local user, though the same effects would be apparent (but more difficult to illustrate) in images of a human user.
- the position of the remote user 1425 (e.g., the position of the remote user's head) is determined using position sensors 1435 at the remote site 1403.
- the position sensors 1435 may be, for example one or more of (or a combination of) stereoscopic cameras, depth cameras, and the like.
- the position of the remote user 1425 is conveyed to the conferencing system at the local site 1401.
- the position, type, and orientation of the remote display 1430 may also be conveyed to the local site 1401.
- the local site 1401 operates to generate a panorama video of the local site 1401, where the panorama video is generated so as to be appropriate to the viewpoint of the remote user 1425.
- the local system may use any of the techniques disclosed herein or other techniques known to those of skill in the art. One such technique is discussed with reference to FIGS. 15 and 16.
- camera C of the local camera array 1420 falls directly on the virtual line of sight between the position of the remote user 1425 and the chair 1410 of the local user.
- camera C is located much nearer to the chair 1410 at the local site 1401 than the viewpoint of the local user (e.g., in the virtual geometry of FIG. 14, the remote user 1425 is approximately twice the distance from the chair 1410 as camera C).
- a video taken by camera C may be taken from the correct angle but would demonstrate substantial foreshortening (as represented in FIG. 15C). This level of foreshortening may appear unrealistic to the remote user 1425.
- the local system may generate a panorama by stitching together, for example, images from cameras B, C, and D of the camera array 1420.
- camera B is the camera that most accurately corresponds to the remote user's viewpoint of the left side of the chair 1410
- camera C is the camera that most accurately corresponds to the remote user's viewpoint of the center of the chair 1410
- camera D is the camera that most accurately corresponds to the remote user's viewpoint of the right side of the chair 1410.
- FIGS. 15A-15D illustrate the views captured by these cameras, with FIG. 15A representing the view captured by camera A, FIG. 15B representing the view captured by camera B, FIG. 15C representing the view captured by camera C, and FIG. 15D representing the view captured by camera D.
- selected segments of the images from cameras B, C, and D are stitched together to construct a panorama view of the chair that is appropriate to the viewpoint of the remote user.
- the view of the chair displays much less foreshortening (as compared to the raw camera image from camera C, depicted in FIG. 15C) and thus may appear more realistic (and more consistent with the virtual geometry) to the remote user.
- the local conferencing system instead of streaming video from all cameras in the local camera array to the remote system—which may consume substantial bandwidth— the local conferencing system streams to the remote system only the constructed video corresponding to the user-centric panorama view.
- FIGS. 14-16 The example described with respect to FIGS. 14-16 has been selected for clarity of illustration and that alternate techniques may be employed for generating a panorama video with a user-dependent viewpoint, including other techniques described herein.
- panorama and its variants as used herein is not limited to wide-angle views but rather refers to a view constructed through a combination of separately-captured views (which may include depth-image views), although such constructed panorama views may indeed be wide- angle views in some embodiments.
- modules include hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits
- Each described module may also include instructions executable for carrying out the one or more functions described as being carried out by the respective module, and it is noted that those instructions may take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and/or the like, and may be stored in any suitable non-transitory computer-readable medium or media, such as commonly referred to as RAM, ROM, etc.
- there is telepresence method comprising, at a local telepresence site: providing a plurality of cameras, wherein the cameras and a local user at the local telepresence site are collectively arranged in a substantially circular configuration; and from each of the plurality of cameras, streaming a view of the local user to a different respective remote user.
- the method may further include receiving views of the respective remote users and displaying those views to the local user, each received view being displayed at a position corresponding to the camera associated with the respective remote user.
- the method may include wherein the displayed views are displayed to the local user using an augmented reality headset.
- the method may include wherein the views of the respective remote users are stitched into a panoramic view.
- the method may include wherein the plurality of cameras includes at least four cameras.
- the method may include wherein the cameras and the local user are substantially evenly spaced around the substantially circular configuration.
- a telepresence method comprising, at a local telepresence site: capturing video of a local user at the local telepresence site using at least two cameras; using the captured video, generating a real-time 3D model of the local user; based on the real-time 3D model, generating a plurality of virtual views of the local user, each virtual view being generated with respect to a virtual viewpoint, wherein the virtual viewpoints and the local user are collectively arranged in a substantially circular configuration; and streaming each of the virtual views to a different respective remote user.
- the method may further include receiving views of the respective remote users and displaying those views to the local user, each received view being displayed at a position corresponding to the virtual viewpoint associated with the respective remote user.
- the method may include wherein the displayed views are displayed to the local user using an augmented reality headset.
- the method may include wherein the views of the respective remote users are stitched into a panoramic view.
- the method may include wherein the plurality of virtual viewpoints includes at least four virtual viewpoints.
- the method may include wherein the virtual viewpoints and the local user are substantially evenly spaced around the substantially circular configuration.
- the method may include wherein at least one of the cameras is a depth camera.
- a method of providing a teleconference showing remote meeting sites from a user-dependent viewpoint comprising: determining a telepresence session virtual geometry for at least a first site and a second site; determining, by a first telepresence system at the first site, the viewpoint of the first telepresence participant at the first site with respect to the telepresence session virtual geometry; providing the determined viewpoint to a second telepresence system at a second site; capturing, using a plurality of cameras at the second site, a plurality of video feeds; rendering, from the plurality of video feeds captured by the plurality of cameras at the second site, a panorama video corresponding to the received viewpoint with respect to the telepresence session virtual geometry of the first telepresence participant at the first site; and sending the rendered panorama video to the first telepresence system at the first site.
- the method may include wherein properties of the display pose, sensor setup, and geometry of a remote user are signaled to the local conferencing system.
- the method may include wherein details of the display pose, sensor setup, geometry, position of a first remote user, and meeting geometry are used to produce a virtual panorama image of the local scene for the first far end user.
- a telepresence method comprising: receiving, at a first telepresence site, information representing a head position of a remote user at a second telepresence site; simultaneously capturing a plurality of video streams of the first telepresence site; combining the captured video streams into a panorama video corresponding to a virtual viewpoint based on the head position of the remote user; and streaming the panorama video to the second telepresence site.
- the method may further include receiving, at the first telepresence site, information representing a configuration of a display at the second telepresence site, wherein the panorama video is generated based at least on part on the configuration of the display and sensor setup (of any or all of the telepresence sites).
- a telepresence method comprising: receiving, at a local telepresence site, information representing a first head position of a first remote user at a first remote telepresence site; receiving, at the local telepresence site, information representing a second head position of a second remote user at a second remote telepresence site; simultaneously capturing a plurality of video streams of the local site; combining the captured video streams into a first panorama video corresponding to a first virtual viewpoint based on the head position of the first remote user; streaming the first panorama video to the first remote site; combining the captured video streams into a second panorama video corresponding to a second virtual viewpoint based on the head position of the second remote user; and streaming the second panorama video to the second remote site.
- the method may further include receiving, at the local telepresence site, information representing a configuration of a first display at the first remote telepresence site, wherein the first panorama video is generated based at least on part on the configuration of the first display; and receiving, at the local telepresence site, information representing a configuration of a second display at the second remote telepresence site, wherein the second panorama video is generated based at least on part on the configuration of the second display.
- a system for providing a teleconference showing remote meeting sites from a user-dependent viewpoint comprising a processor and a non- transitory computer-readable medium storing instructions operative, when executed on the processor, to perform functions comprising: determining a telepresence session virtual geometry for at least a first site and a second site; determining, by a first telepresence system at the first site, the viewpoint of the first telepresence participant at the first site with respect to the telepresence session virtual geometry; providing the determined viewpoint to a second telepresence system at a second site; capturing, using a plurality of cameras at the second site, a plurality of video feeds; rendering, from the plurality of video feeds captured by the plurality of cameras at the second site, a panorama video corresponding to the received viewpoint with respect to the telepresence session virtual geometry of the first telepresence participant at the first site; and sending the rendered panorama video to the first telepresence system at the first site.
- a telepresence system comprising a processor and a non- transitory computer-readable medium storing instructions operative, when executed on the processor, to perform functions comprising: receiving, at a first telepresence site, information representing a head position of a remote user at a second telepresence site; simultaneously capturing a plurality of video streams of the first telepresence site; combining the captured video streams into a panorama video corresponding to a virtual viewpoint based on the head position of the remote user; and streaming the panorama video to the second telepresence site.
- a telepresence system comprising a processor and a non- transitory computer-readable medium storing instructions operative, when executed on the processor, to perform functions comprising: receiving, at a local telepresence site, information representing a first head position of a first remote user at a first remote telepresence site; receiving, at the local telepresence site, information representing a second head position of a second remote user at a second remote telepresence site; simultaneously capturing a plurality of video streams of the local site; combining the captured video streams into a first panorama video corresponding to a first virtual viewpoint based on the head position of the first remote user; streaming the first panorama video to the first remote site; combining the captured video streams into a second panorama video corresponding to a second virtual viewpoint based on the head position of the second remote user; and streaming the second panorama video to the second remote site.
- there is a method comprising: receiving, at a local telepresence site, information representing head positions of a plurality of remote users at remote telepresence sites; calculating a plurality of rendering viewpoints for each of the plurality of remote users based on the head positions of the plurality of remote users; transmitting, to the plurality of remote telepresence sites, the plurality of rendering viewpoints for each of the plurality of remote users; receiving, at the local telepresence site, a plurality of video streams captured from the rendering viewpoint for each remote user; capturing, simultaneously, a plurality of video streams of the local telepresence site for each of a plurality of rendering viewpoints; streaming to the plurality of remote users a video stream of the local telepresence site captured from the respective rendering viewpoint for each of the plurality of remote users; combining the plurality of received video streams into a combined panorama video for the local telepresence site; and displaying the combined panorama video at the local telepresence site.
- the method may include wherein the rendering viewpoints for each of the plurality of remote users are calculated for an equal angle distribution.
- the method may further include determining to calculate a video stream for each of a plurality of rendering viewpoints when the plurality of remote users exceeds a virtual rendering threshold; capturing three-dimensional (3D) image data of the local telepresence site; calculating, simultaneously, a video stream of the local telepresence site for each of a plurality of rendering viewpoints; and streaming to the plurality of remote users a video stream of the local telepresence site calculated for the respective rendering viewpoint for each of the plurality of remote users.
- 3D three-dimensional
- a method of providing a teleconference showing remote meeting sites from a user-dependent viewpoint comprising: defining a telepresence session virtual geometry for a telepresence session involving a local site and at least a first remote site; determining a viewpoint position of a local user at the local site; providing the local user's viewpoint position to the first remote site; capturing video feeds from a plurality of cameras at the first remote site; generating from the video feeds a panorama video corresponding to the viewpoint position of the local user; and sending the panorama video to the local site for viewing by the local user.
- Exemplary embodiments disclosed herein are implemented using one or more wired and/or wireless network nodes, such as a wireless transmit/receive unit (WTRU) or other network entity.
- WTRU wireless transmit/receive unit
- FIG. 17A is a diagram illustrating an example communications system 1700 in which one or more disclosed embodiments may be implemented.
- the communications system 1700 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users.
- the communications system 1700 may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth.
- the communications systems 1700 may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single- carrier FDMA (SC-FDMA), zero-tail unique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block-filtered OFDM, filter bank multicarrier (FBMC), and the like.
- CDMA code division multiple access
- TDMA time division multiple access
- FDMA frequency division multiple access
- OFDMA orthogonal FDMA
- SC-FDMA single- carrier FDMA
- ZT UW DTS-s OFDM zero-tail unique-word DFT-Spread OFDM
- UW-OFDM unique word OFDM
- FBMC filter bank multicarrier
- the communications system 1700 may include wireless transmit/receive units (WTRUs) 1702a, 1702b, 1702c, 1702d, a RAN 1704, a CN 1706, a public switched telephone network (PSTN) 1708, the Internet 1710, and other networks 1712, though it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and/or network elements.
- WTRUs 1702a, 1702b, 1702c, 1702d may be any type of device configured to operate and/or communicate in a wireless environment.
- the WTRUs 1702a, 1702b, 1702c, 1702d may be configured to transmit and/or receive wireless signals and may include a user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription- based unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smartphone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (IoT) device, a watch or other wearable, a head-mounted display (HMD), a vehicle, a drone, a medical device and applications (e.g., remote surgery), an industrial device and applications (e.g., a robot and/or other wireless devices operating in an industrial and/or an automated processing chain contexts), a consumer electronics device, a device operating on commercial and/or industrial wireless networks, and the like.
- UE user equipment
- PDA personal digital assistant
- smartphone a laptop
- a netbook a personal
- the communications systems 1700 may also include a base station 1714a and/or a base station 1714b.
- Each of the base stations 1714a, 1714b may be any type of device configured to wirelessly interface with at least one of the WTRUs 1702a, 1702b, 1702c, 1702d to facilitate access to one or more communication networks, such as the CN 1706, the Internet 1710, and/or the other networks 1712.
- the base stations 1714a, 1714b may be a base transceiver station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a gNB, a NR NodeB, a site controller, an access point (AP), a wireless router, and the like. While the base stations 1714a, 1714b are each depicted as a single element, it will be appreciated that the base stations 1714a, 1714b may include any number of interconnected base stations and/or network elements.
- the base station 1714a may be part of the RAN 1704, which may also include other base stations and/or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc.
- BSC base station controller
- RNC radio network controller
- the base station 1714a and/or the base station 1714b may be configured to transmit and/or receive wireless signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum.
- a cell may provide coverage for a wireless service to a specific geographical area that may be relatively fixed or that may change over time.
- the cell may further be divided into cell sectors. For example, the cell associated with the base station 1714a may be divided into three sectors.
- the base station 1714a may include three transceivers, i.e., one for each sector of the cell.
- the base station 1714a may employ multiple-input multiple output (MTMO) technology and may utilize multiple transceivers for each sector of the cell.
- MTMO multiple-input multiple output
- beamforming may be used to transmit and/or receive signals in desired spatial directions.
- the base stations 1714a, 1714b may communicate with one or more of the WTRUs 1702a, 1702b, 1702c, 1702d over an air interface 1716, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (TR), ultraviolet (UV), visible light, etc.).
- the air interface 1716 may be established using any suitable radio access technology (RAT).
- RAT radio access technology
- the communications system 1700 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like.
- the base station 1714a in the RAN 1704 and the WTRUs 1702a, 1702b, 1702c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 1716 using wideband CDMA (WCDMA).
- WCDMA may include communication protocols such as High-Speed Packet Access (HSPA) and/or Evolved HSPA (HSPA+).
- HSPA may include High-Speed Downlink (DL) Packet Access (HSDPA) and/or High-Speed UL Packet Access (HSUPA).
- the base station 1714a and the WTRUs 1702a, 1702b, 1702c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 1716 using Long Term Evolution (LTE) and/or LTE- Advanced (LTE- A) and/or LTE- Advanced Pro (LTE-A Pro).
- E-UTRA Evolved UMTS Terrestrial Radio Access
- LTE Long Term Evolution
- LTE- A LTE- Advanced
- LTE-A Pro LTE- Advanced Pro
- the base station 1714a and the WTRUs 1702a, 1702b, 1702c may implement a radio technology such as NR Radio Access, which may establish the air interface 1716 using New Radio (NR).
- a radio technology such as NR Radio Access, which may establish the air interface 1716 using New Radio (NR).
- the base station 1714a and the WTRUs 1702a, 1702b, 1702c may implement multiple radio access technologies.
- the base station 1714a and the WTRUs 1702a, 1702b, 1702c may implement LTE radio access and NR radio access together, for instance using dual connectivity (DC) principles.
- DC dual connectivity
- the air interface utilized by WTRUs 1702a, 1702b, 1702c may be characterized by multiple types of radio access technologies and/or transmissions sent to/from multiple types of base stations (e.g., a eNB and a gNB).
- the base station 1714a and the WTRUs 1702a, 1702b, 1702c may implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 IX, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS- 95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and the like.
- IEEE 802.11 i.e., Wireless Fidelity (WiFi)
- IEEE 802.16 i.e., Worldwide Interoperability for Microwave Access (WiMAX)
- CDMA2000, CDMA2000 IX, CDMA2000 EV-DO Code Division Multiple Access 2000
- IS-95 IS- 95
- IS-856 Interim Standard 856
- GSM Global System for Mobile
- the base station 1714b in FIG. 17A may be a wireless router, Home Node B, Home eNode B, or access point, for example, and may utilize any suitable RAT for facilitating wireless connectivity in a localized area, such as a place of business, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a roadway, and the like.
- the base station 1714b and the WTRUs 1702c, 1702d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN).
- WLAN wireless local area network
- the base station 1714b and the WTRUs 1702c, 1702d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN).
- the base station 1714b and the WTRUs 1702c, 1702d may utilize a cellular- based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR etc.) to establish a picocell or femtocell.
- the base station 1714b may have a direct connection to the Internet 1710.
- the base station 1714b may not be required to access the Internet 1710 via the CN 1706.
- the RAN 1704 may be in communication with the CN 1706, which may be any type of network configured to provide voice, data, applications, and/or voice over internet protocol (VoIP) services to one or more of the WTRUs 1702a, 1702b, 1702c, 1702d.
- the data may have varying quality of service (QoS) requirements, such as differing throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, and the like.
- QoS quality of service
- the CN 1706 may provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, etc., and/or perform high-level security functions, such as user authentication.
- the RAN 1704 and/or the CN 1706 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 1704 or a different RAT.
- the CN 1706 may also be in communication with another RAN (not shown) employing a GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.
- the CN 1706 may also serve as a gateway for the WTRUs 1702a, 1702b, 1702c, 1702d to access the PSTN 1708, the Internet 1710, and/or the other networks 1712.
- the PSTN 1708 may include circuit-switched telephone networks that provide plain old telephone service (POTS).
- POTS plain old telephone service
- the Internet 1710 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the transmission control protocol (TCP), user datagram protocol (UDP) and/or the internet protocol (IP) in the TCP/IP internet protocol suite.
- TCP transmission control protocol
- UDP user datagram protocol
- IP internet protocol
- the networks 1712 may include wired and/or wireless communications networks owned and/or operated by other service providers.
- the networks 1712 may include another CN connected to one or more RANs, which may employ the same RAT as the RAN 1704 or a different RAT.
- Some or all of the WTRUs 1702a, 1702b, 1702c, 1702d in the communications system 1700 may include multi-mode capabilities (e.g., the WTRUs 1702a, 1702b, 1702c, 1702d may include multiple transceivers for communicating with different wireless networks over different wireless links).
- the WTRU 1702c shown in FIG. 17A may be configured to communicate with the base station 1714a, which may employ a cellular-based radio technology, and with the base station 1714b, which may employ an IEEE 802 radio technology.
- FIG. 17B is a system diagram illustrating an example WTRU 1202.
- the WTRU 1202 may include a processor 1718, a transceiver 1720, a transmit/receive element 1722, a speaker/microphone 1724, a keypad 1726, a display/touchpad 1728, nonremovable memory 1730, removable memory 1732, a power source 1734, a global positioning system (GPS) chipset 1736, and/or other peripherals 1738, among others.
- GPS global positioning system
- the WTRU 1702 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.
- the processor 1718 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like.
- the processor 1718 may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the WTRU 1702 to operate in a wireless environment.
- the processor 1718 may be coupled to the transceiver 1720, which may be coupled to the transmit/receive element 1722.
- the transmit/receive element 1722 may be configured to transmit signals to, or receive signals from, a base station (e.g., the base station 1714a) over the air interface 1716.
- a base station e.g., the base station 1714a
- the transmit/receive element 1722 may be an antenna configured to transmit and/or receive RF signals.
- the transmit/receive element 1722 may be an emitter/detector configured to transmit and/or receive IR, UV, or visible light signals, for example.
- the transmit/receive element 1722 may be configured to transmit and/or receive both RF and light signals. It will be appreciated that the transmit/receive element 1722 may be configured to transmit and/or receive any combination of wireless signals.
- the WTRU 1702 may include any number of transmit/receive elements 1722. More specifically, the WTRU 1702 may employ MTMO technology. Thus, in one embodiment, the WTRU 1702 may include two or more transmit/receive elements 1722 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 1716.
- the WTRU 1702 may include two or more transmit/receive elements 1722 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 1716.
- the transceiver 1720 may be configured to modulate the signals that are to be transmitted by the transmit/receive element 1722 and to demodulate the signals that are received by the transmit/receive element 1722.
- the WTRU 1702 may have multi-mode capabilities.
- the transceiver 1720 may include multiple transceivers for enabling the WTRU 1702 to communicate via multiple RATs, such as R and IEEE 802.11, for example.
- the processor 1718 of the WTRU 1702 may be coupled to, and may receive user input data from, the speaker/microphone 1724, the keypad 1726, and/or the display/touchpad 1728 (e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit).
- the processor 1718 may also output user data to the speaker/microphone 1724, the keypad 1726, and/or the display/touchpad 1728.
- the processor 1718 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 1730 and/or the removable memory 1732.
- the non-removable memory 1730 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device.
- the removable memory 1732 may include a subscriber identity module (SFM) card, a memory stick, a secure digital (SD) memory card, and the like.
- the processor 1718 may access information from, and store data in, memory that is not physically located on the WTRU 1702, such as on a server or a home computer (not shown).
- the processor 1718 may receive power from the power source 1734, and may be configured to distribute and/or control the power to the other components in the WTRU 1702.
- the power source 1734 may be any suitable device for powering the WTRU 1702.
- the power source 1734 may include one or more dry cell batteries (e.g., nickel -cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.
- the processor 1718 may also be coupled to the GPS chipset 1736, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 1702.
- location information e.g., longitude and latitude
- the WTRU 1702 may receive location information over the air interface 1716 from a base station (e.g., base stations 1714a, 1714b) and/or determine its location based on the timing of the signals being received from two or more nearby base stations. It will be appreciated that the WTRU 1702 may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment.
- the processor 1718 may further be coupled to other peripherals 1738, which may include one or more software and/or hardware modules that provide additional features, functionality and/or wired or wireless connectivity.
- the peripherals 1738 may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photographs and/or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a Virtual Reality and/or Augmented Reality (VR/AR) device, an activity tracker, and the like.
- FM frequency modulated
- the peripherals 1738 may include one or more sensors, the sensors may be one or more of a gyroscope, an accelerometer, a hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and/or a humidity sensor.
- a gyroscope an accelerometer, a hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and/or a humidity sensor.
- the WTRU 1702 may include a full duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for both the UL (e.g., for transmission) and downlink (e.g., for reception) may be concurrent and/or simultaneous.
- the full duplex radio may include an interference management unit 1739 to reduce and or substantially eliminate self-interference via either hardware (e.g., a choke) or signal processing via a processor (e.g., a separate processor (not shown) or via processor 1718).
- the WRTU 1702 may include a half-duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for either the UL (e.g., for transmission) or the downlink (e.g., for reception)).
- a half-duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for either the UL (e.g., for transmission) or the downlink (e.g., for reception)).
- ROM read only memory
- RAM random access memory
- register cache memory
- semiconductor memory devices magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD- ROM disks, and digital versatile disks (DVDs).
- a processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
Telepresence sessions may be used which utilize user-centric views of local and remote users. A virtual geometry is defined for two or more telepresence sites. A first telepresence site receives information representing a viewpoint position of a remote user at a second telepresence site, where the viewpoint position may be expressed as a position in the virtual geometry. Cameras (at least some of which may be depth cameras) at the first telepresence site simultaneously capture a plurality of video streams of the first telepresence site. At the first telepresence site, the captured video streams are combined into a panorama video corresponding to a virtual viewpoint based on the viewpoint position of the remote user, and the panorama video is streamed to the second telepresence site.
Description
METHOD AND SYSTEM FOR SHOWING REMOTE MEETING SITES FROM USER- DEPENDENT VIEWPOINTS
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] The present application is a non-provisional filing of, and claims benefit under 35 U.S.C. §119(c) from, U.S. Provisional Patent Application Serial No. 62/378,630, filed August 23, 2016, entitled "METHOD AND SYSTEM FOR SHOWING REMOTE MEETING SITES FROM USER-DEPENDENT VIEWPOINTS", and U.S. Provisional Patent Application Serial No. 62/428,301, filed November 30, 2016, entitled "METHOD AND SYSTEM FOR SHOWING REMOTE MEETING SITES FROM USER-DEPENDENT VIEWPOINTS", each of which is incorporated herein by reference in its entirety.
TECHNICAL FIELD
[0002] This disclosure relates to systems and methods for telepresence systems. More specifically, this disclosure relates to systems and methods for formation of user-centric views of local and remote users within a consistent virtual geometry.
BACKGROUND
[0003] Telepresence (remote interaction) systems give users the feeling of more personal interaction. In basic telepresence systems, this is accomplished by one camera in each local meeting room shared by all remote participants. In more advanced systems supporting directions and spatiality, multiple physical or virtual cameras are used in each meeting space.
[0004] Problems of mismatch may arise in telepresence sessions between the position of a camera (which serves as the position of the counterparty's eyes) and the position of the counterparty's face on a display. This mismatch may exist along a vertical axis or along a horizontal axis. The mismatch, referred to as camera-display parallax, may result in poor eye contact and may detract from the realism of telepresence interactions.
[0005] If a large number of people and sites are participating in a telepresence session, a wide video display may be used for showing all of the participants at each meeting site. In advanced systems supporting directions and spatiality, multiple physical or virtual cameras are used in each meeting space, with different participants being shown in different displays. Showing each participant or group of participants in separate video windows, however, is unsatisfactory due to not supporting enough the feeling of continuity and immersion.
SUMMARY
[0006] Methods and systems are described that support formation of remote user-centric panorama views of a local meeting room depicted by a sensor arrangement, for example multiple cameras or depth sensors. In exemplary embodiments, a panorama from the correct viewpoint is provided individually for all participants of the telepresence or interaction session. In some embodiments, a panorama is formed locally, and derived views are sent to each receiver as a panorama video. A front-end system may omit transmitting excessive amounts of real-time 3D data for deriving panoramas at each receiving site.
[0007] In an exemplary embodiment, a method is provided for conducting a teleconference showing remote meeting sites from a user-dependent viewpoint. An exemplary teleconference system operates to determine a multi-party, multi -location telepresence session geometry. Systems and methods operate to determine and track the location of a far-end (remote) user. The location of the far-end user is provided to the local conferencing system. The local system captures video feeds of local participants. The local system further operates to construct a panorama video corresponding to the viewpoint of a first far-end user using the location of the far-end user, the session geometry, and video feeds of local participants. The local system communicates the constructed panorama video to the remote user. In some embodiments, properties of the display pose and the geometry of a far end user are signaled to the local conferencing system. In some embodiments, details of the display pose, geometry, position of a first far end user, and meeting geometry are used to produce a virtual panorama image of the local scene for the first far end user.
BRIEF DESCRIPTION OF THE DRAWINGS
[0008] FIG. 1 is a schematic plan view illustrating use of camera-centric viewpoints in panorama generation.
[0009] FIG. 2 is a schematic plan view illustrating formation of a user-centric panorama in a three-party meeting, where user positions are tracked and combined in a unified geometry (coordinate system).
[0010] FIG. 3 is a schematic functional block diagram of a system architecture employed in some embodiments.
[0011] FIG. 4 is a schematic plan view illustrating an example of a shared geometry for a meeting between five participants at three local sites.
[0012] FIG. 5 is a schematic plan view illustrating an example of a linear sensor arrangement to support each remote user centric view.
[0013] FIG. 6 is a schematic plan view illustrating an embodiment in which restrictions to the visibility of users are caused either by the span of the camera array or the display.
[0014] FIG. 7 is a schematic plan view illustrating the use of separate user centric camera views in accordance with an embodiment.
[0015] FIG. 8A is a schematic plan view of camera views illustrating equal distribution of cameras around a circular constellation.
[0016] FIG. 8B is a schematic plan view of camera views illustrating equal angular distribution of local cameras.
[0017] FIG. 8C is a schematic plan view of camera views illustrating equal angular distribution of cameras that capture the view seen by each user's eyes.
[0018] FIG. 8D is a table of camera and position angles for an example embodiment with six users and telepresence sites.
[0019] FIG. 9A is a schematic plan view of camera views, where each camera represents a view seen by a user's eyes.
[0020] FIG. 9B is a schematic perspective view of camera views with one camera on the circular geometric setup replaced by a user.
[0021] FIG. 10 is a schematic plan view of camera views where each physical camera is replaced with a virtual camera.
[0022] FIG. 11 is a flow chart illustrating a method performed in an exemplary embodiment.
[0023] FIG. 12 is a flow chart illustrating a method performed in an exemplary embodiment.
[0024] FIG. 13 is a message flow diagram illustrating information exchanged in an exemplary embodiment.
[0025] FIG. 14 is a schematic plan view illustrating an exemplary virtual geometry.
[0026] FIGS. 15A-15D are perspective views illustrating images captured from respective cameras in the virtual geometry of FIG. 14.
[0027] FIG. 16 is a schematic illustration of a user-dependent panorama view constructed from the captured images illustrated in FIGS. 15B-15D.
[0028] FIG. 17A is a system diagram illustrating an example communications system in which one or more disclosed embodiments may be implemented.
[0029] FIG. 17B is a system diagram illustrating an example wireless transmit/receive unit (WTRU) that may be used within the communications system illustrated in FIG. 17A according to an embodiment.
DETAILED DESCRIPTION
[0030] FIG. 1 illustrates two issues that are addressed in some embodiments by the generation of a user-centric panorama video stream. The first issue is the use of camera-centric viewpoints (e.g., cameras on the wall 103 capture views 111 along lines of sight 109 from a set point 107) instead of user-centric viewpoints (e.g., a user 105 looking at a display with field of view 106). The second issue is the formation of the panorama from an arbitrary or even ambiguous eye-point (e.g., point 107). In FIG. 1, an environment with two users (125a, 125b) is captured by an array of cameras on the wall 103 (heavy dashed line), which are used for forming a camera centric panorama (from fixed point 107). For simplicity, showing the panorama is seen from one participant's (105) viewpoint only. Furthermore, in current systems, due to using separate camera viewpoints, stitching sub-views into a panorama induces visible distortions on sub-view edges. One technique that may be used to improve panorama appearance is to employ a normalized, evenly colored background at each meeting room, which reduces discontinuity if bringing them side by side on a large screen. This works even if views from different meeting rooms are taken side by side.
[0031] Defects in panorama formation stem in part from failure to track a viewer's position. If a system does not have information regarding the viewpoint of the user, an accurate remote user- centered panorama, in general, cannot be generated.
[0032] If multiple camera views are taken from the same point to various directions, the views may be stitched into a large panorama. Generation of a panorama is more complicated if images of one physical space are collected using multiple spatially dispersed cameras, but various techniques have been developed for generating such a panorama. In some embodiments, such a panorama may be generated using approximation, using the views from the spatially dispersed cameras. In such embodiments, neighboring camera views are stitched into a panorama so that the distortions on image borders stay small enough. Discontinuities on sub-view edges may be reduced by various filtering and interpolation operations. The viewpoint of the resulting panorama, however, may be an ambiguous average viewpoint chosen on a viewing line perpendicular to the panorama image's plane.
[0033] In cases where a physical view is captured by multiple cameras, all captured views include different parts of the same view. As the views come from different cameras from different
viewpoints (even from slightly different distances), those parts appear in different perspectives and scales. Various techniques are available for stitching these views together into a geometrically consistent panorama.
[0034] In one such technique, forming a 3D reconstruction using different camera views is used to obtain the combined panorama view from a correct view-point, for example as described in T. Moons et al., 3D Reconstruction from Multiple Images Part 1: Principles, FOUNDATIONS AND TRENDS IN COMPUTER GRAPHICS AND VISION, Vol. 4, No. 4, 287-398 (2008). In some embodiments, accuracy may be improved by hybrid embodiments using depth sensors in addition to visual cameras, as described in Tola et al., Virtual View Generation with a Hybrid Camera Array, CVLAB-REPORT-2009-001, EPFL (2009). To use such a panorama for telepresence purposes, a far enough viewpoint is chosen on a viewing line perpendicular to the panorama image's plane.
[0035] In some embodiments, techniques are used to compile a consistent panorama for a remote viewer in a specific location. Such embodiments are particularly useful in telepresence systems that provide various viewpoints to remote sites, e.g., for supporting spatial orientation.
[0036] To provide individual user-centric views and panoramas for a flexible number of sites and users, positions of the users are tracked using one or more known position-tracking techniques, and the positions of the users are brought into a common coordinate system, referred to as a common geometry.
[0037] In embodiments described herein, the formation of a 3D reconstruction from multiple camera and sensor views is one available technique to support formation of panoramas from desired viewpoints.
[0038] Known panorama formation embodiments generally provide a (one) perspective approximation to each local site and its participants. The panorama is formed from a viewpoint which is typically further away than one or more system camera(s). Even though true eye- directions or eye-contact are not supported, there is less horizontal parallax distortion, and the result is more acceptable if viewed on a large display. In a networked setting, like a telepresence system, panorama formation may be made either in a system front-end component, e.g., in the transmitting terminals or in the receiving devices, provided that the information for forming the panorama is received for this purpose. The chosen embodiment may have considerable effects to the required bitrate and other system properties.
[0039] However, in some embodiments, a user-centered perspective may be used to view an environment through a display with an attached camera and/or an attached 3D sensor, e.g., a device
such as a tablet computer. One system for providing a user-centered perspective is described in Domagoj Bancevic, Cha Lee, Matthew Turk, Tobias Hollerer, Doug A. Bowman, A Hand-Held AR Magic Lens with User-Perspective Rendering, ISMAR 10 (2012).
[0040] In some embodiments, methods and systems described in Moons et al. may be used for forming a correct user's perspective. In particular, a depth sensor and user tracking may be used to generate the video from the user's viewpoint. In a local case where cameras and the viewer co- locate in same environment, a panorama of several camera views is created by, e.g., positioning tablets showing the video with corrected viewpoints side by side. Analogous embodiments may generate panoramas in remote settings where the viewer and camera are not physically in the same environment.
[0041] One known technique for addressing horizontal parallax distortion is a virtual vectoring system, such as described in PCT Application WO 2008101117 (filed Feb. 14, 2008). Virtual vectoring stitches frontal views of remote participants into a panorama without regard to what happens to the background. If an evenly coloured background is used, little distortion may be visible.
[0042] A technique for panorama generation that may be employed in embodiments herein is described in Dou et al., Room-Sized Informal Telepresence System, IEEE VIRTUAL REALITY 4 (2012). In such methods, a frontal image of a person is stitched into a consistent background panorama (for example, the person may be turned to appear more front-facing in the central panorama). The results, however, may still be approximate. For example, if a user is looking towards the camera, all viewers may feel looked at (e.g., the Mona Lisa effect). Analogous technologies are described in US Patent Application 2012/0050458 (filed Aug. 31, 2010).
[0043] In some embodiments, a 3D reconstruction of each local site and its participants is formed, and the reconstruction is used for forming a virtual panorama perspective from an arbitrary viewpoint. Systems and methods that may be used for generating such 3D reconstructions are described, for example, in Zhang et al., Viewport: A Distributed, Immersive Teleconferencing System with Infrared Dot Pattern, ΓΕΕΕ MULTIMEDIA, Vol. 20, No. 1, 17-27 (2013). In most cases which are similar, all 3D information is captured and transmitted to a receiving terminal for deriving user-centered views or panoramas. In alternative embodiments, an arbitrary viewpoint of a scene depicted by an array of cameras may be generated by forming a high-quality depth image of the scene using the array of cameras. This technique may involve identifying the same feature points in separate camera views and deriving the 3D position of the feature points using tri angulation.
[0044] In some embodiments, where the density of cameras depicting a remote site is high enough, the methods of light field rendering (LFR) are used for forming a perspective view from a desired location. According to LFR, the panorama view from a user' s viewpoint may be sampled from those camera views— e.g., one pixel per each view— depicting towards the scene along user- centred lines-of-sight.
[0045] One technique for the generation of a panorama view may be a variant of the light field rendering method, as described in Yang, et al., Creating Adaptive Views for Group Video Teleconferencing- An Image-Based Approach, INTERNATIONAL WORKSHOP ON IMMERSIVE TELEPRESENCE (2002). In such embodiments, the panorama is collated from a set of vertical image slices received from an array of cameras. The panorama is generated using a collection of pixel columns from various camera views on each user centric line-of-sight. In some embodiments, the generation of a view may be performed using a crossed slits projection technique as described in the Zomet et al., Mosaicing New Views: The Crossed-Slits Projection, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, Vol. 25, No. 6, 741-754 (2003). A horizontal camera array may provide only one fixed vertical viewpoint (and focal length), but in the horizontal direction a number of viewpoints may be provided (using a modified focal length).
[0046] Some embodiments may make use of a hybrid sensor, such as by complementing video cameras with depth sensors. Such systems may improve results for producing 3D surfaces or light field renderings, and may also be used to improve the view-point calculation using techniques such as those described in Tola et al.
[0047] In exemplary embodiments, the positions of a camera and a viewer are determined in order to calculate an accurate user perspective. In exemplary embodiments, the viewer and camera are placed in a virtual geometry that is shared by all the local and remote sites. Generation of such a shared virtual geometry is described in International Application No. PCT/US 16/46848, filed Aug. 12, 2016, entitled "System and Method for Augmented Reality Multi-View Telepresence," and U.S. Provisional Patent Application No. 62/357,060, filed June 30, 2016, entitled "System and Method for Spatial Interaction Using Automatically Positioned Cameras," both of which are incorporated herein by reference in their entirety.
[0048] In various embodiments, different techniques may be used to stream 3D content.
Methods for representing 3D information may be adapted from 3DTV technologies and from game development technologies. Techniques for presenting 3D content with one monocular video and the respective depth map are available, although such techniques have problems, for example, in edge regions revealing content behind occluding objects. This representation is supported, for example, by the Kinect sensor. Several 3D formats may be used, including, for example, Dolby
3D, XpanD 3D, Panavision 3D, Masterlmage 3D, and ΓΜΑΧ 3D. Better quality with more free view-point selection generally calls for more complex methods. Recently, a specific compression standard H.264 MVC (multiple view video coding) has been published for the purpose. In Blu-ray disks, the standard is used for coding stereoscopic 3D content. Exemplary methods that may be used include those described in Chen and Vetro, Next-Generation 3d Formats with Depth Map Support, IEEE MULTIMEDIA, Vol. 21, No. 2, 90-94 (2014).
[0049] Exemplary embodiments described herein operate to form consistent panoramas with accurate individual viewpoints to one or more remote viewers. Disclosed embodiments further operate to tracking the users' positions and place them in a shared virtual geometry to provide for consistent tracking of user viewpoints.
[0050] Methods and systems are described supporting formation of remote user-centric panorama views of a local meeting room depicted by an array of cameras. A panorama from the correct viewpoint is provided individually for each of the participants of the telepresence or interaction session.
[0051] FIG. 2 illustrates an embodiment of generating a user-centric panorama in a meeting taking place in three locations (201, 202, 203) (although participants in location 203 are not illustrated), where user positions (such as those of user 225a and 225b at location 202) are tracked and combined in a unified geometry (coordinate system). For simplicity, the panorama (e.g., views 207) from only one remote participant's (user 205) viewpoint is shown. The herein disclosed systems may be implemented so that the panorama (207) is formed locally (at the location of the cameras), and the derived views are sent to each remote receiver as a panorama video. A front-end system may omit transmitting excessive amounts of real-time 3D data to be used for deriving panoramas at each remote site.
[0052] An exemplary system architecture using components described herein is shown in FIG. 3. A system' s architecture may however be implemented in various ways, for example using client- side or server-based components. In the embodiment illustrated in FIG. 3, a user 303 at a first site is using a user terminal 335 to view video of a second user 305 at a second site. Communications between sites may occur over the internet and/or through servers 301.
[0053] In an exemplary embodiment, a telepresence system may include, but is not limited to, the following components. A sensor arrangement 325 is provided for supporting remote viewpoint selection. The sensor arrangement 325 may include, for example, a set of depth sensors and a set of 2D video cameras. The video cameras may be arranged in a horizontal array (e.g., a linear array). Software and hardware may be provided to track position of each participant, such as
tracking systems 310 and 312. The software and hardware may operate to use the sensor arrangement 325 for remote view point selection. Determining the position of each participant may be performed, for example, based on tracking features in the videos filmed by the cameras in augmented reality (AR) glasses used by local users.
[0054] Software may be provided to form a shared virtual geometry 315 containing the participants with respect to the sensor arrangement. The software may further be operative to form a virtual panorama 330 from a remote user's viewpoint in a virtual geometry (which may be retrieved from a database for virtual geometry 320). Software is further provided that is capable of manipulating the panorama video 340, e.g., augmenting virtual objects to the video or changing background of the video (such as from a database of objects 345). Software and appropriate communication interface hardware is provided to transmit panorama video between the participant sites.
[0055] Aterminal 335 may be provided for displaying the user-centric panorama. The terminal 335 may be, for example, augmented reality glasses (or other augmented reality display system, such as other head-mounted displays), used by the collaboration participants. Alternatively, the terminal 335 may be a multiview display, showing different view to each local participant. An audio system may be provided to produce and capture spatial audio so that remote participant's sound seems to be coming from direction of the corresponding video stream. A session manager may be provided for organization of sessions, sites and users.
[0056] Exemplary embodiments of the disclosed systems and methods operate to support formation of a consistent panorama of each local meeting site from a freely selected viewpoint and to support meeting the natural focal length of a human observer. Exemplary embodiments avoid typical panorama distortions caused by stitching multiple uncalibrated camera views from arbitrary or ambiguous viewpoints.
[0057] Exemplary systems and methods employ detection and tracking of user (e.g., head) positions during system use, such as by tracking modules 310 and 312. Any suitable method of position tracking may be used. In an exemplary implementation, user position is tracked by an embedded camera in AR glasses capturing graphical markers or other known features in the environment.
[0058] In an embodiment, there is a method of providing a teleconference showing remote meeting sites from a user-dependent viewpoint, using the system as discussed above. The telepresence session virtual geometry may be defined for a telepresence session involving a local site having a local telepresence system and at least a first remote site having a first remote
telepresence system. The viewpoint position of a first remote user at the first remote site may be determined at the remote telepresence system, with respect to the telepresence session virtual geometry, and communicated to the local telepresence system. The local telepresence system may capture video or image feeds of at least a first local user at the local site using a plurality of cameras or other image detection means. In some instances, the local telepresence system may generate, from the plurality of captured video or image feeds, a video (such as a panorama video, but not necessarily) corresponding to the viewpoint position of the first remote user with respect to the telepresence session virtual geometry. The local telepresence system may then send or otherwise communicate the generated panorama video to the first remote telepresence system at the first remote site, for viewing by the first remote user (such as with a set of AR glasses, or other display system).
[0059] In another embodiment, the local telepresence system may communicate the captured video or image data to the remote telepresence system, which may then generate the appropriate panorama video.
[0060] Similarly, the local telepresence system may determine the viewpoint position of at least a first local user at the local site, and communicate this viewpoint position to one or more remote telepresence systems at one or more remote sites. These remote telepresence systems may use the local user's viewpoint position to generate videos or images (such as panorama videos, but not necessarily) which are then communicated back to the local telepresence system for display to the local user (such as with AR glasses, or some other display system).
[0061] At each meeting room participating in the session, users select their respective positions (e.g., particular seats), which - after user positions are being detected - leaves a system with a mutual, virtual orientation of meeting sites. The local geometries are configured into a desired constellation, with the number of local geometries being the number of meeting sites. The number of sites (and people's locations in them) suggests a type of virtual geometry to be used, e.g., three sites with participants in a row suggest a triangular type of geometry. If the local participants are sitting more in an arc form, a circular type of virtual geometry may be selected. In FIG. 4, an example of a shared geometry for a meeting between five participants 401a-e at three sites 403, 405, 407 is shown.
[0062] In the example of FIG. 4, an exemplary geometry optimization results in a virtual meeting geometry where the participants 401a-e are in a circle form 410 (dashed circle). The room walls 415 with camera arrays are shown by dashed lines. The shaded triangle 420 in the middle is a virtual space between real physical spaces. As a system defines its dimensions, virtual physical
positions of all participants are determined and may be used for calculating view-points (such as views 430 from each user to the users at different sites).
[0063] After selection of the shape type of the shared geometry, further optimizing may be made by adjusting its actual form (deviations from symmetry) and size. An optimization step is that the average distance of each participant from the virtual table's circumference is minimized. Many different optimization techniques may be used.
[0064] After defining the virtual geometry, individual view-points (virtual lines-of-sight) may be supported for all participants. Note, however, that the dimensions (span) of the sensor arrays or hardware displays (if in use instead of AR glasses) may at times restrict capturing or showing remote counterparts. This may happen, for example, if a user is located far sideward from the center of a capture or display setup. The situation is described in greater detail with respect to FIG. 6.
[0065] In some embodiments, the shared geometry is recalculated during an ordinary session management process, e.g., if more sites with user(s) enter or leave the meeting. This may be compared to a physical meeting, where people may relocate after changes in participation. The policy in triggering geometry changes may vary, as it may not be desirable to change the geometry for every move of individual participants.
[0066] In exemplary embodiments, consistent high-quality panorama viewpoints are generated for a plurality of viewpoints based on the positions of remote viewers in the defined meeting geometry.
[0067] FIG. 5 illustrates an example of a linear sensor arrangement to support each remote user centric view. As depicted, a user 501 at a first site 505 may observe a view 503 that is captured by a plurality of sensors 510 at a second site 507. There may be two users 515a and 515b at the second site 507.
[0068] A linear array of cameras, such as those described in PCT/US 16/46848, may be used in some embodiments. Another linear array of cameras that may be used in some embodiments is described in R. Yang, et al. Such linear camera arrays may be used to forming remote user centered panoramas based on either camera based 3D reconstruction or light-field-rendering. Note that a horizontal camera array provides only one fixed vertical viewpoint (and focal length), but in horizontal direction, a number of viewpoints is supported (using a modified focal length). A linear camera array may be described as an application of the crossed-slits projection described in Zomet et al.
[0069] In some embodiments, a two-dimensional light-field capture setup may be employed, such as an 8x8 array of sixty-four cameras, such as that described in J. Yang et al., A Real-Time Distributed Light Field Camera, EUROGRAPHICS WORKSHOP ON RENDERING 1-10 (2002). A two- dimensional light-field-camera may naturally provide symmetrical perspectives in horizontal and vertical directions.
[0070] Using captured images, known methods of multiple camera based 3D reconstruction, such as those described above may be employed. If a linear arrangement of cameras with coinciding image planes is used, the techniques described in Moons et al. may be used for triangulating common features in the scene.
[0071] A linear camera array may be employed in some embodiments for light-field capture. For capturing high enough quality light-fields, the cameras may have a sufficiently high density, as known to those of skill in the art.
[0072] In different embodiments, rendering the remote user centered panorama may be done either at the local site (if the user position is made available) or optionally at the receiving site (if a complete 3D reconstruction or light-field is transmitted for this purpose, or all video views required for their construction). Rendering at the local site may result in significant savings in communication bandwidth, but may also result in additional latency caused by collection and transmission of information regarding remote user positions.
[0073] Perspective to 3D reconstruction. In order to form a desired remote perspective to a 3D reconstruction, the reconstruction is aligned with the local physical view. After that, forming the panorama may be performed using one or more of the techniques described herein.
[0074] Deriving a viewpoint from light-field. Providing that the light-field camera captures the scene without disturbing occlusions, a panorama from the desired remote viewpoint may be formed using the known principles of light-field-rendering.
[0075] Various coding methods and standards described herein may be used for coding and transmission of the information required for remote user-centered panorama. The chosen architecture affects to the choice, as the information may be sent as separate videos, multi-view video (such as H.264 MVC), depth plus video, mesh plus texture, light-field, and so on.
[0076] In an exemplary system implementation, multiple remote user centric panoramas are formed in a local front end and are transmitted using H.264 MVC type of coding method exploiting their remaining redundancies (due to views showing the same environment from slightly different viewpoints).
[0077] In exemplary embodiments, an individualized panorama is displayed to each remote user. The display may be provided using Augmented Reality glasses or HMDs, among other options. The supported field-of-view of existing AR glasses is typically rather narrow. To accommodate this issue, the glasses may display a correct sub-view of a wider panorama, according to the user's viewing direction. In such cases, tracking of both user's position and viewing direction may be performed. This tracking may be implemented either by the HMD camera (e.g., by tracking markers or features) or by some other suitable tracking arrangement. Technologies such as those used by the Oculus Rift may be employed to support sub-view formation and position tracking.
[0078] In some embodiments, multi-view displays may be used, with the information provided to each viewing segment being changed according to users' movements in the space, including the users' distance from the display. Embodiments disclosed herein thus provide a more realistic viewpoint than a viewpoint that would result from merely rendering a constant 3D or stereoscopic information to each viewing segment.
[0079] Multiscopic displays may serve multiple users, showing different content to different display directions. However, they do not typically support naturally changing parallax for a user moving further or closer to the display, inside his/her viewing segment. Embodiments disclosed herein control the multi-view content according to a user's distance as part of the user tracking process.
[0080] Systems set forth herein may use an array of cameras (and/or other sensors) for capturing data for panoramas. In order to avoid camera-display parallax, the array may be positioned on the eye-level of the viewing participants. If using a hardware display, positioning of the array without blocking the display, or vice versa, may require additional effort.
[0081] Imaging through a display has been used to solve the camera positioning problem (and the implied camera-display parallax problem). In addition to the conventional half-silvered mirror solutions, this has more recently been addressed by time-multiplexing an LCD screen between display and see-through modes and/or by integrating a matrix of cameras inside the display. Techniques that may be used in embodiments disclosed herein include techniques described in US Patent Application 2014/0146127 (Filed Nov. 29, 2012) and US Patent 7,808,540 (Filed Jan. 11, 2007). In some embodiments, an analogous imaging system is implemented through an auto- multiscopic display. Particularly, this may be done using a display based on parallax barriers instead of lenticular lenses.
[0082] One type of display that may be employed for embodiments disclosed herein is the BiDi (for bi-directional) of image capturing display as described in Hirsch et al., BiDi Screen: A Thin, Depth-Sensing LCD for 3D Interaction Using Light Fields, ACM TRANSACTIONS ON GRAPHICS (TOG), Vol. 28, No. 5, 9 (2009). In this type of display, a dense matrix of camera elements is integrated into the display for the light field capture. In embodiments that employ hardware displays instead of AR glasses, the size of the display may be selected in order to support the defined (wide) geometry; the display orientation with respect to remote user(s) is determined (e.g., for showing content in the correct viewing segment). The view may be restricted if the hardware display is too narrow to support the full geometry. In order to support the use of a hardware display, the properties (size and type) and pose of each display in the unified meeting geometry are determined. The system may then serve users with correct content and lines-of-sight.
[0083] FIG. 6 illustrates an embodiment in which restrictions to the visibility of users (605, 606, 607) are caused either by the span of the camera array 610 or the display 615. For simplification, the situation is shown in meeting in three locations 601, 602, 603 (although participants at location 603 are not illustrated). Users 605 and 606 do not see each other without changing their positions. A respective restriction may happen if a camera array 610 is too short for the used geometry. This situation may be compared to a physical window between two rooms, where seeing users in the other room depends on the viewer's position and the window size. Differing from a physical window, viewing restrictions in the disclosed system may also be asymmetrical. A system may also make use of information regarding the span of the camera array 610 in use. In cases where a camera array 610 cannot support lines-of-sight to all remote participants (analogous to a case where a window is too small), panorama views may be cropped respectively.
[0084] With reference to FIG. 3, the properties and pose of each display are included in "user terminal parameters" transmitted to a system front-end for panorama formation 330. Respectively, the properties and pose of each camera/sensor array 325 are included in "sensor positions and parameters" used by a system in forming a unified meeting geometry 315 and in a system front- end for panorama formation 330. The span of both the display and camera setups (local and/or remote) may be taken into consideration whether the panorama display is based on AR glasses or flat panels.
[0085] In exemplary embodiments, panorama formation has the basic benefit of more complete and consistent perception of the received far-end view. In addition to visual consistency, panorama formation operates to reduce redundancy and bitrate for transmitting remote user-centric views. Redundancy is reduced whether the panorama is formed by making 3D reconstruction from
multi-camera views, or by using light-field-rendering techniques. In the latter, however, the amount of redundancy is considerably large and so is the benefit of forming the panorama at a system front-end.
[0086] In an alternative embodiment, as illustrated in FIG. 7, instead of a combined panorama, separate user-centric camera views may be, in effect, brought to a remote user's eye-point. In practice, this may be implemented by the same components and procedures as any of the above discussed embodiments using panorama formation. For example, the sub-views 705 may be taken along the line-of-sight of a remote viewer 701, and be made correct from the viewer's perspective, rather than being camera centric (as in FIG. 1). This variation may be preferred by a user in some circumstances. For example, the remote user may not want to cover his/her view with large panoramas (e.g., due to using a small display, or wanting to save display space for showing some other information). In another example, the local user may want to protect his/her privacy by restricting the view. In some embodiments, a system may support users viewing either sub-views or panoramas according to their preferences.
[0087] Additional embodiments are discussed in relation to FIGS. 8-10. One embodiment uses a circular meeting table with participants placed uniformly around the table. For one embodiment, the table is a virtual, circular table. In one embodiment, users may see each other in a spatially faithful way, where users and cameras are placed in a unified meeting setting with a global coordinate system. Cameras are remote users' proxies (eyes) in each local user's environment. Each view is rendered respectively in these proxy eye positions, resulting in natural eye-contact and directions between users.
[0088] FIGS. 8 A to 8C show one embodiment with a spatially-oriented telepresence between multiple persons in a circular constellation. For example, the person on the lower left of FIG. 8 A may be the local participant 801 and may see five remote participants 805a-e distributed spatially around him/her.
[0089] FIG. 8 A shows remote users 805a-e seen by a local participant 801 in a circular constellation 810 (dashed circle). A local participant 801 is shown at the lower left of FIG. 8A. For one embodiment, remote users 805a-e are distributed around a virtual circle 810, where each of the remote users 805a-e is displayed locally looking at the local participant 801. The solid, straight lines 815 that go from a remote user (as rendered locally) to the local participant 801 show line-of-sight lines reflecting the view that remote users 805a-e would have if those remote users 805a-e were present locally.
[0090] FIG. 8B depicts an exemplary scenario in which remote users occupy equally-sized segments of 30 degrees. The example of FIG. 8B shows six users distributed 30° apart. FIG. 8B is similar to FIG. 8A, with the addition of the example 30° angular distribution. For one embodiment, the equal angular distributions for each wedge segment are distributed evenly on either side of a line-of-sight between the local participant 801 in the lower left of FIG. 8B and each remote user 805a-e as rendered locally. For this example, with five remote users 805a-e rendered 30° apart, two smaller wedges, each 15° wide, are shown between a line 820 perpendicular to the local participant 801 and an edge of one of the outermost 30° wedges.
[0091] FIG. 8C shows an example respective camera capture setup. Each of the six users may be captured by a capture setup as depicted in FIG. 8C. For an example embodiment with six users, video data is captured for each user from five different viewpoints. For the example of FIG. 8C, depicting a single user 801 of the six users, each user is captured from 30°, 60°, 90°, 120°, and 150°. The field-of-view for each camera/user 806a-e may depend on the maximum number of remote participants. For one embodiment, to perceive remote partners and spaces from a natural perspective, the focal length for each camera 806a-e is set the same as a human eye, which means cameras with an "normal" objective (e.g., 50 mm for a full-frame, sensor-equipped DSLR camera) are used. The diameter of the capture setup affects the distances of users at the virtual table in the global coordinate system. For some embodiments, the diameter of the capture circle 810 may be selected based on personal and cultural preferences. In order for the users to have the same perception of distances, the diameter should be the same for all user environments. For example, if the main purpose of a system is personal communication, the diameter may be relatively small (for example, 1.5 meters). For some embodiments, the diameter of the circle 810 may be used to create a virtual table.
[0092] FIG. 8D shows a table of user positions and camera angles for an example set of six positions. For the embodiment shown in FIG. 8D, a local user is in position 1 and remote users are in positions 2 to 6. For one embodiment, if user A is a local user, user A is in position 1 and user B is in position 2. This pattern is repeated for the rest of the remote users, with user F in position 6 if user A is a local user. For one embodiment, this paradigm is repeated if user B is a local user, with user B in position 1 and user C in position 2. Again, this patter is repeated for the rest of the remote users, with user F in position 5 and user A in position 6. For one embodiment, these patterns are repeated if any of the other users is a local user, as shown in the table. This table is an example embodiment for a set of six locations with one user at each location. The paradigm shown in FIG. 8D for six users may be repeated if the number of users or telepresence sites is less or more than six users or six sites.
[0093] FIG. 9A shows an embodiment of camera capture locations for a circular table geometry 900. The example capture set-up uses a set of cameras 905 in a circular arrangement, with equal distances in between each camera 905. For one embodiment, a local user 901 replaces one of the cameras in the circle 900. For the example embodiment shown in FIG. 9A, each user is distributed an equal angle from each other, measured from the center of the circle. For the example of FIG. 9A, each user is separated by 60° as measured from the center of the circle. In other embodiments, any number of users may be evenly distributed around a circle.
[0094] FIG. 9B shows an embodiment where each camera 905 represents a user's eyes. For one embodiment, a number of cameras are chosen to support a predefined number of users, but the number of users supported may reach a maximum number, above which, the field-of-view for each camera becomes too narrow. Similar to FIG. 8C, the embodiment of FIG. 9B shows a camera setup that captures video data of a user 901 from five different locations. For one embodiment, the number of cameras used is N-l, where N is the number of total users.
[0095] FIG. 10 shows a camera setup using virtual cameras. For one embodiment, if the number of cameras exceeds a virtual rendering threshold, physical cameras may be replaced with virtual cameras. For one embodiment, such as the one shown in FIG. 10, a three-dimensional (3D) reconstruction model may be captured for a user's environment 1000 (or a local telepresence site), and virtual viewpoints 1008 (or virtual camera video streams) may be calculated for locations of virtual cameras 1010. For one embodiment, depth sensors are calibrated and used to make 3D depth measurements of a user's environment. For the example embodiment shown in FIG. 10, four depth sensors 1005 are shown to create a 3D model of the environment 1000. Four depth sensors 1005 are shown, with two depth sensors 1005 on the top side wall 1020 and two depth sensors 1005 on the right side wall 1022. Other embodiments may have more or fewer depth sensors 1005. Once a 3D model of the user's environment 1000 is created, virtual camera viewpoints 1008 may be calculated from the 3D model for each of the plurality of virtual cameras 1010.
[0096] Physical cameras may be replaced with virtual cameras (such as by using a calibrated set of depth sensors). Even though physical cameras may be small, such cameras (along with their supporting frames) occupy and occlude part of the meeting space. Virtual camera views allow the number and positioning of remote users (or cameras) in each local space to be flexible.
[0097] FIG. 11 is a flow chart illustrating steps in an exemplary embodiment for defining the virtual meeting geometry in embodiments used herein. FIG. 12 is a flow chart (continuing from FIG. 11) illustrating steps in an exemplary embodiment for providing each remote user with a user-centered panorama including augmented objects (if any).
[0098] As shown in FIG. 11, for each site, user, and AR object, a user's position may be tracked relative to their camera setup, and an AR object's position (for example, given by a user) may be read or otherwise obtained. These positions may be stored until all sites, users, and AR objects have been analyzed. Using the stored positions, a geometry for each meeting site may be formed. Such geometry may consider display and sensor setup parameters at each site.
[0099] As shown in FIG. 12, for each site and each remote user, a user centered panorama may be formed. This may comprise capturing videos of each user, and using the geometry of each meeting site to form the remote user centered panorama. Also, the user centered panorama may be augmented with any AR objects (if any) that were added by a user. The created (and possibly augmented) user centered panorama may then be transmitted to other users and displayed, in some cases in consideration of the display setup parameters (and in some cases sensor setup parameters) for each receiving user (e.g., type, size, etc.). Panoramas may be created and transmitted until all sites and all users have been processed. The panoramas may be maintained until a change in the meeting setup is triggered (e.g., a user leaves the meeting, etc.). If a change is triggered, if the meeting has not ended, the meeting site geometry may be accordingly updated, and new user centered panoramas may be formed.
[0100] FIG. 13 is a message flow diagram illustrating a process performed in an exemplary embodiment. A local and remote conferencing system may communicate to calibrate a conference makeup, e.g., determining a virtual geometry and locations of the users in the virtual geometry (such as based at least in part on the display and/or sensor setup parameters at each site or system). The remote conferencing system sends to the local conferencing system information on its properties (e.g., display/sensor type, size, and position). The remote conferencing system also determines the location of the remote user (e.g., the position of the remote user's head or eyes) and conveys that location information to the local conferencing system. The location may be determined by, for example, a camera at the remote conference site. The location information may be conveyed in a variety of formats such as, for example, as coordinates in a common coordinate system of the virtual geometry. An array of local cameras at the local conference site collect video feeds of a local participant. Based on these video feeds, the local conferencing system constructs a panorama view representing the viewpoint of the remote user. The local conferencing system supplies a video feed of the constructed panorama view to the remote conferencing system for display on the remote display.
[0101] One exemplary method for constructing a panorama video based on a viewpoint of a remote user is described with respect to FIGS. 14-16. For the purposes of illustration, reference is made to an exemplary layout in FIG. 14. The example of FIG. 14 involves only two sites— a single
local site 1401 and a single remote site 1403— although the principles disclosed herein may be implemented in circumstances with different numbers of sites. For the sake of simplicity, components are shown only for displaying a local site 1401 to a remote user 1425 (e.g., cameras A-E of camera array 1420 at the local site 1401 and a display 1430 at the remote site 1403). Components for displaying a remote site 1403 to a local user (e.g., cameras at the remote site 1403 and a display at the local site 1401) may be employed in embodiments but are not illustrated in FIG. 14. The display 1430 at the remote site 1403 is depicted as a wall-mounted display screen, although other display types as described herein may alternatively be used. The local site 1401 and remote site 1403 are arranged in a virtual geometry that simulates a face-to-face meeting. For ease of illustration of parallax and foreshortening effects (discussed with respect to FIGS. 15 and 16), an empty chair is depicted in the position of the local user, though the same effects would be apparent (but more difficult to illustrate) in images of a human user.
[0102] In an exemplary embodiment, the position of the remote user 1425 (e.g., the position of the remote user's head) is determined using position sensors 1435 at the remote site 1403. The position sensors 1435 may be, for example one or more of (or a combination of) stereoscopic cameras, depth cameras, and the like. The position of the remote user 1425 is conveyed to the conferencing system at the local site 1401. The position, type, and orientation of the remote display 1430 may also be conveyed to the local site 1401. The local site 1401 operates to generate a panorama video of the local site 1401, where the panorama video is generated so as to be appropriate to the viewpoint of the remote user 1425. To generate an appropriate panorama, the local system may use any of the techniques disclosed herein or other techniques known to those of skill in the art. One such technique is discussed with reference to FIGS. 15 and 16.
[0103] It may be noted that camera C of the local camera array 1420 falls directly on the virtual line of sight between the position of the remote user 1425 and the chair 1410 of the local user. However, camera C is located much nearer to the chair 1410 at the local site 1401 than the viewpoint of the local user (e.g., in the virtual geometry of FIG. 14, the remote user 1425 is approximately twice the distance from the chair 1410 as camera C). Thus, a video taken by camera C may be taken from the correct angle but would demonstrate substantial foreshortening (as represented in FIG. 15C). This level of foreshortening may appear unrealistic to the remote user 1425.
[0104] To provide a more viewpoint-appropriate view for the remote user 1425, the local system may generate a panorama by stitching together, for example, images from cameras B, C, and D of the camera array 1420. As seen from the dashed lines of sight 1440 illustrated in FIG. 14, camera B is the camera that most accurately corresponds to the remote user's viewpoint of the
left side of the chair 1410, camera C is the camera that most accurately corresponds to the remote user's viewpoint of the center of the chair 1410, and camera D is the camera that most accurately corresponds to the remote user's viewpoint of the right side of the chair 1410. FIGS. 15A-15D illustrate the views captured by these cameras, with FIG. 15A representing the view captured by camera A, FIG. 15B representing the view captured by camera B, FIG. 15C representing the view captured by camera C, and FIG. 15D representing the view captured by camera D.
[0105] As illustrated in FIG. 16, selected segments of the images from cameras B, C, and D are stitched together to construct a panorama view of the chair that is appropriate to the viewpoint of the remote user. In particular, as seen in the constructed panorama view of FIG. 16, the view of the chair displays much less foreshortening (as compared to the raw camera image from camera C, depicted in FIG. 15C) and thus may appear more realistic (and more consistent with the virtual geometry) to the remote user. In an exemplary embodiment, instead of streaming video from all cameras in the local camera array to the remote system— which may consume substantial bandwidth— the local conferencing system streams to the remote system only the constructed video corresponding to the user-centric panorama view.
[0106] The example described with respect to FIGS. 14-16 has been selected for clarity of illustration and that alternate techniques may be employed for generating a panorama video with a user-dependent viewpoint, including other techniques described herein.
[0107] The term "panorama" and its variants as used herein is not limited to wide-angle views but rather refers to a view constructed through a combination of separately-captured views (which may include depth-image views), although such constructed panorama views may indeed be wide- angle views in some embodiments.
[0108] Note that various hardware elements of one or more of the described embodiments are referred to as "modules" that carry out (i.e., perform, execute, and the like) various functions that are described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits
(ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) deemed suitable by those of skill in the relevant art for a given implementation. Each described module may also include instructions executable for carrying out the one or more functions described as being carried out by the respective module, and it is noted that those instructions may take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and/or the like, and may be stored in any suitable non-transitory computer-readable medium or media, such as commonly referred to as RAM, ROM, etc.
[0109] In an embodiment, there is telepresence method comprising, at a local telepresence site: providing a plurality of cameras, wherein the cameras and a local user at the local telepresence site are collectively arranged in a substantially circular configuration; and from each of the plurality of cameras, streaming a view of the local user to a different respective remote user. The method may further include receiving views of the respective remote users and displaying those views to the local user, each received view being displayed at a position corresponding to the camera associated with the respective remote user. The method may include wherein the displayed views are displayed to the local user using an augmented reality headset. The method may include wherein the views of the respective remote users are stitched into a panoramic view. The method may include wherein the plurality of cameras includes at least four cameras. The method may include wherein the cameras and the local user are substantially evenly spaced around the substantially circular configuration.
[0110] In an embodiment, there is a telepresence method comprising, at a local telepresence site: capturing video of a local user at the local telepresence site using at least two cameras; using the captured video, generating a real-time 3D model of the local user; based on the real-time 3D model, generating a plurality of virtual views of the local user, each virtual view being generated with respect to a virtual viewpoint, wherein the virtual viewpoints and the local user are collectively arranged in a substantially circular configuration; and streaming each of the virtual views to a different respective remote user. The method may further include receiving views of the respective remote users and displaying those views to the local user, each received view being displayed at a position corresponding to the virtual viewpoint associated with the respective remote user. The method may include wherein the displayed views are displayed to the local user using an augmented reality headset. The method may include wherein the views of the respective remote users are stitched into a panoramic view. The method may include wherein the plurality of virtual viewpoints includes at least four virtual viewpoints. The method may include wherein the virtual viewpoints and the local user are substantially evenly spaced around the substantially circular configuration. The method may include wherein at least one of the cameras is a depth camera.
[0111] In an embodiment, there is a method of providing a teleconference showing remote meeting sites from a user-dependent viewpoint, the method comprising: determining a telepresence session virtual geometry for at least a first site and a second site; determining, by a first telepresence system at the first site, the viewpoint of the first telepresence participant at the first site with respect to the telepresence session virtual geometry; providing the determined viewpoint to a second telepresence system at a second site; capturing, using a plurality of cameras at the second site, a plurality of video feeds; rendering, from the plurality of video feeds captured
by the plurality of cameras at the second site, a panorama video corresponding to the received viewpoint with respect to the telepresence session virtual geometry of the first telepresence participant at the first site; and sending the rendered panorama video to the first telepresence system at the first site. The method may include wherein properties of the display pose, sensor setup, and geometry of a remote user are signaled to the local conferencing system. The method may include wherein details of the display pose, sensor setup, geometry, position of a first remote user, and meeting geometry are used to produce a virtual panorama image of the local scene for the first far end user.
[0112] In an embodiment, there is a telepresence method comprising: receiving, at a first telepresence site, information representing a head position of a remote user at a second telepresence site; simultaneously capturing a plurality of video streams of the first telepresence site; combining the captured video streams into a panorama video corresponding to a virtual viewpoint based on the head position of the remote user; and streaming the panorama video to the second telepresence site. The method may further include receiving, at the first telepresence site, information representing a configuration of a display at the second telepresence site, wherein the panorama video is generated based at least on part on the configuration of the display and sensor setup (of any or all of the telepresence sites).
[0113] In an embodiment, there is a telepresence method comprising: receiving, at a local telepresence site, information representing a first head position of a first remote user at a first remote telepresence site; receiving, at the local telepresence site, information representing a second head position of a second remote user at a second remote telepresence site; simultaneously capturing a plurality of video streams of the local site; combining the captured video streams into a first panorama video corresponding to a first virtual viewpoint based on the head position of the first remote user; streaming the first panorama video to the first remote site; combining the captured video streams into a second panorama video corresponding to a second virtual viewpoint based on the head position of the second remote user; and streaming the second panorama video to the second remote site. The method may further include receiving, at the local telepresence site, information representing a configuration of a first display at the first remote telepresence site, wherein the first panorama video is generated based at least on part on the configuration of the first display; and receiving, at the local telepresence site, information representing a configuration of a second display at the second remote telepresence site, wherein the second panorama video is generated based at least on part on the configuration of the second display.
[0114] In an embodiment, there is a system for providing a teleconference showing remote meeting sites from a user-dependent viewpoint, the system comprising a processor and a non-
transitory computer-readable medium storing instructions operative, when executed on the processor, to perform functions comprising: determining a telepresence session virtual geometry for at least a first site and a second site; determining, by a first telepresence system at the first site, the viewpoint of the first telepresence participant at the first site with respect to the telepresence session virtual geometry; providing the determined viewpoint to a second telepresence system at a second site; capturing, using a plurality of cameras at the second site, a plurality of video feeds; rendering, from the plurality of video feeds captured by the plurality of cameras at the second site, a panorama video corresponding to the received viewpoint with respect to the telepresence session virtual geometry of the first telepresence participant at the first site; and sending the rendered panorama video to the first telepresence system at the first site.
[0115] In an embodiment, there is a telepresence system comprising a processor and a non- transitory computer-readable medium storing instructions operative, when executed on the processor, to perform functions comprising: receiving, at a first telepresence site, information representing a head position of a remote user at a second telepresence site; simultaneously capturing a plurality of video streams of the first telepresence site; combining the captured video streams into a panorama video corresponding to a virtual viewpoint based on the head position of the remote user; and streaming the panorama video to the second telepresence site.
[0116] In an embodiment, there is a telepresence system comprising a processor and a non- transitory computer-readable medium storing instructions operative, when executed on the processor, to perform functions comprising: receiving, at a local telepresence site, information representing a first head position of a first remote user at a first remote telepresence site; receiving, at the local telepresence site, information representing a second head position of a second remote user at a second remote telepresence site; simultaneously capturing a plurality of video streams of the local site; combining the captured video streams into a first panorama video corresponding to a first virtual viewpoint based on the head position of the first remote user; streaming the first panorama video to the first remote site; combining the captured video streams into a second panorama video corresponding to a second virtual viewpoint based on the head position of the second remote user; and streaming the second panorama video to the second remote site.
[0117] In an embodiment, there is a method, comprising: receiving, at a local telepresence site, information representing head positions of a plurality of remote users at remote telepresence sites; calculating a plurality of rendering viewpoints for each of the plurality of remote users based on the head positions of the plurality of remote users; transmitting, to the plurality of remote telepresence sites, the plurality of rendering viewpoints for each of the plurality of remote users; receiving, at the local telepresence site, a plurality of video streams captured from the rendering
viewpoint for each remote user; capturing, simultaneously, a plurality of video streams of the local telepresence site for each of a plurality of rendering viewpoints; streaming to the plurality of remote users a video stream of the local telepresence site captured from the respective rendering viewpoint for each of the plurality of remote users; combining the plurality of received video streams into a combined panorama video for the local telepresence site; and displaying the combined panorama video at the local telepresence site. The method may include wherein the rendering viewpoints for each of the plurality of remote users are calculated for an equal angle distribution. The method may further include determining to calculate a video stream for each of a plurality of rendering viewpoints when the plurality of remote users exceeds a virtual rendering threshold; capturing three-dimensional (3D) image data of the local telepresence site; calculating, simultaneously, a video stream of the local telepresence site for each of a plurality of rendering viewpoints; and streaming to the plurality of remote users a video stream of the local telepresence site calculated for the respective rendering viewpoint for each of the plurality of remote users.
[0118] In an embodiment, there is a method of providing a teleconference showing remote meeting sites from a user-dependent viewpoint comprising: defining a telepresence session virtual geometry for a telepresence session involving a local site and at least a first remote site; determining a viewpoint position of a local user at the local site; providing the local user's viewpoint position to the first remote site; capturing video feeds from a plurality of cameras at the first remote site; generating from the video feeds a panorama video corresponding to the viewpoint position of the local user; and sending the panorama video to the local site for viewing by the local user.
[0119] Exemplary embodiments disclosed herein are implemented using one or more wired and/or wireless network nodes, such as a wireless transmit/receive unit (WTRU) or other network entity.
[0120] FIG. 17A is a diagram illustrating an example communications system 1700 in which one or more disclosed embodiments may be implemented. The communications system 1700 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communications system 1700 may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communications systems 1700 may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single- carrier FDMA (SC-FDMA), zero-tail unique-word DFT-Spread OFDM (ZT UW DTS-s OFDM),
unique word OFDM (UW-OFDM), resource block-filtered OFDM, filter bank multicarrier (FBMC), and the like.
[0121] As shown in FIG. 17A, the communications system 1700 may include wireless transmit/receive units (WTRUs) 1702a, 1702b, 1702c, 1702d, a RAN 1704, a CN 1706, a public switched telephone network (PSTN) 1708, the Internet 1710, and other networks 1712, though it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and/or network elements. Each of the WTRUs 1702a, 1702b, 1702c, 1702d may be any type of device configured to operate and/or communicate in a wireless environment. By way of example, the WTRUs 1702a, 1702b, 1702c, 1702d, any of which may be referred to as a "station" and/or a "STA", may be configured to transmit and/or receive wireless signals and may include a user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription- based unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smartphone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (IoT) device, a watch or other wearable, a head-mounted display (HMD), a vehicle, a drone, a medical device and applications (e.g., remote surgery), an industrial device and applications (e.g., a robot and/or other wireless devices operating in an industrial and/or an automated processing chain contexts), a consumer electronics device, a device operating on commercial and/or industrial wireless networks, and the like. Any of the WTRUs 1702a, 1702b, 1702c and 1702d may be interchangeably referred to as a UE.
[0122] The communications systems 1700 may also include a base station 1714a and/or a base station 1714b. Each of the base stations 1714a, 1714b may be any type of device configured to wirelessly interface with at least one of the WTRUs 1702a, 1702b, 1702c, 1702d to facilitate access to one or more communication networks, such as the CN 1706, the Internet 1710, and/or the other networks 1712. By way of example, the base stations 1714a, 1714b may be a base transceiver station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a gNB, a NR NodeB, a site controller, an access point (AP), a wireless router, and the like. While the base stations 1714a, 1714b are each depicted as a single element, it will be appreciated that the base stations 1714a, 1714b may include any number of interconnected base stations and/or network elements.
[0123] The base station 1714a may be part of the RAN 1704, which may also include other base stations and/or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 1714a and/or the base station 1714b may be configured to transmit and/or receive wireless signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be in licensed spectrum,
unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage for a wireless service to a specific geographical area that may be relatively fixed or that may change over time. The cell may further be divided into cell sectors. For example, the cell associated with the base station 1714a may be divided into three sectors. Thus, in one embodiment, the base station 1714a may include three transceivers, i.e., one for each sector of the cell. In an embodiment, the base station 1714a may employ multiple-input multiple output (MTMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and/or receive signals in desired spatial directions.
[0124] The base stations 1714a, 1714b may communicate with one or more of the WTRUs 1702a, 1702b, 1702c, 1702d over an air interface 1716, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (TR), ultraviolet (UV), visible light, etc.). The air interface 1716 may be established using any suitable radio access technology (RAT).
[0125] More specifically, as noted above, the communications system 1700 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, the base station 1714a in the RAN 1704 and the WTRUs 1702a, 1702b, 1702c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 1716 using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed Packet Access (HSPA) and/or Evolved HSPA (HSPA+). HSPA may include High-Speed Downlink (DL) Packet Access (HSDPA) and/or High-Speed UL Packet Access (HSUPA).
[0126] In an embodiment, the base station 1714a and the WTRUs 1702a, 1702b, 1702c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 1716 using Long Term Evolution (LTE) and/or LTE- Advanced (LTE- A) and/or LTE- Advanced Pro (LTE-A Pro).
[0127] In an embodiment, the base station 1714a and the WTRUs 1702a, 1702b, 1702c may implement a radio technology such as NR Radio Access, which may establish the air interface 1716 using New Radio (NR).
[0128] In an embodiment, the base station 1714a and the WTRUs 1702a, 1702b, 1702c may implement multiple radio access technologies. For example, the base station 1714a and the WTRUs 1702a, 1702b, 1702c may implement LTE radio access and NR radio access together, for instance using dual connectivity (DC) principles. Thus, the air interface utilized by WTRUs 1702a,
1702b, 1702c may be characterized by multiple types of radio access technologies and/or transmissions sent to/from multiple types of base stations (e.g., a eNB and a gNB).
[0129] In other embodiments, the base station 1714a and the WTRUs 1702a, 1702b, 1702c may implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 IX, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS- 95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and the like.
[0130] The base station 1714b in FIG. 17A may be a wireless router, Home Node B, Home eNode B, or access point, for example, and may utilize any suitable RAT for facilitating wireless connectivity in a localized area, such as a place of business, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a roadway, and the like. In one embodiment, the base station 1714b and the WTRUs 1702c, 1702d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In an embodiment, the base station 1714b and the WTRUs 1702c, 1702d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 1714b and the WTRUs 1702c, 1702d may utilize a cellular- based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR etc.) to establish a picocell or femtocell. As shown in FIG. 17 A, the base station 1714b may have a direct connection to the Internet 1710. Thus, the base station 1714b may not be required to access the Internet 1710 via the CN 1706.
[0131] The RAN 1704 may be in communication with the CN 1706, which may be any type of network configured to provide voice, data, applications, and/or voice over internet protocol (VoIP) services to one or more of the WTRUs 1702a, 1702b, 1702c, 1702d. The data may have varying quality of service (QoS) requirements, such as differing throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, and the like. The CN 1706 may provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, etc., and/or perform high-level security functions, such as user authentication. Although not shown in FIG. 17A, it will be appreciated that the RAN 1704 and/or the CN 1706 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 1704 or a different RAT. For example, in addition to being connected to the RAN 1704, which may be utilizing a NR radio technology, the CN 1706 may also be in communication with another RAN (not shown) employing a GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.
[0132] The CN 1706 may also serve as a gateway for the WTRUs 1702a, 1702b, 1702c, 1702d to access the PSTN 1708, the Internet 1710, and/or the other networks 1712. The PSTN 1708 may include circuit-switched telephone networks that provide plain old telephone service (POTS). The Internet 1710 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the transmission control protocol (TCP), user datagram protocol (UDP) and/or the internet protocol (IP) in the TCP/IP internet protocol suite. The networks 1712 may include wired and/or wireless communications networks owned and/or operated by other service providers. For example, the networks 1712 may include another CN connected to one or more RANs, which may employ the same RAT as the RAN 1704 or a different RAT.
[0133] Some or all of the WTRUs 1702a, 1702b, 1702c, 1702d in the communications system 1700 may include multi-mode capabilities (e.g., the WTRUs 1702a, 1702b, 1702c, 1702d may include multiple transceivers for communicating with different wireless networks over different wireless links). For example, the WTRU 1702c shown in FIG. 17A may be configured to communicate with the base station 1714a, which may employ a cellular-based radio technology, and with the base station 1714b, which may employ an IEEE 802 radio technology.
[0134] FIG. 17B is a system diagram illustrating an example WTRU 1202. As shown in FIG. 17B, the WTRU 1202 may include a processor 1718, a transceiver 1720, a transmit/receive element 1722, a speaker/microphone 1724, a keypad 1726, a display/touchpad 1728, nonremovable memory 1730, removable memory 1732, a power source 1734, a global positioning system (GPS) chipset 1736, and/or other peripherals 1738, among others. It will be appreciated that the WTRU 1702 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.
[0135] The processor 1718 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 1718 may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the WTRU 1702 to operate in a wireless environment. The processor 1718 may be coupled to the transceiver 1720, which may be coupled to the transmit/receive element 1722. While FIG. 17B depicts the processor 1718 and the transceiver 1720 as separate components, it will be appreciated that the processor 1718 and the transceiver 1720 may be integrated together in an electronic package or chip.
[0136] The transmit/receive element 1722 may be configured to transmit signals to, or receive signals from, a base station (e.g., the base station 1714a) over the air interface 1716. For example, in one embodiment, the transmit/receive element 1722 may be an antenna configured to transmit and/or receive RF signals. In an embodiment, the transmit/receive element 1722 may be an emitter/detector configured to transmit and/or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit/receive element 1722 may be configured to transmit and/or receive both RF and light signals. It will be appreciated that the transmit/receive element 1722 may be configured to transmit and/or receive any combination of wireless signals.
[0137] Although the transmit/receive element 1722 is depicted in FIG. 17B as a single element, the WTRU 1702 may include any number of transmit/receive elements 1722. More specifically, the WTRU 1702 may employ MTMO technology. Thus, in one embodiment, the WTRU 1702 may include two or more transmit/receive elements 1722 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 1716.
[0138] The transceiver 1720 may be configured to modulate the signals that are to be transmitted by the transmit/receive element 1722 and to demodulate the signals that are received by the transmit/receive element 1722. As noted above, the WTRU 1702 may have multi-mode capabilities. Thus, the transceiver 1720 may include multiple transceivers for enabling the WTRU 1702 to communicate via multiple RATs, such as R and IEEE 802.11, for example.
[0139] The processor 1718 of the WTRU 1702 may be coupled to, and may receive user input data from, the speaker/microphone 1724, the keypad 1726, and/or the display/touchpad 1728 (e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit). The processor 1718 may also output user data to the speaker/microphone 1724, the keypad 1726, and/or the display/touchpad 1728. In addition, the processor 1718 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 1730 and/or the removable memory 1732. The non-removable memory 1730 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 1732 may include a subscriber identity module (SFM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 1718 may access information from, and store data in, memory that is not physically located on the WTRU 1702, such as on a server or a home computer (not shown).
[0140] The processor 1718 may receive power from the power source 1734, and may be configured to distribute and/or control the power to the other components in the WTRU 1702. The power source 1734 may be any suitable device for powering the WTRU 1702. For example, the power source 1734 may include one or more dry cell batteries (e.g., nickel -cadmium (NiCd),
nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.
[0141] The processor 1718 may also be coupled to the GPS chipset 1736, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 1702. In addition to, or in lieu of, the information from the GPS chipset 1736, the WTRU 1702 may receive location information over the air interface 1716 from a base station (e.g., base stations 1714a, 1714b) and/or determine its location based on the timing of the signals being received from two or more nearby base stations. It will be appreciated that the WTRU 1702 may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment.
[0142] The processor 1718 may further be coupled to other peripherals 1738, which may include one or more software and/or hardware modules that provide additional features, functionality and/or wired or wireless connectivity. For example, the peripherals 1738 may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photographs and/or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a Virtual Reality and/or Augmented Reality (VR/AR) device, an activity tracker, and the like. The peripherals 1738 may include one or more sensors, the sensors may be one or more of a gyroscope, an accelerometer, a hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and/or a humidity sensor.
[0143] The WTRU 1702 may include a full duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for both the UL (e.g., for transmission) and downlink (e.g., for reception) may be concurrent and/or simultaneous. The full duplex radio may include an interference management unit 1739 to reduce and or substantially eliminate self-interference via either hardware (e.g., a choke) or signal processing via a processor (e.g., a separate processor (not shown) or via processor 1718). In an embodiment, the WRTU 1702 may include a half-duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for either the UL (e.g., for transmission) or the downlink (e.g., for reception)).
[0144] Although features and elements are described above in particular combinations, one of ordinary skill in the art will appreciate that each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein may
be implemented in a computer program, software, or firmware incorporated in a computer- readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD- ROM disks, and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.
Claims
1. A method of providing a teleconference showing remote meeting sites from a user- dependent viewpoint comprising:
defining a telepresence session virtual geometry for a telepresence session involving a local site having a local telepresence system and a first remote site having a first remote telepresence system;
receiving, at the local telepresence system, a viewpoint position of a first remote user at the first remote site with respect to the telepresence session virtual geometry; capturing video feeds from a plurality of cameras of a first local user of the local telepresence system at the local site;
generating, from the plurality of captured video feeds, a panorama video corresponding to the viewpoint position of the first remote user with respect to the telepresence session virtual geometry; and
sending, from the local telepresence system, the generated panorama video to the first remote telepresence system at the first remote site, for viewing by the first remote user.
2. The method of claim 1 , further comprising receiving information representing a
configuration of a display of the first remote telepresence system at the first remote site; and wherein the panorama video is generated based at least on part on the configuration of the display of the first remote telepresence system at the first remote site.
3. The method of any of claims 1 -2, wherein generating the panorama video corresponding to the viewpoint position of the first remote user comprises:
using the plurality of captured video feeds, generating at the local telepresence system a real-time 3D model of the first local user; and based on the generated real-time 3D model, generating at the local telepresence system a virtual view of the first local user with respect to the received viewpoint position of the first remote user.
4. The method of any of claims 1 -2, wherein generating the panorama video corresponding to the viewpoint position of the first remote user comprises:
selecting at least a first and a second captured video feed based at least in part on the viewpoint position of the first remote user, wherein the first and second captured video feeds are captured by a first and a second camera of the plurality of cameras; and stitching together the selected at least first and second captured video feeds to generate the panorama video.
5. The method of any of claims 1-2, wherein the plurality of cameras of the local
telepresence system are configured for light-field capture; and wherein generating the panorama video corresponding to the viewpoint position of the first remote user comprises light-field-rendering.
6. The method of any of claims 1-5, wherein sending the generated panorama video to the first remote telepresence system comprises transmitting the generated panorama video from the local telepresence system using a H.264 MVC type coding method.
7. The method of any of claims 1-6, wherein the telepresence session further involves a second remote site having a second remote telepresence system, the method further comprising:
receiving, at the local telepresence system, a viewpoint position of a second remote user at the second remote site with respect to the telepresence session virtual geometry;
generating, from the plurality of captured video feeds, a second panorama video corresponding to the viewpoint position of the second remote user with respect to the telepresence session virtual geometry; and
sending, from the local telepresence system, the generated second panorama video to the second remote telepresence system for viewing by the second remote user.
8. The method of claim 7, further comprising receiving information representing a
configuration of a display of the second remote telepresence system at the second remote site; and wherein the second panorama video is generated based at least on part on the configuration of the display of the second remote telepresence system at the second remote site.
The method of any of claims 1-8, further compri
receiving at the local telepresence system views of at least the first remote user; and
displaying the received views to the first local user, wherein each received view is displayed at a position corresponding to a viewpoint position of the first local user with respect to the telepresence session virtual geometry.
10. The method of claim 9, wherein the displayed views are displayed to the first local user using an augmented reality headset.
11. The method of claim 9, wherein the received views of the respective remote users are stitched into a panoramic view for display to the first local user.
12. The method of claim 9, wherein the received views are displayed to the first local user using a multi-view display.
13. The method of claim 9, wherein the viewpoint position of the first local user with respect to the telepresence session virtual geometry is determined by a tracking system of the local telepresence system.
14. A system for providing a teleconference showing remote meeting sites from a user- dependent viewpoint, the system comprising a processor and a non-transitory computer- readable medium storing instructions operative, when executed on the processor, to perform functions comprising:
defining a telepresence session virtual geometry for a telepresence session involving a local site having a local telepresence system and a first remote site having a first remote telepresence system;
receiving, at the local telepresence system, a viewpoint position of a first remote user at the first remote site with respect to the telepresence session virtual geometry; capturing video feeds from a plurality of cameras of a first local user of the local telepresence system at the local site;
generating, from the plurality of captured video feeds, a panorama video corresponding to the viewpoint position of the first remote user with respect to the telepresence session virtual geometry; and
sending, from the local telepresence system, the generated panorama video to the first remote telepresence system at the first remote site, for viewing by the first remote user.
15. The system of claim 14, wherein the system further comprises an augmented reality display used by the first local user, and wherein the instructions are further operative to perform functions comprising:
receiving at the local telepresence system views of at least the first remote user; and
displaying the received views to the first local user via the augmented reality display, wherein each received view is displayed at a position corresponding to a viewpoint position of the first local user with respect to the telepresence session virtual geometry.
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201662378630P | 2016-08-23 | 2016-08-23 | |
US62/378,630 | 2016-08-23 | ||
US201662428301P | 2016-11-30 | 2016-11-30 | |
US62/428,301 | 2016-11-30 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2018039071A1 true WO2018039071A1 (en) | 2018-03-01 |
Family
ID=59738502
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2017/047605 WO2018039071A1 (en) | 2016-08-23 | 2017-08-18 | Method and system for showing remote meeting sites from user-dependent viewpoints |
Country Status (1)
Country | Link |
---|---|
WO (1) | WO2018039071A1 (en) |
Cited By (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2019204026A1 (en) * | 2018-04-17 | 2019-10-24 | Microsoft Technology Licensing, Llc | Telepresence device |
WO2019204027A1 (en) * | 2018-04-17 | 2019-10-24 | Microsoft Technology Licensing, Llc | Telepresence devices operation methods |
WO2021011083A1 (en) * | 2019-07-18 | 2021-01-21 | Microsoft Technology Licensing, Llc | Dynamic detection and correction of light field camera array miscalibration |
WO2021011087A1 (en) * | 2019-07-18 | 2021-01-21 | Microsoft Technology Licensing, Llc | Device pose detection and pose-related image capture and processing for light field based telepresence communications |
WO2021130406A1 (en) * | 2019-12-23 | 2021-07-01 | Teknologian Tutkimuskeskus Vtt Oy | Method for a telepresence system |
US11082659B2 (en) | 2019-07-18 | 2021-08-03 | Microsoft Technology Licensing, Llc | Light field camera modules and light field camera module arrays |
US11336833B1 (en) | 2021-02-26 | 2022-05-17 | Microsoft Technology Licensing, Llc | Remote user field of view-based camera orienting |
WO2022171767A1 (en) * | 2021-02-12 | 2022-08-18 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Video conference apparatus, video conference method and computer program using a spatial virtual reality environment |
US11528451B2 (en) | 2018-06-18 | 2022-12-13 | Eyecon As | Video conferencing system |
US11553123B2 (en) | 2019-07-18 | 2023-01-10 | Microsoft Technology Licensing, Llc | Dynamic detection and correction of light field camera array miscalibration |
US20230388454A1 (en) * | 2021-02-12 | 2023-11-30 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Video conference apparatus, video conference method and computer program using a spatial virtual reality environment |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030067536A1 (en) * | 2001-10-04 | 2003-04-10 | National Research Council Of Canada | Method and system for stereo videoconferencing |
WO2008101117A1 (en) | 2007-02-14 | 2008-08-21 | Teliris, Inc. | Telepresence conference room layout, dynamic scenario manager, diagnostics and control system and method |
US20090207233A1 (en) * | 2008-02-14 | 2009-08-20 | Mauchly J William | Method and system for videoconference configuration |
US7808540B2 (en) | 2007-01-09 | 2010-10-05 | Eastman Kodak Company | Image capture and integrated display apparatus |
US20120050458A1 (en) | 2010-08-31 | 2012-03-01 | Cisco Technology, Inc. | System and method for providing depth adaptive video conferencing |
US20140104368A1 (en) * | 2011-07-06 | 2014-04-17 | Kar-Han Tan | Telepresence portal system |
US20140146127A1 (en) | 2012-11-29 | 2014-05-29 | Cisco Technology, Inc. | Capturing Video through a Display |
US20150213650A1 (en) * | 2014-01-24 | 2015-07-30 | Avaya Inc. | Presentation of enhanced communication between remote participants using augmented and virtual reality |
-
2017
- 2017-08-18 WO PCT/US2017/047605 patent/WO2018039071A1/en active Application Filing
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030067536A1 (en) * | 2001-10-04 | 2003-04-10 | National Research Council Of Canada | Method and system for stereo videoconferencing |
US7808540B2 (en) | 2007-01-09 | 2010-10-05 | Eastman Kodak Company | Image capture and integrated display apparatus |
WO2008101117A1 (en) | 2007-02-14 | 2008-08-21 | Teliris, Inc. | Telepresence conference room layout, dynamic scenario manager, diagnostics and control system and method |
US20090207233A1 (en) * | 2008-02-14 | 2009-08-20 | Mauchly J William | Method and system for videoconference configuration |
US20120050458A1 (en) | 2010-08-31 | 2012-03-01 | Cisco Technology, Inc. | System and method for providing depth adaptive video conferencing |
US20140104368A1 (en) * | 2011-07-06 | 2014-04-17 | Kar-Han Tan | Telepresence portal system |
US20140146127A1 (en) | 2012-11-29 | 2014-05-29 | Cisco Technology, Inc. | Capturing Video through a Display |
US20150213650A1 (en) * | 2014-01-24 | 2015-07-30 | Avaya Inc. | Presentation of enhanced communication between remote participants using augmented and virtual reality |
Non-Patent Citations (10)
Title |
---|
CHEN; VETRO: "Next-Generation 3d Formats with Depth Map Support", IEEE MULTIMEDIA, vol. 21, no. 2, 2014, pages 90 - 94, XP011549002, DOI: doi:10.1109/MMUL.2014.31 |
DOMAGOJ BARICEVIC; CHA LEE; MATTHEW TURK; TOBIAS HOLLERER; DOUG A. BOWMAN: "A Hand-Held AR Magic Lens with User-Perspective Rendering", ISMAR, vol. 10, 2012 |
DOU ET AL.: "Room-Sized Informal Telepresence System", IEEE VIRTUAL REALITY, vol. 4, 2012 |
HIRSCH ET AL.: "BiDi Screen: A Thin, Depth-Sensing LCD for 3D Interaction Using Light Fields", ACM TRANSACTIONS ON GRAPHICS (TOG, vol. 28, no. 5, 2009, pages 9, XP002645234, DOI: doi:10.1145/1618452.1618505 |
J. YANG ET AL.: "A Real-Time Distributed Light Field Camera", EUROGRAPHICS WORKSHOP ON RENDERING, vol. 1-10, 2002 |
T. MOONS ET AL.: "3D Reconstruction from Multiple Images Part 1: Principles", FOUNDATIONS AND TRENDS IN COMPUTER GRAPHICS AND VISION, vol. 4, no. 4, 2008, pages 287 - 398 |
TOLA ET AL.: "Virtual View Generation with a Hybrid Camera Array", CVLAB-REPORT-2009-001, EPFL, 2009 |
YANG ET AL.: "Creating Adaptive Views for Group Video Teleconferencing-An Image-Based Approach", INTERNATIONAL WORKSHOP ON IMMERSIVE TELEPRESENCE, 2002 |
ZHANG ET AL.: "Viewport: A Distributed, Immersive Teleconferencing System with Infrared Dot Pattern", IEEE MULTIMEDIA, vol. 20, no. 1, 2013, pages 17 - 27, XP011493977, DOI: doi:10.1109/MMUL.2013.12 |
ZOMET ET AL.: "Mosaicing New Views: The Crossed-Slits Projection", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, vol. 25, no. 6, 2003, pages 741 - 754, XP001185047, DOI: doi:10.1109/TPAMI.2003.1201823 |
Cited By (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2019204026A1 (en) * | 2018-04-17 | 2019-10-24 | Microsoft Technology Licensing, Llc | Telepresence device |
WO2019204027A1 (en) * | 2018-04-17 | 2019-10-24 | Microsoft Technology Licensing, Llc | Telepresence devices operation methods |
US10554928B2 (en) | 2018-04-17 | 2020-02-04 | Microsoft Technology Licensing, Llc | Telepresence device |
US11089265B2 (en) | 2018-04-17 | 2021-08-10 | Microsoft Technology Licensing, Llc | Telepresence devices operation methods |
US11528451B2 (en) | 2018-06-18 | 2022-12-13 | Eyecon As | Video conferencing system |
US11064154B2 (en) | 2019-07-18 | 2021-07-13 | Microsoft Technology Licensing, Llc | Device pose detection and pose-related image capture and processing for light field based telepresence communications |
US11082659B2 (en) | 2019-07-18 | 2021-08-03 | Microsoft Technology Licensing, Llc | Light field camera modules and light field camera module arrays |
WO2021011087A1 (en) * | 2019-07-18 | 2021-01-21 | Microsoft Technology Licensing, Llc | Device pose detection and pose-related image capture and processing for light field based telepresence communications |
US11270464B2 (en) | 2019-07-18 | 2022-03-08 | Microsoft Technology Licensing, Llc | Dynamic detection and correction of light field camera array miscalibration |
WO2021011083A1 (en) * | 2019-07-18 | 2021-01-21 | Microsoft Technology Licensing, Llc | Dynamic detection and correction of light field camera array miscalibration |
US11553123B2 (en) | 2019-07-18 | 2023-01-10 | Microsoft Technology Licensing, Llc | Dynamic detection and correction of light field camera array miscalibration |
WO2021130406A1 (en) * | 2019-12-23 | 2021-07-01 | Teknologian Tutkimuskeskus Vtt Oy | Method for a telepresence system |
WO2022171767A1 (en) * | 2021-02-12 | 2022-08-18 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Video conference apparatus, video conference method and computer program using a spatial virtual reality environment |
US20230388454A1 (en) * | 2021-02-12 | 2023-11-30 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Video conference apparatus, video conference method and computer program using a spatial virtual reality environment |
US11336833B1 (en) | 2021-02-26 | 2022-05-17 | Microsoft Technology Licensing, Llc | Remote user field of view-based camera orienting |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10841537B2 (en) | Spatially faithful telepresence supporting varying geometries and moving users | |
WO2018039071A1 (en) | Method and system for showing remote meeting sites from user-dependent viewpoints | |
US11363240B2 (en) | System and method for augmented reality multi-view telepresence | |
WO2018005235A1 (en) | System and method for spatial interaction using automatically positioned cameras | |
US11711504B2 (en) | Enabling motion parallax with multilayer 360-degree video | |
EP3219098B1 (en) | System and method for 3d telepresence | |
US12047552B2 (en) | Method and system for near-eye focal plane overlays for 3D perception of content on 2D displays | |
JP7263451B2 (en) | Layered Enhanced Entertainment Experience | |
WO2018175335A1 (en) | Method and system for discovering and positioning content into augmented reality space | |
US20160353058A1 (en) | Method and apparatus to present three-dimensional video on a two-dimensional display driven by user interaction | |
WO2018200337A1 (en) | System and method for simulating light transport between virtual and real objects in mixed reality | |
US11431901B2 (en) | Aggregating images to generate content | |
WO2021131992A1 (en) | Terminal device, application server, reception method, and transmission method | |
JP3488096B2 (en) | Face image control method in three-dimensional shared virtual space communication service, three-dimensional shared virtual space communication device, and program recording medium therefor | |
US10482671B2 (en) | System and method of providing a virtual environment | |
KR20220046356A (en) | A method and an apparatus for rendering a content in a mobile communication system | |
JP2019513310A (en) | Holographic video capture and telepresence system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 17758754 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 17758754 Country of ref document: EP Kind code of ref document: A1 |