US20220005215A1 - Efficient encoding of depth data across devices - Google Patents
Efficient encoding of depth data across devices Download PDFInfo
- Publication number
- US20220005215A1 US20220005215A1 US17/361,143 US202117361143A US2022005215A1 US 20220005215 A1 US20220005215 A1 US 20220005215A1 US 202117361143 A US202117361143 A US 202117361143A US 2022005215 A1 US2022005215 A1 US 2022005215A1
- Authority
- US
- United States
- Prior art keywords
- camera
- user
- depth
- minimum
- equipped device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/162—User input
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
- H04N21/4223—Cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/632—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing using a connection between clients on a wide area network, e.g. setting up a peer-to-peer communication via Internet for retrieving video segments from the hard-disk of other client devices
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/845—Structuring of content, e.g. decomposing content into time segments
- H04N21/8451—Structuring of content, e.g. decomposing content into time segments using Advanced Video Coding [AVC]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/15—Conference systems
- H04N7/157—Conference systems defining a virtual conference space and using avatars or agents
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B2027/0178—Eyeglass type
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0179—Display position adjusting means not related to the information to be displayed
- G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
Definitions
- One or more embodiments relate generally to augmented reality, and in particular, generating efficient encoding of depth data across devices.
- Augmented reality is an enhanced version of the real physical world that is achieved through the use of digital visual elements, sound, or other sensory stimuli delivered via technology.
- One embodiment provides a method comprising determining a spatial relationship between an augmented reality (AR) device and a camera-equipped device.
- the AR device is worn by a user.
- the camera-equipped device is positioned within proximity of the user.
- the method comprises determining, based on the spatial relationship, minimum and maximum boundaries of a user volume encompassing a head and one or more extremities of the user.
- the minimum and maximum boundaries represent a reduced depth range that is narrower than a full depth range of a camera frustum of the camera-equipped device.
- the method further comprises encoding, based on the minimum and maximum boundaries, a depth image frame captured via the camera-equipped device, and transmitting the encoded depth image frame over a remote network connection.
- a reconstructed 3D image based in part on the encoded depth image frame is rendered by a different AR device.
- FIG. 1 illustrates an example computing architecture for implementing efficient encoding of depth data across devices, in one or more embodiments
- FIG. 2 illustrates an example system for efficient encoding of depth data across devices, in one or more embodiments
- FIG. 3 illustrates an example workflow for the system, in one or more embodiments
- FIG. 4 is a flowchart of an example process for calibrating a position of the camera-equipped device in 3D coordinate space
- FIG. 5 is a flowchart of an example process for calculating z values for different parts (e.g., head and hands/feet) of the local user relative to the position of the camera-equipped device, in one or more embodiments;
- FIG. 6 illustrates an example encoding process, in one or more embodiments
- FIG. 7 illustrates an example decoding process, in one or more embodiments
- FIG. 8 illustrates an example exchange of data between the AR device and the camera-equipped device, in one or more embodiments
- FIG. 9A illustrates an example full depth range, in one or more embodiment
- FIG. 9B illustrates an example reduced depth range, in one or more embodiments.
- FIG. 10A illustrates an example user volume encompassing the local user at the start of a holocall, in one or more embodiments
- FIG. 10B illustrates another example user volume encompassing the local user raising a hand during the holocall, in one or more embodiments
- FIG. 10C illustrates another example user volume encompassing the local user with a hand extended forward during the holocall, in one or more embodiments
- FIG. 10D illustrates another example user volume encompassing the local user lowering a raised hand during the holocall, in one or more embodiments
- FIG. 10E illustrates another example user volume encompassing the local user positioning a lowered hand on the side of the body of the local user during the holocall, in one or more embodiments
- FIG. 10F illustrates another example user volume encompassing the local user moving closer forward to the camera-equipped device during the holocall, in one or more embodiments
- FIG. 10G illustrates another example user volume encompassing the local user moving further closer forward to the camera-equipped device during the holocall, in one or more embodiments
- FIG. 10H illustrates another example user volume encompassing the local user moving further backwards from the camera-equipped device during the holocall, in one or more embodiment
- FIG. 11 illustrates another example system for efficient encoding of depth data across devices, in one or more embodiments
- FIG. 12 illustrates another example system for efficient encoding of depth data across devices, in one or more embodiments
- FIG. 13 is a flowchart of an example process for efficient encoding of depth data across devices, in one or more embodiments.
- FIG. 14 is an exemplary high-level block diagram showing an information processing system comprising a computer system useful for implementing disclosed embodiments.
- One or more embodiments relate generally to augmented reality, and in particular, generating efficient encoding of depth data across devices.
- One embodiment provides a method comprising determining a spatial relationship between an augmented reality (AR) device and a camera-equipped device.
- the AR device is worn by a user.
- the camera-equipped device is positioned within proximity of the user.
- the method comprises determining, based on the spatial relationship, minimum and maximum boundaries of a user volume encompassing a head and one or more extremities of the user.
- the minimum and maximum boundaries represent a reduced depth range that is narrower than a full depth range of a camera frustum of the camera-equipped device.
- the method further comprises encoding, based on the minimum and maximum boundaries, a depth image frame captured via the camera-equipped device, and transmitting the encoded depth image frame over a remote network connection.
- a reconstructed 3D image based in part on the encoded depth image frame is rendered by a different AR device.
- video call generally refers to reception and transmission of audio-video signals by users in different locations, for communication between people in real time.
- Holographic calls are example augmented reality (AR) application use cases that involve AR, immersive, volumetric video calls between multiple users.
- a holocall involves streaming RGB and depth image frames from a first device to a second device across a network.
- a holographic call typically involves an AR/VR headset and a mobile phone.
- One user captures real-time images of themselves via a camera of the mobile phone, and another user views the user in stereoscopic 3D via the AR/VR headset.
- the AR/VR headset renders images of the user at a streaming rate of about 30 fps to enable holocalls to become more broadly available as a consumer product, 3D data about a user must be encoded efficiently to allow for low-latency transport of information about the user across a wide variety of network conditions.
- Lossy codecs i.e., lossless methods
- H264/H265 interframe
- JPEG intraframe
- Existing lossless depth compression algorithms are inflexible with respect to handling fluctuations in quality given network conditions.
- CV/AI-based methods for determining a relevant range of values that a user occupies are not the most effective. For example, in terms of computational costs, processing time increases for higher-resolution images when utilizing AI-based methods for real-time human segmentation (e.g., DeepLab3). If a pair of AR glasses is responsible for rendering images at 60 fps, processing time may be better allocated for rendering. Further, AI-based methods for real-time human segmentation perform less well when part of the face is occluded by sunglasses or a large AR/VR headset, resulting in video streaming instability for users with occluded faces and/or in noisy environments.
- AI-based methods for real-time human segmentation perform less well when part of the face is occluded by sunglasses or a large AR/VR headset, resulting in video streaming instability for users with occluded faces and/or in noisy environments.
- One or more embodiments provide a system for improving efficiency of transporting depth image frames across a network.
- the system is configured to track a position of a device (e.g., a mobile phone) and a position of a user of the device, and automatically adjust encoding of 3D data about the user, such as depth image frames captured by the device, to a volume around the user based on the tracked positions.
- the resulting encoded 3D data is transported from the device across the network with low-latency.
- the system reduces a quantization error associated with quantizing depth range (i.e., range of depth values). To reduce the quantization error, the system determines a range of depth values that the user occupies.
- One or more embodiments are compatible with existing image/video lossy codecs or lossless depth compression techniques, such as JPEG or H264/H265.
- One embodiment provides a single calibration-based method for improving the efficiency and accuracy of person-centric depth data transmission in the context of holocalls.
- FIG. 1 illustrates an example computing architecture 100 for implementing efficient encoding of depth data across devices, in one or more embodiments.
- the computer architecture 100 provides a framework for a holocall between a local user 10 and a remote user 20 .
- the computer architecture 100 comprises at least an AR device 110 and a camera-equipped device 130 .
- the AR device 110 and the camera-equipped device 130 are paired devices.
- the computer architecture 100 comprises at least a remote AR device 160 .
- the AR device 110 includes resources, such as one or more processor units 111 and one or more storage units 112 .
- One or more applications 116 may execute/operate on the AR device 110 utilizing the resources of the AR device 110 .
- the AR device 110 comprises one or more sensor units 114 integrated in or coupled to the AR device 110 , such as a GPS, a motion sensor, etc.
- the AR device 110 comprises one or more I/O units 113 integrated in or coupled to the AR device 110 .
- the one or more I/O units 113 include, but are not limited to, an AR display, etc. Examples of an AR device 110 include, but are not limited to, a pair of AR glasses, an AR headset, etc. During the holocall, the AR device 110 is worn by the local user 10 .
- the AR device 110 comprises a communications unit 115 configured to exchange data with the camera-equipped device 130 over a local network connection (e.g., a wireless connection such as a Wi-Fi connection or a cellular data connection, a wired connection, or a combination of the two).
- the communications unit 115 may comprise any suitable communications circuitry operative to connect to a communications network and to exchange communications operations and media between the AR device 110 and other devices connected to the same communications network.
- the communications unit 115 may be operative to interface with a communications network using any suitable communications protocol such as, for example, Wi-Fi (e.g., an IEEE 802.11 protocol), Bluetooth®, high frequency systems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communication systems), infrared, GSM, GSM plus EDGE, CDMA, quadband, and other cellular protocols, VOIP, TCP-IP, or any other suitable protocol.
- Wi-Fi e.g., an IEEE 802.11 protocol
- Bluetooth® high frequency systems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communication systems), infrared, GSM, GSM plus EDGE, CDMA, quadband, and other cellular protocols, VOIP, TCP-IP, or any other suitable protocol.
- the camera-equipped device 130 includes resources, such as one or more processor units 131 and one or more storage units 132 .
- One or more applications 136 may execute/operate on the camera-equipped device 130 utilizing the resources of the camera-equipped device 130 .
- the camera-equipped device 130 comprises one or more sensor units 134 integrated in or coupled to the camera-equipped device 130 , such as a camera 137 , a microphone, a GPS, a motion sensor, etc.
- the camera-equipped device 130 comprises one or more I/O units 133 integrated in or coupled to the camera-equipped device 130 .
- the one or more I/O units 133 include, but are not limited to, a physical user interface (PUI) and/or a GUI, such as a keyboard, a keypad, a touch interface, a touch screen, a knob, a button, a display screen, etc.
- the local user 10 can utilize at least one I/O unit 133 to configure one or more user preferences, configure one or more parameters, provide input, etc.
- Examples of a camera-equipped device 130 include, but are not limited to, a television (e.g., a smart television), a mobile electronic device (e.g., a tablet, a smart phone, a laptop, etc.), a wearable device (e.g., a smart watch, a smart band, a head-mounted display, smart glasses, etc.), an Internet of things (IoT) device, etc.
- a television e.g., a smart television
- a mobile electronic device e.g., a tablet, a smart phone, a laptop, etc.
- a wearable device e.g., a smart watch, a smart band, a head-mounted display, smart glasses, etc.
- IoT Internet of things
- the camera 137 includes both a depth camera 180 ( FIG. 6 ) and a RGB camera 190 ( FIG. 6 ).
- the camera-equipped device 130 is positioned within proximity of the local user 10 to capture RGB and depth image frames of the local user 10 .
- the camera-equipped device 130 comprises a communications unit 135 configured to exchange data with the AR device 110 over a local network connection (e.g., a wireless connection such as a Wi-Fi connection or a cellular data connection, a wired connection, or a combination of the two).
- the communications unit 135 is further configured to exchange data with a remote device, such as the remote AR device 160 , over a remote network connection (e.g., a wireless connection such as a Wi-Fi connection or a cellular data connection, a wired connection, or a combination of the two).
- the communications unit 135 may comprise any suitable communications circuitry operative to connect to a communications network and to exchange communications operations and media between the camera-equipped device 130 and other devices connected to the same communications network.
- the communications unit 135 may be operative to interface with a communications network using any suitable communications protocol such as, for example, Wi-Fi (e.g., an IEEE 802.11 protocol), Bluetooth high frequency systems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communication systems), infrared, GSM, GSM plus EDGE, CDMA, quadband, and other cellular protocols, VOIP, TCP-IP, or any other suitable protocol.
- Wi-Fi e.g., an IEEE 802.11 protocol
- Bluetooth high frequency systems e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communication systems
- infrared GSM
- GSM plus EDGE Code Division Multiple Access
- CDMA Code Division Multiple Access
- quadband Code Division Multiple Access
- VOIP Voice over IP
- TCP-IP Transmission Control Protocol
- the one or more applications 136 on the camera-equipped device 130 may further include one or more software mobile applications loaded onto or downloaded to the camera-equipped device 130 , such as a camera application, a social media application, a video streaming application, an AR application, etc.
- the remote AR device 160 includes resources, such as one or more processor units 161 and one or more storage units 162 .
- One or more applications 166 may execute/operate on the remote AR device 160 utilizing the resources of the remote AR device 160 .
- the remote AR device 160 comprises one or more sensor units 164 integrated in or coupled to the remote AR device 160 , such as a GPS, a motion sensor, etc.
- the remote AR device 160 comprises one or more I/O units 163 integrated in or coupled to the remote AR device 160 .
- the one or more I/O units 163 include, but are not limited to, an AR display, etc. Examples of a remote AR device 160 include, but are not limited to, a pair of AR glasses, an AR headset, etc. During the holocall, the remote AR device 160 is worn by the remote user 20 .
- the remote AR device 160 comprises a communications unit 165 configured to exchange data with the camera-equipped device 130 over the remote network connection.
- the communications unit 165 may comprise any suitable communications circuitry operative to connect to a communications network and to exchange communications operations and media between the remote AR device 160 and other devices connected to the same communications network.
- the communications unit 165 may be operative to interface with a communications network using any suitable communications protocol such as, for example, Wi-Fi (e.g., an IEEE 802.11 protocol), Bluetooth®, high frequency systems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communication systems), infrared, GSM, GSM plus EDGE, CDMA, quadband, and other cellular protocols, VOIP, TCP-IP, or any other suitable protocol.
- Wi-Fi e.g., an IEEE 802.11 protocol
- Bluetooth® high frequency systems
- high frequency systems e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communication systems
- infrared GSM, G
- FIG. 2 illustrates an example system 550 for efficient encoding of depth data across devices, in one or more embodiments.
- the system 550 comprises: (1) an AR device tracking module 120 , (2) a depth camera module 140 , (3) a local server 150 , and (4) a remote client 170 .
- the one or more applications 116 ( FIG. 1 ) on the AR device 100 include the AR device tracking module 120
- the one or more applications 136 ( FIG. 1 ) on the camera-equipped device 130 include the depth camera module 140 and the local server 150
- the one or more applications 166 ( FIG. 1 ) on the remote AR device 160 include the remote client 170 .
- the AR device tracking module 120 is configured to perform an initial calibration against a position of the camera-equipped device 130 to determine a spatial relationship between the paired devices. During the holocall, for each depth image frame, the AR device tracking module 120 is further configured to: (1) track positions of different parts of the local user 10 , wherein the different parts include the head and one or more extremities (e.g., hands, feet) of the local user 10 , and (2) determine, based on the positions tracked, a reduced depth range for the depth image frame.
- a reduced depth range is narrower than a full depth range (i.e., all depth values within the frustum of the camera 137 ).
- the reduced depth range contains only relevant depth values, i.e., depth values representing a volume the local user 10 (“user volume”) occupies, wherein the user volume encompasses most of the body of the local user 10 including the different parts tracked.
- the AR device tracking module 120 tracks the different parts of the local user 10 to dynamically update the relevant range of depth values for each depth image frame.
- the AR device tracking module 120 is further configured to: (1) track additional positions of different elements in the environment of the local user 10 , wherein the different elements include one or more images or objects within proximity of the local user 10 , and (2) determine, based on the additional positions tracked, a reduced depth range for the depth image frame.
- the one or more images or objects are tracked optically or via other means utilizing either the camera 137 or the AR device 110 .
- the additional positions tracked may expand the reduced depth range (i.e., expand minimum and maximum boundaries).
- the depth camera module 140 is configured to capture, via the camera 137 , RGB and depth image frames of the local user 10 during the holocall.
- the depth camera module 140 is further configured to perform an encoding process that encodes each depth image frame captured based on a reduced depth range (determined via the AR device tracking module 120 ) in the depth image frame.
- the depth camera module 140 is further configured to apply a lossy codec (e.g., JPEG, H264, or H265) to each RGB image frame captured and each resulting encoded depth image frame to better accommodate network bandwidth constraints.
- a lossy codec e.g., JPEG, H264, or H265
- the local server 150 is configured to send/relay a compressed RGB image frame and a compressed, encoded depth image frame (compressed via the depth camera module 140 ) to the remote client 170 as an output stream transported over the remote network connection.
- the remote client 170 is configured to: (1) receive an input stream from the local server 150 over the remote network connection, (2) perform a decoding process on an encoded depth image frame included in the input stream to decode a reduced depth range in the depth image frame, (3) reconstruct a 3D image of the local user 10 based a RGB image frame included in the input stream and the reduced depth range, and (4) render the resulting 3D image on an AR display of the remote AR device 160 .
- FIG. 3 illustrates an example workflow for the system 550 , in one or more embodiments.
- each of the AR device tracking module 120 , the depth camera module 140 , the local server 150 , and the remote client 170 has at least two operating phrases: (1) a setup and calibration phase, and (2) an iterative loop phase.
- the AR device tracking module 120 , the depth camera module 140 , the local server 150 , and the remote client 170 operate as follows: First, the depth camera module 140 establishes a server. Next, the AR device tracking module 120 establishes a tracking client, and connects to the server (established by the depth camera module 140 ). The AR device tracking module 120 then calibrates a position of the camera-equipped device 130 in 3D coordinate space with three axes, i.e., the x-axis, the y-axis, and the z-axis. Next, the depth camera module 140 provides a set K of intrinsic parameters for the camera (“camera intrinsics”) of the camera-equipped device 120 to the local server 150 .
- the local server 150 relays the camera intrinsics K to the remote client 170 over the remote network connection.
- the remote client 170 caches or saves the camera intrinsics K received from the local server 150 .
- the remote client 170 utilizes the camera intrinsics K for 3D reprojection.
- depth pixel depth value
- z value z value
- the AR device tracking module 120 In the iterative loop phase, the AR device tracking module 120 , the depth camera module 140 , the local server 150 , and the remote client 170 operate as follows: For each image frame, the AR device tracking module 120 tracks different parts (e.g., head and hands/feet) of the local user 10 , and calculates z values for the different parts relative to the position of the camera-equipped device 130 , wherein each z value is a point along the z-axis. Next, the AR device tracking module 120 determines, based on the z values, a reduced depth range in the depth image frame. Specifically, the AR device tracking module 120 determines z min and z max , wherein z min and z max are minimum and maximum boundaries of a user volume in the depth image frame.
- different parts e.g., head and hands/feet
- the AR device tracking module 120 determines, based on the z values, a reduced depth range in the depth image frame. Specifically, the AR device tracking module 120
- z min and z max represent minimum and maximum distances, respectively, of the body of the local user 10 relative to the z-axis of the camera-equipped device 130 .
- the AR device tracking module 120 sends z min and z max to the depth camera module 140 .
- the depth camera module 140 In response to receiving z min and z max , the depth camera module 140 captures RGB and depth image frames via the camera. The depth camera module 140 then performs an encoding process on the depth image frame captured based on z min and z max .
- the local server 150 applies a lossy codec (e.g., JPEG, H264, or H265) to the RGB image frame and the resulting encoded depth image frame (from the depth camera module 140 ), and sends/relays, over the remote network connection, the resulting compressed image frames to the remote client 170 as an output stream with z min and z max included in header values.
- a lossy codec e.g., JPEG, H264, or H265
- the reduced depth range reduces quantization error, thereby allowing for higher fidelity reconstruction on the remote client 170 and improving perceptual video quality while streaming on the remote AR device 160 .
- the remote client 170 receives, over the remote network connection, from the local server 150 , an input stream comprising the compressed images frames, z min , and z max .
- the remote client 170 reconstructs a 3D image based on z min and z max , and reprojects the 3D image based on the camera intrinsics K (cached or saved during the setup and calibration phase).
- the reprojected 3D image is rendered in the AR display of the remote AR device 160 .
- Simultaneous localization and mapping is a computational technology involving constructing or updating a map of an unknown environment while simultaneously keeping track of an agent's location within it.
- a device performing SLAM is configured to map its location, create a 3D virtual map, and locate itself inside the map.
- Visual SLAM is a particular type of SLAM that uses only visual inputs to perform location and mapping.
- visual SLAM involves calculating the position and orientation of a device with respect to its surroundings, while mapping the environment at the same time, using only visual inputs from a camera.
- FIG. 4 is a flowchart of an example process 200 for calibrating a position of the camera-equipped device 130 in 3D coordinate space, in one or more embodiments.
- Process block 210 includes using visual slam to calculate a pose of the AR device 110 relative to an arbitrary world-space origin in virtual coordinate space.
- Process block 220 includes aligning the physical camera-equipped device 130 in the virtual coordinate space. The aligning is performed in accordance with process block 230 , process block 240 , process block 250 , or process block 260 .
- Process block 230 includes the local user 10 manually aligning a virtual transform represented by an AR user interface (UI) (rendered by the AR device 110 ) with a physical pose P pos of the camera-equipped device 130 .
- UI AR user interface
- Process block 240 includes the local user 10 aligning the camera-equipped device 130 with a virtual transform represented in an AR UI (rendered by the AR device 110 ).
- Process block 250 includes a fiducial displayed on a display screen of the camera-equipped device 130 automatically aligning a virtual transform with the physical pose P pos of the camera-equipped device 130 .
- Process block 260 includes using an object detection and pose estimation algorithm to estimate the physical pose P pos of the camera-equipped device 130 .
- process block 230 Each of process block 230 , process block 240 , process block 250 , and process block 260 returns to process block 210 .
- process blocks 210 - 260 are performed by the AR device tracking module 120 during the setup and calibration phase.
- the AR device tracking module 120 triggers the AR device 110 to render the AR UI representing the virtual transform.
- the AR device tracking module 120 triggers the camera-equipped device 130 to display the fiducial.
- FIG. 5 is a flowchart of an example process 300 for calculating z values for different parts (e.g., head and hands/feet) of the local user 10 relative to the position of the camera-equipped device 130 , in one or more embodiments.
- Process block 310 includes using visual slam to calculate a position H pos of the head (“head position”) of the local user 10 .
- Process block 320 includes using system/OS/visual slam methods to calculate positions L pos and R pos of the left hand and right hand (“hand positions”) (or the left foot and right foot), respectively, of the local user 10 .
- Process block 330 includes transforming the head position H pos relative to the physical pose P pos of the camera-equipped device 130 to obtain H′ pos .
- Process block 340 includes transforming the hand positions L pos and R pos relative to the physical pose P pos of the camera-equipped device 130 to obtain L′ pos and R′ pos .
- Process block 350 includes iterating through z components of H′ pos , L′ pos , and R′ pos to determine z min and z max .
- Process block 360 includes adding standard buffer amounts B front and B back to z min and z max to account for the body thickness of the local user 10 exceeding the tracked parts (i.e., head and hands/feet).
- Process block 370 includes converting z min and z max to units matching a depth camera format (e.g., m to mm), wherein z min and z max are sent to the depth camera module 140 as the units.
- a depth camera format e.g., m to mm
- process blocks 310 - 370 are performed by the AR device tracking module 120 during the iterative loop phase.
- the AR device tracking module 120 is configured to transform the head position H pos relative to the physical pose P pos of the camera-equipped device 130 to obtain H′ pos , in accordance with equation (1) provided below:
- H′ pos P rot ⁇ 1 *( H pos ⁇ P pos ) (1)
- P rot ⁇ 1 is the inverse of a rotation matrix of the camera-equipped device 130 .
- the AR device tracking module 120 is configured to transform the hand positions L pos and R pos relative to the physical pose P pos of the camera-equipped device 130 to obtain L′ pos and R′ pos , in accordance with equations (2)-(3) provided below:
- R′ pos P rot ⁇ 1 *( R pos ⁇ P pos ) (3).
- the AR device tracking module 120 is configured to iterate through z components of H′ pos , L′ pos , and R′ pos to determine z min and z max , in accordance with equations (4)-(5) provided below:
- the AR device tracking module 120 is configured to add standard buffer amounts B front and B back to z min and z max to account for the body thickness of the local user 10 exceeding the tracked parts (i.e., head and hands/feet), in accordance with equations (6)-(7) provided below:
- FIG. 6 illustrates an example encoding process, in one or more embodiments.
- the depth camera module 140 is configured to receive, as inputs, the following: (1) an array 410 of depth pixels from the depth camera 180 , and (2) z min and z max from the AR device tracking module 120 .
- the array 410 represents a full depth range, i.e., all depth values within the frustum of the depth camera 180 .
- the depth camera module 140 is configured to convert the array 410 to a depth image frame by: (1) performing an encoding process on the array 410 based on z min and z max , and (2) creating a depth bitmap 420 representing the depth image frame based on encoded values resulting from the encoding process.
- the encoding process involves the following operations: For each depth pixel z p of the array 410 , the depth camera module 140 determines a normalized value z′ of the depth pixel z p in accordance with equation (8) provided below:
- the depth camera module 140 performs int8 quantization to convert the normalized value z′ from a higher-precision format into an int8 value z i in a lower-precision format, in accordance with equation (9) provided below:
- the depth camera module 140 encodes the int8 value z i as a 24-bit color value z c , in accordance with equation (10) provided below:
- the depth camera module 140 creates the depth bitmap 420 by setting colors of the depth bitmap 420 that correspond to the depth pixel z p to the 24-bit color value z c .
- the local server 150 is configured to receive, as inputs, the following: (1) a RGB bitmap 400 from the RGB camera 190 , wherein the RGB bitmap 400 represents an RGB image frame captured by the RGB camera 190 , and (2) the depth bitmap 420 from the depth camera module 140 .
- the local server 150 is configured to: (1) compress the RGB bitmap 400 by applying lossy codec to the RGB bitmap 400 , resulting in a compressed RGB bitmap 430 , and (2) compress the depth bitmap 420 by applying lossy codec to the depth bitmap 420 , resulting in a compressed depth bitmap 440 .
- the local server 150 is configured to: (1) convert the compressed RGB bitmap 430 into a first byte array 450 , and (2) convert the compressed depth bitmap 440 into a second byte array 460 that includes z min and z max as header values, and (3) write the first byte array 450 and the second byte array 460 to an output stream.
- the output stream is relayed/transported to the remote client 170 over the remote network connection.
- FIG. 7 illustrates an example decoding process, in one or more embodiments.
- the remote client 170 is configured to read, from an input stream, a first byte array 500 and a second byte array 510 .
- the input stream is relayed/transported from the local server 150 over the remote network connection.
- the second byte array 510 includes z min and z max as header values.
- the remote client 170 is configured to: (1) convert the first byte array 500 to a RGB bitmap 520 representing an RGB image frame, and (2) convert the second byte array 510 to a depth bitmap representing a depth image frame.
- the remote client 170 is configured to perform a decoding process on the depth bitmap based on z min and z max to obtain decoded z values.
- the decoding process involves the following operations: For each 24-bit color value z c of the depth bitmap, the remote client 170 extracts an int8 value z i from the 24-bit color value z c , in accordance with equation (11) provided below:
- the remote client 170 decodes a normalized value z′ from the int8 value z i , in accordance with equation (12) provided below:
- the remote client 170 de-normalizes the normalized value z′ based on z min and z max to obtain a decoded depth pixel z p , in accordance with equation (13) provided below:
- the remote client 170 is configured to perform a 3D reprojection based on RGB bitmap 520 , the decoded z values resulting from the decoding process, and the camera intrinsics K of the camera-equipped device 120 (cached/saved at the start of the session).
- the 3D reprojection involves the remote client 170 reprojecting RGB pixels from the RGB bitmap 520 into space using the decoded z values and intrinsic parameters from K such as U0, V0, Fx, Fy, resulting in a 3D image 530 of the local user 10 that is rendered on the viewport of the remote AR device 160 .
- FIG. 8 illustrates an example exchange of data between the AR device 110 and the camera-equipped device 130 , in one or more embodiments.
- the AR device tracking module 120 operates on the AR device 110
- the depth camera module 140 and the local server 150 operate on the camera-equipped device 130 .
- the AR device 110 is a pair of AR glasses worn on the head of the local user 10
- the camera-equipped device 130 is a mobile phone.
- the AR device tracking module 120 calculates z values for the head and hands of the local user 10 relative to the position of the camera-equipped device 130 , then determines, based on the z values, a user volume 550 encompassing most of the local user 10 including the heads and hands.
- the user volume represents a reduced depth range containing only relevant depth values (as compared to the full depth range containing all depth values within the frustum of the camera).
- the minimum and maximum boundaries of the user volume are z min and z max , wherein z min is a minimum distance of the body of the local user 10 relative to the z-axis of the camera-equipped device 130 , and z max is a maximum distance of the body of the local user 10 relative to the z-axis of the camera-equipped device 130 .
- the AR device 110 sends z min and z max (determined by the AR device tracking module 120 ) to the camera-equipped device 130 .
- each of z min and z max is 16 bits in size, such that a total size of bits sent from the AR device 110 for each depth image frame is 32 bits.
- the camera-equipped device 130 captures RGB and depth image frames via the camera.
- the depth camera module 140 performs an encoding process on the depth image frame based on z min and z max . For example, in one embodiment, the depth camera module 140 performs 8-bit encoding to convert each depth pixel from a higher-precision format (DEPTH16) to a lower-precision format (8-bit).
- the camera-equipped device 130 sends, via the local server 150 , compressed RGB and encoded depth image frames to the remote client 170 over the remote network connection.
- the computational costs of the system 550 is less than conventional CV/AI-based methods (e.g., AI-based methods for real-time human segmentation).
- FIG. 9A illustrates an example full depth range, in one or more embodiments.
- the full depth range contains all depth values within the frustum of the camera 137 of the camera-equipped device 130 .
- FIG. 9B illustrates an example reduced depth range, in one or more embodiments.
- the AR device tracking module 120 reduces depth range by culling irrelevant depth values, such that the reduced depth range contains only relevant depth values representing a user volume 550 encompassing most of the body of the local user 10 including the different parts (e.g., head and hands/feet) tracked.
- z min 900 mm
- z max 1500 mm.
- FIGS. 10A-10H illustrate changes to z min and z max during a holocall, in one or more embodiments.
- FIG. 10A illustrates an example user volume 600 encompassing the local user 10 at the start of a holocall, in one or more embodiments. As shown in FIG. 10A , at the start of the holocall, the hands of the local user 10 are lowered and positioned on the sides of the body of the local user 10 .
- FIG. 10B illustrates another example user volume 610 encompassing the local user 10 raising a hand during the holocall, in one or more embodiments.
- the minimum distance of the body of the local user 10 relative to the z-axis of the camera-equipped device 130 decreases, resulting in a larger/wider user volume 610 that encompasses the raised hand.
- FIG. 10C illustrates another example user volume 620 encompassing the local user 10 with a hand extended forward during the holocall, in one or more embodiments.
- the minimum distance of the body of the local user 10 relative to the z-axis of the camera-equipped device 130 decreases even further, resulting in an even larger/wider user volume 620 that encompasses the extended hand.
- FIG. 10D illustrates another example user volume 630 encompassing the local user 10 lowering a raised hand during the holocall, in one or more embodiments.
- the minimum distance of the body of the local user 10 relative to the z-axis of the camera-equipped device 130 increases, resulting in a smaller/narrower user volume 630 that encompasses the lowered hand.
- FIG. 10E illustrates another example user volume 640 encompassing the local user 10 positioning a lowered hand on the side of the body of the local user 10 during the holocall, in one or more embodiments.
- the minimum distance of the body of the local user 10 relative to the z-axis of the camera-equipped device 130 increases even further, resulting in a smaller/narrower user volume 640 .
- FIG. 10F illustrates another example user volume 650 encompassing the local user 10 moving closer forward to the camera-equipped device 130 during the holocall, in one or more embodiments.
- FIG. 10G illustrates another example user volume 660 encompassing the local user 10 moving further closer forward to the camera-equipped device 130 during the holocall, in one or more embodiments.
- FIG. 10H illustrates another example user volume 670 encompassing the local user 10 moving further backwards from the camera-equipped device 130 during the holocall, in one or more embodiments.
- FIG. 11 illustrates another example system 560 for efficient encoding of depth data across devices, in one or more embodiments.
- the system 560 splits computation between the AR device 110 and the camera-equipped device 130 .
- the one or more applications 116 ( FIG. 1 ) on the AR device 100 include the AR device tracking module 120 and the local server 150
- the one or more applications 136 ( FIG. 1 ) on the camera-equipped device 130 include the depth camera module 140
- the one or more applications 166 ( FIG. 1 ) on the remote AR device 160 include the remote client 170 .
- FIG. 12 illustrates another example system 570 for efficient encoding of depth data across devices, in one or more embodiments.
- the system 560 performs most computation on the camera-equipped device 130 .
- the one or more applications 136 ( FIG. 1 ) on the camera-equipped device 130 include the AR device tracking module 120 , the depth camera module 140 , and the local server 150 .
- the system 570 includes a remote user device 195 on which the remote client 170 executes/operates.
- the AR device 110 relays sensor data to the AR device tracking module 120 on the camera-equipped device 130 over the local network connection.
- the remote AR device 160 receives a 3D image (reconstructed by the remote client 170 ) from the remote user device 195 over a local network connection, and renders the 3D image on the AR display of the remote AR device 160 .
- FIG. 13 is a flowchart of an example process 800 for efficient encoding of depth data across devices, in one or more embodiments.
- Process block 810 includes determining a spatial relationship between an AR device (e.g., AR device 110 ) and a camera-equipped device (e.g., camera-equipped device 130 ), wherein the AR device is worn by a user (e.g., local user 10 ), and the camera-equipped device is positioned within proximity of the user.
- an AR device e.g., AR device 110
- a camera-equipped device e.g., camera-equipped device 130
- the AR device is worn by a user (e.g., local user 10 )
- the camera-equipped device is positioned within proximity of the user.
- Process block 820 includes determining, based on the spatial relationship, minimum and maximum boundaries (z min and z max ) of a user volume encompassing a head and one or more extremities of the user, wherein the minimum and maximum boundaries represent a reduced depth range that is narrower than a full depth range of a camera frustum of the camera-equipped device.
- Process block 830 includes encoding, based on the minimum and maximum boundaries, a depth image frame captured via the camera-equipped device.
- Process block 840 includes transmitting the encoded depth image frame over a remote network connection, wherein a reconstructed 3D image based in part on the encoded depth image frame is rendered by a different AR device (e.g., remote AR device 160 ).
- process blocks 810 - 840 may be performed one or more components of the system 550 , 560 , 570 , such as the AR device tracking module 120 , the depth camera module 140 , the local server 150 , and the remote client 170 .
- FIG. 14 is a high-level block diagram showing an information processing system comprising a computer system 900 useful for implementing the disclosed embodiments.
- Computer system 900 may be incorporated the AR device 110 , the camera-equipped device 130 , the remote AR device 160 , and/or the remote user device 195 .
- the computer system 900 includes one or more processors 901 , and can further include an electronic display device 902 (for displaying video, graphics, text, and other data), a main memory 903 (e.g., random access memory (RAM)), storage device 904 (e.g., hard disk drive), removable storage device 905 (e.g., removable storage drive, removable memory module, a magnetic tape drive, optical disk drive, computer readable medium having stored therein computer software and/or data), viewer interface device 906 (e.g., keyboard, touch screen, keypad, pointing device), and a communication interface 907 (e.g., modem, a network interface (such as an Ethernet card), a communications port, or a PCMCIA slot and card).
- a communication interface 907 e.g., modem, a network interface (such as an Ethernet card), a communications port, or a PCMCIA slot and card.
- the communication interface 907 allows software and data to be transferred between the computer system and external devices.
- the system 900 further includes a communications infrastructure 908 (e.g., a communications bus, cross-over bar, or network) to which the aforementioned devices/modules 901 through 907 are connected.
- a communications infrastructure 908 e.g., a communications bus, cross-over bar, or network
- Information transferred via communications interface 907 may be in the form of signals such as electronic, electromagnetic, optical, or other signals capable of being received by communications interface 907 , via a communication link that carries signals and may be implemented using wire or cable, fiber optics, a phone line, a cellular phone link, an radio frequency (RF) link, and/or other communication channels.
- Computer program instructions representing the block diagram and/or flowcharts herein may be loaded onto a computer, programmable data processing apparatus, or processing devices to cause a series of operations performed thereon to generate a computer implemented process.
- processing instructions for process 200 ( FIG. 4 ), processing instructions for process 300 ( FIG. 5 ), and/or processing instructions for process 800 ( FIG. 13 ) may be stored as program instructions on the memory 903 , storage device 904 and the removable storage device 905 for execution by the processor 901 .
- Embodiments have been described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products. Each block of such illustrations/diagrams, or combinations thereof, can be implemented by computer program instructions.
- the computer program instructions when provided to a processor produce a machine, such that the instructions, which execute via the processor create means for implementing the functions/operations specified in the flowchart and/or block diagram.
- Each block in the flowchart/block diagrams may represent a hardware and/or software module or logic. In alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures, concurrently, etc.
- the terms “computer program medium,” “computer usable medium,” “computer readable medium”, and “computer program product,” are used to generally refer to media such as main memory, secondary memory, removable storage drive, a hard disk installed in hard disk drive, and signals. These computer program products are means for providing software to the computer system.
- the computer readable medium allows the computer system to read data, instructions, messages or message packets, and other computer readable information from the computer readable medium.
- the computer readable medium may include non-volatile memory, such as a floppy disk, ROM, flash memory, disk drive memory, a CD-ROM, and other permanent storage. It is useful, for example, for transporting information, such as data and computer instructions, between computer systems.
- Computer program instructions may be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
- aspects of the embodiments may be embodied as a system, method or computer program product. Accordingly, aspects of the embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, aspects of the embodiments may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.
- the computer readable medium may be a computer readable storage medium.
- a computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.
- a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
- Computer program code for carrying out operations for aspects of one or more embodiments may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages.
- the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
- the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
- LAN local area network
- WAN wide area network
- Internet Service Provider for example, AT&T, MCI, Sprint, EarthLink, MSN, GTE, etc.
- These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
- the computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
- each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s).
- the functions noted in the block may occur out of the order noted in the figures.
- two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Theoretical Computer Science (AREA)
- Optics & Photonics (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
Description
- This application claims priority from U.S. Provisional Patent Application Ser. No. 63/047,142, filed on Jul. 1, 2020, incorporated herein by reference.
- One or more embodiments relate generally to augmented reality, and in particular, generating efficient encoding of depth data across devices.
- Augmented reality (AR) is an enhanced version of the real physical world that is achieved through the use of digital visual elements, sound, or other sensory stimuli delivered via technology.
- One embodiment provides a method comprising determining a spatial relationship between an augmented reality (AR) device and a camera-equipped device. The AR device is worn by a user. The camera-equipped device is positioned within proximity of the user. The method comprises determining, based on the spatial relationship, minimum and maximum boundaries of a user volume encompassing a head and one or more extremities of the user. The minimum and maximum boundaries represent a reduced depth range that is narrower than a full depth range of a camera frustum of the camera-equipped device. The method further comprises encoding, based on the minimum and maximum boundaries, a depth image frame captured via the camera-equipped device, and transmitting the encoded depth image frame over a remote network connection. A reconstructed 3D image based in part on the encoded depth image frame is rendered by a different AR device.
- These and other features, aspects and advantages of the one or more embodiments will become understood with reference to the following description, appended claims and accompanying figures.
-
FIG. 1 illustrates an example computing architecture for implementing efficient encoding of depth data across devices, in one or more embodiments; -
FIG. 2 illustrates an example system for efficient encoding of depth data across devices, in one or more embodiments; -
FIG. 3 illustrates an example workflow for the system, in one or more embodiments; -
FIG. 4 is a flowchart of an example process for calibrating a position of the camera-equipped device in 3D coordinate space; -
FIG. 5 is a flowchart of an example process for calculating z values for different parts (e.g., head and hands/feet) of the local user relative to the position of the camera-equipped device, in one or more embodiments; -
FIG. 6 illustrates an example encoding process, in one or more embodiments; -
FIG. 7 illustrates an example decoding process, in one or more embodiments; -
FIG. 8 illustrates an example exchange of data between the AR device and the camera-equipped device, in one or more embodiments; -
FIG. 9A illustrates an example full depth range, in one or more embodiment; -
FIG. 9B illustrates an example reduced depth range, in one or more embodiments; -
FIG. 10A illustrates an example user volume encompassing the local user at the start of a holocall, in one or more embodiments; -
FIG. 10B illustrates another example user volume encompassing the local user raising a hand during the holocall, in one or more embodiments; -
FIG. 10C illustrates another example user volume encompassing the local user with a hand extended forward during the holocall, in one or more embodiments; -
FIG. 10D illustrates another example user volume encompassing the local user lowering a raised hand during the holocall, in one or more embodiments; -
FIG. 10E illustrates another example user volume encompassing the local user positioning a lowered hand on the side of the body of the local user during the holocall, in one or more embodiments; -
FIG. 10F illustrates another example user volume encompassing the local user moving closer forward to the camera-equipped device during the holocall, in one or more embodiments; -
FIG. 10G illustrates another example user volume encompassing the local user moving further closer forward to the camera-equipped device during the holocall, in one or more embodiments; -
FIG. 10H illustrates another example user volume encompassing the local user moving further backwards from the camera-equipped device during the holocall, in one or more embodiment; -
FIG. 11 illustrates another example system for efficient encoding of depth data across devices, in one or more embodiments; -
FIG. 12 illustrates another example system for efficient encoding of depth data across devices, in one or more embodiments; -
FIG. 13 is a flowchart of an example process for efficient encoding of depth data across devices, in one or more embodiments; and -
FIG. 14 is an exemplary high-level block diagram showing an information processing system comprising a computer system useful for implementing disclosed embodiments. - The following description is made for the purpose of illustrating the general principles of one or more embodiments and is not meant to limit the inventive concepts claimed herein. Further, particular features described herein can be used in combination with other described features in each of the various possible combinations and permutations. Unless otherwise specifically defined herein, all terms are to be given their broadest possible interpretation including meanings implied from the specification as well as meanings understood by those skilled in the art and/or as defined in dictionaries, treatises, etc.
- One or more embodiments relate generally to augmented reality, and in particular, generating efficient encoding of depth data across devices. One embodiment provides a method comprising determining a spatial relationship between an augmented reality (AR) device and a camera-equipped device. The AR device is worn by a user. The camera-equipped device is positioned within proximity of the user. The method comprises determining, based on the spatial relationship, minimum and maximum boundaries of a user volume encompassing a head and one or more extremities of the user. The minimum and maximum boundaries represent a reduced depth range that is narrower than a full depth range of a camera frustum of the camera-equipped device. The method further comprises encoding, based on the minimum and maximum boundaries, a depth image frame captured via the camera-equipped device, and transmitting the encoded depth image frame over a remote network connection. A reconstructed 3D image based in part on the encoded depth image frame is rendered by a different AR device.
- For expository purposes, the term “video call” as used in this specification generally refers to reception and transmission of audio-video signals by users in different locations, for communication between people in real time.
- Holographic calls (“holocalls”) are example augmented reality (AR) application use cases that involve AR, immersive, volumetric video calls between multiple users. A holocall involves streaming RGB and depth image frames from a first device to a second device across a network. A holographic call typically involves an AR/VR headset and a mobile phone. One user captures real-time images of themselves via a camera of the mobile phone, and another user views the user in stereoscopic 3D via the AR/VR headset. For example, the AR/VR headset renders images of the user at a streaming rate of about 30 fps to enable holocalls to become more broadly available as a consumer product, 3D data about a user must be encoded efficiently to allow for low-latency transport of information about the user across a wide variety of network conditions.
- Lossy codecs (i.e., lossless methods), such as H264/H265 (interframe) or JPEG (intraframe), are desirable as such codecs allow for fluctuations in quality given network conditions. Existing lossless depth compression algorithms, however, are inflexible with respect to handling fluctuations in quality given network conditions.
- To improve perceptual quality of video depth while streaming, conventional technologies reduce full range of depth values (i.e., values of depth image frames). Quantizing the full range of depth values, however, results in more artifacts in regions of the face and/or body rendered in an AR/VR display of the AR/VR headset.
- In the context of AR holographic calls, existing CV/AI-based methods for determining a relevant range of values that a user occupies are not the most effective. For example, in terms of computational costs, processing time increases for higher-resolution images when utilizing AI-based methods for real-time human segmentation (e.g., DeepLab3). If a pair of AR glasses is responsible for rendering images at 60 fps, processing time may be better allocated for rendering. Further, AI-based methods for real-time human segmentation perform less well when part of the face is occluded by sunglasses or a large AR/VR headset, resulting in video streaming instability for users with occluded faces and/or in noisy environments.
- One or more embodiments provide a system for improving efficiency of transporting depth image frames across a network. In one embodiment, the system is configured to track a position of a device (e.g., a mobile phone) and a position of a user of the device, and automatically adjust encoding of 3D data about the user, such as depth image frames captured by the device, to a volume around the user based on the tracked positions. The resulting encoded 3D data is transported from the device across the network with low-latency.
- In one embodiment, to improve perceptual quality of video depth while streaming, the system reduces a quantization error associated with quantizing depth range (i.e., range of depth values). To reduce the quantization error, the system determines a range of depth values that the user occupies.
- One or more embodiments are compatible with existing image/video lossy codecs or lossless depth compression techniques, such as JPEG or H264/H265.
- One embodiment provides a single calibration-based method for improving the efficiency and accuracy of person-centric depth data transmission in the context of holocalls.
-
FIG. 1 illustrates anexample computing architecture 100 for implementing efficient encoding of depth data across devices, in one or more embodiments. Thecomputer architecture 100 provides a framework for a holocall between alocal user 10 and aremote user 20. In the environment of thelocal user 10, thecomputer architecture 100 comprises at least anAR device 110 and a camera-equippeddevice 130. TheAR device 110 and the camera-equippeddevice 130 are paired devices. In the environment of theremote user 20, thecomputer architecture 100 comprises at least aremote AR device 160. - The
AR device 110 includes resources, such as one ormore processor units 111 and one ormore storage units 112. One ormore applications 116 may execute/operate on theAR device 110 utilizing the resources of theAR device 110. In one embodiment, theAR device 110 comprises one ormore sensor units 114 integrated in or coupled to theAR device 110, such as a GPS, a motion sensor, etc. In one embodiment, theAR device 110 comprises one or more I/O units 113 integrated in or coupled to theAR device 110. In one embodiment, the one or more I/O units 113 include, but are not limited to, an AR display, etc. Examples of anAR device 110 include, but are not limited to, a pair of AR glasses, an AR headset, etc. During the holocall, theAR device 110 is worn by thelocal user 10. - In one embodiment, the
AR device 110 comprises acommunications unit 115 configured to exchange data with the camera-equippeddevice 130 over a local network connection (e.g., a wireless connection such as a Wi-Fi connection or a cellular data connection, a wired connection, or a combination of the two). Thecommunications unit 115 may comprise any suitable communications circuitry operative to connect to a communications network and to exchange communications operations and media between theAR device 110 and other devices connected to the same communications network. Thecommunications unit 115 may be operative to interface with a communications network using any suitable communications protocol such as, for example, Wi-Fi (e.g., an IEEE 802.11 protocol), Bluetooth®, high frequency systems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communication systems), infrared, GSM, GSM plus EDGE, CDMA, quadband, and other cellular protocols, VOIP, TCP-IP, or any other suitable protocol. - The camera-equipped
device 130 includes resources, such as one ormore processor units 131 and one ormore storage units 132. One ormore applications 136 may execute/operate on the camera-equippeddevice 130 utilizing the resources of the camera-equippeddevice 130. In one embodiment, the camera-equippeddevice 130 comprises one ormore sensor units 134 integrated in or coupled to the camera-equippeddevice 130, such as acamera 137, a microphone, a GPS, a motion sensor, etc. In one embodiment, the camera-equippeddevice 130 comprises one or more I/O units 133 integrated in or coupled to the camera-equippeddevice 130. In one embodiment, the one or more I/O units 133 include, but are not limited to, a physical user interface (PUI) and/or a GUI, such as a keyboard, a keypad, a touch interface, a touch screen, a knob, a button, a display screen, etc. In one embodiment, thelocal user 10 can utilize at least one I/O unit 133 to configure one or more user preferences, configure one or more parameters, provide input, etc. Examples of a camera-equippeddevice 130 include, but are not limited to, a television (e.g., a smart television), a mobile electronic device (e.g., a tablet, a smart phone, a laptop, etc.), a wearable device (e.g., a smart watch, a smart band, a head-mounted display, smart glasses, etc.), an Internet of things (IoT) device, etc. - In one embodiment, the
camera 137 includes both a depth camera 180 (FIG. 6 ) and a RGB camera 190 (FIG. 6 ). During the holocall, the camera-equippeddevice 130 is positioned within proximity of thelocal user 10 to capture RGB and depth image frames of thelocal user 10. - In one embodiment, the camera-equipped
device 130 comprises acommunications unit 135 configured to exchange data with theAR device 110 over a local network connection (e.g., a wireless connection such as a Wi-Fi connection or a cellular data connection, a wired connection, or a combination of the two). Thecommunications unit 135 is further configured to exchange data with a remote device, such as theremote AR device 160, over a remote network connection (e.g., a wireless connection such as a Wi-Fi connection or a cellular data connection, a wired connection, or a combination of the two). Thecommunications unit 135 may comprise any suitable communications circuitry operative to connect to a communications network and to exchange communications operations and media between the camera-equippeddevice 130 and other devices connected to the same communications network. Thecommunications unit 135 may be operative to interface with a communications network using any suitable communications protocol such as, for example, Wi-Fi (e.g., an IEEE 802.11 protocol), Bluetooth high frequency systems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communication systems), infrared, GSM, GSM plus EDGE, CDMA, quadband, and other cellular protocols, VOIP, TCP-IP, or any other suitable protocol. - In one embodiment, the one or
more applications 136 on the camera-equippeddevice 130 may further include one or more software mobile applications loaded onto or downloaded to the camera-equippeddevice 130, such as a camera application, a social media application, a video streaming application, an AR application, etc. - The
remote AR device 160 includes resources, such as one ormore processor units 161 and one ormore storage units 162. One ormore applications 166 may execute/operate on theremote AR device 160 utilizing the resources of theremote AR device 160. In one embodiment, theremote AR device 160 comprises one ormore sensor units 164 integrated in or coupled to theremote AR device 160, such as a GPS, a motion sensor, etc. In one embodiment, theremote AR device 160 comprises one or more I/O units 163 integrated in or coupled to theremote AR device 160. In one embodiment, the one or more I/O units 163 include, but are not limited to, an AR display, etc. Examples of aremote AR device 160 include, but are not limited to, a pair of AR glasses, an AR headset, etc. During the holocall, theremote AR device 160 is worn by theremote user 20. - In one embodiment, the
remote AR device 160 comprises acommunications unit 165 configured to exchange data with the camera-equippeddevice 130 over the remote network connection. Thecommunications unit 165 may comprise any suitable communications circuitry operative to connect to a communications network and to exchange communications operations and media between theremote AR device 160 and other devices connected to the same communications network. Thecommunications unit 165 may be operative to interface with a communications network using any suitable communications protocol such as, for example, Wi-Fi (e.g., an IEEE 802.11 protocol), Bluetooth®, high frequency systems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communication systems), infrared, GSM, GSM plus EDGE, CDMA, quadband, and other cellular protocols, VOIP, TCP-IP, or any other suitable protocol. -
FIG. 2 illustrates anexample system 550 for efficient encoding of depth data across devices, in one or more embodiments. In one embodiment, thesystem 550 comprises: (1) an ARdevice tracking module 120, (2) adepth camera module 140, (3) alocal server 150, and (4) aremote client 170. As shown inFIG. 2 , in one embodiment, the one or more applications 116 (FIG. 1 ) on theAR device 100 include the ARdevice tracking module 120, the one or more applications 136 (FIG. 1 ) on the camera-equippeddevice 130 include thedepth camera module 140 and thelocal server 150, and the one or more applications 166 (FIG. 1 ) on theremote AR device 160 include theremote client 170. - In one embodiment, the AR
device tracking module 120 is configured to perform an initial calibration against a position of the camera-equippeddevice 130 to determine a spatial relationship between the paired devices. During the holocall, for each depth image frame, the ARdevice tracking module 120 is further configured to: (1) track positions of different parts of thelocal user 10, wherein the different parts include the head and one or more extremities (e.g., hands, feet) of thelocal user 10, and (2) determine, based on the positions tracked, a reduced depth range for the depth image frame. A reduced depth range is narrower than a full depth range (i.e., all depth values within the frustum of the camera 137). The reduced depth range contains only relevant depth values, i.e., depth values representing a volume the local user 10 (“user volume”) occupies, wherein the user volume encompasses most of the body of thelocal user 10 including the different parts tracked. The ARdevice tracking module 120 tracks the different parts of thelocal user 10 to dynamically update the relevant range of depth values for each depth image frame. - In one embodiment, during the holocall, for each depth image frame, the AR
device tracking module 120 is further configured to: (1) track additional positions of different elements in the environment of thelocal user 10, wherein the different elements include one or more images or objects within proximity of thelocal user 10, and (2) determine, based on the additional positions tracked, a reduced depth range for the depth image frame. In one embodiment, the one or more images or objects are tracked optically or via other means utilizing either thecamera 137 or theAR device 110. The additional positions tracked may expand the reduced depth range (i.e., expand minimum and maximum boundaries). - In one embodiment, the
depth camera module 140 is configured to capture, via thecamera 137, RGB and depth image frames of thelocal user 10 during the holocall. - In one embodiment, the
depth camera module 140 is further configured to perform an encoding process that encodes each depth image frame captured based on a reduced depth range (determined via the AR device tracking module 120) in the depth image frame. - In one embodiment, the
depth camera module 140 is further configured to apply a lossy codec (e.g., JPEG, H264, or H265) to each RGB image frame captured and each resulting encoded depth image frame to better accommodate network bandwidth constraints. - In one embodiment, the
local server 150 is configured to send/relay a compressed RGB image frame and a compressed, encoded depth image frame (compressed via the depth camera module 140) to theremote client 170 as an output stream transported over the remote network connection. - In one embodiment, the
remote client 170 is configured to: (1) receive an input stream from thelocal server 150 over the remote network connection, (2) perform a decoding process on an encoded depth image frame included in the input stream to decode a reduced depth range in the depth image frame, (3) reconstruct a 3D image of thelocal user 10 based a RGB image frame included in the input stream and the reduced depth range, and (4) render the resulting 3D image on an AR display of theremote AR device 160. -
FIG. 3 illustrates an example workflow for thesystem 550, in one or more embodiments. In one embodiment, each of the ARdevice tracking module 120, thedepth camera module 140, thelocal server 150, and theremote client 170 has at least two operating phrases: (1) a setup and calibration phase, and (2) an iterative loop phase. - In the setup and calibration phase, the AR
device tracking module 120, thedepth camera module 140, thelocal server 150, and theremote client 170 operate as follows: First, thedepth camera module 140 establishes a server. Next, the ARdevice tracking module 120 establishes a tracking client, and connects to the server (established by the depth camera module 140). The ARdevice tracking module 120 then calibrates a position of the camera-equippeddevice 130 in 3D coordinate space with three axes, i.e., the x-axis, the y-axis, and the z-axis. Next, thedepth camera module 140 provides a set K of intrinsic parameters for the camera (“camera intrinsics”) of the camera-equippeddevice 120 to thelocal server 150. Thelocal server 150 relays the camera intrinsics K to theremote client 170 over the remote network connection. Theremote client 170 caches or saves the camera intrinsics K received from thelocal server 150. As described in detail later, in the iterative loop phase, theremote client 170 utilizes the camera intrinsics K for 3D reprojection. - In this specification, the terms “depth pixel”, “depth value”, and “z value” are used interchangeably.
- In the iterative loop phase, the AR
device tracking module 120, thedepth camera module 140, thelocal server 150, and theremote client 170 operate as follows: For each image frame, the ARdevice tracking module 120 tracks different parts (e.g., head and hands/feet) of thelocal user 10, and calculates z values for the different parts relative to the position of the camera-equippeddevice 130, wherein each z value is a point along the z-axis. Next, the ARdevice tracking module 120 determines, based on the z values, a reduced depth range in the depth image frame. Specifically, the ARdevice tracking module 120 determines zmin and zmax, wherein zmin and zmax are minimum and maximum boundaries of a user volume in the depth image frame. zmin and zmax represent minimum and maximum distances, respectively, of the body of thelocal user 10 relative to the z-axis of the camera-equippeddevice 130. The ARdevice tracking module 120 sends zmin and zmax to thedepth camera module 140. - In response to receiving zmin and zmax, the
depth camera module 140 captures RGB and depth image frames via the camera. Thedepth camera module 140 then performs an encoding process on the depth image frame captured based on zmin and zmax. Thelocal server 150 applies a lossy codec (e.g., JPEG, H264, or H265) to the RGB image frame and the resulting encoded depth image frame (from the depth camera module 140), and sends/relays, over the remote network connection, the resulting compressed image frames to theremote client 170 as an output stream with zmin and zmax included in header values. Therefore, instead of transporting all depth values captured within the camera frustum across the network, encoding only the reduced range of depth values culls irrelevant depth values, such that only the relevant depth values are networked. The reduced depth range reduces quantization error, thereby allowing for higher fidelity reconstruction on theremote client 170 and improving perceptual video quality while streaming on theremote AR device 160. - The
remote client 170 receives, over the remote network connection, from thelocal server 150, an input stream comprising the compressed images frames, zmin, and zmax. Theremote client 170 reconstructs a 3D image based on zmin and zmax, and reprojects the 3D image based on the camera intrinsics K (cached or saved during the setup and calibration phase). The reprojected 3D image is rendered in the AR display of theremote AR device 160. - Simultaneous localization and mapping (SLAM) is a computational technology involving constructing or updating a map of an unknown environment while simultaneously keeping track of an agent's location within it. For example, a device performing SLAM is configured to map its location, create a 3D virtual map, and locate itself inside the map. Visual SLAM is a particular type of SLAM that uses only visual inputs to perform location and mapping. For example, visual SLAM involves calculating the position and orientation of a device with respect to its surroundings, while mapping the environment at the same time, using only visual inputs from a camera.
-
FIG. 4 is a flowchart of anexample process 200 for calibrating a position of the camera-equippeddevice 130 in 3D coordinate space, in one or more embodiments. Process block 210 includes using visual slam to calculate a pose of theAR device 110 relative to an arbitrary world-space origin in virtual coordinate space. Process block 220 includes aligning the physical camera-equippeddevice 130 in the virtual coordinate space. The aligning is performed in accordance with process block 230, process block 240,process block 250, or process block 260. - Process block 230 includes the
local user 10 manually aligning a virtual transform represented by an AR user interface (UI) (rendered by the AR device 110) with a physical pose Ppos of the camera-equippeddevice 130. - Process block 240 includes the
local user 10 aligning the camera-equippeddevice 130 with a virtual transform represented in an AR UI (rendered by the AR device 110). -
Process block 250 includes a fiducial displayed on a display screen of the camera-equippeddevice 130 automatically aligning a virtual transform with the physical pose Ppos of the camera-equippeddevice 130. - Process block 260 includes using an object detection and pose estimation algorithm to estimate the physical pose Ppos of the camera-equipped
device 130. - Each of process block 230, process block 240,
process block 250, and process block 260 returns to process block 210. - In one embodiment, process blocks 210-260 are performed by the AR
device tracking module 120 during the setup and calibration phase. For example, in one embodiment, the ARdevice tracking module 120 triggers theAR device 110 to render the AR UI representing the virtual transform. As another example, in one embodiment, the ARdevice tracking module 120 triggers the camera-equippeddevice 130 to display the fiducial. -
FIG. 5 is a flowchart of anexample process 300 for calculating z values for different parts (e.g., head and hands/feet) of thelocal user 10 relative to the position of the camera-equippeddevice 130, in one or more embodiments. Process block 310 includes using visual slam to calculate a position Hpos of the head (“head position”) of thelocal user 10. Process block 320 includes using system/OS/visual slam methods to calculate positions Lpos and Rpos of the left hand and right hand (“hand positions”) (or the left foot and right foot), respectively, of thelocal user 10.Process block 330 includes transforming the head position Hpos relative to the physical pose Ppos of the camera-equippeddevice 130 to obtain H′pos.Process block 340 includes transforming the hand positions Lpos and Rpos relative to the physical pose Ppos of the camera-equippeddevice 130 to obtain L′pos and R′pos. -
Process block 350 includes iterating through z components of H′pos, L′pos, and R′pos to determine zmin and zmax.Process block 360 includes adding standard buffer amounts Bfront and Bback to zmin and zmax to account for the body thickness of thelocal user 10 exceeding the tracked parts (i.e., head and hands/feet).Process block 370 includes converting zmin and zmax to units matching a depth camera format (e.g., m to mm), wherein zmin and zmax are sent to thedepth camera module 140 as the units. - In one embodiment, process blocks 310-370 are performed by the AR
device tracking module 120 during the iterative loop phase. For example, in one embodiment, the ARdevice tracking module 120 is configured to transform the head position Hpos relative to the physical pose Ppos of the camera-equippeddevice 130 to obtain H′pos, in accordance with equation (1) provided below: -
H′ pos =P rot −1*(H pos −P pos) (1), - wherein Prot −1 is the inverse of a rotation matrix of the camera-equipped
device 130. - As another example, in one embodiment, the AR
device tracking module 120 is configured to transform the hand positions Lpos and Rpos relative to the physical pose Ppos of the camera-equippeddevice 130 to obtain L′pos and R′pos, in accordance with equations (2)-(3) provided below: -
L′ pos =P rot −1(L pos −P pos) (2), and -
R′ pos =P rot −1*(R pos −P pos) (3). - As another example, in one embodiment, the AR
device tracking module 120 is configured to iterate through z components of H′pos, L′pos, and R′pos to determine zmin and zmax, in accordance with equations (4)-(5) provided below: -
z min=min(H′ posz ,L′ posz ,R′ posz ) (4), and -
z max=max(H′ posz ,L′ posz ,R′ posz ) (5). - As another example, in one embodiment, the AR
device tracking module 120 is configured to add standard buffer amounts Bfront and Bback to zmin and zmax to account for the body thickness of thelocal user 10 exceeding the tracked parts (i.e., head and hands/feet), in accordance with equations (6)-(7) provided below: -
z min =z min +B front (6), and -
z max =z max +B back (7) -
FIG. 6 illustrates an example encoding process, in one or more embodiments. In one embodiment, during each iteration of the iterative loop phase, thedepth camera module 140 is configured to receive, as inputs, the following: (1) anarray 410 of depth pixels from thedepth camera 180, and (2) zmin and zmax from the ARdevice tracking module 120. In one embodiment, zmin and zmax are dynamically set for each depth image frame (e.g., zmin=900 mm, and zmax=1500 mm). Thearray 410 represents a full depth range, i.e., all depth values within the frustum of thedepth camera 180. Each element zp of thearray 410 is a measurement of a single depth pixel (e.g., zp=1245 mm). - In one embodiment, the
depth camera module 140 is configured to convert thearray 410 to a depth image frame by: (1) performing an encoding process on thearray 410 based on zmin and zmax, and (2) creating adepth bitmap 420 representing the depth image frame based on encoded values resulting from the encoding process. In one embodiment, the encoding process involves the following operations: For each depth pixel zp of thearray 410, thedepth camera module 140 determines a normalized value z′ of the depth pixel zp in accordance with equation (8) provided below: -
- To reduce a number of bits (i.e., bit depth) representing the normalized value z′, the
depth camera module 140 performs int8 quantization to convert the normalized value z′ from a higher-precision format into an int8 value zi in a lower-precision format, in accordance with equation (9) provided below: -
└z′*28 ┐≈z i (9). - The
depth camera module 140 encodes the int8 value zi as a 24-bit color value zc, in accordance with equation (10) provided below: -
(z i ,z i ,z i)=z c (10). - The
depth camera module 140 creates thedepth bitmap 420 by setting colors of thedepth bitmap 420 that correspond to the depth pixel zp to the 24-bit color value zc. - For example, if zp=1245 mm, zmin=900 mm, and zmax=1500 mm, z′=
-
- In one embodiment, the
local server 150 is configured to receive, as inputs, the following: (1) aRGB bitmap 400 from theRGB camera 190, wherein theRGB bitmap 400 represents an RGB image frame captured by theRGB camera 190, and (2) thedepth bitmap 420 from thedepth camera module 140. - In one embodiment, the
local server 150 is configured to: (1) compress theRGB bitmap 400 by applying lossy codec to theRGB bitmap 400, resulting in acompressed RGB bitmap 430, and (2) compress thedepth bitmap 420 by applying lossy codec to thedepth bitmap 420, resulting in acompressed depth bitmap 440. - In one embodiment, the
local server 150 is configured to: (1) convert thecompressed RGB bitmap 430 into afirst byte array 450, and (2) convert thecompressed depth bitmap 440 into asecond byte array 460 that includes zmin and zmax as header values, and (3) write thefirst byte array 450 and thesecond byte array 460 to an output stream. The output stream is relayed/transported to theremote client 170 over the remote network connection. -
FIG. 7 illustrates an example decoding process, in one or more embodiments. In one embodiment, during each iteration of the iterative loop phase, theremote client 170 is configured to read, from an input stream, afirst byte array 500 and asecond byte array 510. The input stream is relayed/transported from thelocal server 150 over the remote network connection. Thesecond byte array 510 includes zmin and zmax as header values. Theremote client 170 is configured to: (1) convert thefirst byte array 500 to aRGB bitmap 520 representing an RGB image frame, and (2) convert thesecond byte array 510 to a depth bitmap representing a depth image frame. - In one embodiment, the
remote client 170 is configured to perform a decoding process on the depth bitmap based on zmin and zmax to obtain decoded z values. In one embodiment, the decoding process involves the following operations: For each 24-bit color value zc of the depth bitmap, theremote client 170 extracts an int8 value zi from the 24-bit color value zc, in accordance with equation (11) provided below: -
z c=(z i ,z i ,z i) (11). - The
remote client 170 decodes a normalized value z′ from the int8 value zi, in accordance with equation (12) provided below: -
- The
remote client 170 de-normalizes the normalized value z′ based on zmin and zmax to obtain a decoded depth pixel zp, in accordance with equation (13) provided below: -
z p=[z′*(z max −z min)]+z min (13). - For example, if zc=147, zmin=900 mm, and zmax=1500 mm, zi=147,
-
- In one embodiment, the
remote client 170 is configured to perform a 3D reprojection based onRGB bitmap 520, the decoded z values resulting from the decoding process, and the camera intrinsics K of the camera-equipped device 120 (cached/saved at the start of the session). In one embodiment, the 3D reprojection involves theremote client 170 reprojecting RGB pixels from theRGB bitmap 520 into space using the decoded z values and intrinsic parameters from K such as U0, V0, Fx, Fy, resulting in a3D image 530 of thelocal user 10 that is rendered on the viewport of theremote AR device 160. -
FIG. 8 illustrates an example exchange of data between theAR device 110 and the camera-equippeddevice 130, in one or more embodiments. In one embodiment, the ARdevice tracking module 120 operates on theAR device 110, and thedepth camera module 140 and thelocal server 150 operate on the camera-equippeddevice 130. In one embodiment, theAR device 110 is a pair of AR glasses worn on the head of thelocal user 10, and the camera-equippeddevice 130 is a mobile phone. - The AR
device tracking module 120 calculates z values for the head and hands of thelocal user 10 relative to the position of the camera-equippeddevice 130, then determines, based on the z values, auser volume 550 encompassing most of thelocal user 10 including the heads and hands. The user volume represents a reduced depth range containing only relevant depth values (as compared to the full depth range containing all depth values within the frustum of the camera). Specifically, the minimum and maximum boundaries of the user volume are zmin and zmax, wherein zmin is a minimum distance of the body of thelocal user 10 relative to the z-axis of the camera-equippeddevice 130, and zmax is a maximum distance of the body of thelocal user 10 relative to the z-axis of the camera-equippeddevice 130. TheAR device 110 sends zmin and zmax (determined by the AR device tracking module 120) to the camera-equippeddevice 130. In one embodiment, each of zmin and zmax is 16 bits in size, such that a total size of bits sent from theAR device 110 for each depth image frame is 32 bits. - The camera-equipped
device 130 captures RGB and depth image frames via the camera. Thedepth camera module 140 performs an encoding process on the depth image frame based on zmin and zmax. For example, in one embodiment, thedepth camera module 140 performs 8-bit encoding to convert each depth pixel from a higher-precision format (DEPTH16) to a lower-precision format (8-bit). The camera-equippeddevice 130 sends, via thelocal server 150, compressed RGB and encoded depth image frames to theremote client 170 over the remote network connection. - As the
system 550 utilizes a single calibration-based method that relies only on an exchange of 32 bits per frame (2×int16) over an existing network connection to communicate a reduced depth range, the computational costs of thesystem 550 is less than conventional CV/AI-based methods (e.g., AI-based methods for real-time human segmentation). -
FIG. 9A illustrates an example full depth range, in one or more embodiments. Before the ARdevice tracking module 120 reduces depth range by culling irrelevant depth values, the full depth range contains all depth values within the frustum of thecamera 137 of the camera-equippeddevice 130. For example, for the full depth range, zmin=0 mm and zmax=∞ mm. -
FIG. 9B illustrates an example reduced depth range, in one or more embodiments. The ARdevice tracking module 120 reduces depth range by culling irrelevant depth values, such that the reduced depth range contains only relevant depth values representing auser volume 550 encompassing most of the body of thelocal user 10 including the different parts (e.g., head and hands/feet) tracked. For example, for the reduced depth range/user volume, zmin=900 mm and zmax=1500 mm. -
FIGS. 10A-10H illustrate changes to zmin and zmax during a holocall, in one or more embodiments.FIG. 10A illustrates anexample user volume 600 encompassing thelocal user 10 at the start of a holocall, in one or more embodiments. As shown inFIG. 10A , at the start of the holocall, the hands of thelocal user 10 are lowered and positioned on the sides of the body of thelocal user 10. The ARdevice tracking module 120 determines minimum and maximum boundaries of theuser volume 600 as follows: zmin=596 mm, and zmax=1342 mm. -
FIG. 10B illustrates anotherexample user volume 610 encompassing thelocal user 10 raising a hand during the holocall, in one or more embodiments. As shown inFIG. 10B , when thelocal user 10 raises a hand during the holocall, the minimum distance of the body of thelocal user 10 relative to the z-axis of the camera-equippeddevice 130 decreases, resulting in a larger/wider user volume 610 that encompasses the raised hand. The ARdevice tracking module 120 determines minimum and maximum boundaries of theuser volume 610 as follows: zmin=530 mm, and zmax=1342 mm. -
FIG. 10C illustrates anotherexample user volume 620 encompassing thelocal user 10 with a hand extended forward during the holocall, in one or more embodiments. As shown inFIG. 10C , when thelocal user 10 extends a hand forward during the holocall, the minimum distance of the body of thelocal user 10 relative to the z-axis of the camera-equippeddevice 130 decreases even further, resulting in an even larger/wider user volume 620 that encompasses the extended hand. The ARdevice tracking module 120 determines minimum and maximum boundaries of theuser volume 620 as follows: zmin=3260 mm, and zmax=1342 mm. -
FIG. 10D illustrates anotherexample user volume 630 encompassing thelocal user 10 lowering a raised hand during the holocall, in one or more embodiments. As shown inFIG. 10D , when thelocal user 10 lowers a raised hand during the holocall, the minimum distance of the body of thelocal user 10 relative to the z-axis of the camera-equippeddevice 130 increases, resulting in a smaller/narrower user volume 630 that encompasses the lowered hand. The ARdevice tracking module 120 determines minimum and maximum boundaries of theuser volume 630 as follows: zmin=408 mm, and zmax=1342 mm. -
FIG. 10E illustrates anotherexample user volume 640 encompassing thelocal user 10 positioning a lowered hand on the side of the body of thelocal user 10 during the holocall, in one or more embodiments. As shown inFIG. 10E , when thelocal user 10 positions a lowered hand on the side of the body of thelocal user 10 during the holocall, the minimum distance of the body of thelocal user 10 relative to the z-axis of the camera-equippeddevice 130 increases even further, resulting in a smaller/narrower user volume 640. The ARdevice tracking module 120 determines minimum and maximum boundaries of theuser volume 640 as follows: zmin=596 mm, and zmax=1342 mm. -
FIG. 10F illustrates anotherexample user volume 650 encompassing thelocal user 10 moving closer forward to the camera-equippeddevice 130 during the holocall, in one or more embodiments. As shown inFIG. 10F , when thelocal user 10 moves closer forward to the camera-equippeddevice 130 during the holocall, both the minimum and maximum distance of the body of thelocal user 10 relative to the z-axis of the camera-equippeddevice 130 decrease. The ARdevice tracking module 120 determines minimum and maximum boundaries of theuser volume 650 as follows: zmin=454 mm, and zmax=1166 mm. -
FIG. 10G illustrates anotherexample user volume 660 encompassing thelocal user 10 moving further closer forward to the camera-equippeddevice 130 during the holocall, in one or more embodiments. As shown inFIG. 10G , when thelocal user 10 moves further closer forward to the camera-equippeddevice 130 during the holocall, both the minimum and maximum distance of the body of thelocal user 10 relative to the z-axis of the camera-equippeddevice 130 decrease even further. The ARdevice tracking module 120 determines minimum and maximum boundaries of theuser volume 660 as follows: zmin=311 mm, and zmax=990 mm. -
FIG. 10H illustrates anotherexample user volume 670 encompassing thelocal user 10 moving further backwards from the camera-equippeddevice 130 during the holocall, in one or more embodiments. As shown inFIG. 10H , when thelocal user 10 moves further backwards from the camera-equippeddevice 130 during the holocall, both the minimum and maximum distance of the body of thelocal user 10 relative to the z-axis of the camera-equippeddevice 130 increase. The ARdevice tracking module 120 determines minimum and maximum boundaries of theuser volume 670 as follows: zmin=431 mm, and zmax=1140 mm. -
FIG. 11 illustrates anotherexample system 560 for efficient encoding of depth data across devices, in one or more embodiments. In one embodiment, thesystem 560 splits computation between theAR device 110 and the camera-equippeddevice 130. Specifically, in thesystem 560, the one or more applications 116 (FIG. 1 ) on theAR device 100 include the ARdevice tracking module 120 and thelocal server 150, the one or more applications 136 (FIG. 1 ) on the camera-equippeddevice 130 include thedepth camera module 140, and the one or more applications 166 (FIG. 1 ) on theremote AR device 160 include theremote client 170. -
FIG. 12 illustrates anotherexample system 570 for efficient encoding of depth data across devices, in one or more embodiments. In one embodiment, thesystem 560 performs most computation on the camera-equippeddevice 130. Specifically, in thesystem 570, the one or more applications 136 (FIG. 1 ) on the camera-equippeddevice 130 include the ARdevice tracking module 120, thedepth camera module 140, and thelocal server 150. Further, thesystem 570 includes aremote user device 195 on which theremote client 170 executes/operates. TheAR device 110 relays sensor data to the ARdevice tracking module 120 on the camera-equippeddevice 130 over the local network connection. Theremote AR device 160 receives a 3D image (reconstructed by the remote client 170) from theremote user device 195 over a local network connection, and renders the 3D image on the AR display of theremote AR device 160. -
FIG. 13 is a flowchart of anexample process 800 for efficient encoding of depth data across devices, in one or more embodiments.Process block 810 includes determining a spatial relationship between an AR device (e.g., AR device 110) and a camera-equipped device (e.g., camera-equipped device 130), wherein the AR device is worn by a user (e.g., local user 10), and the camera-equipped device is positioned within proximity of the user.Process block 820 includes determining, based on the spatial relationship, minimum and maximum boundaries (zmin and zmax) of a user volume encompassing a head and one or more extremities of the user, wherein the minimum and maximum boundaries represent a reduced depth range that is narrower than a full depth range of a camera frustum of the camera-equipped device.Process block 830 includes encoding, based on the minimum and maximum boundaries, a depth image frame captured via the camera-equipped device.Process block 840 includes transmitting the encoded depth image frame over a remote network connection, wherein a reconstructed 3D image based in part on the encoded depth image frame is rendered by a different AR device (e.g., remote AR device 160). - In one embodiment, process blocks 810-840 may be performed one or more components of the
system device tracking module 120, thedepth camera module 140, thelocal server 150, and theremote client 170. -
FIG. 14 is a high-level block diagram showing an information processing system comprising acomputer system 900 useful for implementing the disclosed embodiments.Computer system 900 may be incorporated theAR device 110, the camera-equippeddevice 130, theremote AR device 160, and/or theremote user device 195. Thecomputer system 900 includes one ormore processors 901, and can further include an electronic display device 902 (for displaying video, graphics, text, and other data), a main memory 903 (e.g., random access memory (RAM)), storage device 904 (e.g., hard disk drive), removable storage device 905 (e.g., removable storage drive, removable memory module, a magnetic tape drive, optical disk drive, computer readable medium having stored therein computer software and/or data), viewer interface device 906 (e.g., keyboard, touch screen, keypad, pointing device), and a communication interface 907 (e.g., modem, a network interface (such as an Ethernet card), a communications port, or a PCMCIA slot and card). Thecommunication interface 907 allows software and data to be transferred between the computer system and external devices. Thesystem 900 further includes a communications infrastructure 908 (e.g., a communications bus, cross-over bar, or network) to which the aforementioned devices/modules 901 through 907 are connected. - Information transferred via
communications interface 907 may be in the form of signals such as electronic, electromagnetic, optical, or other signals capable of being received bycommunications interface 907, via a communication link that carries signals and may be implemented using wire or cable, fiber optics, a phone line, a cellular phone link, an radio frequency (RF) link, and/or other communication channels. Computer program instructions representing the block diagram and/or flowcharts herein may be loaded onto a computer, programmable data processing apparatus, or processing devices to cause a series of operations performed thereon to generate a computer implemented process. In one embodiment, processing instructions for process 200 (FIG. 4 ), processing instructions for process 300 (FIG. 5 ), and/or processing instructions for process 800 (FIG. 13 ) may be stored as program instructions on thememory 903,storage device 904 and theremovable storage device 905 for execution by theprocessor 901. - Embodiments have been described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products. Each block of such illustrations/diagrams, or combinations thereof, can be implemented by computer program instructions. The computer program instructions when provided to a processor produce a machine, such that the instructions, which execute via the processor create means for implementing the functions/operations specified in the flowchart and/or block diagram. Each block in the flowchart/block diagrams may represent a hardware and/or software module or logic. In alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures, concurrently, etc.
- The terms “computer program medium,” “computer usable medium,” “computer readable medium”, and “computer program product,” are used to generally refer to media such as main memory, secondary memory, removable storage drive, a hard disk installed in hard disk drive, and signals. These computer program products are means for providing software to the computer system. The computer readable medium allows the computer system to read data, instructions, messages or message packets, and other computer readable information from the computer readable medium. The computer readable medium, for example, may include non-volatile memory, such as a floppy disk, ROM, flash memory, disk drive memory, a CD-ROM, and other permanent storage. It is useful, for example, for transporting information, such as data and computer instructions, between computer systems. Computer program instructions may be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
- As will be appreciated by one skilled in the art, aspects of the embodiments may be embodied as a system, method or computer program product. Accordingly, aspects of the embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, aspects of the embodiments may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.
- Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
- Computer program code for carrying out operations for aspects of one or more embodiments may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
- Aspects of one or more embodiments are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
- These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
- The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
- The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
- References in the claims to an element in the singular is not intended to mean “one and only” unless explicitly so stated, but rather “one or more.” All structural and functional equivalents to the elements of the above-described exemplary embodiment that are currently known or later come to be known to those of ordinary skill in the art are intended to be encompassed by the present claims. No claim element herein is to be construed under the provisions of 35 U.S.C.
section 112, sixth paragraph, unless the element is expressly recited using the phrase “means for” or “step for.” - The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
- The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the embodiments has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the embodiments in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention.
- Though the embodiments have been described with reference to certain versions thereof; however, other versions are possible. Therefore, the spirit and scope of the appended claims should not be limited to the description of the preferred versions contained herein.
Claims (20)
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/361,143 US12026901B2 (en) | 2020-07-01 | 2021-06-28 | Efficient encoding of depth data across devices |
PCT/KR2021/008355 WO2022005226A1 (en) | 2020-07-01 | 2021-07-01 | Efficient encoding of depth data across devices |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US202063047142P | 2020-07-01 | 2020-07-01 | |
US17/361,143 US12026901B2 (en) | 2020-07-01 | 2021-06-28 | Efficient encoding of depth data across devices |
Publications (2)
Publication Number | Publication Date |
---|---|
US20220005215A1 true US20220005215A1 (en) | 2022-01-06 |
US12026901B2 US12026901B2 (en) | 2024-07-02 |
Family
ID=79167797
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/361,143 Active 2042-07-31 US12026901B2 (en) | 2020-07-01 | 2021-06-28 | Efficient encoding of depth data across devices |
Country Status (2)
Country | Link |
---|---|
US (1) | US12026901B2 (en) |
WO (1) | WO2022005226A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11475652B2 (en) | 2020-06-30 | 2022-10-18 | Samsung Electronics Co., Ltd. | Automatic representation toggling based on depth camera field of view |
WO2024232882A1 (en) * | 2023-05-10 | 2024-11-14 | Innopeak Technology, Inc. | Systems and methods for multi-view depth estimation using simultaneous localization and mapping |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9417754B2 (en) * | 2011-08-05 | 2016-08-16 | P4tents1, LLC | User interface system, method, and computer program product |
Family Cites Families (32)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2008049916A1 (en) | 2006-10-26 | 2008-05-02 | Seereal Technologies S.A. | Mobile tlephony system comprising holographic display |
US9727473B2 (en) | 2008-09-30 | 2017-08-08 | Intel Corporation | Methods to communicate a timestamp to a storage system |
US8379101B2 (en) | 2009-05-29 | 2013-02-19 | Microsoft Corporation | Environment and/or target segmentation |
KR101705025B1 (en) | 2009-11-20 | 2017-02-09 | 삼성전자주식회사 | Apparatus and method for compressing 3d video using edge information |
US8897491B2 (en) | 2011-06-06 | 2014-11-25 | Microsoft Corporation | System for finger recognition and tracking |
US9386268B2 (en) | 2012-04-09 | 2016-07-05 | Intel Corporation | Communication using interactive avatars |
US9191619B2 (en) | 2012-08-01 | 2015-11-17 | Google Inc. | Using an avatar in a videoconferencing system |
EP2708982B1 (en) | 2012-09-18 | 2019-09-11 | Samsung Electronics Co., Ltd | Method for guiding the user of a controller of a multimedia apparatus to move within recognizable range of the multimedia apparatus, the multimedia apparatus, and target tracking apparatus thereof |
US8970656B2 (en) | 2012-12-20 | 2015-03-03 | Verizon Patent And Licensing Inc. | Static and dynamic video calling avatars |
WO2014204330A1 (en) | 2013-06-17 | 2014-12-24 | 3Divi Company | Methods and systems for determining 6dof location and orientation of head-mounted display and associated user movements |
US10175483B2 (en) | 2013-06-18 | 2019-01-08 | Microsoft Technology Licensing, Llc | Hybrid world/body locked HUD on an HMD |
KR102063076B1 (en) | 2013-07-10 | 2020-01-07 | 엘지전자 주식회사 | The mobile device and controlling method thereof, the head mounted display and controlling method thereof |
US9524588B2 (en) | 2014-01-24 | 2016-12-20 | Avaya Inc. | Enhanced communication between remote participants using augmented and virtual reality |
US10311638B2 (en) | 2014-07-25 | 2019-06-04 | Microsoft Technology Licensing, Llc | Anti-trip when immersed in a virtual reality environment |
US9728010B2 (en) | 2014-12-30 | 2017-08-08 | Microsoft Technology Licensing, Llc | Virtual representations of real-world objects |
US9690103B2 (en) | 2015-02-16 | 2017-06-27 | Philip Lyren | Display an image during a communication |
US10175492B2 (en) | 2015-04-24 | 2019-01-08 | Eon Reality, Inc. | Systems and methods for transition between augmented reality and virtual reality |
KR20160133230A (en) | 2015-05-12 | 2016-11-22 | 엘지전자 주식회사 | Mobile terminal |
DE102015210453B3 (en) | 2015-06-08 | 2016-10-13 | Bitmanagement Software GmbH | METHOD AND APPARATUS FOR GENERATING DATA FOR A TWO OR THREE-DIMENSIONAL PRESENTATION OF AT LEAST ONE PART OF AN OBJECT AND FOR PRODUCING THE TWO OR THREE-DIMENSIONAL PRESENTATION AT LEAST THE PART OF THE OBJECT |
US10354449B2 (en) | 2015-06-12 | 2019-07-16 | Hand Held Products, Inc. | Augmented reality lighting effects |
KR101940718B1 (en) | 2015-09-04 | 2019-01-22 | 한국전자통신연구원 | Apparatus and method for extracting person domain based on RGB-Depth image |
KR101763636B1 (en) | 2015-10-15 | 2017-08-02 | 한국과학기술원 | Method for collaboration using head mounted display |
KR20180038175A (en) | 2016-10-06 | 2018-04-16 | 주식회사 케이티 | Server, device and method for providing virtual reality service |
WO2018106198A1 (en) | 2016-12-10 | 2018-06-14 | Yasar Universitesi | Viewing three-dimensional models through mobile-assisted virtual reality (vr) glasses |
KR102424814B1 (en) | 2018-01-17 | 2022-07-26 | 한국전력공사 | Apparatus and method for encording kinect video data |
US10375313B1 (en) | 2018-05-07 | 2019-08-06 | Apple Inc. | Creative camera |
JP2020136921A (en) | 2019-02-20 | 2020-08-31 | 株式会社オプテージ | Video call system and computer program |
KR102665643B1 (en) | 2019-02-20 | 2024-05-20 | 삼성전자 주식회사 | Method for controlling avatar display and electronic device thereof |
JP7382620B2 (en) | 2019-05-08 | 2023-11-17 | サン電子株式会社 | Virtual experience system and computer program |
US11164383B2 (en) | 2019-08-30 | 2021-11-02 | Lg Electronics Inc. | AR device and method for controlling the same |
AU2021290132C1 (en) | 2020-06-08 | 2024-04-18 | Apple Inc. | Presenting avatars in three-dimensional environments |
US11475652B2 (en) | 2020-06-30 | 2022-10-18 | Samsung Electronics Co., Ltd. | Automatic representation toggling based on depth camera field of view |
-
2021
- 2021-06-28 US US17/361,143 patent/US12026901B2/en active Active
- 2021-07-01 WO PCT/KR2021/008355 patent/WO2022005226A1/en active Application Filing
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9417754B2 (en) * | 2011-08-05 | 2016-08-16 | P4tents1, LLC | User interface system, method, and computer program product |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11475652B2 (en) | 2020-06-30 | 2022-10-18 | Samsung Electronics Co., Ltd. | Automatic representation toggling based on depth camera field of view |
WO2024232882A1 (en) * | 2023-05-10 | 2024-11-14 | Innopeak Technology, Inc. | Systems and methods for multi-view depth estimation using simultaneous localization and mapping |
Also Published As
Publication number | Publication date |
---|---|
US12026901B2 (en) | 2024-07-02 |
WO2022005226A1 (en) | 2022-01-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN111580765B (en) | Screen projection method, screen projection device, storage medium, screen projection equipment and screen projection equipment | |
US10712565B2 (en) | Method and system for streaming information in wireless virtual reality | |
CN112887739B (en) | Electronic device, system and control method thereof | |
CN107258086B (en) | Method, apparatus and system for static frame image quality improvement for sink display | |
US8842154B2 (en) | Systems and methods for providing personal video services | |
US20240098298A1 (en) | Segmentation-based parameterized motion models | |
KR20190015093A (en) | Reference frame reprojection for improved video coding | |
CN112887728A (en) | Electronic device, control method and system of electronic device | |
CN112449140B (en) | Video super-resolution processing method and device | |
US10853975B2 (en) | Hybrid projection-based point cloud texture coding | |
US11812066B2 (en) | Methods, devices and stream to encode global rotation motion compensated images | |
US20220005215A1 (en) | Efficient encoding of depth data across devices | |
US20160088298A1 (en) | Video coding rate control including target bitrate and quality control | |
JP2019534616A (en) | Method and apparatus for encoding and decoding omnidirectional video | |
CN109413152B (en) | Image processing method, image processing device, storage medium and electronic equipment | |
US20240233190A1 (en) | Partial decoding and reconstruction of submeshes | |
WO2024114475A1 (en) | Video transcoding method and apparatus, electronic device, computer readable storage medium, and computer program product | |
US10841549B2 (en) | Methods and apparatus to facilitate enhancing the quality of video | |
CN116033155A (en) | Binocular image compression method, device and readable storage medium | |
US20220303555A1 (en) | Combining high-quality foreground with enhanced low-quality background | |
WO2021057464A1 (en) | Video processing method and apparatus, and storage medium and electronic device | |
US20230386086A1 (en) | Video compression | |
US20250024068A1 (en) | Motion coding for vertices in inter-coded basemesh frame for dynamic mesh coding | |
JP2017520871A (en) | System and method for transmitting, receiving and restoring thinned color channels in image data | |
US20240022700A1 (en) | Immersive display controller interface |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SAMSUNG ELECTRONICS CO., LTD., KOREA, REPUBLIC OF Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:EVANGELISTA, EDGAR C.;FAUCHER, AARON S.;KIM, JAEHYUN;AND OTHERS;SIGNING DATES FROM 20210626 TO 20210628;REEL/FRAME:056694/0241 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |