TW202344064A - Systems and methods of signaling information for holographic communications - Google Patents

Abstract

Systems and methods for signaling information for holographic communications include one or more servers which maintain a first session with a first device of a first user and one or more second sessions with one or more second devices of one or more second users. The server(s) may receive, via the first session from the first device, audio/video (A/V) data of a first user and scaling data for the first user. The server(s) may modify a scale of the first user represented in video data of the A/V data according to the scaling data. The server(s) may transmit, via the one or more second sessions, modified A/V data of the first user to the one or more second devices, for rendering to the one or more second users.

Description

System and method for sending information for holographic communication

The present invention generally relates to wireless communications between devices, including but not limited to systems and methods for signaling information in holographic communications. Cross-references to related applications

This application claims the rights and priorities of U.S. Patent Application No. 63/322,851 filed on March 23, 2022 and U.S. Non-Provisional Patent Application No. 18/081,958 filed on December 15, 2022. These applications The contents of the case are incorporated into this article by reference in full.

Augmented reality (AR), virtual reality (VR) and mixed reality (MR) have become more common, and the technology is supported on a wider variety of platforms and devices. Some devices may be configured for video or audio calls and/or conferencing.

Various aspects of the invention relate to systems, methods, and computer-readable media for signaling information in holographic communications. One or more servers may maintain a first session with a first device of a first user and one or more second sessions with one or more second devices of one or more second users. The server may receive audio/video (A/V) data of the first user and zoom data for the first user from the first device through the first session. The server may modify the proportion of the first user represented in the video data of the A/V data based on the scaling data. The server may transmit the first user's modified A/V data to one or more second devices via one or more second sessions for presentation to one or more second users.

In some embodiments, the server may establish a first session and one or more second devices based on a request for a three-dimensional (3D) communication session between a first device and one or more second devices. session. In some examples, the server may receive second A/V data of one or more second users from one or more second devices via one or more second sessions and use it for one or more second devices. The user's second zoom data. The server may modify the proportion of one or more second users represented in the second video data of the second A/V data based on the second scaling data. The server may transmit the modified second A/V data of one or more second users to the first device via the first session for presentation to the first user of the first device. In some embodiments, the server may modify the proportion of the first video data by modifying the proportion of the user represented in the first video data based on the first scaling data and the second scaling data, and may modify the proportion of the first video data by modifying the proportion of the user represented in the first video data based on the first scaling data and the second scaling data. A scaling data and a second scaling data to modify the proportion of one or more second users represented in the second video data and modify the proportion of one or more second users represented in the second video data. In some specific examples, the server may modify the proportion of the first user represented in the first video data to match the one or more first user represented in the second video data according to the first zoom data and the second zoom data. 2. Ratio of users.

In some specific examples, A/V data includes three-dimensional (3D) video data and spatial audio data. In some embodiments, the server may receive data indicating the first user's field-of-view (FOV). In some embodiments, data indicating (FOV) is determined by the first device based on data received from a third device communicatively coupled to the first device. In some embodiments, the server may maintain a first user, a second user, and a third user for a first device and at least a second user for a second device and a third user for a third device. The relative position of each of the three users relative to the local map. The server may determine the first user's FOV based on the direction data and the first user's relative position relative to the local map. In some embodiments, the server may transmit the second A/V data of the second user to the first device via the first session at a first element rate based on data indicative of the first user's FOV. The server may transmit the third A/V data of the third user to the first device via the first session at a second bit rate based on the data indicative of the first user's FOV.

Before turning to the drawings, which illustrate certain specific examples in detail, it is to be understood that the invention is not limited to the details or methodology set forth in the description or illustrated in the drawings. It is also to be understood that the terminology used herein is for the purpose of description only and should not be regarded as limiting.

Figure 1 illustrates an exemplary wireless communications system 100. The wireless communication system 100 may include a base station 110 (also referred to as a "wireless communication node 110" or "station 110") and one or more user equipment (UE) 120 (also referred to as a "wireless communication device 120") or "terminal device 120"). The base station 110 and the UE 120 can communicate via wireless communication links 130A, 130B, and 130C. The wireless communication link 130 may be a cellular communication link that complies with 3G, 4G, 5G or other cellular communication protocols or WiFi communication protocols. In one example, wireless communication link 130 supports, employs, or is based on orthogonal frequency division multiple access (OFDMA). In one aspect, UE 120 is located within geographic boundaries relative to base station 110 and may communicate with or through base station 110 . In some embodiments, wireless communication system 100 includes more, fewer, or different components than shown in FIG. 1 . For example, the wireless communication system 100 may include one or more additional base stations 110 in addition to those shown in FIG. 1 .

In some specific examples, the UE 120 may be a user device, such as a mobile phone, a smart phone, a personal digital assistant (PDA), a tablet computer, a laptop computer, a portable computing device, etc. Each UE 120 may communicate with the base station 110 via a corresponding communication link 130. For example, the UE 120 can transmit data to the base station 110 via the wireless communication link 130 and receive data from the base station 110 via the wireless communication link 130 . Example data may include audio data, image data, text, etc. The communication or transmission of data from the UE 120 to the base station 110 may be referred to as uplink communication. The transmission or reception of data by the UE 120 from the base station 110 may be referred to as downlink communication. In some examples, UE 120A includes wireless interface 122, processor 124, memory device 126, and one or more antennas 128. These components may be implemented as hardware, software, firmware, or a combination thereof. In some embodiments, UE 120A includes more, fewer, or different components than shown in FIG. 1 . For example, UE 120 may include an electronic display and/or input device. For example, UE 120 may include additional antennas 128 and wireless interfaces 122 than those shown in FIG. 1 .

The antenna 128 may be a component that receives radio frequency (RF) signals and/or transmits RF signals via wireless media. RF signals can be at frequencies between 200 MHz and 100 GHz. RF signals may have packets, symbols or frames corresponding to the data used for communication. Antenna 128 may be a dipole antenna, a patch antenna, a loop antenna, or any suitable antenna for wireless communications. In one aspect, a single antenna 128 is used for both transmitting and receiving RF signals. In one aspect, different antennas 128 are used to transmit RF signals and receive RF signals. In one aspect, multiple antennas 128 are used to support multiple-in, multiple-out (MIMO) communications.

Wireless interface 122 includes or is implemented as a transceiver for transmitting and receiving RF signals over a wireless medium. The wireless interface 122 can communicate with the wireless interface 112 of the base station 110 via the wireless communication link 130A. In one configuration, wireless interface 122 is coupled to one or more antennas 128 . In one aspect, wireless interface 122 may receive RF signals via the RF frequency received by antenna 128 and down-convert the RF signals to a baseband frequency (eg, 0 to 1 GHz). Wireless interface 122 may provide the down-converted signal to processor 124 . In one aspect, wireless interface 122 may receive a baseband signal for transmission at a baseband frequency from processor 124 and upconvert the baseband signal to generate an RF signal. Wireless interface 122 may transmit RF signals via antenna 128 .

Processor 124 is a component that processes data. The processor 124 may be implemented as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a logic circuit, or the like. Processor 124 may obtain instructions from memory device 126 and execute the instructions. In one aspect, processor 124 may receive down-converted data from wireless interface 122 at a baseband frequency and decode or process the down-converted data. For example, the processor 124 may generate audio data or image data based on the down-converted data and present the audio indicated by the audio data and/or the image indicated by the image data to the user of UE 120A. In one aspect, processor 124 may generate or obtain data for transmission at baseband frequencies and encode or process the data. For example, the processor 124 may encode or process image data or audio data at a baseband frequency and provide the encoded or processed data to the wireless interface 122 for transmission.

Memory device 126 is a component that stores data. The memory device 126 may be implemented as random access memory (RAM), flash memory, read only memory (ROM), erasable programmable memory (ROM), or erasable programmable memory (ROM). read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), scratchpad, hard disk, removable disk, CD-ROM, or Any device that stores data. Memory device 126 may be implemented as a non-transitory computer-readable medium that stores instructions executable by processor 124 to perform the various functions of UE 120A disclosed herein. In some embodiments, memory device 126 and processor 124 are integrated into a single component.

In some embodiments, each of UEs 120B...120N includes similar components of UE 120A used to communicate with base station 110 . Therefore, detailed descriptions of repeated parts are omitted in this article for the sake of brevity.

In some specific examples, the base station 110 may be an evolved node B (eNB), a serving eNB, a target eNB, a pico station or a pico station. The base station 110 may be communicatively coupled to another base station 110 or other communication device via a wireless communication link and/or a wired communication link. Base station 110 may receive data (or RF signals) from UE 120 in uplink communications. Additionally or alternatively, base station 110 may provide the data to another UE 120, another base station, or another communication device. Thus, base station 110 allows communication among UEs 120 associated with base station 110 or other UEs associated with different base stations. In some embodiments, base station 110 includes wireless interface 112, processor 114, memory device 116, and one or more antennas 118. These components may be implemented as hardware, software, firmware, or a combination thereof. In some embodiments, base station 110 includes more, fewer, or different components than shown in FIG. 1 . For example, base station 110 may include an electronic display and/or input device. For example, base station 110 may include additional antennas 118 and wireless interfaces 112 other than those shown in FIG. 1 .

Antenna 118 may be a component that receives radio frequency (RF) signals and/or transmits RF signals via wireless media. Antenna 118 may be a dipole antenna, a patch antenna, a loop antenna, or any suitable antenna for wireless communications. In one aspect, a single antenna 118 is used for both transmitting and receiving RF signals. In one aspect, different antennas 118 are used to transmit RF signals and receive RF signals. In one aspect, multiple antennas 118 are used to support multiple-input multiple-output (MIMO) communications.

Wireless interface 112 includes or is implemented as a transceiver for transmitting and receiving RF signals over a wireless medium. The wireless interface 112 can communicate with the wireless interface 122 of the UE 120 via the wireless communication link 130 . In one configuration, wireless interface 112 is coupled to one or more antennas 118 . In one aspect, wireless interface 112 may receive RF signals via the RF frequency received by antenna 118 and down-convert the RF signals to a baseband frequency (eg, 0 to 1 GHz). Wireless interface 112 may provide the down-converted signal to processor 124 . In one aspect, wireless interface 122 may receive a baseband signal from processor 114 for transmission at a baseband frequency and upconvert the baseband signal to generate an RF signal. Wireless interface 112 may transmit RF signals via antenna 118 .

Processor 114 is a component that processes data. Processor 114 may be implemented as an FPGA, ASIC, logic circuit, or the like. Processor 114 may obtain instructions from memory device 116 and execute the instructions. In one aspect, processor 114 may receive down-converted data from wireless interface 112 at a baseband frequency and decode or process the down-converted data. For example, the processor 114 may generate audio data or image data based on the down-converted data. In one aspect, processor 114 may generate or obtain data for transmission at baseband frequencies and encode or process the data. For example, the processor 114 may encode or process image data or audio data at baseband frequencies and provide the encoded or processed data to the wireless interface 112 for transmission. In one aspect, processor 114 may configure, assign, schedule, or allocate communication resources to different UEs 120 . For example, the processor 114 may set different modulation schemes, time slots, channels, frequency bands, etc. for the UE 120 to avoid interference. Processor 114 may generate data (or UL CG) indicating the configuration of communication resources and provide the data (or UL CG) to wireless interface 112 for transmission to UE 120 .

Memory device 116 is a component that stores data. Memory device 116 may be implemented as RAM, flash memory, ROM, EPROM, EEPROM, scratchpad, hard drive, removable disk, CD-ROM, or any device capable of storing data. Memory device 116 may be implemented as a non-transitory computer-readable medium that stores instructions executable by processor 114 to perform the various functions of base station 110 disclosed herein. In some embodiments, memory device 116 and processor 114 are integrated into a single component.

In some specific examples, communication between the base station 110 and the UE 120 is based on one or more layers of the Open Systems Interconnection (Open Systems Interconnection; OSI) model. The OSI model may include the following layers: physical layer, Medium Access Control (MAC) layer, Radio Link Control (RLC) layer, Packet Data Convergence Protocol (PDCP) layer, Radio Resource Control (RRC) layer, Non Access Stratum (NAS) layer or Internet Protocol (IP) layer and other layers.

Figure 2 is a block diagram of an exemplary artificial reality system environment 200. In some specific examples, the artificial reality system environment 200 includes a HWD 250 worn by a user, and controls that provide artificial reality (eg, augmented reality, virtual reality, mixed reality) content to the HWD 250 Station 210. Each of HWD 250 and console 210 may be a separate UE 120. HWD 250 may be called, include, or be part of the following: head mounted display (HMD), head mounted device (HMD), head wearable device (HWD) , head worn display (head worn display; HWD) or head worn device (head worn device; HWD). The HWD 250 can detect the position and/or orientation of the HWD 250 and the shape, position and/or orientation of the user's body/hands/face, and convert the detected position/or orientation and/or the HWD 250 Or tracking information indicating the shape, position and/or orientation of the body/hand/face is provided to the console 210 . The console 210 can be based on the detected position and/or orientation of the HWD 250, the detected shape, position and/or orientation of the user's body/hand/face, and/or for use in artificial reality. The user inputs image data that generates images used to indicate artificial reality, and transmits the image data to the HWD 250 for presentation. In some embodiments, artificial reality system environment 200 includes more, fewer, or different components than shown in FIG. 2 . In some embodiments, the functionality of one or more components of artificial reality system environment 200 may be distributed among the components in a manner different from that described herein. For example, some of the functionality of console 210 may be performed by HWD 250. For example, some of the functionality of HWD 250 may be performed by console 210. In some embodiments, console 210 is integrated as part of HWD 250.

In some embodiments, the HWD 250 is an electronic component that can be worn by the user and can present or provide an artificial reality experience to the user. The HWD 250 can display one or more images, videos, audio, or a combination thereof to provide users with an artificial reality experience. In some embodiments, the audio is presented via an external device (eg, speakers and/or headphones) that receives audio information from HWD 250, console 210, or both, and presents the audio based on the audio information. In some embodiments, HWD 250 includes a sensor 255 , a wireless interface 265 , a processor 270 , an electronic display 275 , a lens 280 and a compensator 285 . These components may operate together to detect the position of the HWD 250 and the gaze direction of the user wearing the HWD 250, and display an image of the field of view within the artificial reality corresponding to the detected position and/or orientation of the HWD 250. In other embodiments, HWD 250 includes more, fewer, or different components than shown in FIG. 2 .

In some embodiments, the sensor 255 includes an electronic component or a combination of an electronic component and a software component that detects the position and orientation of the HWD 250 . Examples of sensors 255 may include: one or more imaging sensors, one or more accelerometers, one or more gyroscopes, one or more magnetometers, or another device that detects motion and/or position. Suitable type of sensor. For example, one or more accelerometers can measure translational movement (e.g., forward/backward, up/down, left/right), and one or more gyroscopes can measure rotational movement (e.g., pitch, yaw, roll). In some embodiments, the sensor 255 detects translational movement and rotational movement, and determines the orientation and position of the HWD 250 . In one aspect, sensor 255 may detect translational and rotational movement relative to a previous orientation and position of HWD 250 and determine HWD by accumulating or integrating the detected translational movement and/or rotational movement. 250's new orientation and/or position. For example, assuming that the HWD 250 is oriented in a direction of 25 degrees from the reference direction, in response to detecting that the HWD 250 has rotated 20 degrees, the sensor 255 can determine that the HWD 250 is now facing in a direction of 45 degrees from the reference direction or in a direction away from the reference direction. Oriented in a direction 45 degrees from the reference direction. For another example, assuming that the HWD 250 is located two feet away from the reference point in the first direction, in response to detecting that the HWD 250 has moved three feet in the second direction, the sensor 255 may determine that the HWD 250 is now located at the first The vector multiplication of two feet in one direction and three feet in the second direction.

In some embodiments, sensor 255 includes an eye tracker. The eye tracker may include electronic components or a combination of electronic components and software components that determine the gaze direction of the user of HWD 250 . In some embodiments, HWD 250, console 210, or combinations thereof may incorporate the gaze direction of a user of HWD 250 to generate image data for artificial reality. In some embodiments, the eye tracker includes two eye trackers, wherein each eye tracker captures an image of a corresponding eye and determines the gaze direction of the eye. In one example, the eye tracker determines an angular rotation of the eye, a translation of the eye, a change in eye torsion, and/or a change in eye shape based on the captured image of the eye, and based on the determined changes in angular rotation, translation, and eye torsion The relative gaze direction relative to HWD 250 is determined. In one approach, the eye tracker can illuminate or project a predetermined reference or structured pattern onto a portion of the eye, and capture an image of the eye to analyze the pattern projected on the portion of the eye to determine the position of the eye relative to HWD 250 Relative gaze direction. In some embodiments, the eye tracker also has the orientation of the HWD 250 and the relative gaze direction relative to the HWD 250 to determine the user's gate direction. For example, assuming that HWD 250 is oriented in a direction 30 degrees from the reference direction, and the relative gaze direction of HWD 250 is -10 degrees (or 350 degrees) relative to HWD 250, the eye tracker can determine the user's gaze direction 20 degrees from the reference direction. In some embodiments, a user of HWD 250 can configure HWD 250 (eg, via user settings) to enable or disable the eye tracker. In some embodiments, the user of the HWD 250 is prompted to enable or disable the eye tracker.

In some embodiments, wireless interface 265 includes electronic components or a combination of electronic components and software components that communicate with console 210 . Wireless interface 265 may be or correspond to wireless interface 122 . The wireless interface 265 can communicate with the wireless interface 215 of the console 210 via the wireless communication link via the base station 110 . Via the communication link, the wireless interface 265 may transmit to the console 210 data indicating the determined position and/or orientation of the HWD 250 and/or the determined gaze direction of the user. Additionally, via the communication link, the wireless interface 265 may receive image data indicating or corresponding to the image to be displayed and additional data associated with the image from the console 210 .

In some embodiments, the processor 270 includes an electronic component or a combination of an electronic component and a software component that is used to display one or more images based on changes in space in view of the artificial reality, for example. In some embodiments, processor 270 is implemented as part of or communicatively coupled to processor 124 . In some embodiments, processor 270 is implemented as a processor (or graphical processing unit (GPU)) that executes instructions to perform various functions described herein. The processor 270 may receive image data describing an image of the artificial reality to be rendered and additional data associated with the image via the wireless interface 265 and render the image for display via the electronic display 275 . In some embodiments, image data from console 210 may be encoded, and processor 270 may decode the image data to display the image. In some embodiments, processor 270 receives additional data from console 210, object information indicative of virtual objects in the artificial reality space, and depth information indicative of the virtual object's depth (or distance from HWD 250). In one aspect, processor 270 may perform rendering, reconstruction, and/or rendering based on images of the artificial reality, object information, depth information from console 210, and/or updated sensor measurements from sensor 255. Projecting and/or blending to update images of the artificial reality to correspond to the updated position and/or orientation of HWD 250 . Assuming that the user rotates his or her head after the initial sensor measurement, processor 270 can generate a small portion (eg, 10%) of the image corresponding to the updated field of view within the artificial reality based on the updated sensor measurement. , and append that portion of the image to the image data from console 210 via reprojection, rather than recreating the entire image in response to updated sensor measurements. Processor 270 may perform shading and/or blending of the additional edges. Therefore, the processor 270 can generate an image of the artificial reality without re-creating the image of the artificial reality based on the updated sensor measurements.

In some embodiments, electronic display 275 is an electronic component that displays images. The electronic display 275 may be, for example, a liquid crystal display or an organic light emitting diode display. Electronic display 275 may be a transparent display that allows the user to see through. In some embodiments, when HWD 250 is worn by a user, electronic display 275 is positioned proximate (eg, less than 3 inches) to the user's eyes. In one aspect, electronic display 275 emits or projects light toward the user's eyes based on images generated by processor 270 .

In some embodiments, lens 280 is a mechanical component that modifies light received from electronic display 275 . Lens 280 can amplify light from electronic display 275 and correct optical errors associated with the light. Lens 280 may be a Fresnel lens, a convex lens, a concave lens, a filter, or any suitable optical component that modifies light from electronic display 275 . Through the lens 280, light from the electronic display 275 can reach the pupil, so that even though the electronic display 275 is very close to the eye, the user can still see the image displayed by the electronic display 275.

In some embodiments, compensator 285 includes an electronic component or a combination of electronic and software components that performs compensation to compensate for any distortion or aberration. In one aspect, lens 280 introduces optical aberrations, such as chromatic aberration, pincushion distortion, barrel distortion, and the like. Compensator 285 may determine compensation (eg, predistortion) to be applied to the image to be displayed from processor 270 to compensate for distortion caused by lens 280 , and apply the determined compensation to the image from processor 270 . Compensator 285 may provide the predistorted image to electronic display 275 .

In some embodiments, console 210 is an electronic component or a combination of electronic components and software components that provides content to be displayed to HWD 250 . In one aspect, console 210 includes wireless interface 215 and processor 230 . These components may operate together to determine a field of view of the artificial reality corresponding to the position of HWD 250 and the direction of gaze of the user of HWD 250 (e.g., the user's FOV), and may generate an indication corresponding to the determined field of view. Image data of images of artificial reality. In addition, these components can operate together to generate additional data associated with the image. The additional data may be information associated with images that present or represent artificial reality rather than artificial reality. Examples of additional data include hand model data, mapping information used to convert the position and orientation of the HWD 250 in physical space to virtual space (or simultaneous localization and mapping (SLAM) data), eyes Tracking data, motion vector information, depth information, edge information, object information, etc. The console 210 can provide image data and additional data to the HWD 250 for rendering artificial reality. In other embodiments, console 210 includes more, fewer, or different components than shown in FIG. 2 . In some embodiments, console 210 is integrated as part of HWD 250.

In some embodiments, the wireless interface 215 is an electronic component or a combination of an electronic component and a software component that communicates with the HWD 250 . Wireless interface 215 may be or correspond to wireless interface 122 . Wireless interface 215 may be a corresponding component of wireless interface 265 to communicate via a communication link (eg, a wireless communication link). Via the communication link, wireless interface 215 may receive data from HWD 250 indicating the determined position and/or orientation of HWD 250 and/or the determined gaze direction of the user. Additionally, via the communication link, the wireless interface 215 may transmit to the HWD 250 image data describing the image to be displayed and additional data associated with the image of the artificial reality.

Processor 230 may include or correspond to components that generate content to be displayed based on the position and/or orientation of HWD 250 . In some embodiments, processor 230 is implemented as part of or communicatively coupled to processor 124 . In some embodiments, the processor 230 may incorporate the user's gaze direction of the HWD 250 . In one aspect, processor 230 determines the field of view of the artificial reality based on the position and/or orientation of HWD 250 . For example, the processor 230 maps the position of the HWD 250 in the physical space to a position in the artificial reality space, and determines the artificial reality along the direction corresponding to the mapped orientation from the mapped position in the artificial reality space. The vision of the surrounding space. The processor 230 may generate image data describing an image of the determined field of view of the artificial reality space, and transmit the image data to the HWD 250 via the wireless interface 215 . In some embodiments, the processor 230 may generate additional data including motion vector information, depth information, edge information, object information, hand model data, etc. associated with the image, and combine the additional data with the image data via the wireless interface 215 transferred together to HWD 250. Processor 230 may encode image data describing the image and may transmit the encoded data to HWD 250 . In some embodiments, the processor 230 generates and provides image data to the HWD 250 periodically (eg, every 11 ms).

In one aspect, the process of detecting the position of the HWD 250 and the gaze direction of the user wearing the HWD 250 and presenting the image to the user should be performed within a frame time (eg, 11 ms or 16 ms). The time delay between the movement of a user wearing HWD 250 and the images displayed corresponding to the user's movement can cause jitter, which can induce motion sickness and can degrade the user experience. In one aspect, the HWD 250 and the console 210 can prioritize AR/VR communication so that the delay between the movement of a user wearing the HWD 250 and the images displayed corresponding to the user's movement can be represented in the message. frame time (for example, 11 ms or 16 ms) to provide a seamless experience.

Figure 3 is a diagram of HWD 250 according to an exemplary embodiment. In some embodiments, HWD 250 includes front rigid body 305 and belt 310 . The front rigid body 305 includes an electronic display 275 (not shown in FIG. 3 ), a lens 280 (not shown in FIG. 3 ), a sensor 255 , a wireless interface 265 and a processor 270 . In the specific example shown in FIG. 3 , the wireless interface 265 , processor 270 and sensor 255 are located within the front rigid body 205 and may not be visible from the outside. In other embodiments, HWD 250 has a different configuration than shown in FIG. 3 . For example, wireless interface 265, processor 270, and/or sensor 255 may be in a different location than shown in Figure 3.

Various operations described in this article can be performed on computer systems. Figure 4 shows a block diagram of a representative computing system 414 that may be used to implement the present invention. In some embodiments, source device 110 , sink device 120 , console 210 , HWD 250 are implemented by computing system 414 . Computing system 414 may be implemented as, for example, a consumer device such as a smartphone, other mobile phone, tablet computer, portable computing device (e.g., smart watch, glasses, head-worn display), desktop computer, laptop computers, or implemented via distributed computing devices. Computing system 414 may be implemented to provide VR, AR, MR experiences. In some embodiments, computing system 414 may include conventional computer components, such as processor 416, storage device 418, network interface 420, user input device 422, and user output device 424.

Network interface 420 may provide a connection to a wide area network (eg, the Internet) to which the WAN interface of the remote server system is also connected. Network interface 420 may include a wired interface (e.g., B Ethernet) and/or wireless interface.

Network interface 420 may include a transceiver that allows computing system 414 to transmit and receive data from remote devices using transmitters and receivers. The transceiver can be configured to support industry standards for transmit/receive support for bidirectional communications. The antenna can be attached to the transceiver housing and electrically coupled to the transceiver. Additionally or alternatively, a multiple antenna array may be electrically coupled to the transceiver such that multiple beams pointed in different directions may facilitate transmission and/or reception of data.

The transmitter may be configured to wirelessly transmit frames, slots, or symbols generated by processor unit 416. Similarly, a receiver may be configured to receive frames, slots, or symbols, and processor unit 416 may be configured to process frames. For example, processor unit 416 may be configured to determine the type of the frame and process the frame and/or fields of the frame accordingly.

User input device 422 may include any device (or devices) through which a user may provide a signal to computing system 414; computing system 414 may interpret the signal as indicative of a specific user request or information. User input device 422 may include a keyboard, trackpad, touch screen, mouse or other pointing device, scroll wheel, click wheel, dial, button, switch, keypad, microphone, sensor (e.g., motion sensor). any or all of the sensors, eye tracking sensors, etc.).

User output device 424 may include any device through which computing system 414 may provide information to a user. For example, user output device 424 may include a display for displaying images generated by or delivered to computing system 414 . Displays can incorporate various image-generating technologies, such as liquid crystal displays (LCDs), light-emitting diodes (LEDs) including organic light-emitting diodes (OLEDs), projection systems, and cathode ray tubes. (cathode ray tube; CRT) or similar, and supporting electronics (e.g., digital to analog or analog to digital converters, signal processors, or similar). Devices that serve as both input and output devices can be used, such as touch screens. In addition to or in lieu of a display, an output device 424 may also be provided. Examples include indicator lights, speakers, tactile "display" devices, printers, etc.

Some implementations include electronic components, such as microprocessors, storage, and memory, which store computer program instructions in a computer-readable storage medium (eg, a non-transitory computer-readable medium). Many of the features described in this specification may be implemented as a program specified as a set of program instructions encoded on a computer-readable storage medium. When these program instructions are executed by one or more processors, they cause the processors to perform various operations indicated in the program instructions. Examples of program instructions or computer code include machine code such as that produced by a compiler, and files including higher-level code that is executed by a computer, electronic component, or microprocessor using an interpreter. Through suitable programming, processor 416 may provide various functionality for computing system 414, including any of the functionality described herein as being performed by a server or client or other functionality associated with information management services. One.

It will be understood that computing system 414 is illustrative and that changes and modifications are possible. Computer systems used in conjunction with the present invention may have other capabilities not specifically described herein. Additionally, although computing system 414 is described with reference to specific blocks, it should be understood that such blocks are defined for convenience of description and are not intended to imply a specific physical arrangement of component parts. For example, different blocks may be located in the same facility, in the same server rack, or on the same motherboard. Additionally, the blocks need not correspond to physically distinct components. The blocks may be configured to perform various operations, such as by programming the processor or providing appropriate control circuitry, and depending on how the initial configuration is obtained, the various blocks may or may not be reconfigurable. Implementations of the invention may be implemented in a variety of devices including electronic devices implemented using any combination of circuitry and software. System and method for holographic communication

Referring now to FIG. 5 , depicted is an exemplary view 500 of a holographic call or communication session via a head wearable device (HWD) 606 in accordance with an exemplary implementation of the present invention. Holographic calling or communication may be or include services provided by XR/AR/VR/MR devices, such as smart glasses or other HWD (such as HWD 606). Services may provide the user with information represented in three-dimensional (3D) graphics overlaid on top of the user's physical/spatial environment visible through the display (e.g., the physical/spatial environment of AR glasses and/or VR headsets). The ability to communicate with others. In various embodiments, the location and orientation of other users or objects displayed to the user may be determined by one or more servers and/or by smart glasses, HWD, or other devices communicatively coupled thereto. determination.

To provide holographic communication, various imagers and/or microphones can capture the user's audio/video (A/V) data (e.g., from various angles, directions, viewing angles, positions). The imager and/or microphone may communicate A/V data to the server (e.g., via a tethered/connected smartphone or other user device). For example, a user device receiving A/V data may compress the A/V data and may transmit the compressed A/V data to a server. Upon receiving the compressed A/V data, the server can transmit the compressed A/V data to another device, such as another smartphone or user associated with another user of the holographic communication session device). Another device can receive and reconstruct the media for presentation on the HWD. For example, another device can transmit the displayed media to the HWD in various formats for presentation to another user. In cases where a holographic communication session includes multiple users (eg, three or more), the server may collect and process A/V data for each user.

In situations where a holographic communication session includes multiple users, the server may apply various considerations as part of processing the A/V data. For example, video data for a specific user or object should be scaled to match the size of the background and other users or objects. Similarly, audio data can be spatially reconstructed/remapped to match the corresponding user's location on the display. Additionally, where multiple users are participating in a holographic communication session, the server can virtually position each user in a fixed location relative to the other users.

Referring now to FIG. 6 , depicted is a block diagram of a system 600 for holographic communications in accordance with an exemplary implementation of the present invention. System 600 may include a plurality of client systems 602 communicatively coupled to one or more servers 604 . The user system 602 may include a respective head wearable device (HWD) 606, such as a virtual reality (VR) headset, augmented reality (AR) smart glasses, or other devices. HWD 606 may be similar to HWD 250 described above (or include hardware/software/components similar to HWD 250). HWD 606 may be configured to be displayed or augmented on an environment including a physical (or real world) environment in embodiments where HWD 606 is AR HWD 606 or a virtual environment in embodiments where HWD 606 is VR HWD 606 graphics. User system 602 may include imaging system 608 . Imaging system 608 may be configured to capture at least video data corresponding to a user of client system 602 . Additional details regarding imaging system 508 are described with reference to FIG. 7 . User system 602 may include user device 610 . User device 610 may be or include hardware similar to UE 120 described above with reference to FIG. 1 . For example, user device 610 may include a smartphone, mobile device, tablet, laptop, or other user device. Although shown as separate devices, in various embodiments, two or more of the devices of respective client systems 602 may be combined into a single device. For example, imaging system 608 may be a component of the hardware of user device 610 .

As described in greater detail below, and according to various embodiments, each of the user systems 602 may establish a hologram communication session with the server 604 . The imaging system 508 of the respective user system 602 may be configured to capture audio/video (A/V) data for the respective user and may transmit the A/V data to the user device 610 (e.g., via local connection or link). In some embodiments, imaging system 608 may be configured to capture scaling data indicative of the size, gravity, dimension, or proportion of a user or object of A/V data. Imaging system 608 may be configured to transmit zoom data to user device 610 . Additionally, and in various embodiments, HWD 606 may be configured to capture directional data indicative of a respective user's gaze, and HWD 606 may be configured to transmit the directional data to user device 610 . User device 610 may be configured to communicate, transmit, send, or otherwise provide A/V data, zoom data, and/or direction data to server 604. Server 604 may be configured to receive A/V data as well as zoom data and direction data from each of client systems 602 . Server 604 may be configured to manage holographic communication sessions by controlling various aspects of A/V data based on scaling data and orientation data, and may communicate modified A/V data corresponding to an individual user To other users' client systems for presentation (e.g., via respective HWD 606).

Server 604 may include one or more processors 612. The processor 612 may be similar to the processors 114, 124, 230, 270 described above with reference to FIGS. 1 and 2 and/or the processing unit 416 described above with reference to FIG. 4. Server 604 may include memory 614. Memory 614 may be similar to memories 116, 126 described above with reference to FIG. 1 and/or storage 418 described above with reference to FIG. 4. Server 604 may include one or more processing engines 616. Processing engine 616 may be or include any device, component, element or hardware designed or configured to perform various functions or tasks associated with server 604 . For example, processing engine 616 may be configured to perform various functions related to establishing and managing holographic communication sessions across multiple client systems 602, as described in greater detail below. It should be noted that the various processing engines 616 described herein may be subdivided into a plurality of additional processing engines 616 and, additionally or alternatively, the various processing engines 616 described herein may be combined into a single processing engine 616.

Server 604 may include session manager engine 618. Session manager engine 618 may be configured to establish and/or maintain holographic communication sessions across multiple client systems 602 . In some embodiments, session manager engine 618 may be configured to establish a session in response to receiving a request to establish a session from each of user devices 610 . For example, a first user of first client system 602(2) may access applications or resources on user device 610(1) to initiate communication with other user devices 610(2) through 610( N) holographic communication session (e.g., by dialing the number or user name or other user identifier associated with user device 610(2) through 610(N)). User device 610(1) may transmit to server 604 a request to initiate a hologram session and identifiers for other user devices 610(2) through 610(N). Session manager engine 618 can be configured to receive a request and can establish a session based on the request. Session manager engine 618 may be configured to transmit or forward requests to join with other user devices 610(2)-610(N) using identifiers for other user devices 610(2)-610(N). ) session.

Referring to FIGS. 6 and 7 , the session manager engine 618 may be configured to maintain for each session a local map 620 that includes an index 622 associated with each client system 602 included in the session. In particular, FIG. 7 depicts a graphical representation 700 of a virtualization map 620 in accordance with an exemplary implementation of the present invention. Local map 620 may include a table that includes the location and relative location of each of the users associated with respective user systems 602, such as Table 1 below. Location Adjacent devices (L, R) Device ID Device name A (UserDevice(N), UserDevice(2)) AAAAAAA User device (1) B (User device (1), User device (3)) BBBBBBB User device (2) … N (User Device (N-1), User Device (1)) NNNNNNN User device (N) Table 1. Partial mapping for holographic communication sessions

In Table 1 above, each column of the table may be an index 622 corresponding to a specific user device 602. Each index may include a location corresponding to the user's location within graphical representation 700 . For example, the user corresponding to user device 610(1) may be located at location A, the user corresponding to user device 610(2) may be located at location B, and so on. The client system 602 may randomly assign locations in the virtualization map on a first-come, first-served basis according to various user-defined rules, etc. Although shown as a round table graphical representation, in various embodiments, the session manager engine 618 can be configured to receive the type or settings of each user system 602 . For example, one user in the first user system 602(1) may be placed at a round table, while another user in the second user system 602(2) may be placed at a rectangular table. As part of joining a session, session manager engine 618 may be configured to receive respective user entity settings from first user device 610(1) and second user device 610(2), for example. Session manager engine 618 can be configured to incorporate this information into the index for the corresponding user.

As shown in Table 1, each index 622 may include data indicating neighboring devices and/or users within local map 620. Therefore, each user can have different physical settings but can be "virtually placed" across sessions or located in close proximity to the same user. Continuing with the example shown in Figure 7 and referring to Table 1, the first user's index may include information identifying devices of other client systems 602 on the right and left of the first user, such as those located at location B. The second user (or a user associated with the second client system 602(2)) and the Nth user (e.g., eight of the users are in the holographic communication session, the Nth user may be located H). Similarly, indexing a second user (e.g., client system 602(2)) may include identifying a first user (e.g., associated with the first user of first client system 602(1)). Data of the user device 610(1)) and the third user located in the third location C. When a new user enters a holographic communication session, the session manager engine 618 may be configured to update the session's partial map 620 and may add the new user's index 622 based on the updated partial map 620.

Referring back to FIG. 6 , server 604 may include session data receiving engine 624 . Session data reception engine 624 may be configured to ingest, identify, manage, or otherwise receive session data from each of plurality of user systems 602 . The session data receiving engine 624 can be configured to maintain or manage the data stream and the session data transmitting engine 638 to maintain the data flow between the user systems 602 to provide a hologram communication session.

Referring now to FIGS. 6 and 8 , the client system 602 may include various devices for capturing session data for transmission to the server 604 . In particular, FIG. 8 shows various views of an imaging system 608 in accordance with an exemplary implementation of the present invention. Although shown as a separate imaging system 608, in various embodiments, each of the components of imaging system 608 may be incorporated into or included with user device 610. In other words, imaging system 608 may be the imaging system of user device 610 .

Imaging system 608 may be configured to capture audio/video (A/V) data of a user of client system 602 . In some embodiments, imaging system 608 may be configured to capture spatial audio via two or more microphones 800 and via various laser emitters 802 , color or imaging (eg, red-green-blue (RGB)) Sensor 804, depth sensor 806, etc. capture three-dimensional (3D) video. Two or more microphones 800 can form a stereo audio capture system. Similarly, although shown using a laser emitter 802, a color or image sensor 804, and a depth sensor 806, in various embodiments, in order to capture 3D video, the imaging system 608 may include two or more A camera configured to form a stereoscopic video capture system configured to capture 3D video. As illustrated in Figure 8, imaging system 608 may include a depth field of view (FOV), image sensor FOV, in both the lateral (or horizontal) direction and the longitudinal (or vertical) direction.

Imaging system 608 may be configured to communicate, transmit, send, or otherwise provide data corresponding to the FOV of imaging system 608 to user device 610 . In some embodiments, imaging system 608 may be configured to receive data as part of establishing or otherwise participating in a session (e.g., as part of negotiating a local link or connection between imaging system 608 and user device 610 part) is provided to the user device 610. Imaging system 608 may be configured to provide data as a range or number of bits or pixels in the longitudinal, lateral, and/or depth directions. Imaging system 608 may be configured to transmit data about the microphones, such as microphone type and orientation (eg, left and right orientation) of a stereo microphone.

Referring now to FIGS. 8 and 9 , imaging system 608 may be configured to transmit, send, or otherwise provide A/V data captured by imaging system 608 to user device 610 . In particular, FIG. 9 shows an example image corresponding to video data that may be captured by imaging system 608 in accordance with an example implementation of the present invention. Imaging system 608 may be configured to capture video data of A/V data as point clouds, grids, RGB-Depth (RGB-D), etc. Imaging system 608 may be configured to capture, detect, or otherwise determine the height and width of users or objects represented by video data. In some embodiments, imaging system 608 may be configured to base the range or distance from imaging system 608 to an object or user (eg, an individual point on the user as detected by imaging system 608 ) and FOV data. Estimating the height and width of the user or object as a percentage or ratio of the object/user relative to the FOV. Imaging system 608 may be configured to compute a value (eg, a 12-bit signed integer) representing the height or width of a user (eg, spanning 0 to 4095 mm). Imaging system 608 may be configured to transmit, send, or otherwise provide values representing height and width (eg, collectively referred to as zoom data) to user device 610 .

In some embodiments, imaging system 608 may be configured to calculate the distance between each point of the point cloud and a vertical plane including the depth sensor based on various intrinsic and extrinsic qualities of imaging system 608 . For example, imaging system 608 may be configured to compute or otherwise determine rotations and translations from the 3D physical world coordinate system to the imager's coordinate system. Imaging system 608 may be configured to determine the intrinsic matrix K for the imaging system based on various parameters of imaging system 608. Imaging system 608 may be configured to determine the intrinsic matrix K as The variables or parameters used to calculate the intrinsic matrix K are shown in Table 2 and may be determined or otherwise provided by the operating system of imaging system 608 . parameters unit definition Comment fx float X-axis focal length (in pixels) fy float Y-axis focal length (in pixels) cx float X-axis principle point (in pixels) cy float Y-axis principle point (in pixels) s float Offset coefficient If the image axis is vertical, it is zero Table 2. Parameters used to determine the intrinsic matrix of the imaging system

Imaging system 608 may be configured to determine or generate a point cloud from a pair of RGB-D frames by using an intrinsic matrix K-inversion transformation. In various embodiments, imaging system 608 may be configured to determine the size of objects or persons in video data using one or more applications or resources of imaging system 608 that provide information around A bounding box or cube of an object to display measurements (for example, length, width, and height). The imaging system 608 can be configured to set a minimum size of the bounding box or cube so that the bounding box fits closely around the object or user, thus providing a more accurate measurement of the size of the object or user.

Imaging system 608 may be configured to transmit zoom data in real-time transport control protocol (RTCP) packets to user device 610 . In some embodiments, imaging system 608 may be configured to transmit zoom data periodically and/or on demand. RTCP packets may include packet type, object identifier, width and height. For example, an RTCP packet may have four bits for the packet type (e.g., to indicate that the RTCP includes size information of a person or object), four bits for an object identifier (e.g., to indicate or identify video data a person or object in) and a format of 12 bits for both the width and height of the person or object (e.g., ranging from 0 to 4095 for both width and height).

HWD 606 may be configured to communicate, transmit, or otherwise provide direction data to user device 610. Directional information may be or include position, direction, direction, or other directional information indicative of the user's wearer's gaze. In some embodiments, HWD 606 may be configured to provide direction data as 8-bit signed bits representing the direction of each axis (eg, on the X-axis or transverse axis, and on the Y-axis or longitudinal axis) integer. HWD 606 may be configured to measure direction data based on relative position relative to true north, based on relative position relative to a fixture or location (such as imaging system 608), etc. For example, HWD 606 may include an accelerometer, a gyroscope, or other motion sensor configured to measure the position or movement of HWD 608 (eg, relative to an axis or plane of HWD 606). HWD 606 may be configured to determine direction data based on or in accordance with measurements from the motion sensor of HWD 606. HWD 606 may be configured to transmit direction data to user device 610.

The HWD 606 can be configured to transmit direction data to the user device in RTCP packets. Similar to RTCP packets that include scaling data, the HWD 606 can be configured to transmit RTCP packets with direction data on demand and/or periodically. RTCP packets may include packet type, HWD identifier, X direction, Y direction and/or Z direction. For example, an RTCP packet may have four bits including four bits for the packet type (e.g., indicating that the RTCP includes size information for a person or object), four bits for a HWD identifier (e.g., indicating or identifying the HWD 606) that captures direction data and/or an 8-bit signed integer value (e.g., spanning between 0 mm and 255 mm) determined for each of the X, Y, and Z directions.

Referring now to Figure 10, depicted is a diagram of an end user system 602 in communication with a server 604 in accordance with an exemplary implementation of the present invention. As illustrated in FIG. 10 , HWD 606 may maintain a wireless local area network (WLAN) (such as Wi-Fi) connection with user device 610 , and user device 610 may maintain a connection with server 604 Cellular connection (shown as a 5G connection, but any type or form of cellular connection may be suitable). In this example, imaging system 606 may be local or local to user device 610 . HWD 606 may be configured to communicate or transmit direction data to user device 610 (eg, via a WLAN connection). User device 610 may be configured to capture A/V data and zoom data via the camera and microphone and transmit the A/V data and zoom data to server 604 (eg, via a cellular connection). As described in greater detail below, server 604 may be configured to transmit other A/V data (eg, of other users in the holographic communication session) back to user device 610 , which may transmit A/V data via a WLAN connection. /V data is transmitted to HWD 606 for presentation via one or more speakers and displays.

Referring back to FIG. 6, the session data receiving engine 624 of the server 604 may be configured to receive A/V data 626 from the respective communications of each of the plurality of client systems 602 in the holographic communication session. The session data receiving engine 624 may be configured to also receive zoom data 628 (eg, from the imaging system 608 of the respective user system 602) and orientation data 630 (eg, from the HWD 606 of the respective user system 602) . As described in greater detail below, server 604 may be configured to use scaling data 628 to generate, determine, derive, or otherwise provide modified video data based on scaling data 628 and may use direction data 630 to select bits. The rate is used to transmit the modified A/V data to the user system 602 for presentation.

The server 604 may include a session data processing engine 632 that includes a scaler 634 and a field of view (FOV) determiner 636 . As a brief overview, the scaler 634 may be configured to change, adjust, normalize, update, or otherwise modify the video of the A/V data based on the scaling data 628 received from the imaging system 608 of the respective user system 602 The proportion, relative size or proportion of the objects or users depicted in them. FOV determiner 636 may be configured to identify, detect, derive, compute, calculate, or otherwise determine each of the respective users based on the direction data 630 received from the HWD 606 of the respective user system 602 field of view.

Referring now to FIG. 6 and FIGS. 11A-11B , scaler 634 may be configured to modify the scale of objects or users depicted in the video data of the A/V data based on scaling data 628 . Specifically, FIGS. 11A and 11B show examples of frames of video data from three different user systems before and after scaling modifications according to an exemplary implementation of the present invention. As illustrated in Figures 11A and 11B, video data may originate from individual client systems 602(1) through 602(3) and may include representations of three different users 1102, 1104, 1106. The first user 1102 may be a child having a height of approximately three feet, the second user 1104 may be an adult having a height of approximately six feet, and the third user 1106 may be an adult having a height of approximately five feet. As mentioned above, client systems 602(1)-602(3) may be configured to provide zoom data for respective users 1102-1106 to server 604.

Scaler 634 may be configured to receive video data (eg, of A/V data 626) and scaling data 628 from user systems 602(1)-602(3). In some embodiments, scaler 634 may be configured to modify the scale of users or objects depicted in the video data based on scaling data 628 from each of the user systems 602 . Scaler 634 may be configured to increase and/or reduce users or objects in the video data based on scaling data 628 from respective client systems 602 relative to scaling data 628 from other client systems 602 Modify the scale of the user or object using its virtual representation. In some embodiments, scaler 634 may be configured to modify video from first user system 602 based on scaling data 628 from the same user system 602 and scaling data 628 from other user systems 628 The proportion of objects or users depicted in the data. Scaler 634 may be configured to modify the scale (eg, size, dimensions) of an object or user to normalize the proportions of an object relative to other objects. For example, assuming that the zoom data 628 from the first user system 602 is 20% larger than the zoom data 628 from the second user system 602, the scaler 630 may be configured to reduce the scale of the object by reducing the scale of the object. Modify the proportion of objects or users depicted in the video data from the first user system 602 (e.g., decrease by 20%) and/or modify the proportion of objects or users from the first client system 602 by increasing the proportion of the objects (e.g., increase by 20%) 2. The proportions or objects depicted in the video data of the user system 602. Thus, in response to modifying the proportions of objects, each of the users or objects represented in the modified video data may have normalized proportions (e.g., across A/V data) to exhibit substantially accurate relative proportions.

Continuing with the example shown in FIG. 11A , scaler 634 may be configured to scale based on scaling data 628 from a first user system 602 (eg, indicating a height of three feet) relative to a height from another user system 602 . Scaling data 628 (e.g., indicating the height of six feet of second user 1104) to modify the scale of first user 1102 to reduce the size represented in the video data 626 of A/V data 626 from first user system 602 The ratio of first users 1102. Similarly, scaler 634 may be configured to scale based on scaling data 628 from a third user system 602 (e.g., indicating a height of six feet) relative to scaling data 628 from another user system 602 (e.g., indicating a height of six feet for the second user 1104 and/or a height of three feet for the first user) to modify the proportions of the third user 1106 to increase the A/ from the third user system 602(3) The proportion of third users 1102 represented in the video data of V data 626. As illustrated in FIG. 11B , after scaler 634 modifies the proportions of users depicted in respective video data of A/V data 626 , the modified video data corresponding to the respective users may have appropriate relative proportions (e.g., The first user 1102 is shown as approximately half as tall as the second user 1104 , the second user 1104 is shown as approximately 20% taller than the third user 1108 , and the third user 1108 is taller than the first user 1102 High approximately 65%).

Referring again to FIGS. 6 and 7 , the FOV determiner 636 may be configured to detect, calculate, operate, and perform operations based on direction data 630 received from the user system 602 (direction data 630 represented as vector 704 in FIG. 7 ). The FOV 702 corresponding to the user of the client system 602 is identified or otherwise determined. FOV determiner 636 may be configured to determine FOV 702 based on direction data 630 and local map 620 maintained by session manager engine 618 . More specifically, the FOV determiner 636 may be configured to determine a particular user's FOV 702 based on the direction data 630 and the user's location in the local map (eg, the user's face, eyes, or gaze). In some embodiments, FOV determiner 636 may be configured to identify or determine a viewing range for application to direction data 630 to determine FOV 702 . The viewing range may be a default or standard viewing range (eg, 20° to the left and right and both the top and bottom sides of the vector 704 defined by the direction data 630). The view scope may be specific to the HWD 604 (eg, and provided to the server 604 as part of establishing a holographic communication session). FOV determiner 636 may be configured to determine FOV 702 as the viewing range applied to vector 704 in the X and Y directions. FOV determiner 636 may be configured to determine the FOV 702 corresponding to each of the users of respective client systems 602 .

FOV determiner 636 may be configured to identify, detect, or otherwise determine the location of an object or user (eg, reflected in local map 620 ) relative to FOV 702 . FOV determiner 636 may be configured to use local map 620 to determine the location of an object or user relative to FOV 702 . FOV determiner 636 may be configured to apply FOV 702 to local map 620 to determine which objects or users are within the respective user's FOV 702. For example, FOV determiner 636 may be configured to project FOV 702 onto local map 620 to determine which objects or users are located at locations that overlap or intersect FOV 702. In the example shown in Figure 7, after projecting the FOV 702 onto the local map 620, the FOV determiner 636 can be configured to determine the FOV of the user located at locations D-F and the user located at location A. 702 intersect or overlap. FOV determiner 636 may be configured to use indices 622(1) through 622(N) to determine the position of each of the users relative to FOV 702.

Referring to Figure 6, server 604 may include session data transfer engine 638. Session data transmission engine 638 may be configured to select a bit rate 640 (eg, from a plurality of bit rates 640(1) to 640(N)) for A/V data for transmission to user system 602 . Session data transfer engine 638 may be configured to select a bit rate 640 for compressing A/V data. The A/V data may be or include modified A/V data (eg, after scaler 634 modifies the scale of the object based on scale data 628). Based on the location assigned to the user corresponding to the source user system 602 relative to the FOV 702 corresponding to the receiving user system 602, the session data transfer engine 638 may be configured to interact with the source user system 602 A bit rate 640 is selected in association with the given A/V data for transmission to the receiving user system 602 . Accordingly, the session data transfer engine 638 may be configured to generate A/V data based on the source of the A/V data assigned (eg, by the session manager engine 618 ) to the FOV 702 with respect to the receiving user system 602 The location of the data source user system 602), and the same A/V data is compressed at different bit rates for different receiving end user systems 602.

In some embodiments, session data transfer engine 638 may be configured to select bits based on whether the location assigned to the user corresponding to source user system 602 is within FOV 702 corresponding to receiving user system 602 Meta rate 640. In other words, the session data transfer engine 638 may be configured to select the first element rate 640 for the A/V data from the source user system 602 having a location within the FOV 702 of the sink user system 602 , and selecting a second bit rate 640 for the A/V data from the source user system 602 having a location outside the FOV 702 of the sink user system 602. In this example, the first bit rate 640 may be higher than the second bit rate 640, thus resulting in higher quality/definition (eg, less pixelation) of A/V data for users within the FOV 702 ) video data. In some embodiments, session data transfer engine 638 may be configured to select A/V data for use with source client system 602 having a location that is not within FOV 702 based on proximity of the location to FOV 702 The bit rate is 640. For example, session data transfer engine 638 may be configured to select a bit rate 640 for A/V data associated with source user system 602 to vary with the location corresponding to source user system 602 Proximity increases closer to FOV 702.

Referring now to FIG. 12 , depicted is a flow diagram illustrating an exemplary method 1200 for updating session conditions of a client system in accordance with an exemplary implementation of the present invention. Method 1200 may be performed by one or more of the devices, components, or hardware of FIG. 6, such as server 604. As a brief overview, at step 1202, method 1200 may begin. At step 1204, server 604 may receive a request to join the session. At step 1206, server 604 may determine whether the session is ongoing. At step 1208, the server 604 may reconfigure the A/V data for other client systems 602. At step 1210, server 604 may transmit the update to client system 602. At step 1212, the server 604 may transmit the update to the requesting client system 602.

At step 1202, method 1200 may begin. Method 1200 may begin when server 604 generates a new session. For example, server 604 may generate new sessions in response to one or more users requesting new sessions on their respective client systems 602 . Server 604 may establish a session in response to a request from user system 602 . Therefore, the server 604 may begin executing steps 1202 to 1212 of the method 1200 in response to the new session being established. Server 604 may be configured to perform method 1200 for each session established by server 604.

At step 1204, server 604 may receive a request to join the session. Server 604 may receive a request to join a session from user system 602 . User system 602 may initiate a request that includes an identifier or other identifying information for a particular session. For example, a user of client system 602 may be configured to control user device 610 to select a link for an invitation to join a session, enter a code or identifier for a particular session, etc. The client system 602 may be configured to transmit a request including an identifier of the session to the server 604 . The user system 602 may be configured to transmit requests to the server 604 using a cellular connection or link between the user system 602 and the server 604 . Server 604 may be configured to receive requests from user system 602 .

At step 1206, server 604 may determine whether the session is ongoing. In some embodiments, server 604 may be configured to determine whether a session is ongoing by performing a lookup of the session using the session identifier. Server 604 may be configured to determine whether a session is ongoing in response to determining that one or more additional client systems 602 are currently active or otherwise included in the session. Server 604 may be configured to determine that the session is not ongoing in response to determining that the identifier is not a known identifier (eg, in response to performing a lookup using the session identifier) and/or that the user is not active on the session. At step 1206, if the server 604 determines that the session (eg, from the request) is not an ongoing session, the method 1200 may continue to step 1212. At step 1206, if the server 604 determines that the session is ongoing, the method 1200 may continue to step 1208.

At step 1208, the server 604 may reconfigure the A/V data for other client systems 602. In some embodiments, server 604 may reconfigure A/V data received from client systems 602 currently active on the session and/or reconfigure A/V data received from client systems 602 joining the session. V information. The server 604 may reconfigure the A/V data by modifying the bit rate used to encode the A/V information sent from the server 604 to the user system. For example, with the limited display or resolution of the HWD 606, the server 604 can reduce or reduce the bit rate of the three-dimensional (3D) video data as more users participate in the session and therefore have fewer displays. Pixels can be assigned to represent each user or object. Server 604 may adjust session conditions (eg, bit rate used to encode A/V data, bit rate used for user system 602 to encode A/V data for decoding at server 604, etc.).

At step 1210, server 604 may transmit the update to client system 602. The server 604 may transmit updates to the client system 602 currently in the session. Server 604 may transmit an update to identify the new bit rate to be used by client system 602 to encode A/V data to server 604 and/or decode A/V data from server 604. Server 604 may transmit updates to each of client systems 602 to set, establish, or otherwise update session conditions for each of client systems 602 . At step 1212, the server 604 may transmit the update to the requesting client system 602. Similar to step 1210, the server 604 may transmit the update to the user system 602 requesting to join the session. Similarly, where a session is not active or ongoing (eg, at step 1206 ), server 604 may transmit updates to client system 602 to establish a session with a single client system 602 .

Referring now to FIG. 13 , depicted is a flow diagram illustrating an example method 1300 for updating a device's field of view (FOV) in accordance with an example implementation of the present invention. Method 1300 may be performed by one or more of the devices, components, or hardware of FIG. 6, such as client system 602 and/or server 604. Although described as being performed by the user system 602 herein, similar functionality and steps may be performed by the server 604 as described above with reference to FIGS. 6-7. As a brief overview, at step 1302, method 1300 may begin. At step 1304, the client system 602 may transmit a request to join the session. At step 1306, the user system 602 may determine the format and bit rate. At step 1308, the user system 602 may receive the index and identifier. At step 1310, the user system 602 may start a session. At step 1310, the user system 602 may determine whether the FOV has changed. At step 1312, user system 602 may transmit the updated FOV.

At step 1302, method 1300 may begin. Method 1300 may begin when user device 606 is turned on. Method 1300 may begin when user device 606 establishes a connection (eg, a local connection) with other components or elements of user system 602 . Method 1300 may begin when user device 606 opens or otherwise launches an application for a holographic communication session.

At step 1304, the client system 602 may transmit a request to join the session. Step 1304 may be similar to step 1204 described above with reference to FIG. 12 . The user system 602 may transmit a request that includes an identifier or other information related to the session. For example, the user system 602 may transmit the request using an application or resource on the user device 606 and include an identifier of the session in the request (e.g., by selecting a session link or typing the session's code, simply to name a few possibilities). User system 602 may be configured to transmit requests to server 604.

At step 1306, the user system 602 may determine the format and bit rate. The user system 602 can identify various formatting information corresponding to the A/V data captured by the user system 602 . For example, client system 602 may recognize formatted information from an application, from server 604, from local information, the operating system stored on one or more components of client system 602, and so on. The client system 602 can identify the media format of the 3D video, such as the codec type, the resolution of the 2D video including projected 3D video elements, the maximum number of points captured and compressed, etc.

At step 1308, the user system 602 may receive the index and identifier. The client system 602 may receive the index and device identifier of the client system 602 from the server 604 . As described above, server 604 may maintain an index for each of the client systems 602 in the session, where each index includes (among other information) a device identifier corresponding to the user system 602. The server 604 may transfer the indexes of each of the client systems 602 in the session to the new client system 602 for use in constructing or otherwise maintaining the local mapping. Therefore, when the current session is active, the server 604 can assign a unique identifier and a location index to the client system 602 to join the current session, and can share the unique identifier and location index with other client systems 602 . The position index may be or include a real number or an integer indicating the order of the 3D objects in a clockwise or counterclockwise manner. The first and last objects may be assigned a specific type of object identifier or location index (eg, object_id_first, object_id_last). The last object can be assigned the maximum position_index value.

At step 1310, client system 602 may begin a session. The user system 602 can transmit the A/V data of each user of the user system 602 to the server 604, and the server 604 can transmit the A/V data of other users of other user systems 602 Return to the user system 602.

At step 1310, the user system 602 may determine whether the FOV has changed. The user system 602 may determine whether the FOV has changed based on or in accordance with the sensor data of the user system 602 . For example, user system 602 may determine that the FOV has changed in response to motion detected from a motion sensor, such as a gyroscope and/or accelerometer. User system 602 may be configured to determine that the FOV has changed in response to data from HWD 606 of user system 602. In other words, the user system 602 may detect the FOV (including changes thereof) based on data from one or more sensors of the HWD 606 .

At step 1312, user system 602 may transmit the updated FOV. In some embodiments, user system 602 may transmit data corresponding to the updated FOV to server 604 . The client system 602 may transmit data corresponding to the object or user currently within the updated FOV (eg, a virtual representation of the object or user). In this regard, the client system 602 may transmit a list or other identifier of the device with a location index within the FOV. In some embodiments, the user system 602 may transmit data corresponding to the coordinates of the FOV to the server 604 . The server 604 can receive data corresponding to the coordinates of the FOV and can determine the user of the user system 602 corresponding to the location within the FOV assigned to the user system 602 .

The method 1300 may loop between steps 1312 and 1314 until the corresponding user system 602 terminates the session (eg, by exiting the session, disconnecting the HWD 606 and/or other components of the user system 602 , etc.). Method 1300 may end when user system 602 terminates the session.

Referring now to FIG. 14, depicted is a flow diagram illustrating an exemplary method 1400 for managing bit rates for objects in a communication session in accordance with an exemplary implementation of the present invention. Method 1400 may be performed by one or more of the devices, components, or hardware of FIG. 6, such as server 604. As a brief overview, at step 1402, method 1400 may begin. At step 1404, server 604 may receive a request to join the session. At step 1406, server 604 may determine the format and bit rate. At step 1408, server 604 may update the index and local mapping. At step 1410, server 604 may continue the session. At step 1412, server 604 may determine whether a new field of view (FOV) has been received. At step 1414, server 504 may adjust the bit rate. At step 1416, server 604 may determine whether a request to terminate the session has been received.

At step 1402, method 1400 may begin. Similar to step 1202, method 1400 may begin when server 604 generates a new session. For example, server 604 may generate new sessions in response to one or more users requesting new sessions on their respective client systems 602 . Server 604 may establish a session in response to a request from user system 602 . Server 604 may be configured to perform method 1400 for each session established by server 604.

At step 1404, server 604 may receive a request to join the session. Step 1404 may be similar to step 1204 described above. At step 1406, server 604 may determine the format and bit rate. Server 604 may determine the media format and bit rate used to send and receive A/V data (eg, to and from client system 602). The server 604 may determine the media format, including the codec type, the resolution of the 2D video, the maximum number of points captured and compressed at the user system 604, and the number of points used for transmission from the user system 602 to the server 604. The encoding bit rate of the data, the decoding bit rate for the data transmitted from the server 604 to the client system 602, etc.

At step 1408, server 604 may update the index and local mapping. The server 604 may update the index and local map to add the client system 602 that generated the request at step 1404. Server 604 may update the index to include the new device identifier at a location index that is not currently used by other client systems 602 . For example, in the case where user system 602 is added to the current session, server 604 may assign a unique identifier and location index, which server 604 may share or otherwise transmit to Other client systems 602. In the event that a new client system is inserted between two location indices, the server 604 may assign the client system a location index that is the average between the new adjacent location indices (e.g., for the corresponding location index). adjacent user system 604). Additionally, in the case where a new object or client system is inserted between the first object and the last object, the new object or client system 602 may be assigned (eg, by server 604 ) to be larger than the previous last object. The value of the position index, and the user system 602 can be the new last object. Additionally, when the client system 602 leaves the conference session that was the previous first object (or the last object), the server 604 can assign the next (or previous) object as the previous object by updating the device identifier and location index. An object (or the last object).

At step 1410, server 604 may continue the session.

At step 1412, server 604 may determine whether a new field of view (FOV) has been received. In some embodiments, server 604 may determine whether a new FOV has been received from one of user systems 602 . Server 604 may receive data indicating the new FOV in response to execution of steps of method 1300 (eg, step 1314). The server 604 identifies or determines the new FOV based on the data received from the user system. For example, the data may be or include directional data. The direction data may be a direction corresponding to the gaze of the user of client system 602, a set of device identifiers included in the FOV of client system 602, etc. The server 604 may identify the user system 602 corresponding to a location within the FOV of the user system 602 (eg, having an updated FOV).

At step 1414, server 504 may adjust the bit rate. The server 604 can adjust the bit rate used to compress the A/V data transmitted from the server 504 to the user system 602 . Server 604 may adjust the bit rate to compress A based on which user systems 602 correspond to locations within the FOV of user system 602 and which user systems 602 correspond to locations outside the FOV of user system 602 /Vdata. Server 604 may adjust the bit rate to compress A/V data from client systems corresponding to locations within the FOV at a higher bit rate than A/V data from client systems 602 corresponding to locations outside the FOV. 602 A/V data. Server 604 may select a bit rate for compressing A/V data from the user system corresponding to a position outside the FOV to gradually become larger as the position approaches the FOV (e.g., closer to corresponding to a position within the FOV). the bit rate of the client system 602). In this regard, the bit rate used to compress the A/V data may gradually increase as the location corresponding to the source user system 602 of the A/V data approaches the FOV. The server 604 may compress the A/V data based on a selected bit rate for the A/V data (e.g., selected based on the location of the source user system 602 relative to the user's FOV of the respective user system 602 ). for transmission to respective user systems 602.

At step 1416, server 604 may determine whether a request to terminate the session has been received. The method 1400 may thus loop between steps 1412 and 1414 until a request to terminate the session is received from the end user system 602 . Upon receipt of the request at step 1416, the method 1400 may loop back to step 1408 to update the index and local map. Method 1400 may continue through steps 1408 to 1416 until there are no more user systems 602 connected to/joining the session.

Referring now to FIG. 15 , depicted is a flowchart illustrating an exemplary method 1500 for signaling information for holographic communications in accordance with an exemplary implementation of the present invention. Method 1500 may be performed by the devices, components, or hardware described above with reference to FIGS. 1-14. As a brief overview, at step 1502, one or more servers may maintain a session. At step 1504, the server may receive audio/video (A/V) data and zoom data. At step 1506, the server may modify the scale. At step 1508, the server may transmit the modified video data.

At step 1502, one or more servers may maintain the session. In some embodiments, the server may maintain a first session with a first device of a first user and one or more second sessions with one or more second devices of one or more second users. In some specific examples, the method 1500 may continue to establish a plurality of sessions (eg, including a first session and a second session) before step 1502. The server may establish a first session with a first device of a first user and one or more second sessions with one or more second devices of one or more second users. The server may establish the first and second sessions in response to or in response to a request for a conference call between the first device and the second device. The server may establish the first and second sessions in response to the first and second devices joining a session (or conference call) between the respective devices.

At step 1504, the server may receive audio/video (A/V) data and zoom data. In some embodiments, the server may receive audio/video (A/V) data for the first user and zoom data for the first user from the first device via the first session. The first device can capture, detect or otherwise identify A/V data and zoom data. For example, the first device may receive A/V data and zoom data from an imaging system communicatively coupled to the first device. The first device may forward, transmit, send or otherwise provide the A/V data and zoom data to the server. The server may receive A/V data from the first user on a periodic or frequency basis, or may create the A/V data as part of a holographic communication session maintained between the devices. The server can receive scaling data on a second frequency. For example, the server may receive scaling data at various intervals, which may be every 10 seconds, every 30 seconds, every minute, etc. The server may thus receive scaled data separate from the first user's A/V data.

In some specific examples, the server may receive the second A/V data and the second zoom data of the second user. The server may receive second A/V data and second zoom data regarding one or more second sessions maintained with the second device (eg, at step 1502). Similar to the first device, the second device may transmit the second A/V data and the second scaling data received from the HWD communicatively coupled to the second device to the server on the second session. The second device may transmit the second A/V data and the second zoom data at intervals similar to the first A/V data and the zoom data (eg, from the first device). In response to the second device transmitting the A/V and zoom data, the server may receive the second A/V data and the second zoom data.

At step 1506, the server may modify the scale. In some embodiments, the server may modify the scale of the first user (or object) represented in the video data of the A/V data based on the scaling data. The server may modify the proportion of the first user represented in the video data based on scaling data from the corresponding first device. The server may modify the first user's scale based on scaling data from each of the devices (eg, the first device and the second device). The server may modify the first user's ratio to normalize the first user's ratio from the first A/V data relative to the second user's ratio represented in the video data of the second A/V data. . Therefore, in some specific examples, the server may modify the proportion of the second user represented in the second video data of the second A/V data based on the second scaling data (and the first scaling data). In some specific examples, the server may modify the proportion of users represented in the first video data based on the first zoom data and the second zoom data. Similarly, the server may modify the proportion of the second user represented in the second video data based on the first zoom data and the second zoom data. The server may modify the proportion of the first user represented in the first video data to match the proportions of one or more second users represented in the second video data based on the first zoom data and the second zoom data. In this regard, user proportions across A/V data from user devices can be normalized by the server based on scaling data from respective user devices.

At step 1508, the server may transmit the modified video data. In some embodiments, the server may transmit the first user's modified A/V data (e.g., after modifying the ratio of the video data) to one or more second devices (e.g., via the second session) to To appear to one or more second users. The server may transmit the modified A/V data to each of the second user's second devices to display the modified visual data to the second user (e.g., in adjusting the visual data to the first user's after proportion). In some specific examples, the server may also transmit the modified second A/V data of the second user to the first device via the first session for display to the first user of the first device.

Referring now to FIG. 16 , depicted is a flow diagram illustrating an exemplary method 1600 for improving a field of view in accordance with an exemplary implementation of the present invention. Method 1600 may be performed by the devices, components, or hardware described above with reference to FIGS. 1-14. As a brief overview, at step 1602, one or more servers may maintain a partial map. At step 1604, the server can receive audio/video data. At step 1606, the server may receive direction data. At step 1608, the server may transmit the displayed data.

At step 1602, one or more servers may maintain the partial mapping. In some embodiments, the server may maintain the relative position of each of the plurality of users relative to the local map. The server may maintain the relative position of the respective devices assigned to each of the plurality of users relative to the local map. In some embodiments, the server may assign a relative location to each user device for the corresponding user included in the session. The server may maintain a local map including device identifiers and corresponding locations assigned to respective corresponding devices. In some embodiments, the server may maintain a local map as a location index for each user. The location index of the index may include or otherwise indicate a location assigned to the user relative to at least some of the plurality of users. In some embodiments, a location index may include or otherwise indicate a location assigned to a respective user relative to each of the plurality of users. In some embodiments, the location index may include or otherwise indicate the location assigned to the respective user relative to the user's nearest neighbor (eg, the user assigned to the location).

At step 1604, the server can receive audio/video data. In some embodiments, the server may receive audio/video (A/V) data of the first user from the first device. The server may receive the user's A/V data over a session maintained between the server and the first device. The first device may detect, determine, identify, or otherwise generate A/V data in response to the A/V data being captured by an imaging system communicatively coupled to the first device. The first device can transmit A/V data to the server via the session. At step 1606, the server may receive direction data. In some embodiments, the server may receive direction data from the second device indicating the gaze of the second user among the plurality of users. Directional data may include gaze vectors or coordinates, identifiers of user devices associated with one or more locations within the second user's field of view (FOV), etc. The second device may detect, determine, or otherwise identify directional data based on data from one or more sensors of a head wearable device (HWD) communicatively coupled to the second device. The second device can transmit the direction data to the server via the second session. The server may determine or identify one or more devices associated with a location within the second user's FOV. In some embodiments, the server may determine the FOV based on directional data (e.g., by applying the viewing range to the vector or coordinates of the gaze) and determine which devices are assigned to locations within the FOV (e.g., based on local mapping ). In some embodiments, the server may determine which devices are assigned to locations within the FOV based on device identifiers included as direction data from the second device.

In some embodiments, the server may select a bit rate from a plurality of bit rates based on the position of the second user relative to data indicative of the direction of the first user's gaze. For example, the server may select the bit rate of the A/V data for the source user device based on the location assigned to the source user device relative to the FOV of the user device determined to receive the A/V data. The server can select the bit rate used to compress A/V data for transmission to the user device. In this regard, for a given source user device, the server may select different bit rates for compressing data from the source user device based on one or more positions assigned to the source user device relative to the FOV of the sink user device. Device A/V data for transmission to different receiving end user devices.

In some specific examples, the server may identify a subset of the position indexes of a subset of users within the second user's field of view (or FOV) based on the second user's position index and direction data. The server may select a bit rate for displaying data for the subset of users based on the location index associated with the user within the second user's FOV. In some embodiments, the server may select a subset of users assigned to locations within the second user's field of view to have a higher bit rate than is used to appear outside the subset (e.g., within the field of view). The bit rate of data from other users (excluding). In some embodiments, the server may select a bit rate for A/V data for other users outside the field of view based on the proximity of the other user's location index relative to the field of view. For example, one or more servers may select a bit rate for displaying data for individual users of other users to increase as the proximity of each user's location index decreases relative to the view range. increase.

At step 1608, the server may transmit the displayed data. In some embodiments, the server may transmit data corresponding to the presentation of the first A/V data to the second device. The server may transmit the displayed data to the second device for compression at a bit rate selected based on the direction data and the relative position of the second user relative to the local map. The server may transmit the displayed data in response to compressing the displayed data at a selected bit rate. The server may transmit the displayed data to the second device for decompression and display (eg, via a HWD communicatively coupled to the second device). The server may similarly receive direction data indicating the other user's gaze, receive A/V data for the second user, and transmit a presentation corresponding to the second A/V data compressed at the second bit rate. information. In this regard, the server may compress the A/V data at different bit rates for transmission to the user based on whether the A/V data corresponds to a location within the FOV of the user device receiving the compressed A/V data. device.

Now that some illustrative implementations have been described, it is apparent that the foregoing is illustrative rather than restrictive and has been presented by way of example. In particular, although many of the examples presented herein involve specific combinations of method acts or system elements, the acts and the elements may be combined in other ways to achieve the same goals. It is not intended that acts, components, and features discussed in connection with one implementation be excluded from similar roles in other implementations or implementations.

The hardware and data processing components used to implement the various procedures, operations, illustrative logic, logic blocks, modules and circuits described in connection with the specific examples disclosed herein may be general purpose single or multi-chip processors, digital signal processors (digital signal processor; DSP), application special integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware component or designed to perform implement or perform any combination of the functionality described herein. A general purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. In some embodiments, specific procedures and methods may be performed by circuitry specific to a given function. Memory (e.g., memory, memory units, storage devices, etc.) may include a or computer program for storing information and/or computer code used to implement or facilitate the various processes, layers, and modules described in this disclosure. Multiple devices (e.g., RAM, ROM, flash memory, hard drive storage, etc.). Memory may be or include volatile memory or non-volatile memory, and may include database components, object code components, script components, or any other components used to support the various activities and information structures described in this invention. Type of information structure. According to an exemplary embodiment, memory is communicatively coupled to the processor via processing circuitry and includes a computer program for executing (e.g., by the processing circuitry and/or the processor) one or more programs described herein code.

The present invention covers methods, systems and program products on any machine-readable media for implementing various operations. Embodiments of the present invention may be implemented using an existing computer processor, or by a dedicated computer processor incorporated into a suitable system for this or another purpose, or by a hardwired system. Specific examples within the scope of the invention include program products including machine-readable media for carrying or having machine-executable instructions or data structures stored thereon. Such machine-readable media can be any available media that can be accessed by a general purpose or special purpose computer or other machine with a processor. By way of example, such machine-readable media may include RAM, ROM, EPROM, EEPROM or other optical storage, magnetic disk storage or other magnetic storage devices, or may be used to carry or store machine-executable instructions or data Any other medium containing the required program code in a structured form and accessible by a general-purpose or special-purpose computer or other machine having a processor. Combinations of the above are also included within the scope of machine-readable media. Machine-executable instructions include, for example, instructions and data that cause a general-purpose computer, special-purpose computer, or special-purpose processor to perform a certain function or group of functions.

The phraseology and terminology used herein are for the purpose of description and should not be regarded as limiting. The use of "includes," "includes," "has," "contains," "involves," "characterized by," "characterized by," and variations thereof herein is meant to encompass the items listed thereafter, as well as their equivalents. and additional items, and alternative implementations consisting of those items exclusively enumerated thereafter. In one implementation, the systems and methods described herein consist of one, any combination, or combination of more than one of the described elements, acts, or components.

Any reference in the singular to implementations or elements or acts of the systems and methods mentioned herein may also encompass implementations that include a plurality of such elements, and any reference herein in the plural to any implementation or element or act also shall include Implementations including only a single component may be covered. References to the singular or plural form are not intended to limit the disclosed systems or methods, components, acts, or elements thereof to the singular or plural configuration. Reference to any act or element based on any information, action, or element may include implementation in which the action or element is based, at least in part, on any information, action, or element.

Any implementation disclosed herein may be combined with any other implementation or specific example, and references to "implementation," "implementations," "an implementation," or the like are not necessarily mutually exclusive and are intended to indicate the specific features, structures described in connection with the implementation Or features may be included in at least one implementation or specific example. Such terms as used herein do not necessarily all refer to the same implementation. Any implementation may be combined, inclusively or exclusively, with any other implementation in any manner consistent with aspects and implementations disclosed herein.

In the case where a technical feature in the drawings, embodiments or any patent application is followed by a reference symbol, the reference symbol has been included to increase the understandability of the drawings, embodiments and patent application. Accordingly, neither the reference sign nor its absence shall have any limiting effect on the scope of any element of the claimed patent scope.

The systems and methods described herein may be implemented in other specific forms without departing from their characteristics. Unless expressly indicated otherwise, references to "approximately," "approximately," "substantially" or other terms of degree include a change of +/-10% from a given measurement, unit or range. Coupling elements may be electrically, mechanically or physically coupled to each other directly or through intervening elements. The scope of the systems and methods described herein is therefore indicated by the appended claims rather than the foregoing description, and changes within the meaning and scope of equivalents to the claims are covered herein.

The term "coupled" and variations thereof include the joining of two components to each other, either directly or indirectly. Such engagement may be stationary (eg, permanent or fixed) or removable (eg, removable or releasable). Such joining may be achieved by coupling the two components directly to each other; coupling the two components to each other using separate intervening components and any additional intermediate components coupled to each other; or using one of the two components. An intervening member formed integrally into a single unit couples the two members to each other. If "coupled" or a variation thereof is modified by an additional term (e.g., directly coupled), then the general definition of "coupled" provided above is modified by the plain meaning of that additional term (e.g., "directly coupled" "joined" means joining two components without any separate intervening components), resulting in a narrower definition than the general definition of "coupled" provided above. Such coupling may be mechanical, electrical or fluid.

References to "or" are to be understood as inclusive such that any term described using "or" may refer to any of a single, more than one, and all of the described terms. References to "at least one of 'A' and 'B'" may include only 'A', only 'B', and both 'A' and 'B'. Such references used in conjunction with "includes" or other open terms may include additional items.

Modifications to the components and actions described, such as changes in the size, dimensions, structure, shape and specific gravity of the various components, values of parameters, mounting configurations, use of materials, colors, and orientations, may be made without materially departing from the teachings herein. Occurs in light of the teaching content and merits of the subject disclosed. For example, an element shown as integrally formed may be constructed from multiple parts or elements, the position of elements may be reversed or otherwise varied, and the nature or number or position of discrete elements may be altered or varied. Other substitutions, modifications, changes and omissions may be made in the design, operating conditions and configuration of the disclosed elements and operations without departing from the scope of the invention.

References herein to the position of elements (e.g., "top," "bottom," "above," "below") are only used to describe the various elements in the drawings. Orientation. The orientation of various elements may vary according to other exemplary embodiments, and such variations are intended to be covered by this invention.

100:Wireless communication system 110: Base station/wireless communication node/station/source device 112:Wireless interface 114: Processor 116:Memory device 118:Antenna 120: User equipment/wireless communication device/terminal device/receiving device 120A:UE 120B:UE 120N:UE 122:Wireless interface 124: Processor 126:Memory device 128:Antenna 130: Wireless communication link 130A: Wireless communication link 130B: Wireless communication link 130C: Wireless communication link 200: Artificial reality system environment 210:Console 215:Wireless interface 230: Processor 250:HWD 255: Sensor 265:Wireless interface 270: Processor 275: Electronic display 280:Lens 285:Compensator 305: Front rigid body 310:bring 414:Computing Systems 416: Processor 418:Storage device 420:Network interface 422: User input device 424:User output device 500:View 600:System 602: User system 602(1): First user system 602(2): Second user system 602(3): User system 604:Server 606:Head wearable device 608: Imaging system 610: User device 610(1): User device 610(2): Other user devices 610(N): Other user devices 612: Processor 614:Memory 616: Processing engine 618: Session Manager Engine 620: Partial mapping/virtualization mapping 622:Index 622(1):Index 622(N):Index 624: Session data receiving engine 626:A/V data 628:Zoom data 630: Direction information 632: Session data processing engine 634: Scaler 636: Field of view determiner 638: Session data transfer engine 640: bit rate 640(1): bit rate 640 (N): bit rate 700: Graphical representation 702:FOV 704:Vector 800:Microphone 802:Laser launcher 804: Color or image sensor 806: Depth sensor 1102:User 1104:User 1106:User 1200:Method 1202: Steps 1204:Step 1206: Steps 1208: Steps 1210: Steps 1212: Steps 1300:Method 1302: Steps 1304: Steps 1306: Steps 1308: Steps 1310: Steps 1312: Steps 1314: Steps 1400:Method 1402: Steps 1404: Step 1406:Step 1408:Step 1410: Steps 1412: Steps 1414: Steps 1416: Steps 1500:Method 1502:Step 1504:Step 1506: Steps 1508:Step 1600:Method 1602: Steps 1604: Steps 1606: Steps 1608: Steps A: Location B: Location C: Location D: location F: location H: location

The accompanying drawings are not intended to be drawn to scale. Similar reference numbers and names in the various drawings identify similar elements. For clarity, not every component may be labeled in every drawing. [Fig. 1] is a diagram of an exemplary wireless communication system according to an exemplary implementation of the present invention. [Fig. 2] is a diagram of a console and a head wearable display for presenting augmented reality or virtual reality according to an exemplary implementation of the present invention. [Fig. 3] is a diagram of a head wearable display according to an exemplary implementation of the present invention. [FIG. 4] is a block diagram of a computing environment according to an exemplary implementation of the present invention. [Fig. 5] is an exemplary view of a holographic call or communication session via a head wearable device (HWD) according to an exemplary implementation of the present invention. [Fig. 6] is a block diagram of a system for holographic communication according to an exemplary implementation of the present invention. [Fig. 7] is a graphical representation of virtualization mapping according to an exemplary implementation of the present invention. [Fig. 8] An illustration including various views of an imaging system according to an exemplary implementation of the present invention. [FIG. 9] includes example images corresponding to video data that may be captured by an imaging system in accordance with an example implementation of the present invention. [Fig. 10] is a diagram of an end user system communicating with a server according to an exemplary implementation of the present invention. [FIG. 11A] and [FIG. 11B] are examples of frames of video data from three different user systems before and after scaling modifications according to an exemplary implementation of the present invention. [FIG. 12] is a flowchart showing an exemplary method of updating session conditions of a user system according to an exemplary implementation of the present invention. [FIG. 13] is a flowchart illustrating an exemplary method of updating a field of view (FOV) of a device according to an exemplary implementation of the present invention. [FIG. 14] is a flowchart illustrating an exemplary method of managing bit rates for objects in a communication session in accordance with an exemplary implementation of the present invention. [Fig. 15] is a flowchart showing an exemplary method for sending information for holographic communication according to an exemplary implementation of the present invention. [FIG. 16] is a flowchart illustrating an exemplary method of improving a field of view according to an exemplary implementation of the present invention.

1500:Method

1502:Step

1504:Step

1506:Step

1508:Step

Claims (20)

A method that contains: Maintaining, by one or more servers, a first session with a first device of a first user and one or more second sessions with one or more second devices of one or more second users; Receive audio/video (A/V) data for the first user and zoom data for the first user from the first device via the first session by the one or more servers; Modify, by the server or servers, the proportion of the first user represented in the video data of the A/V data based on the scaling data; and The modified A/V data of the first user is transmitted by the one or more servers to the one or more second devices via the one or more second sessions for use by the one or more second devices. Two users appear. The method of claim 1, further comprising establishing, by the one or more servers, the first device based on a request for a three-dimensional (3D) communication session between the first device and the one or more second devices. session and the one or more second sessions. For example, the method of request item 1 further includes: Second A/V data of the one or more second users is received from the one or more second devices via the one or more second sessions by the one or more servers and used for the one or more second users. the second zoom data of the second user; modify, by the one or more servers, the proportion of the one or more second users represented in the second video data of the second A/V data based on the second scaling data; and Transmitting the modified second A/V data of the one or more second users to the first device via the first session by the one or more servers for use by the first device The user appears. The method of claim 3, wherein modifying the proportion of the video data includes modifying, by the one or more servers, the proportion of the first user represented in the video data based on the zoom data and the second zoom data. proportion, and wherein modifying the proportion of the one or more second users represented in the second video data includes modifying, by the one or more servers, the second zoom data based on the scaling data and the second scaling data. The proportion of the one or more second users represented in the video data. The method of claim 4, wherein modifying the ratio includes modifying the ratio of the first user represented in the video data according to the zoom data and the second zoom data to match the ratio represented in the second video data. the proportion of the one or more second users. Such as the method of claim 1, wherein the A/V data includes three-dimensional (3D) video data and spatial audio data. For example, the method of request item 1 further includes: Data indicating the first user's field of view (FOV) is received from the one or more servers. The method of claim 7, wherein the data indicating the (FOV) is determined by the first device based on data received from a third device communicatively coupled to the first device. For example, the method of request item 7 further includes: Maintaining the first user, the third user for the first device and at least one second user for the second device and a third user of the third device through the one or more servers the relative position of each of the two users and the third user with respect to the local map; and The FOV of the first user is determined by the one or more servers based on the direction data and the relative position of the first user with respect to the local map. For example, the method of request item 9 further includes: Transmitting second A/V data of the at least one second user via the first session at a first element rate based on the data indicating the FOV of the first user. the first device; and Based on the data indicating the FOV of the first user, the third A/V data of the third user is transmitted by the one or more servers to the third user via the first session at a second bit rate. A device. One or more servers containing: One or more processors configured to: maintaining a first session with a first device of a first user and a second session with one or more second devices of one or more second users; receiving audio/video (A/V) data for the first user and zoom data for the first user from the first device via the first session; Modify the proportion of the first user's video data represented in the A/V data based on the scaling data; and The modified A/V data of the first user is transmitted to the one or more second devices via the one or more second sessions for presentation to the one or more second users. The one or more servers of claim 11, wherein the one or more processors are configured to respond based on a response to a three-dimensional (3D) communication session between the first device and the one or more second devices. A request is made to establish the first session and the one or more second sessions. As in claim 11, one or more servers, wherein the one or more processors are configured to: Receive second A/V data for the one or more second users from the one or more second devices via the one or more second sessions and second data for the one or more second users. Zoom data; Modify the proportion of the second video data of the second A/V data based on the second scaling data; and Modified second A/V data of the one or more second users is transmitted to the first device via the first session for presentation to the first user of the first device. If requesting one or more servers of item 13, wherein the one or more processors are configured to modify the ratio of the video data based on the scaling data and the second scaling data, and based on the scaling data and the second scaling data The second scaling data is used to modify the ratio of the second video data. Such as requesting one or more servers of item 14, wherein the one or more processors are configured to modify the ratio of the video data based on the scaling data and the second scaling data to match the ratio of the second video data the ratio. If requesting one or more servers in item 11, the A/V data includes three-dimensional (3D) video data and spatial audio data. As in claim 11, one or more servers, wherein the one or more processors are configured to: Data indicating a field of view (FOV) of the first user is received, the data being determined by the first device based on data received from a third device communicatively coupled to the first device. As in claim 17, one or more servers, wherein the one or more processors are configured to: Maintaining the first user, the second user and the third user for the first device and at least one second user for the second device and a third user for the third device The relative position of each of relative to the local map; and The first user's field of view is determined based on the data indicating the FOV and the relative position of the first user relative to the local map. As requested in item 18, one or more servers, wherein the one or more processors are configured to: transmitting second A/V data of the at least one second user to the first device via the first session at a first element rate based on the data indicative of the FOV of the first user; and Transmit third A/V data of the third user to the first device via the first session at a second bit rate based on the data indicative of the FOV of the first user. A device containing: one or more cameras; and One or more processors configured to: determining zoom data for a user of the first device based on image data captured via the one or more cameras of the first device; Receive audio/video (A/V) data from the user; and transmit the A/V data and the scaling data of the user to one or more servers, so that the one or more servers modify the A/V data of the user based on the scaling data, and The user's modified A/V data is transmitted to one or more second devices.

Images