US20220342365A1 - System and method for holographic communication - Google Patents
System and method for holographic communication Download PDFInfo
- Publication number
- US20220342365A1 US20220342365A1 US17/861,652 US202217861652A US2022342365A1 US 20220342365 A1 US20220342365 A1 US 20220342365A1 US 202217861652 A US202217861652 A US 202217861652A US 2022342365 A1 US2022342365 A1 US 2022342365A1
- Authority
- US
- United States
- Prior art keywords
- images
- cameras
- display
- image
- views
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 230000006854 communication Effects 0.000 title claims abstract description 42
- 238000004891 communication Methods 0.000 title claims abstract description 42
- 238000000034 method Methods 0.000 title abstract description 52
- 230000006870 function Effects 0.000 description 32
- 238000012545 processing Methods 0.000 description 16
- 230000003287 optical effect Effects 0.000 description 11
- 230000007613 environmental effect Effects 0.000 description 7
- 239000011159 matrix material Substances 0.000 description 7
- 230000011218 segmentation Effects 0.000 description 7
- 230000008901 benefit Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 238000013528 artificial neural network Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 210000003128 head Anatomy 0.000 description 5
- 238000009877 rendering Methods 0.000 description 4
- 238000000926 separation method Methods 0.000 description 4
- 239000013598 vector Substances 0.000 description 4
- 239000011521 glass Substances 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 2
- 210000000887 face Anatomy 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000003595 spectral effect Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000003416 augmentation Effects 0.000 description 1
- 230000007175 bidirectional communication Effects 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 230000001815 facial effect Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000005305 interferometry Methods 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000007176 multidirectional communication Effects 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 230000005236 sound signal Effects 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 229920002803 thermoplastic polyurethane Polymers 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
- 239000012780 transparent material Substances 0.000 description 1
- 238000004800 variational method Methods 0.000 description 1
- 230000001755 vocal effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/302—Image reproducers for viewing without the aid of special glasses, i.e. using autostereoscopic displays
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
- G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
- G03H1/0005—Adaptation of holography to specific applications
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
- G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
- G03H1/04—Processes or apparatus for producing holograms
- G03H1/08—Synthesising holograms, i.e. holograms synthesized from objects or objects from holograms
- G03H1/0866—Digital holographic imaging, i.e. synthesizing holobjects from holograms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/012—Head tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/243—Image signal generators using stereoscopic image cameras using three or more 2D image sensors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/349—Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking
- H04N13/351—Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking for displaying simultaneously
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/366—Image reproducers using viewer tracking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/388—Volumetric displays, i.e. systems where the image is built up from picture elements distributed through a volume
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B30/00—Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images
- G02B30/20—Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images by providing first and second parallax images to an observer's left and right eyes
- G02B30/26—Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images by providing first and second parallax images to an observer's left and right eyes of the autostereoscopic type
- G02B30/27—Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images by providing first and second parallax images to an observer's left and right eyes of the autostereoscopic type involving lenticular arrays
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
- G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
- G03H1/0005—Adaptation of holography to specific applications
- G03H2001/0088—Adaptation of holography to specific applications for video-holography, i.e. integrating hologram acquisition, transmission and display
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
- G03H2226/00—Electro-optic or electronic components relating to digital holography
- G03H2226/04—Transmission or communication means, e.g. internet protocol
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
- G03H2226/00—Electro-optic or electronic components relating to digital holography
- G03H2226/05—Means for tracking the observer
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
Definitions
- This invention relates generally to the communications field, and more specifically to a new and useful system and method in the communications field.
- Image displays are an integral part of modern life. From televisions to monitors to smartphone and tablet screens, image displays provide users with the ability to view and interact with information presented in a variety of forms.
- FIG. 1 is a schematic representation of an example of a communications system.
- FIG. 2 is a schematic representation of an example of a communications method.
- FIG. 3 is a schematic representation of an example of a communications system.
- FIG. 4 is a schematic representation of an example flow chart of a method.
- FIGS. 5A and 5B are schematic representations of variants of the method.
- FIG. 6 is a schematic representation of an example of compressing different images with different fidelities.
- FIG. 7 is a schematic representation of an example of compressing image(s) using machine learning.
- FIG. 8 is a schematic representation of an example of capturing a subset of views at a higher frame rate than other views, segmenting the higher frame rate views into a foreground and background, using the segmentation to segment the remaining views, and transmitting the foreground from the image capture system to the image display system.
- the method can include determining one or more user parameters S 200 , transmitting the user parameter S 300 , generating views based on the user parameter(s) S 400 , transmitting the generated views S 500 , and displaying the received views S 600 .
- the method can optionally include providing a communicable link S 100 .
- the method can any suitable steps.
- a communication system can include a display subsystem 100 and an image capture subsystem 200 .
- the display subsystem can include a lightfield display 130 , a computing system 160 , one or more sensors 190 , and/or any component.
- the image capture subsystem can include an image acquisition system 230 , a computing system 260 , and/or any component.
- the method and system preferably function to capture and display lightfield images in real or near-real time.
- a lightfield image can be a still image, a frame of a light field video, and/or be any suitable image.
- the method and system can be particularly beneficial for providing visual (and audio) communication between two or more users (e.g., in different locations), and enable unidirectional, bidirectional, and/or multi-directional communication (e.g., between N communication endpoints).
- the method and system can additionally or alternatively be used to generate and display for photography, security, video monitoring, and/or for any application.
- the method and system preferably capture and display the lightfield images at a frame rate of at least 30 frames per second, but can capture and display the lightfield images at any suitable frame rate.
- multiple users can concurrently view images sent by different endpoints on the same lightfield display.
- a first and second user can concurrently communicate with a first and second remote endpoint, respectively, using the same lightfield display.
- variants of the technology can enable real and/or near-real time capture and display of lightfield images (also referred to as holographic images, hologrammatic image, three dimensional image, etc.), for example by decreasing the bandwidth requirements for transferring the lightfield image, decreasing the processing requirements for transferring the lightfield image, and/or otherwise increase the speed of and/or decrease the processing necessary to capture or display lightfield images.
- generating and/or transmitting only views based on a user parameter can greatly decrease (e.g., by a factor of 2 ⁇ , 4 ⁇ , 10 ⁇ , 40 ⁇ , 100 ⁇ , etc.) the processing power necessary to generate, transmit, and display the lightfield images.
- Second variants of the technology can display lightfield images that have few or no visual artifacts.
- the lightfield images can avoid visual artifacts resulting from shadows (e.g., the appearance of holes in a face, loss of information, etc.), from accessories (such as the perception of glasses that are molded to a person's face), from scene features (e.g., facial hair, transparent materials in the scene, etc.), and/or from other sources.
- the technology can minimize the presence of artifacts by representing the lightfield image using a set of images (e.g., as opposed to generating a mesh or other three dimensional representation of the subject of the lightfield image).
- a communication system 10 can include a display subsystem 100 and an image capture subsystem 200 .
- the display subsystem 100 can include a lightfield display 130 , a computing system 160 , one or more sensors 190 , and/or any component.
- the image capture subsystem 200 can include an image acquisition system 230 , a computing system 260 , one or more sensors 290 , and/or any component.
- Each capture and/or viewing endpoint can include one or more communication systems or components thereof.
- each end point can include a display subsystem and an image capture subsystem.
- one endpoint can include a display subsystem and another endpoint can include an image capture subsystem.
- the display subsystem and the image capture subsystem can be associated with (e.g., proximal to, collocated with, in use by, etc.) the same user and/or with different users.
- a display subsystem can be associated with one or more viewers (e.g., receivers) and an image capture subsystem can be associated with one or more viewers (e.g., senders).
- a display subsystem can be associated with one or more viewers (e.g., receivers) and an image capture subsystem can be associated with a scene and/or location (e.g., one that does not necessarily include any sender(s), one that periodically includes sender(s), etc.).
- the display subsystem and the image capture subsystem are generally arranged in different locations (e.g., different rooms, different states, different countries, etc.), but can be collocated (e.g., within the same room, within a predetermined distance of one another, etc.).
- the display subsystem 100 functions to display lightfield images.
- the display subsystem can additionally or alternatively function to determine one or more user parameters, receive lightfield images, and/or perform any function.
- the lightfield image is preferably represented by a set of views (e.g., images of the scene such as showing different perspectives, renderings of the scene, processed images, interpolated images, etc.).
- the lightfield image can be represented using a mesh (e.g., polygonal mesh), polygonal representation, curve representation, voxel representation, represented in a format as disclosed in U.S. patent application Ser. No. 17/226,404 filed 9 Apr. 2021 entitled ‘SYSTEM AND METHOD FOR GENERATING LIGHT FIELD IMAGES’ incorporated in its entirety by this reference, and/or using any suitable representation.
- the set of views can include any suitable number of views between 1-1000 views such as 2, 4, 8, 12, 20, 45, 50, 75, 90, 100, 135, and/or 150 images.
- the lightfield image can include greater than 1000 views and/or any suitable number of views.
- the display subsystem 100 can include a lightfield display 130 , one or more sensors 190 , and a computing system (e.g., a display computing system 160 ), and/or any components.
- a computing system e.g., a display computing system 160
- the lightfield display 130 functions to display the lightfield image.
- the viewers e.g., receivers
- the lightfield display is preferably operable without (e.g., the lightfield images are perceived as three dimensional without using) peripherals (such as head mounted devices, glasses, polarizers, etc.), but can be operable with (e.g., require) peripherals.
- the lightfield display preferably simultaneously displays each view associated with a lightfield image, but can display any views with any suitable timing.
- a lightfield display can include one or more: panel, optical element, parallax generator, optical volume, and/or any component.
- the display can be and/or include components of a ‘superstereoscopic display,’ a ‘retroreflecting display,’ and/or holographic display as disclosed in U.S. patent application Ser. No. 16/374,955 titled ‘SUPERSTEREOSCOPIC DISPLAY WITH ENHANCED OFF-ANGLE SEPARATION’ filed 4 Apr. 2019, U.S. Pat. No. 10,241,344 titled ‘ADVANCED RETROREFLECTING AERIAL DISPLAYS’ filed 28 Nov. 2018, U.S. patent application Ser. No.
- the sensors 190 function to determine (e.g., measure) and/or track one or more user (or viewer, receiver, sender, etc.) parameter.
- the sensors can additionally or alternatively function to determine one or more environmental parameter (e.g., property of the viewed scene; property of an environment proximal a receiver, display subsystem, image capture subsystem, etc.; etc.) and/or perform any suitable function.
- Examples of user parameters can include: user pose, user head pose, user eye pose (e.g., position and/or orientation such as x/y/z/ ⁇ / ⁇ / ⁇ for each of a user's eyes), user eye vector, user gaze direction, kinematics (e.g., user motion), user gestures and/or interactions (such as at an input device), number of users (e.g., viewers, receivers, senders, etc.), intentional sounds (e.g., talking, laughing, verbal communication, etc.), and/or any suitable user parameters.
- Examples of environmental parameters can include ambient lighting, ambient sounds, and/or any suitable environmental parameter.
- the sensor(s) preferably function without using markers (e.g., without requiring special glasses and/or targets), but can use (e.g., require) markers.
- the sensor(s) can include tracking sensors 195 (e.g., optical sensors such as depth cameras, stereo cameras, interferometry, cameras, etc.; inertial trackers; magnetic trackers; wireless trackers such as using a set of anchors; eye and/or gaze trackers such as eye-attached trackers, optical gaze trackers, electrical potential gaze trackers, etc.; audio trackers such as ultrasonic sensors; etc.), audio sensors (e.g., microphones, speakers, transducers, etc.), lighting sensors, and/or any suitable sensors.
- tracking sensors 195 e.g., optical sensors such as depth cameras, stereo cameras, interferometry, cameras, etc.
- inertial trackers e.g., inertial trackers
- magnetic trackers such as using a set of anchors
- eye and/or gaze trackers such as eye-attache
- the sensor(s) can be mounted below the lightfield display (e.g., below optical elements of the light field display, as shown for example in FIG. 3 , etc.), above the lightfield display (e.g., above optical elements of the lightfield display), next to the lightfield display, and/or have any arrangement relative to the display.
- the sensors can be separate from the lightfield display.
- the senor can be the same as an image capture subsystem (e.g., the image acquisition system of the image capture subsystem). However, the sensor can alternatively be different from an image capture subsystem.
- a communication system can include a depth camera (e.g., a time of flight camera, stereo camera, LIDAR integrated camera, a camera with a processor configured to extract a depth of an image, etc.) that can be used for viewer tracking and a camera array (e.g., with or without a depth sensor) for capturing image or views of a subject.
- a depth camera e.g., a time of flight camera, stereo camera, LIDAR integrated camera, a camera with a processor configured to extract a depth of an image, etc.
- a camera array e.g., with or without a depth sensor
- the image capture subsystem 200 functions to acquire one or more images (e.g., views) of a target proximal the image capture subsystem, generate a lightfield image of the scene, transmit the lightfield image, and/or perform any suitable steps.
- the target can be an object (e.g., one or more users such as senders, wildlife, plants, inanimate objects, etc.) and/or the scene within the field-of-view of the image capture subsystem.
- the image capture subsystem can include an image acquisition system 230 , a computing system (e.g., a view computing system 260 ), sensors 290 , and/or any suitable components.
- the image acquisition system 230 functions to capture images of the target.
- the image acquisition system is preferably an array of cameras 235 (e.g., a plurality of cameras), but may additionally or alternatively include a camera (e.g., a camera configured to capture images of the target from a plurality of perspectives such as a moveable camera, a camera with an image sensor (e.g., CMOS detector, CCD detector, etc.) arranged to capture images of the target from different perspectives, etc.), a plenoptic camera, a computing system (e.g., that retrieves the images from a storage module, that stores the images in a buffer of a storage module, that renders the images, that generates the views, etc.), and/or be otherwise arranged.
- a camera e.g., a camera configured to capture images of the target from a plurality of perspectives such as a moveable camera, a camera with an image sensor (e.g., CMOS detector, CCD detector, etc.) arranged to capture
- the camera frame rate is preferably at least 24 frames per second (e.g., 30 fps, 45 fps, 60 fps, 120 fps, 240 fps, etc.), but can be less than 24 fps and/or any suitable frame rate.
- the camera array can be a one-dimensional camera array (e.g., where the image sensor for each camera of the camera array is aligned to a reference axis such as a horizontal reference axis, a vertical reference axis, a straight reference line, a curved reference line, along an edge of a display, etc.), a two dimensional camera array (e.g., where the cameras are arranged on a two-dimensional grid, a rectilinear grid, a curvilinear grid, etc.), a three dimensional camera array (e.g., where the cameras are placed with a predetermined arrangement in three dimensional space; to match a pixel or screen shape such as to define a spherical spatial distribution to match a spherical screen or pixel of a display; etc.), and/or otherwise be arranged.
- a reference axis such as a horizontal reference axis, a vertical reference axis, a straight reference line, a curved reference line, along an edge of
- the number of cameras in the camera array can depend on the user parameters (e.g., the number of users; the distance such as an average distance, optimal viewing distance, focal distance, maximal distance, minimal distance, etc. between the user and the display; etc.), an environmental parameter (e.g., a distance of a target from the image capture system, a number of targets, etc.), views (e.g., the number of views that can be displayed, the number of views that need to be displayed for the viewers or receivers to perceive the scene as three dimensional or with predetermined quality, etc.), a camera parameter (e.g., the camera frame rate, the camera resolution, the camera field of view, a stereo-camera baseline, frame rate, image resolution, etc.), a computing system property (e.g., bandwidth of information transfer, processing bandwidth, etc.), and/or depend on any property.
- the user parameters e.g., the number of users; the distance such as an average distance, optimal viewing distance, focal distance, maximal distance, minimal distance, etc. between the user and the display
- the number of cameras can be any value or range thereof between about 1 and 100 cameras; however, the number of cameras can be greater than 100.
- the camera array can include 8 cameras arranged in a 4 ⁇ 2 array.
- a camera array can include at least 45 cameras (for example, when a display preferably displays up to 45 views).
- the camera array can include 45 cameras arranged along a horizontal extent (e.g., in a given row, 45 columns of cameras, etc.) and 2 or more rows of cameras.
- the camera array can include any suitable number of cameras.
- Each camera of the camera array can be the same (e.g., same image sensor, same image sensor size, same frame rate, same spectral response, etc.) or different (different image sensors, different image sensor sizes, different frame rates, different spectral response, etc.).
- cameras of the camera array can be separated into camera subsets.
- the camera array can include primary, secondary, tertiary, and so on cameras.
- the camera subsets can be fixed and/or varying. Each camera of a camera subset can be the same or different. Subsets of cameras can be determined based on camera properties (e.g., frame rate, image sensor size, camera perspective, etc.), sensor data (e.g., tracking information, etc.), user parameters, environmental parameters, and/or otherwise be determined.
- Subsets of cameras can be arranged in a group (e.g., be physically grouped together), be interspersed with other subsets of cameras (e.g., every fourth camera can belong to a camera subset with the remaining cameras belonging to a second camera subset), be randomly distributed, and/or otherwise be distributed or arranged.
- a subset of the cameras can be used (e.g., used exclusively, used for purposes in addition to acquiring images, etc.) to track a target and/or viewer, segment an image (e.g., a collected image, a image acquired using the same or a different camera or image sensor, etc.), classify an image, and/or otherwise be used.
- a subset of cameras e.g., secondary cameras
- a foreground and background can be determined for images associated with the separate camera.
- a camera from the subset of cameras e.g., a secondary camera
- a separate camera e.g., a primary camera
- a foreground and background can be determined for images associated with the separate camera.
- only the foreground can be transmitted to the receiver.
- the background can additionally or alternatively be transmitted to the receiver.
- a first set of cameras (e.g., primary cameras, view acquisition cameras) of the camera array can acquire images at a first frame rate (e.g., approximately 30 fps) and a second set of cameras (e.g., secondary cameras, segmentation cameras) of the camera array can acquire images at a second frame rate (e.g., 120 fps) that is greater than the first frame rate.
- the second set of cameras can acquire images with a lower resolution than the first set of cameras (e.g., at 1080 pp vs 4 k or 8 k resolution for the first set of cameras).
- the second set of cameras can acquire images with an equal or greater resolution (e.g., as a cost savings by including fewer higher quality cameras) and/or any suitable resolution or frame rate.
- the second set of cameras generally has fewer cameras than the first set of cameras.
- the second set of cameras can have the same number of and/or more cameras than the first set of cameras.
- a first set of cameras can be determined based on tracking information for a viewer using a display.
- the first set of cameras can be cameras that are at an equivalent (e.g., nearest to the equivalent; adjacent to; within a threshold distance such as 1 mm, 2 mm, 5 mm, 1 cm, 2 cm, 5 cm, 10 cm, 20 cm, etc. of; etc.) pose as the viewer eye pose (and/or eye vector) were the viewer in approximately the same position as the camera array and/or for a virtual viewer collocated with (or proximal to) the camera array.
- an equivalent e.g., nearest to the equivalent; adjacent to; within a threshold distance such as 1 mm, 2 mm, 5 mm, 1 cm, 2 cm, 5 cm, 10 cm, 20 cm, etc. of; etc.
- the first set of cameras can be aligned to the position of (e.g., closest to, within a threshold distance or position of, etc.) the viewer(s)' eyes (e.g., as determined via tracking the viewer's eyes). As the viewer moves, the first set of cameras can change.
- the set of primary cameras can move (e.g., be physically moved, change which cameras are included in the set of primary cameras by selecting cameras that are offset from the initial primary cameras by a distance approximately equal to the distance the viewer moves, etc.) in the same direction (e.g., by approximately the same amount) or opposite direction (e.g., depending on whether the camera array is mirrored or not to viewer direction).
- the second set of cameras e.g., auxiliary cameras
- the first set of cameras can function to provide auxiliary views such as in anticipation of or to account for motion of a viewer, to provide additional perspectives or views of the target (e.g., sender), and/or otherwise function.
- the camera array can include a third (or more) set of cameras. For example, cameras adjacent to the second set of cameras, distal the first set of cameras, can form a third set of cameras.
- images associated with the third set of cameras are not transmitted to the receiver (and/or the third set of cameras can not be used to acquire images), but they can be transmitted.
- the set of cameras e.g., primary, secondary, and/or tertiary cameras
- the first and second variants can be used in isolation and/or combination.
- the cameras of the camera array are preferably equally spaced, but can be unevenly spaced.
- the separation distance between adjacent cameras within the camera array is preferably between about 1 inch and 24 inches. However, the separation distance between adjacent cameras can be greater than 24 inches and/or less than 1 inch.
- the camera array can be attached to a lightfield display.
- the camera array can be mounted above the lightfield display (e.g., above optical elements of the light field display, as shown for example in FIG. 3 , etc.), below the lightfield display (e.g., below optical elements of the lightfield display), next to the lightfield display, and/or have any arrangement relative to the display (e.g., a subset of cameras above the display and a subset of cameras below the display, a subset of cameras on a first side of the display and a subset of cameras on a second side of the display, etc.).
- the camera array and display can be on the same or different faces (e.g., surfaces, broad faces, etc.) of a housing (e.g., can be arranged in a manner similar to a selfie or front facing camera, be arranged in a manner similar to a rear facing camera, etc.).
- the camera array can be attached to (e.g., mounted to) a computing system, sensor, viewer, be separated from other components (e.g., separate from a display), and/or have any suitable arrangement.
- the image capture sensors 290 can be the same and/or different from the image capture sensors 190 . However, any suitable sensors can be used.
- An optional display of an image capture subsystem 200 can be the same or different from a display of a display subsystem 100 .
- An optional image acquisition system or camera array of a display subsystem 100 can be the same or different from an image acquisition subsystem system 200 .
- the computing system 300 can function to process views (e.g., compress or decompress views), generate views, select views, control system operation, receive and transmit data (e.g., user parameters, views, environmental parameters, etc.), and/or perform any function.
- the computing system can be local (e.g., to the lightfield display, to the image acquisition system, to the display subsystem, to the image capture subsystem, to a sensor, etc.), remote (e.g., cloud computing, server, etc.), and/or otherwise be distributed.
- the computing system can be distributed between a display computing system 160 and an image capture or view computing system 260 (e.g., associated with an image acquisition system).
- the computing system can include one or more: GPUs, CPUs, TPUs, microprocessors, and/or any other suitable processor.
- the communication module 350 functions to communicate user parameters, images (e.g., still frames, videos, timeseries, etc.), and/or other data between communication systems and/or other endpoints.
- Each system or subsystem can include one or more communication modules.
- the communication modules can be wireless modules (e.g., WiFi, cellular, satellite, etc.), wired modules (e.g., coaxial cable, fiber, etc.), and/or other communication modules.
- the communication e.g., between receiver and sender
- the computing system e.g., an image capture computing system
- the computing system can include a multiplexer, which can function to selectively activate (e.g., send instructions to capture image(s) using) specific cameras of the image acquisition system.
- the method functions to capture lightfield images of a target and display the lightfield images to one or more viewers (e.g., viewers).
- the method can additionally or alternatively function to enable communication (e.g., telecommunication) between users (e.g., teleconferencing), viewing 3D imagery of a target, and/or any suitable function.
- the method is preferably performed in real- or near-real time, but additionally or alternatively be delayed (e.g., a light field image or video can be stored for access or viewing later), and/or be performed with any suitable timing.
- the method can include: establishing a communicable link S 100 , determining one or more user parameters S 200 , transmitting the user parameter(s) S 300 , generating views based on the user parameter(s) S 400 , transmitting the generated views S 500 , and displaying the received views S 600 .
- the method can optionally include processing (e.g., pre and/or post processing) the views, and/or any suitable steps.
- One or more instances of the method can be performed sequentially, simultaneously, contemporaneously, within a predetermined time period, synchronously, asynchronously, and/or with any timing.
- two instances of the method can be performed contemporaneously for each of two users.
- the method is preferably performed using a system as described above, but can be performed by any suitable system.
- the method can be performed between two communications system where each communication system can include a lightfield display, a tracking sensor, camera array (or other image acquisition system), and a computing system.
- the method can be performed using a display subsystem and a separate image capture subsystem.
- Establishing a communicable link S 100 functions to connect a display subsystem to an image capture subsystem enabling communication (e.g., data, such as images and/or audio, transmission between them).
- the communicable link can be a network connection, radio, Bluetooth (e.g., BLE), optical communication, wired connection, and/or any suitable data connection can be used.
- Establishing the communicable link S 100 is preferably performed between a display computing system and an image capture computing system (e.g., communication modules of the respective systems), but can be between any suitable components.
- Establishing the communicable link preferably includes identifying and/or verifying the transmitter and receiver, but can include any suitable steps.
- the communicable link can be automatically established (e.g., upon entering a communication room, upon requesting information from an image capture subsystem, based on user preferences, upon detection of motion proximal an image acquisition subsystem, etc.) and/or manually (e.g., by accepting an invitation, by accepting a request, by entering a communication room, etc.).
- the communicable link is preferably wireless (e.g., optical, transmitting data over the internet, etc.), but can be wired.
- Determining a user parameter S 200 preferably functions to determine (e.g., measure, estimate, calculate, etc.) one or more user parameters associated with a display user or viewer (e.g., a receiver). Determining a user parameter is preferably performed by a sensor (e.g., a tracking sensor, a sensor of a display subsystem, etc.), but can be performed by any component. S 200 can be performed before, during, and/or after S 200 .
- a sensor e.g., a tracking sensor, a sensor of a display subsystem, etc.
- user parameters can include: user pose (e.g., of a viewer's head, nose, face, glabella, hairline, etc.), user head pose, user eye pose (e.g., position and/or orientation such as x/y/z/ ⁇ / ⁇ / ⁇ for each of a user's eyes), user gaze direction, user eye vectors, kinematics (e.g., user motion), user gestures and/or interactions (such as at an input device), number of users, a bandwidth (e.g., processing bandwidth, transmission bandwidth, receipt bandwidth, display bandwidth, etc. associated with a computing system associated with the user), and/or any suitable user parameters.
- user pose e.g., of a viewer's head, nose, face, glabella, hairline, etc.
- user eye pose e.g., position and/or orientation such as x/y/z/ ⁇ / ⁇ / ⁇ for each of a user's eyes
- user gaze direction e.g., user gaze direction
- user eye vectors
- the pose can refer to an absolute position, relative positions (e.g., relative to the display, a virtual user location such as where the display user would be if they were collocated with the image capture subsystem, relative to the image capture subsystem, etc.), coordinates, one or more camera of a camera array (e.g., a primary camera), and/or be provided in any manner.
- the user parameter can include a set of view indices, which function to specify the views requested from the opposing communication system (e.g., which views should be captured and/or transmitted back to the display).
- the view indices are preferably determined based on the current or estimated user parameter (e.g., user pose, eye pose, etc.), but can be determined based on the viewing target's current or estimated position, calibration matrices (e.g., between the viewing display and the capturing camera system), or the otherwise determined.
- the view indices can identify the main views corresponding to the user parameter (e.g., the user's position), buffer views (e.g., surrounding the primary views), and/or other views.
- determining a user parameter can include predicting the value for the user parameter.
- the viewer pose can be estimated: until viewer tracking is redetermined (e.g., sensor can identify a viewer feature or pose), for a predetermined amount of time (e.g., 1 s, 2 s, 4 s, 6 s, 10 s, 20 s, values therebetween, >20 s, ⁇ 1 s, etc.), at a second time after the user parameter has been measured, and/or for any suitable condition.
- the user parameter can be predicted based on sensor data (e.g., IMU measurements associated with a user), based on images of the user (e.g., using optic flow), and/or based on any data.
- the user parameters at the second time can be predicted using a dead reckoning, a Kalman filter, an extended Kalman filter, kinematic equations, a particle filter, and/or be otherwise predicted.
- Transmitting the one or more user parameters functions to provide the image capture subsystem with the user parameters.
- the user parameter(s) are preferably transmitted using the communicable link, but can be transmitted in any manner.
- the user parameters are preferably transferred from the display subsystem to the image capture subsystem, but can be transferred between any systems.
- S 300 is preferably performed after S 200 , but can be performed at the same time as and/or before S 200 (e.g., estimated user parameters can be transmitted before user parameters have been measured or updated).
- Generating one or more views S 400 functions to acquire images of a scene and/or target and/or prepare (e.g., process, select, etc.) images to be displayed at a display.
- the views can correspond to images, processed images, selected images, renderings, and/or other data structures.
- the views are preferably generated by an image capture subsystem, but can be generated by a computing system (e.g., a memory module of a computing system, a view generation module of a computing system, etc.) and/or by any component.
- S 400 is preferably performed after S 300 , but can be performed before and/or during S 300 (e.g., a set of images can be acquired by the image acquisition system before and/or at the same time as the user parameters are transmitted to the image capture subsystem).
- the views can be generated based on the user parameters (e.g., as measured in S 200 , estimated user parameters, measured user parameters, etc.), the communicable link (e.g., a bandwidth, a quality of the connection, a stability of the connection, etc.), a computing system (e.g., a processor bandwidth, a processor speed, a GPU, etc.), and/or any suitable property.
- the views to be generated can be specified by the requesting system (e.g., viewing system, display subsystem), the image capture system, and/or another system.
- generating one or more views can include: acquiring images, identifying one or more targets, tracking one or more targets (e.g., using a sensor, using based on one or more image acquired using a camera, etc.), adjusting an image capture parameter (e.g., which camera(s) are used to acquire images, which images or views are selected, a frame rate, an image resolution, etc.), selecting images, processing images, and/or any suitable step(s).
- generating one or more views can include: acquiring images, identifying one or more targets, tracking one or more targets (e.g., using a sensor, using based on one or more image acquired using a camera, etc.), adjusting an image capture parameter (e.g., which camera(s) are used to acquire images, which images or views are selected, a frame rate, an image resolution, etc.), selecting images, processing images, and/or any suitable step(s).
- Generating views preferably includes acquiring a plurality of images of a target.
- the images are preferably acquired by the image acquisition system (e.g., camera array, primary cameras of the camera array, primary and secondary cameras of the camera array, secondary cameras of the camera array, etc.).
- Each image is preferably acquired substantially simultaneously with other images (e.g., within at most 1 ms 10 ms, 20 ms, 100 ms, 1 s, etc. of each other); however, the images can be acquired with any timing.
- the focus of the image acquisition system can be set automatically (e.g., based on a and/or manually.
- Acquiring the plurality of images can include focusing the image acquisition system (e.g., focusing each camera of a camera array) on a target. Each camera is preferably focused on the same target; however, one or more cameras can focus on different targets.
- acquiring the plurality of images can include capturing the images with a predetermined focal point; optionally, determining a target within the images; and adjusting the focal plane of the images (e.g., such that the target is in focus).
- Determining the target can include segmenting the image (e.g., semantic segmentation, foreground/background segmentation, etc.), classifying the image segments, identifying a target of interest based on the classification (e.g., manually such as the display user identifying the target; automatically such as using a neural network, based on a set of rules, etc.), and tracking the target (e.g., across frames such as using optic flow; between images such as using disparity maps; etc.), and/or any suitable steps.
- segmenting the image e.g., semantic segmentation, foreground/background segmentation, etc.
- classifying the image segments e.g., identifying a target of interest based on the classification
- identifying a target of interest e.g., manually such as the display user identifying the target; automatically such as using a neural network, based on a set of rules, etc.
- tracking the target e.g., across frames such as using optic flow; between images such as using disparity maps; etc.
- Adjusting the focal plane of the images can include deconvolving the image, determining a depth map associated with the image, applying a blurring filter to the images (e.g., to adjust a lens blur, based on the depth to a target and/or pixel of the image, etc.), generating a 3D reconstruction (e.g., based on the depth map) generating a view using the 3D reconstruction (e.g., by capturing an image of the 3D reconstruction using virtual cameras), and/or any suitable steps.
- a focus of the images can otherwise be determined or set.
- acquiring a plurality of images can include acquiring a plurality of images using a subset of the camera array.
- the subset of cameras can include any number of cameras between 1 and the total number of cameras in the camera array such as 2, 3, 4, 8, 10, 15, 20, 25, 30, 40, 45, and/or any number of cameras.
- the number of cameras in the subset of cameras matches the number of distinct eye poses of display users.
- the number of cameras in the subset can include buffer cameras (e.g., two, four, six, etc. buffer cameras per camera matching the distinct eye poses of the display users) adjacent to (e.g., surrounding) the primary cameras.
- the subset of the camera array is preferably determined based on the display user parameters, but the subset of the camera array can be predetermined (e.g., based on a set-up procedure), based on the target, and/or be otherwise determined.
- the subset of cameras can be chosen to be the cameras that are closest to the eye pose of a virtual user, where the eye pose of the virtual user is determined based on the eye pose measured for the display user.
- the subset of cameras can include the pair of cameras that are closest to the eye pose of the virtual user.
- the subset of cameras can correspond to the view indices received from the (viewing) display subsystem.
- the cameras in the subset of cameras can correspond to the three or four cameras that are closest each eye pose of a virtual user.
- the subset of cameras can include cameras adjacent to the cameras determined based on the eye pose.
- the cameras in the subset of cameras can be determined based on a mapping (e.g., equation, look-up table, etc.) from the viewer's eye position (e.g., tracked position) to cameras of the camera array.
- the subset of cameras can include any suitable cameras.
- acquiring a plurality of images can include acquiring a plurality of images using every camera of the camera array.
- a subset of the images is preferably selected from the plurality of images.
- the subset of images is preferably selected based on the display user parameters (e.g., user pose, view indices), but can be selected based on the target and/or any suitable property.
- Selected images can include: images associated with primary and/or second cameras, images associated with a subset of cameras, a In an illustrative example as shown in FIG. 5B , the subset of images can be selected based on the eye pose for the display user.
- acquiring a plurality of images can include acquiring a plurality of images using a camera (or other image recording device), where the camera is translated to a plurality of positions, where the plurality of positions are determined based on (e.g., match) a display user parameter (e.g., eye pose).
- a camera or other image recording device
- the plurality of positions are determined based on (e.g., match) a display user parameter (e.g., eye pose).
- Processing the images preferably functions to reduce a size of an image, but can additionally or alternatively function to reduce or remove one or more optical artifact, augment the images (e.g., using digital content), and/or otherwise function.
- Processing the images can include: cropping images, aligning images, rectifying images, transforming images (e.g., using an affine transformation), filtering images, compressing images, applying a shader, reprojecting the images (e.g., interpolating between images, determining an image intermediate between two or more images, etc.), correcting images (e.g., correcting each image based on a camera pose for the respective camera), generating a quilt image (e.g., arranging the set of images in a predetermined order or manner), and/or any suitable image manipulations.
- cropping images aligning images, rectifying images, transforming images (e.g., using an affine transformation), filtering images, compressing images, applying a shader, reprojecting the images (e.g., interpolating
- a first set of images can be compressed using a first codec and/or set of codex parameters and a second set of images can be compressed using a second codec and/or set of codec parameters.
- the first set of images can be associated with primary cameras and can be compressed with a higher quality and/or fidelity (e.g., based on one or more of: objective quality, subjective quality, sharpness, noise, tone reproduction, dynamic range, contrast color accuracy, distortion, vignetting, exposure accuracy, lateral chromatic aberration, lens flare, color moiré, artifacts, etc.) relative to the second set of images (e.g., associated with secondary cameras).
- the first set of images can be compressed with a lower quality and/or fidelity relative to the second set of images
- the first and second set of images can be compressed in different file wrappers, and/or the image can otherwise be compressed.
- Processing the images preferably does not include rendering the images to generate a three dimensional representation of the target and/or scene.
- the images can be processed to generate a three dimensional representation of the target and/or scene.
- processing the images can include warping a projection matrix which functions to accommodate (e.g., correct for) changes in a user parameter (e.g., head orientation, eye orientation, distance from the display, position relative to the display, etc.).
- the warped projection matrix e.g., perspective correct information, perspective projection
- the warped projection matrix is preferably used to determine the views that need to be projected onto the display (e.g., a screen thereof) based on the user's eye position(s), but can be used to determine the images (e.g., additional images) to be captured by the camera array, camera indices (e.g., cameras in addition to the cameras capturing the primary images, secondary cameras, etc.) for additional image capture, and/or otherwise used.
- the warping can be determined based on measurements from the tracking sensor and/or any suitable component.
- the warping can be relative to an on-axis viewer perspective, an initial viewer perspective (e.g., initial viewer pose relative to the display, initial viewer eye pose, etc.), and/or relative to any suitable perspective.
- the projection matrix e.g., warped projection matrix
- the projection matrix can be applied to captured images, interpolated images, perspective corrected images, 3D representation (e.g., mesh), and/or any suitable image or scene representation.
- a plurality of images e.g., for each eye of the viewer
- the projection matrix is used to determine how the images should be processed to determine the projected views.
- the projection matrix is used to determine which pixels, corresponding to a mesh of the scene (e.g., 2D mesh, 3D mesh, polygonal mesh, etc.), should be rendered.
- the projection matrix can be otherwise used.
- S 400 can include segmenting one or more image or view. Segmenting the image(s) can be beneficial as segments or portions of the images can be transmitted (e.g., rather than full images), which can decrease a bandwidth required and/or enable higher resolution images to be transferred.
- the image(s) can be segmented using artificial intelligence (e.g., machine learning; a neural network such as generative adversarial networks, convolutional neural networks, deep neural networks, etc.; trainable methods; etc.), an image classification and/or segmentation algorithm, using computer vision (e.g., thresholding, clustering, edge detection, histogram methods, region-growing methods, partial differential equations, compression methods, variational methods, graph partitioning methods, watershed transformations, multi-scale segmentation, etc.) and/or using any suitable method(s).
- the image can be segmented into key points (e.g., as shown for example in FIG. 7 ), into a foreground and background (e.g., as shown for example in in FIG. 8 ), into one or more subjects and not subjects, into a target and other information, and/or otherwise be segmented.
- one or more images can be segmented into key points.
- the images can be segmented into key points, for instance as shown in FIG. 7 , using a neural network (e.g., a GAN, segmentation network, etc.).
- the key points can be features, pixels, superpixels, coordinates, and/or other distinctive points within the image(s).
- the key points can be transmitted to the display subsystem.
- the key points can be used (e.g., at the display subsystem) to process (e.g., transform, reproject, etc.) a reference image (e.g., based on a change in position of the key points between the reference image and the currently measured key points) to accommodate or display changes or updates to a subject.
- the key points can otherwise be used.
- a first subset of the images can be segmented into a foreground (e.g., the subject such as the sender) and background (e.g., environment or context surrounding the subject).
- the secondary images are preferably acquired at a higher frame rate than a second subset of images (e.g., primary images such as images that are intended to be transmitted), but can be acquired at the same or a lower frame rate than the primary images.
- the secondary images can be segmented using machine learning (e.g., a neural network as shown for example in FIG. 8 ), based on motion (e.g., a difference between the frames), and/or otherwise be segmented.
- the segments of the secondary images can be used to segment the primary images (e.g., based on a known pose or relationship between the secondary cameras and the primary cameras). However, the images can otherwise be segmented or used.
- S 400 can include acquiring auxiliary dataset(s).
- the auxiliary datasets can be acquired simultaneously with, contemporaneously with, before, after, and/or with any suitable timing relative to the images.
- Examples of auxiliary datasets can include: audio datasets (e.g., subject audio, subject speaking, etc.), depth (e.g., distances between the image acquisition system and the subject(s)), environmental parameters, and/or any suitable auxiliary dataset(s).
- the auxiliary dataset(s) can be acquired using a sensor, a computing system (e.g., configured to generate an auxiliary dataset), and/or be acquired using any suitable component.
- the auxiliary datasets can be processed, for example, to change a perceived audio by a receiver (such as to cause the audio perceived by the receiver to be perceived as coming from a predetermined location or distance, to modify a frequency spectrum of the audio, to synchronize the audio with the light field image or frame, etc.), to modify a focus of the lightfield image (e.g., based on depth data), and/or otherwise processing the auxiliary dataset(s).
- a perceived audio by a receiver such as to cause the audio perceived by the receiver to be perceived as coming from a predetermined location or distance, to modify a frequency spectrum of the audio, to synchronize the audio with the light field image or frame, etc.
- a focus of the lightfield image e.g., based on depth data
- Transmitting the one or more views S 500 functions to provide the views (e.g., processed images, selected images, raw images, difference views, key points, primary views, secondary views, etc.) to the display subsystem.
- S 500 is preferably performed after S 400 , but can be performed at the same time as and/or before S 400 (e.g., where the display subsystem can generate and/or select views to be displayed).
- the views are preferably transmitted using the communicable connection, but can be transmitted in any manner.
- views that will be displayed to or perceived by a user are transmitted.
- the set of transmitted (or generated) views can exclude images acquired by cameras of the camera array associated with a camera pose that differs from the eye poses by at least a threshold amount (e.g., a threshold distance, a threshold number of cameras between an excluded camera and a primary or secondary camera, etc.).
- a threshold amount e.g., a threshold distance, a threshold number of cameras between an excluded camera and a primary or secondary camera, etc.
- all or any subset of views can be transmitted.
- the views are preferably transmitted from the image capture subsystem (e.g., a communication module of the image capture subsystem) to the display subsystem (e.g., a communication module of the display subsystem), but can be transmitted between any systems.
- transmitting the one or more views can include transmitting a reference image (e.g., reference light field image) of the subject and/or scene.
- the reference image can be transmitted once (e.g., at a first time point such as when the method is first initiated, during S 100 , etc.), with each set of key points, randomly, when a threshold change in the reference image occurs (e.g., number of subjects changes, reference image is no longer representative of the subject, etc.), at predetermined times or frequencies, and/or with any suitable timing or frequency.
- S 500 can include transmitting auxiliary dataset(s).
- the auxiliary dataset(s) can be transmitted in the same container as the views (e.g., as metadata, embedded in the views, etc.) and/or in a separate container.
- Displaying the received views S 600 functions to display the received views to the display user. Displaying the received views is preferably performed by the display subsystem (e.g., a lightfield display) requesting the views (e.g., tracking the first user), but can be performed by any system.
- the views can be directed to a user, displayed within a viewcone of the display, and/or otherwise be displayed. In specific examples, the views can be displayed in a similar manner as the display operation as disclosed in U.S. patent application Ser. No. 16/374,955 titled ‘SUPERSTEREOSCOPIC DISPLAY WITH ENHANCED OFF-ANGLE SEPARATION’ filed 4 Apr. 2019, U.S. Pat. No.
- S 600 can include processing the views (e.g., using a computing system of the display subsystem). Processing the views to be displayed can include formatting the views to be displayed (e.g., lenticularizing or arranging the views as disclosed in U.S. patent application Ser. No. 17/226,404 filed 9 Apr.
- S 600 can include presenting auxiliary data.
- audio datasets can be presented to the users (e.g., receivers).
- the audio datasets can be presented using a single channel (e.g., mono) and/or a plurality of channels (e.g., stereo).
- the audio data can be modified or processed.
- the audio dataset can be presented as a three dimensional or dynamic audio signal that is perceivable as changing a direction and/or other quality (e.g., frequency, loudness, etc.) as a subject moves.
- the audio can be presented as a static signal (e.g., content is presented as it is recorded by the image capture subsystem or sensor thereof) and/or as any suitable signal.
- the method is described between two systems (e.g., two communication systems, a display subsystem and an image capture subsystem, etc.).
- the steps and/or substeps of the method may be applied (e.g., simultaneously, contemporaneously, in series, in parallel, etc.) for any number of users and/or systems (e.g., display subsystems, image capture subsystems, communication systems, etc.).
- the total number of systems (and/or users) can be bounded by a threshold (for example, based on a computing system of associated with a display, on a computing system associated with an image capture subsystem, based on a bandwidth of a computing system, etc.; a threshold number of users that can be tracked simultaneously; a threshold number of users based on the number of views that can be displayed; etc.), but the total number of displays (and/or users) can be unbounded.
- a method for lightfield communication can include: establishing a communicable link between a first display subsystem associated with a first user and a second image capture subsystem associated with a second user; determining a user parameter (e.g., user pose, user eye pose, gaze tracking, selected view indices, etc.) associated with the first user; transmitting the user parameter to the second image capture subsystem; acquiring images using the second image capture subsystem based on the user parameter, where at least a subset of the images that are acquired correspond to the user parameter; optionally, processing the images (e.g., cropping the images, aligning the images, rendering the images, reprojecting the images, etc.); transmitting the images to the first display subsystem; and displaying the images at the first display subsystem.
- a user parameter e.g., user pose, user eye pose, gaze tracking, selected view indices, etc.
- the process can be repeated for a second user using a second display subsystem (connected to the second image capture subsystem) and a first image capture subsystem (connected to the first display subsystem) to enable bidirectional communication.
- the user parameter can be automatically determined by the first display subsystem or first image capture subsystem (e.g., based on a user pose relative to the respective subsystem).
- the images corresponding to the user parameter can be determined by the first display subsystem or by the second display subsystem (e.g., by mapping the user parameter to a set of cameras).
- the second communication system can capture images at each camera of the second communication system, select a subset of the images based on the first user parameter, and transmit the subset of the images.
- an eye tracking module can be used to track the viewer(s) (e.g., eye position, gaze, etc. thereof).
- the eye tracking module can include a processor (e.g., separate from the display or image acquisition processor, integrated with the display or image acquisition processor, etc.) configured to determine at least one of an eye position or an eye vector of the one or more viewers based on one or more tracking images.
- the processor can additionally or alternatively estimate and/or predict a location of the viewer(s) (e.g., by performing dead reckoning relative to previous tracked positions) and/or otherwise function.
- the tracking image(s) can be acquired using the camera array, a dedicated tracking sensor (e.g., a camera that is separate from the camera array), a depth camera, and/or any suitable tracking sensor.
- Embodiments of the system and/or method can include every combination and permutation of the various system components and the various method processes, wherein one or more instances of the method and/or processes described herein can be performed asynchronously (e.g., sequentially), concurrently (e.g., in parallel), or in any other suitable order by and/or using one or more instances of the systems, elements, and/or entities described herein.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- General Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Human Computer Interaction (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Ophthalmology & Optometry (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Optics & Photonics (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
- Studio Devices (AREA)
Abstract
A holographic communication system and method can include: determining a user parameter for one or more users, generating a set of views based on the user parameter, and displaying the set of generated views.
Description
- This application is a continuation of U.S. application Ser. No. 17/356,129, filed 23 Jun. 2021, which claims the benefit of U.S. Provisional Application No. 63/042,862, filed 23 Jun. 2020, each of which is incorporated in its entirety by this reference.
- This invention relates generally to the communications field, and more specifically to a new and useful system and method in the communications field.
- Image displays are an integral part of modern life. From televisions to monitors to smartphone and tablet screens, image displays provide users with the ability to view and interact with information presented in a variety of forms.
- The advent of three-dimensional displays has enabled users to experience images with higher realism than would be possible with their two-dimensional counterparts. However, the image sizes can be prohibitively large making real or near-real time 3D image display challenging.
- Thus, there is a need in the communications field to create a new and useful system and method. This invention provides such new and useful systems and methods.
-
FIG. 1 is a schematic representation of an example of a communications system. -
FIG. 2 is a schematic representation of an example of a communications method. -
FIG. 3 is a schematic representation of an example of a communications system. -
FIG. 4 is a schematic representation of an example flow chart of a method. -
FIGS. 5A and 5B are schematic representations of variants of the method. -
FIG. 6 is a schematic representation of an example of compressing different images with different fidelities. -
FIG. 7 is a schematic representation of an example of compressing image(s) using machine learning. -
FIG. 8 is a schematic representation of an example of capturing a subset of views at a higher frame rate than other views, segmenting the higher frame rate views into a foreground and background, using the segmentation to segment the remaining views, and transmitting the foreground from the image capture system to the image display system. - The following description of the preferred embodiments of the invention is not intended to limit the invention to these preferred embodiments, but rather to enable any person skilled in the art to make and use this invention.
- As shown in
FIG. 2 , the method can include determining one or more user parameters S200, transmitting the user parameter S300, generating views based on the user parameter(s) S400, transmitting the generated views S500, and displaying the received views S600. The method can optionally include providing a communicable link S100. However, the method can any suitable steps. - As shown in
FIG. 1 , a communication system can include adisplay subsystem 100 and animage capture subsystem 200. The display subsystem can include alightfield display 130, acomputing system 160, one ormore sensors 190, and/or any component. The image capture subsystem can include animage acquisition system 230, acomputing system 260, and/or any component. - The method and system preferably function to capture and display lightfield images in real or near-real time. A lightfield image can be a still image, a frame of a light field video, and/or be any suitable image. The method and system can be particularly beneficial for providing visual (and audio) communication between two or more users (e.g., in different locations), and enable unidirectional, bidirectional, and/or multi-directional communication (e.g., between N communication endpoints). However, the method and system can additionally or alternatively be used to generate and display for photography, security, video monitoring, and/or for any application. The method and system preferably capture and display the lightfield images at a frame rate of at least 30 frames per second, but can capture and display the lightfield images at any suitable frame rate.
- Furthermore, multiple users can concurrently view images sent by different endpoints on the same lightfield display. In examples, a first and second user can concurrently communicate with a first and second remote endpoint, respectively, using the same lightfield display.
- Variations of the technology can confer several benefits and/or advantages.
- First, variants of the technology can enable real and/or near-real time capture and display of lightfield images (also referred to as holographic images, hologrammatic image, three dimensional image, etc.), for example by decreasing the bandwidth requirements for transferring the lightfield image, decreasing the processing requirements for transferring the lightfield image, and/or otherwise increase the speed of and/or decrease the processing necessary to capture or display lightfield images. In a specific example, generating and/or transmitting only views based on a user parameter can greatly decrease (e.g., by a factor of 2×, 4×, 10×, 40×, 100×, etc.) the processing power necessary to generate, transmit, and display the lightfield images.
- Second variants of the technology can display lightfield images that have few or no visual artifacts. For example, the lightfield images can avoid visual artifacts resulting from shadows (e.g., the appearance of holes in a face, loss of information, etc.), from accessories (such as the perception of glasses that are molded to a person's face), from scene features (e.g., facial hair, transparent materials in the scene, etc.), and/or from other sources. In specific examples, the technology can minimize the presence of artifacts by representing the lightfield image using a set of images (e.g., as opposed to generating a mesh or other three dimensional representation of the subject of the lightfield image).
- However, variants of the technology can confer any other suitable benefits and/or advantages.
- As shown in
FIG. 1 , acommunication system 10 can include adisplay subsystem 100 and animage capture subsystem 200. Thedisplay subsystem 100 can include alightfield display 130, acomputing system 160, one ormore sensors 190, and/or any component. Theimage capture subsystem 200 can include animage acquisition system 230, acomputing system 260, one ormore sensors 290, and/or any component. Each capture and/or viewing endpoint can include one or more communication systems or components thereof. For example, to enable two directional communication, each end point can include a display subsystem and an image capture subsystem. In a second example, such as to provide unidirectional communication (e.g., for security or monitoring), one endpoint can include a display subsystem and another endpoint can include an image capture subsystem. - The display subsystem and the image capture subsystem can be associated with (e.g., proximal to, collocated with, in use by, etc.) the same user and/or with different users. In an illustrative example, a display subsystem can be associated with one or more viewers (e.g., receivers) and an image capture subsystem can be associated with one or more viewers (e.g., senders). In a second illustrative example, a display subsystem can be associated with one or more viewers (e.g., receivers) and an image capture subsystem can be associated with a scene and/or location (e.g., one that does not necessarily include any sender(s), one that periodically includes sender(s), etc.). The display subsystem and the image capture subsystem are generally arranged in different locations (e.g., different rooms, different states, different countries, etc.), but can be collocated (e.g., within the same room, within a predetermined distance of one another, etc.).
- The
display subsystem 100 functions to display lightfield images. The display subsystem can additionally or alternatively function to determine one or more user parameters, receive lightfield images, and/or perform any function. The lightfield image is preferably represented by a set of views (e.g., images of the scene such as showing different perspectives, renderings of the scene, processed images, interpolated images, etc.). However, the lightfield image can be represented using a mesh (e.g., polygonal mesh), polygonal representation, curve representation, voxel representation, represented in a format as disclosed in U.S. patent application Ser. No. 17/226,404 filed 9 Apr. 2021 entitled ‘SYSTEM AND METHOD FOR GENERATING LIGHT FIELD IMAGES’ incorporated in its entirety by this reference, and/or using any suitable representation. - The set of views can include any suitable number of views between 1-1000 views such as 2, 4, 8, 12, 20, 45, 50, 75, 90, 100, 135, and/or 150 images. However, the lightfield image can include greater than 1000 views and/or any suitable number of views.
- The
display subsystem 100 can include alightfield display 130, one ormore sensors 190, and a computing system (e.g., a display computing system 160), and/or any components. - The
lightfield display 130 functions to display the lightfield image. The viewers (e.g., receivers) preferably perceive the lightfield image (e.g., as presented by the display) as a holographic or three dimensional image. However, one or more viewers can additionally or alternatively perceive the lightfield image as two dimensional, 2.5 dimensional, and/or as having any suitable dimensionality. The lightfield display is preferably operable without (e.g., the lightfield images are perceived as three dimensional without using) peripherals (such as head mounted devices, glasses, polarizers, etc.), but can be operable with (e.g., require) peripherals. The lightfield display preferably simultaneously displays each view associated with a lightfield image, but can display any views with any suitable timing. In a specific example, a lightfield display can include one or more: panel, optical element, parallax generator, optical volume, and/or any component. In a second specific example, the display can be and/or include components of a ‘superstereoscopic display,’ a ‘retroreflecting display,’ and/or holographic display as disclosed in U.S. patent application Ser. No. 16/374,955 titled ‘SUPERSTEREOSCOPIC DISPLAY WITH ENHANCED OFF-ANGLE SEPARATION’ filed 4 Apr. 2019, U.S. Pat. No. 10,241,344 titled ‘ADVANCED RETROREFLECTING AERIAL DISPLAYS’ filed 28 Nov. 2018, U.S. patent application Ser. No. 17/326,857 filed 21 May 2021 titled ‘SYSTEM AND METHOD FOR HOLOGRAPHIC IMAGE DISPLAY,’ and/or U.S. patent application Ser. No. 17/332,479 filed 27 May 2021 titled ‘SYSTEM AND METHOD FOR HOLOGRAPHIC DISPLAYS,’ each of which is incorporated in its entirety by this reference. - The
sensors 190 function to determine (e.g., measure) and/or track one or more user (or viewer, receiver, sender, etc.) parameter. The sensors can additionally or alternatively function to determine one or more environmental parameter (e.g., property of the viewed scene; property of an environment proximal a receiver, display subsystem, image capture subsystem, etc.; etc.) and/or perform any suitable function. Examples of user parameters can include: user pose, user head pose, user eye pose (e.g., position and/or orientation such as x/y/z/θ/φ/ψ for each of a user's eyes), user eye vector, user gaze direction, kinematics (e.g., user motion), user gestures and/or interactions (such as at an input device), number of users (e.g., viewers, receivers, senders, etc.), intentional sounds (e.g., talking, laughing, verbal communication, etc.), and/or any suitable user parameters. Examples of environmental parameters can include ambient lighting, ambient sounds, and/or any suitable environmental parameter. The sensor(s) preferably function without using markers (e.g., without requiring special glasses and/or targets), but can use (e.g., require) markers. In variants, the sensor(s) can include tracking sensors 195 (e.g., optical sensors such as depth cameras, stereo cameras, interferometry, cameras, etc.; inertial trackers; magnetic trackers; wireless trackers such as using a set of anchors; eye and/or gaze trackers such as eye-attached trackers, optical gaze trackers, electrical potential gaze trackers, etc.; audio trackers such as ultrasonic sensors; etc.), audio sensors (e.g., microphones, speakers, transducers, etc.), lighting sensors, and/or any suitable sensors. - The sensor(s) can be mounted below the lightfield display (e.g., below optical elements of the light field display, as shown for example in
FIG. 3 , etc.), above the lightfield display (e.g., above optical elements of the lightfield display), next to the lightfield display, and/or have any arrangement relative to the display. However, the sensors can be separate from the lightfield display. - In some embodiments, the sensor can be the same as an image capture subsystem (e.g., the image acquisition system of the image capture subsystem). However, the sensor can alternatively be different from an image capture subsystem. For instance, a communication system can include a depth camera (e.g., a time of flight camera, stereo camera, LIDAR integrated camera, a camera with a processor configured to extract a depth of an image, etc.) that can be used for viewer tracking and a camera array (e.g., with or without a depth sensor) for capturing image or views of a subject.
- The
image capture subsystem 200 functions to acquire one or more images (e.g., views) of a target proximal the image capture subsystem, generate a lightfield image of the scene, transmit the lightfield image, and/or perform any suitable steps. The target can be an object (e.g., one or more users such as senders, wildlife, plants, inanimate objects, etc.) and/or the scene within the field-of-view of the image capture subsystem. The image capture subsystem can include animage acquisition system 230, a computing system (e.g., a view computing system 260),sensors 290, and/or any suitable components. - The
image acquisition system 230 functions to capture images of the target. The image acquisition system is preferably an array of cameras 235 (e.g., a plurality of cameras), but may additionally or alternatively include a camera (e.g., a camera configured to capture images of the target from a plurality of perspectives such as a moveable camera, a camera with an image sensor (e.g., CMOS detector, CCD detector, etc.) arranged to capture images of the target from different perspectives, etc.), a plenoptic camera, a computing system (e.g., that retrieves the images from a storage module, that stores the images in a buffer of a storage module, that renders the images, that generates the views, etc.), and/or be otherwise arranged. The camera frame rate is preferably at least 24 frames per second (e.g., 30 fps, 45 fps, 60 fps, 120 fps, 240 fps, etc.), but can be less than 24 fps and/or any suitable frame rate. - The camera array can be a one-dimensional camera array (e.g., where the image sensor for each camera of the camera array is aligned to a reference axis such as a horizontal reference axis, a vertical reference axis, a straight reference line, a curved reference line, along an edge of a display, etc.), a two dimensional camera array (e.g., where the cameras are arranged on a two-dimensional grid, a rectilinear grid, a curvilinear grid, etc.), a three dimensional camera array (e.g., where the cameras are placed with a predetermined arrangement in three dimensional space; to match a pixel or screen shape such as to define a spherical spatial distribution to match a spherical screen or pixel of a display; etc.), and/or otherwise be arranged. The number of cameras in the camera array can depend on the user parameters (e.g., the number of users; the distance such as an average distance, optimal viewing distance, focal distance, maximal distance, minimal distance, etc. between the user and the display; etc.), an environmental parameter (e.g., a distance of a target from the image capture system, a number of targets, etc.), views (e.g., the number of views that can be displayed, the number of views that need to be displayed for the viewers or receivers to perceive the scene as three dimensional or with predetermined quality, etc.), a camera parameter (e.g., the camera frame rate, the camera resolution, the camera field of view, a stereo-camera baseline, frame rate, image resolution, etc.), a computing system property (e.g., bandwidth of information transfer, processing bandwidth, etc.), and/or depend on any property. The number of cameras can be any value or range thereof between about 1 and 100 cameras; however, the number of cameras can be greater than 100. In an illustrative example, where the target is a user such as for teleconferencing, the camera array can include 8 cameras arranged in a 4×2 array. In a second illustrative example, a camera array can include at least 45 cameras (for example, when a display preferably displays up to 45 views). In a variation of the second example, the camera array can include 45 cameras arranged along a horizontal extent (e.g., in a given row, 45 columns of cameras, etc.) and 2 or more rows of cameras. However, the camera array can include any suitable number of cameras.
- Each camera of the camera array can be the same (e.g., same image sensor, same image sensor size, same frame rate, same spectral response, etc.) or different (different image sensors, different image sensor sizes, different frame rates, different spectral response, etc.).
- In some embodiments, cameras of the camera array can be separated into camera subsets. For example, the camera array can include primary, secondary, tertiary, and so on cameras. The camera subsets can be fixed and/or varying. Each camera of a camera subset can be the same or different. Subsets of cameras can be determined based on camera properties (e.g., frame rate, image sensor size, camera perspective, etc.), sensor data (e.g., tracking information, etc.), user parameters, environmental parameters, and/or otherwise be determined. Subsets of cameras can be arranged in a group (e.g., be physically grouped together), be interspersed with other subsets of cameras (e.g., every fourth camera can belong to a camera subset with the remaining cameras belonging to a second camera subset), be randomly distributed, and/or otherwise be distributed or arranged.
- In a first variant of the camera array, a subset of the cameras (e.g., secondary cameras) can be used (e.g., used exclusively, used for purposes in addition to acquiring images, etc.) to track a target and/or viewer, segment an image (e.g., a collected image, a image acquired using the same or a different camera or image sensor, etc.), classify an image, and/or otherwise be used. For example, a subset of cameras (e.g., secondary cameras) can be used to segment an image into a foreground and a background. Based on a relative pose between a camera from the subset of cameras (e.g., a secondary camera) and a separate camera (e.g., a primary camera), a foreground and background can be determined for images associated with the separate camera. In this specific example, only the foreground can be transmitted to the receiver. However, the background can additionally or alternatively be transmitted to the receiver.
- As shown for example in
FIG. 8 , a first set of cameras (e.g., primary cameras, view acquisition cameras) of the camera array can acquire images at a first frame rate (e.g., approximately 30 fps) and a second set of cameras (e.g., secondary cameras, segmentation cameras) of the camera array can acquire images at a second frame rate (e.g., 120 fps) that is greater than the first frame rate. In this specific example, the second set of cameras can acquire images with a lower resolution than the first set of cameras (e.g., at 1080 pp vs 4 k or 8 k resolution for the first set of cameras). However, the second set of cameras can acquire images with an equal or greater resolution (e.g., as a cost savings by including fewer higher quality cameras) and/or any suitable resolution or frame rate. The second set of cameras generally has fewer cameras than the first set of cameras. However, the second set of cameras can have the same number of and/or more cameras than the first set of cameras. - In a second variant of the camera array, a first set of cameras can be determined based on tracking information for a viewer using a display. For example, the first set of cameras can be cameras that are at an equivalent (e.g., nearest to the equivalent; adjacent to; within a threshold distance such as 1 mm, 2 mm, 5 mm, 1 cm, 2 cm, 5 cm, 10 cm, 20 cm, etc. of; etc.) pose as the viewer eye pose (and/or eye vector) were the viewer in approximately the same position as the camera array and/or for a virtual viewer collocated with (or proximal to) the camera array. Alternatively stated, the first set of cameras can be aligned to the position of (e.g., closest to, within a threshold distance or position of, etc.) the viewer(s)' eyes (e.g., as determined via tracking the viewer's eyes). As the viewer moves, the first set of cameras can change. For instance, as a viewer moves in a particular direction (e.g., up, down, left, right), the set of primary cameras can move (e.g., be physically moved, change which cameras are included in the set of primary cameras by selecting cameras that are offset from the initial primary cameras by a distance approximately equal to the distance the viewer moves, etc.) in the same direction (e.g., by approximately the same amount) or opposite direction (e.g., depending on whether the camera array is mirrored or not to viewer direction). In this variant the second set of cameras (e.g., auxiliary cameras) can be one or more cameras adjacent to (e.g., above, below, right, left, behind, in front of, between primary cameras, etc. such as immediately next to, within a threshold number of cameras of, etc.) the first set of cameras, cameras that are not in the first set of cameras, cameras that are not adjacent to the first set of cameras, and/or any suitable cameras of the camera array. The second set of cameras can function to provide auxiliary views such as in anticipation of or to account for motion of a viewer, to provide additional perspectives or views of the target (e.g., sender), and/or otherwise function. In some variations of this variant, the camera array can include a third (or more) set of cameras. For example, cameras adjacent to the second set of cameras, distal the first set of cameras, can form a third set of cameras. Typically images associated with the third set of cameras are not transmitted to the receiver (and/or the third set of cameras can not be used to acquire images), but they can be transmitted. The set of cameras (e.g., primary, secondary, and/or tertiary cameras) can be offset (e.g., vertically offset) relative to the viewers' positions, coincident with the viewers' positions, and/or can have any suitable orientation.
- The first and second variants can be used in isolation and/or combination.
- The cameras of the camera array are preferably equally spaced, but can be unevenly spaced. The separation distance between adjacent cameras within the camera array is preferably between about 1 inch and 24 inches. However, the separation distance between adjacent cameras can be greater than 24 inches and/or less than 1 inch.
- In variants, the camera array can be attached to a lightfield display. The camera array can be mounted above the lightfield display (e.g., above optical elements of the light field display, as shown for example in
FIG. 3 , etc.), below the lightfield display (e.g., below optical elements of the lightfield display), next to the lightfield display, and/or have any arrangement relative to the display (e.g., a subset of cameras above the display and a subset of cameras below the display, a subset of cameras on a first side of the display and a subset of cameras on a second side of the display, etc.). The camera array and display can be on the same or different faces (e.g., surfaces, broad faces, etc.) of a housing (e.g., can be arranged in a manner similar to a selfie or front facing camera, be arranged in a manner similar to a rear facing camera, etc.). However, the camera array can be attached to (e.g., mounted to) a computing system, sensor, viewer, be separated from other components (e.g., separate from a display), and/or have any suitable arrangement. - The
image capture sensors 290 can be the same and/or different from theimage capture sensors 190. However, any suitable sensors can be used. - An optional display of an
image capture subsystem 200 can be the same or different from a display of adisplay subsystem 100. - An optional image acquisition system or camera array of a
display subsystem 100 can be the same or different from an imageacquisition subsystem system 200. - The
computing system 300 can function to process views (e.g., compress or decompress views), generate views, select views, control system operation, receive and transmit data (e.g., user parameters, views, environmental parameters, etc.), and/or perform any function. The computing system can be local (e.g., to the lightfield display, to the image acquisition system, to the display subsystem, to the image capture subsystem, to a sensor, etc.), remote (e.g., cloud computing, server, etc.), and/or otherwise be distributed. In a specific example, the computing system can be distributed between adisplay computing system 160 and an image capture or view computing system 260 (e.g., associated with an image acquisition system). The computing system can include one or more: GPUs, CPUs, TPUs, microprocessors, and/or any other suitable processor. - The communication module 350 functions to communicate user parameters, images (e.g., still frames, videos, timeseries, etc.), and/or other data between communication systems and/or other endpoints. Each system or subsystem can include one or more communication modules. The communication modules can be wireless modules (e.g., WiFi, cellular, satellite, etc.), wired modules (e.g., coaxial cable, fiber, etc.), and/or other communication modules. For example, the communication (e.g., between receiver and sender) can be transmitted over internet, over a cellular network, over a local area network, and/or over any suitable data network.
- In some variants, the computing system (e.g., an image capture computing system) can include a multiplexer, which can function to selectively activate (e.g., send instructions to capture image(s) using) specific cameras of the image acquisition system.
- The method functions to capture lightfield images of a target and display the lightfield images to one or more viewers (e.g., viewers). The method can additionally or alternatively function to enable communication (e.g., telecommunication) between users (e.g., teleconferencing), viewing 3D imagery of a target, and/or any suitable function. The method is preferably performed in real- or near-real time, but additionally or alternatively be delayed (e.g., a light field image or video can be stored for access or viewing later), and/or be performed with any suitable timing.
- As shown for example in
FIG. 4 , the method can include: establishing a communicable link S100, determining one or more user parameters S200, transmitting the user parameter(s) S300, generating views based on the user parameter(s) S400, transmitting the generated views S500, and displaying the received views S600. The method can optionally include processing (e.g., pre and/or post processing) the views, and/or any suitable steps. - One or more instances of the method (and/or steps thereof) can be performed sequentially, simultaneously, contemporaneously, within a predetermined time period, synchronously, asynchronously, and/or with any timing. In an illustrative example, two instances of the method can be performed contemporaneously for each of two users. The method is preferably performed using a system as described above, but can be performed by any suitable system. In a first specific example, the method can be performed between two communications system where each communication system can include a lightfield display, a tracking sensor, camera array (or other image acquisition system), and a computing system. In a second specific example, the method can be performed using a display subsystem and a separate image capture subsystem.
- Establishing a communicable link S100 functions to connect a display subsystem to an image capture subsystem enabling communication (e.g., data, such as images and/or audio, transmission between them). The communicable link can be a network connection, radio, Bluetooth (e.g., BLE), optical communication, wired connection, and/or any suitable data connection can be used. Establishing the communicable link S100 is preferably performed between a display computing system and an image capture computing system (e.g., communication modules of the respective systems), but can be between any suitable components. Establishing the communicable link preferably includes identifying and/or verifying the transmitter and receiver, but can include any suitable steps. The communicable link can be automatically established (e.g., upon entering a communication room, upon requesting information from an image capture subsystem, based on user preferences, upon detection of motion proximal an image acquisition subsystem, etc.) and/or manually (e.g., by accepting an invitation, by accepting a request, by entering a communication room, etc.). The communicable link is preferably wireless (e.g., optical, transmitting data over the internet, etc.), but can be wired.
- Determining a user parameter S200 preferably functions to determine (e.g., measure, estimate, calculate, etc.) one or more user parameters associated with a display user or viewer (e.g., a receiver). Determining a user parameter is preferably performed by a sensor (e.g., a tracking sensor, a sensor of a display subsystem, etc.), but can be performed by any component. S200 can be performed before, during, and/or after S200.
- In specific examples, user parameters can include: user pose (e.g., of a viewer's head, nose, face, glabella, hairline, etc.), user head pose, user eye pose (e.g., position and/or orientation such as x/y/z/θ/φ/ψ for each of a user's eyes), user gaze direction, user eye vectors, kinematics (e.g., user motion), user gestures and/or interactions (such as at an input device), number of users, a bandwidth (e.g., processing bandwidth, transmission bandwidth, receipt bandwidth, display bandwidth, etc. associated with a computing system associated with the user), and/or any suitable user parameters. In variants where the user parameters include a pose (e.g., head pose, eye pose, gaze, etc.), the pose can refer to an absolute position, relative positions (e.g., relative to the display, a virtual user location such as where the display user would be if they were collocated with the image capture subsystem, relative to the image capture subsystem, etc.), coordinates, one or more camera of a camera array (e.g., a primary camera), and/or be provided in any manner. Additionally or alternatively, the user parameter can include a set of view indices, which function to specify the views requested from the opposing communication system (e.g., which views should be captured and/or transmitted back to the display). The view indices are preferably determined based on the current or estimated user parameter (e.g., user pose, eye pose, etc.), but can be determined based on the viewing target's current or estimated position, calibration matrices (e.g., between the viewing display and the capturing camera system), or the otherwise determined. The view indices can identify the main views corresponding to the user parameter (e.g., the user's position), buffer views (e.g., surrounding the primary views), and/or other views.
- In variants (for example, when one or more viewers lose tracking but are still expected or known to be viewing the display), determining a user parameter can include predicting the value for the user parameter. The viewer pose can be estimated: until viewer tracking is redetermined (e.g., sensor can identify a viewer feature or pose), for a predetermined amount of time (e.g., 1 s, 2 s, 4 s, 6 s, 10 s, 20 s, values therebetween, >20 s, <1 s, etc.), at a second time after the user parameter has been measured, and/or for any suitable condition. The user parameter can be predicted based on sensor data (e.g., IMU measurements associated with a user), based on images of the user (e.g., using optic flow), and/or based on any data. The user parameters at the second time can be predicted using a dead reckoning, a Kalman filter, an extended Kalman filter, kinematic equations, a particle filter, and/or be otherwise predicted.
- Transmitting the one or more user parameters S300 functions to provide the image capture subsystem with the user parameters. The user parameter(s) are preferably transmitted using the communicable link, but can be transmitted in any manner. The user parameters are preferably transferred from the display subsystem to the image capture subsystem, but can be transferred between any systems. S300 is preferably performed after S200, but can be performed at the same time as and/or before S200 (e.g., estimated user parameters can be transmitted before user parameters have been measured or updated).
- Generating one or more views S400 functions to acquire images of a scene and/or target and/or prepare (e.g., process, select, etc.) images to be displayed at a display. The views can correspond to images, processed images, selected images, renderings, and/or other data structures. The views are preferably generated by an image capture subsystem, but can be generated by a computing system (e.g., a memory module of a computing system, a view generation module of a computing system, etc.) and/or by any component. S400 is preferably performed after S300, but can be performed before and/or during S300 (e.g., a set of images can be acquired by the image acquisition system before and/or at the same time as the user parameters are transmitted to the image capture subsystem). The views can be generated based on the user parameters (e.g., as measured in S200, estimated user parameters, measured user parameters, etc.), the communicable link (e.g., a bandwidth, a quality of the connection, a stability of the connection, etc.), a computing system (e.g., a processor bandwidth, a processor speed, a GPU, etc.), and/or any suitable property. The views to be generated can be specified by the requesting system (e.g., viewing system, display subsystem), the image capture system, and/or another system.
- In variants, generating one or more views can include: acquiring images, identifying one or more targets, tracking one or more targets (e.g., using a sensor, using based on one or more image acquired using a camera, etc.), adjusting an image capture parameter (e.g., which camera(s) are used to acquire images, which images or views are selected, a frame rate, an image resolution, etc.), selecting images, processing images, and/or any suitable step(s).
- Generating views preferably includes acquiring a plurality of images of a target. The images are preferably acquired by the image acquisition system (e.g., camera array, primary cameras of the camera array, primary and secondary cameras of the camera array, secondary cameras of the camera array, etc.). Each image is preferably acquired substantially simultaneously with other images (e.g., within at most 1
ms 10 ms, 20 ms, 100 ms, 1 s, etc. of each other); however, the images can be acquired with any timing. - The focus of the image acquisition system can be set automatically (e.g., based on a and/or manually. Acquiring the plurality of images can include focusing the image acquisition system (e.g., focusing each camera of a camera array) on a target. Each camera is preferably focused on the same target; however, one or more cameras can focus on different targets. However, acquiring the plurality of images can include capturing the images with a predetermined focal point; optionally, determining a target within the images; and adjusting the focal plane of the images (e.g., such that the target is in focus). Determining the target can include segmenting the image (e.g., semantic segmentation, foreground/background segmentation, etc.), classifying the image segments, identifying a target of interest based on the classification (e.g., manually such as the display user identifying the target; automatically such as using a neural network, based on a set of rules, etc.), and tracking the target (e.g., across frames such as using optic flow; between images such as using disparity maps; etc.), and/or any suitable steps. Adjusting the focal plane of the images can include deconvolving the image, determining a depth map associated with the image, applying a blurring filter to the images (e.g., to adjust a lens blur, based on the depth to a target and/or pixel of the image, etc.), generating a 3D reconstruction (e.g., based on the depth map) generating a view using the 3D reconstruction (e.g., by capturing an image of the 3D reconstruction using virtual cameras), and/or any suitable steps. However, a focus of the images can otherwise be determined or set.
- In a first embodiment, acquiring a plurality of images can include acquiring a plurality of images using a subset of the camera array. The subset of cameras can include any number of cameras between 1 and the total number of cameras in the camera array such as 2, 3, 4, 8, 10, 15, 20, 25, 30, 40, 45, and/or any number of cameras. In a specific example, the number of cameras in the subset of cameras matches the number of distinct eye poses of display users. In a variation of this specific example, the number of cameras in the subset can include buffer cameras (e.g., two, four, six, etc. buffer cameras per camera matching the distinct eye poses of the display users) adjacent to (e.g., surrounding) the primary cameras. The subset of the camera array is preferably determined based on the display user parameters, but the subset of the camera array can be predetermined (e.g., based on a set-up procedure), based on the target, and/or be otherwise determined. In an illustrative example, as shown in
FIG. 5A , the subset of cameras can be chosen to be the cameras that are closest to the eye pose of a virtual user, where the eye pose of the virtual user is determined based on the eye pose measured for the display user. In a related example, the subset of cameras can include the pair of cameras that are closest to the eye pose of the virtual user. In a third specific example, the subset of cameras can correspond to the view indices received from the (viewing) display subsystem. In a second specific example, the cameras in the subset of cameras can correspond to the three or four cameras that are closest each eye pose of a virtual user. In variations of the specific examples, the subset of cameras can include cameras adjacent to the cameras determined based on the eye pose. In a third specific example, the cameras in the subset of cameras can be determined based on a mapping (e.g., equation, look-up table, etc.) from the viewer's eye position (e.g., tracked position) to cameras of the camera array. However the subset of cameras can include any suitable cameras. - In a second embodiment, acquiring a plurality of images can include acquiring a plurality of images using every camera of the camera array. In this embodiment, a subset of the images is preferably selected from the plurality of images. The subset of images is preferably selected based on the display user parameters (e.g., user pose, view indices), but can be selected based on the target and/or any suitable property. Selected images can include: images associated with primary and/or second cameras, images associated with a subset of cameras, a In an illustrative example as shown in
FIG. 5B , the subset of images can be selected based on the eye pose for the display user. - In a third embodiment, acquiring a plurality of images can include acquiring a plurality of images using a camera (or other image recording device), where the camera is translated to a plurality of positions, where the plurality of positions are determined based on (e.g., match) a display user parameter (e.g., eye pose).
- Processing the images preferably functions to reduce a size of an image, but can additionally or alternatively function to reduce or remove one or more optical artifact, augment the images (e.g., using digital content), and/or otherwise function. Processing the images can include: cropping images, aligning images, rectifying images, transforming images (e.g., using an affine transformation), filtering images, compressing images, applying a shader, reprojecting the images (e.g., interpolating between images, determining an image intermediate between two or more images, etc.), correcting images (e.g., correcting each image based on a camera pose for the respective camera), generating a quilt image (e.g., arranging the set of images in a predetermined order or manner), and/or any suitable image manipulations.
- Each image of the set of images can be processed in the same or different manner. For example, as shown in
FIG. 6 , a first set of images can be compressed using a first codec and/or set of codex parameters and a second set of images can be compressed using a second codec and/or set of codec parameters. In this example, the first set of images can be associated with primary cameras and can be compressed with a higher quality and/or fidelity (e.g., based on one or more of: objective quality, subjective quality, sharpness, noise, tone reproduction, dynamic range, contrast color accuracy, distortion, vignetting, exposure accuracy, lateral chromatic aberration, lens flare, color moiré, artifacts, etc.) relative to the second set of images (e.g., associated with secondary cameras). However, the first set of images can be compressed with a lower quality and/or fidelity relative to the second set of images, the first and second set of images can be compressed in different file wrappers, and/or the image can otherwise be compressed. - Processing the images preferably does not include rendering the images to generate a three dimensional representation of the target and/or scene. However, the images can be processed to generate a three dimensional representation of the target and/or scene.
- In variants, processing the images can include warping a projection matrix which functions to accommodate (e.g., correct for) changes in a user parameter (e.g., head orientation, eye orientation, distance from the display, position relative to the display, etc.). The warped projection matrix (e.g., perspective correct information, perspective projection) is preferably used to determine the views that need to be projected onto the display (e.g., a screen thereof) based on the user's eye position(s), but can be used to determine the images (e.g., additional images) to be captured by the camera array, camera indices (e.g., cameras in addition to the cameras capturing the primary images, secondary cameras, etc.) for additional image capture, and/or otherwise used. The warping can be determined based on measurements from the tracking sensor and/or any suitable component. The warping can be relative to an on-axis viewer perspective, an initial viewer perspective (e.g., initial viewer pose relative to the display, initial viewer eye pose, etc.), and/or relative to any suitable perspective. The projection matrix (e.g., warped projection matrix) can be applied to captured images, interpolated images, perspective corrected images, 3D representation (e.g., mesh), and/or any suitable image or scene representation. In some examples, a plurality of images (e.g., for each eye of the viewer) can be used to generate the perspective corrected image. In a first variation, the projection matrix is used to determine how the images should be processed to determine the projected views. In a second variation, the projection matrix is used to determine which pixels, corresponding to a mesh of the scene (e.g., 2D mesh, 3D mesh, polygonal mesh, etc.), should be rendered. However, the projection matrix can be otherwise used.
- In some embodiments, S400 can include segmenting one or more image or view. Segmenting the image(s) can be beneficial as segments or portions of the images can be transmitted (e.g., rather than full images), which can decrease a bandwidth required and/or enable higher resolution images to be transferred. The image(s) can be segmented using artificial intelligence (e.g., machine learning; a neural network such as generative adversarial networks, convolutional neural networks, deep neural networks, etc.; trainable methods; etc.), an image classification and/or segmentation algorithm, using computer vision (e.g., thresholding, clustering, edge detection, histogram methods, region-growing methods, partial differential equations, compression methods, variational methods, graph partitioning methods, watershed transformations, multi-scale segmentation, etc.) and/or using any suitable method(s). The image can be segmented into key points (e.g., as shown for example in
FIG. 7 ), into a foreground and background (e.g., as shown for example in inFIG. 8 ), into one or more subjects and not subjects, into a target and other information, and/or otherwise be segmented. - In a first variant, one or more images can be segmented into key points. The images can be segmented into key points, for instance as shown in
FIG. 7 , using a neural network (e.g., a GAN, segmentation network, etc.). The key points can be features, pixels, superpixels, coordinates, and/or other distinctive points within the image(s). The key points can be transmitted to the display subsystem. The key points can be used (e.g., at the display subsystem) to process (e.g., transform, reproject, etc.) a reference image (e.g., based on a change in position of the key points between the reference image and the currently measured key points) to accommodate or display changes or updates to a subject. However, the key points can otherwise be used. - In a second variant, a first subset of the images (e.g., secondary images) can be segmented into a foreground (e.g., the subject such as the sender) and background (e.g., environment or context surrounding the subject). The secondary images are preferably acquired at a higher frame rate than a second subset of images (e.g., primary images such as images that are intended to be transmitted), but can be acquired at the same or a lower frame rate than the primary images. The secondary images can be segmented using machine learning (e.g., a neural network as shown for example in
FIG. 8 ), based on motion (e.g., a difference between the frames), and/or otherwise be segmented. The segments of the secondary images can be used to segment the primary images (e.g., based on a known pose or relationship between the secondary cameras and the primary cameras). However, the images can otherwise be segmented or used. - S400 can include acquiring auxiliary dataset(s). The auxiliary datasets can be acquired simultaneously with, contemporaneously with, before, after, and/or with any suitable timing relative to the images. Examples of auxiliary datasets can include: audio datasets (e.g., subject audio, subject speaking, etc.), depth (e.g., distances between the image acquisition system and the subject(s)), environmental parameters, and/or any suitable auxiliary dataset(s). The auxiliary dataset(s) can be acquired using a sensor, a computing system (e.g., configured to generate an auxiliary dataset), and/or be acquired using any suitable component. In some variants, the auxiliary datasets can be processed, for example, to change a perceived audio by a receiver (such as to cause the audio perceived by the receiver to be perceived as coming from a predetermined location or distance, to modify a frequency spectrum of the audio, to synchronize the audio with the light field image or frame, etc.), to modify a focus of the lightfield image (e.g., based on depth data), and/or otherwise processing the auxiliary dataset(s).
- Transmitting the one or more views S500 functions to provide the views (e.g., processed images, selected images, raw images, difference views, key points, primary views, secondary views, etc.) to the display subsystem. S500 is preferably performed after S400, but can be performed at the same time as and/or before S400 (e.g., where the display subsystem can generate and/or select views to be displayed). The views are preferably transmitted using the communicable connection, but can be transmitted in any manner.
- In variants, only views that will be displayed to or perceived by a user (e.g., views that are selected and/or captured based on the display user parameters, views corresponding to a left and right eye of a display user, buffer views, primary views, secondary views, etc.) are transmitted. For instance, the set of transmitted (or generated) views can exclude images acquired by cameras of the camera array associated with a camera pose that differs from the eye poses by at least a threshold amount (e.g., a threshold distance, a threshold number of cameras between an excluded camera and a primary or secondary camera, etc.). However, all or any subset of views can be transmitted. The views are preferably transmitted from the image capture subsystem (e.g., a communication module of the image capture subsystem) to the display subsystem (e.g., a communication module of the display subsystem), but can be transmitted between any systems.
- In variants, particularly but not exclusively where the image is segmented (such as into key points), transmitting the one or more views can include transmitting a reference image (e.g., reference light field image) of the subject and/or scene. The reference image can be transmitted once (e.g., at a first time point such as when the method is first initiated, during S100, etc.), with each set of key points, randomly, when a threshold change in the reference image occurs (e.g., number of subjects changes, reference image is no longer representative of the subject, etc.), at predetermined times or frequencies, and/or with any suitable timing or frequency.
- S500 can include transmitting auxiliary dataset(s). The auxiliary dataset(s) can be transmitted in the same container as the views (e.g., as metadata, embedded in the views, etc.) and/or in a separate container.
- Displaying the received views S600 functions to display the received views to the display user. Displaying the received views is preferably performed by the display subsystem (e.g., a lightfield display) requesting the views (e.g., tracking the first user), but can be performed by any system. The views can be directed to a user, displayed within a viewcone of the display, and/or otherwise be displayed. In specific examples, the views can be displayed in a similar manner as the display operation as disclosed in U.S. patent application Ser. No. 16/374,955 titled ‘SUPERSTEREOSCOPIC DISPLAY WITH ENHANCED OFF-ANGLE SEPARATION’ filed 4 Apr. 2019, U.S. Pat. No. 10,241,344 titled ‘ADVANCED RETROREFLECTING AERIAL DISPLAYS’ filed 28 Nov. 2018, U.S. patent application Ser. No. 17/326,857 filed 21 May 2021 titled ‘SYSTEM AND METHOD FOR HOLOGRAPHIC IMAGE DISPLAY,’ and/or U.S. patent application Ser. No. 17/332,479 filed 27 May 2021 titled ‘SYSTEM AND METHOD FOR HOLOGRAPHIC DISPLAYS,’ each of which is incorporated in its entirety by this reference, and/or otherwise be displayed.
- S600 can include processing the views (e.g., using a computing system of the display subsystem). Processing the views to be displayed can include formatting the views to be displayed (e.g., lenticularizing or arranging the views as disclosed in U.S. patent application Ser. No. 17/226,404 filed 9 Apr. 2021 entitled ‘SYSTEM AND METHOD FOR GENERATING LIGHT FIELD IMAGES’ incorporated in its entirety by this reference), modifying a focus of the lightfield image, applying augmentation content (e.g., digital content) to the lightfield image, modifying a reference image (e.g., updating a reference image based on a set of key points), decompressing the views (e.g., using a codex, using a decoder, etc.), and/or otherwise processing the views.
- S600 can include presenting auxiliary data. For example, audio datasets can be presented to the users (e.g., receivers). The audio datasets can be presented using a single channel (e.g., mono) and/or a plurality of channels (e.g., stereo). In some variants, the audio data can be modified or processed. For instance, the audio dataset can be presented as a three dimensional or dynamic audio signal that is perceivable as changing a direction and/or other quality (e.g., frequency, loudness, etc.) as a subject moves. However, the audio can be presented as a static signal (e.g., content is presented as it is recorded by the image capture subsystem or sensor thereof) and/or as any suitable signal.
- For the sake of clarity, the method is described between two systems (e.g., two communication systems, a display subsystem and an image capture subsystem, etc.). However, the steps and/or substeps of the method may be applied (e.g., simultaneously, contemporaneously, in series, in parallel, etc.) for any number of users and/or systems (e.g., display subsystems, image capture subsystems, communication systems, etc.). The total number of systems (and/or users) can be bounded by a threshold (for example, based on a computing system of associated with a display, on a computing system associated with an image capture subsystem, based on a bandwidth of a computing system, etc.; a threshold number of users that can be tracked simultaneously; a threshold number of users based on the number of views that can be displayed; etc.), but the total number of displays (and/or users) can be unbounded.
- In an illustrative example as shown in
FIG. 5A , a method for lightfield communication can include: establishing a communicable link between a first display subsystem associated with a first user and a second image capture subsystem associated with a second user; determining a user parameter (e.g., user pose, user eye pose, gaze tracking, selected view indices, etc.) associated with the first user; transmitting the user parameter to the second image capture subsystem; acquiring images using the second image capture subsystem based on the user parameter, where at least a subset of the images that are acquired correspond to the user parameter; optionally, processing the images (e.g., cropping the images, aligning the images, rendering the images, reprojecting the images, etc.); transmitting the images to the first display subsystem; and displaying the images at the first display subsystem. The process can be repeated for a second user using a second display subsystem (connected to the second image capture subsystem) and a first image capture subsystem (connected to the first display subsystem) to enable bidirectional communication. The user parameter can be automatically determined by the first display subsystem or first image capture subsystem (e.g., based on a user pose relative to the respective subsystem). The images corresponding to the user parameter can be determined by the first display subsystem or by the second display subsystem (e.g., by mapping the user parameter to a set of cameras). - In a related example as shown in
FIG. 5B , the second communication system can capture images at each camera of the second communication system, select a subset of the images based on the first user parameter, and transmit the subset of the images. - In variations of these examples, an eye tracking module can be used to track the viewer(s) (e.g., eye position, gaze, etc. thereof). The eye tracking module can include a processor (e.g., separate from the display or image acquisition processor, integrated with the display or image acquisition processor, etc.) configured to determine at least one of an eye position or an eye vector of the one or more viewers based on one or more tracking images. In variants, the processor can additionally or alternatively estimate and/or predict a location of the viewer(s) (e.g., by performing dead reckoning relative to previous tracked positions) and/or otherwise function. The tracking image(s) can be acquired using the camera array, a dedicated tracking sensor (e.g., a camera that is separate from the camera array), a depth camera, and/or any suitable tracking sensor.
- Embodiments of the system and/or method can include every combination and permutation of the various system components and the various method processes, wherein one or more instances of the method and/or processes described herein can be performed asynchronously (e.g., sequentially), concurrently (e.g., in parallel), or in any other suitable order by and/or using one or more instances of the systems, elements, and/or entities described herein.
- As a person skilled in the art will recognize from the previous detailed description and from the figures and claims, modifications and changes can be made to the preferred embodiments of the invention without departing from the scope of this invention defined in the following claims.
Claims (1)
1. A holographic communication device operable to communicate with one or more communication systems, the holographic communication system comprising:
a first camera array configured to sample images of a scene;
a holographic display mounted proximal the camera array;
an eye tracking module configured to track an eye position of a user proximal the holographic display; and
a second camera array, distal from the first camera array, that is configured to capture images of a second scene based on the eye position, wherein the images of the second scene are perceivable as a holographic image when displayed by the holographic display.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/861,652 US20220342365A1 (en) | 2020-06-23 | 2022-07-11 | System and method for holographic communication |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US202063042862P | 2020-06-23 | 2020-06-23 | |
US17/356,129 US11415935B2 (en) | 2020-06-23 | 2021-06-23 | System and method for holographic communication |
US17/861,652 US20220342365A1 (en) | 2020-06-23 | 2022-07-11 | System and method for holographic communication |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/356,129 Continuation US11415935B2 (en) | 2020-06-23 | 2021-06-23 | System and method for holographic communication |
Publications (1)
Publication Number | Publication Date |
---|---|
US20220342365A1 true US20220342365A1 (en) | 2022-10-27 |
Family
ID=79023438
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/356,129 Active US11415935B2 (en) | 2020-06-23 | 2021-06-23 | System and method for holographic communication |
US17/861,652 Abandoned US20220342365A1 (en) | 2020-06-23 | 2022-07-11 | System and method for holographic communication |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/356,129 Active US11415935B2 (en) | 2020-06-23 | 2021-06-23 | System and method for holographic communication |
Country Status (2)
Country | Link |
---|---|
US (2) | US11415935B2 (en) |
WO (1) | WO2021262860A1 (en) |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20230134663A1 (en) * | 2021-11-02 | 2023-05-04 | Steven Roskowski | Transforming Surveillance Sensor Data into Event Metadata, Bounding Boxes, Recognized Object Classes, Learning Density Patterns, Variation Trends, Normality, Projections, Topology; Determining Variances Out of Normal Range and Security Events; and Initiating Remediation and Actuating Physical Access Control Facilitation |
US11962867B2 (en) * | 2021-11-04 | 2024-04-16 | Tencent America LLC | Asset reusability for lightfield/holographic media |
US11785262B1 (en) * | 2022-03-16 | 2023-10-10 | International Business Machines Corporation | Dynamic compression of audio-visual data |
Family Cites Families (124)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US2631496A (en) | 1947-08-08 | 1953-03-17 | Miles P Rehorn | Stereoscopic viewing method and apparatus |
US5264964A (en) | 1991-12-18 | 1993-11-23 | Sades Faris | Multi-mode stereoscopic imaging system |
US5521724A (en) | 1993-11-15 | 1996-05-28 | Shires; Mark R. | Real-time automultiscopic 3D video display using holographic optical elements (HOEs) |
JPH10510400A (en) | 1994-12-09 | 1998-10-06 | ゴウドレアウ、ジーン−エティエン | Stereoscopic display method and apparatus |
JP3096613B2 (en) | 1995-05-30 | 2000-10-10 | 三洋電機株式会社 | 3D display device |
JPH0915532A (en) | 1995-06-29 | 1997-01-17 | Canon Inc | Stereoscopic image display method and stereoscopic image display device using the method |
US5886675A (en) | 1995-07-05 | 1999-03-23 | Physical Optics Corporation | Autostereoscopic display system with fan-out multiplexer |
US7190518B1 (en) | 1996-01-22 | 2007-03-13 | 3Ality, Inc. | Systems for and methods of three dimensional viewing |
US6064424A (en) | 1996-02-23 | 2000-05-16 | U.S. Philips Corporation | Autostereoscopic display apparatus |
DE59708088D1 (en) | 1996-12-18 | 2002-10-02 | Dresden 3D Gmbh | METHOD AND ARRANGEMENT FOR THE THREE-DIMENSIONAL PRESENTATION OF INFORMATION |
JP3595645B2 (en) | 1997-02-18 | 2004-12-02 | キヤノン株式会社 | 3D image display device |
US6097394A (en) | 1997-04-28 | 2000-08-01 | Board Of Trustees, Leland Stanford, Jr. University | Method and system for light field rendering |
EP0881844B1 (en) | 1997-05-27 | 2002-07-24 | Sanyo Electric Co. Ltd | Head tracking stereoscopic display |
US6798390B1 (en) | 1997-08-29 | 2004-09-28 | Canon Kabushiki Kaisha | 3D image reconstructing apparatus and 3D object inputting apparatus |
US6195184B1 (en) | 1999-06-19 | 2001-02-27 | The United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration | High-resolution large-field-of-view three-dimensional hologram display system and method thereof |
US7227526B2 (en) | 2000-07-24 | 2007-06-05 | Gesturetek, Inc. | Video-based image control system |
US20020141635A1 (en) | 2001-01-24 | 2002-10-03 | Swift David C. | Method and system for adjusting stereoscopic image to optimize viewing for image zooming |
ES2180445B1 (en) | 2001-06-29 | 2004-04-01 | Jose Javier Alejo Trevijano | AUTOESTEREOSCOPIC VISION SYSTEM. |
US8369607B2 (en) | 2002-03-27 | 2013-02-05 | Sanyo Electric Co., Ltd. | Method and apparatus for processing three-dimensional images |
JP2003299121A (en) | 2002-04-05 | 2003-10-17 | Canon Inc | Three-dimensional image display apparatus and system thereof |
GB2387664B (en) | 2002-04-17 | 2005-08-24 | Philip Anthony Surman | Autostereoscopic display |
NZ521505A (en) | 2002-09-20 | 2005-05-27 | Deep Video Imaging Ltd | Multi-view display |
US7705935B2 (en) | 2004-01-20 | 2010-04-27 | Ecrans Polaires Inc. | Stereoscopic display system |
US20070247590A1 (en) | 2004-08-10 | 2007-10-25 | Seereal Technologies Gmbh | Autostereoscopic Multi-User Display with Directed Illumination |
CA2520385A1 (en) | 2004-09-20 | 2006-03-20 | Kenneth Tetterington | Three dimensional image generator |
US7903332B2 (en) | 2004-10-13 | 2011-03-08 | Koninklijke Philips Electronics N.V. | Stereoscopic display apparatus |
KR100728113B1 (en) | 2005-10-20 | 2007-06-13 | 삼성에스디아이 주식회사 | Stereoscopic display device and driving method thereof |
JP4650279B2 (en) | 2006-01-19 | 2011-03-16 | エプソンイメージングデバイス株式会社 | Electro-optical device and electronic apparatus |
JP4215058B2 (en) | 2006-01-19 | 2009-01-28 | エプソンイメージングデバイス株式会社 | Image display device and electronic device |
US7916934B2 (en) | 2006-04-04 | 2011-03-29 | Mitsubishi Electric Research Laboratories, Inc. | Method and system for acquiring, encoding, decoding and displaying 3D light fields |
FR2906899B1 (en) | 2006-10-05 | 2009-01-16 | Essilor Int | DISPLAY DEVICE FOR STEREOSCOPIC VISUALIZATION. |
CN101568888B (en) | 2006-10-26 | 2013-01-30 | 视瑞尔技术公司 | Mobile tlephony system comprising holographic display |
US8559705B2 (en) | 2006-12-01 | 2013-10-15 | Lytro, Inc. | Interactive refocusing of electronic images |
US7792423B2 (en) | 2007-02-06 | 2010-09-07 | Mitsubishi Electric Research Laboratories, Inc. | 4D light field cameras |
WO2009085581A1 (en) | 2007-12-21 | 2009-07-09 | 3M Innovative Properties Company | Light control film |
KR100893618B1 (en) | 2008-03-07 | 2009-04-20 | 삼성모바일디스플레이주식회사 | Electronic display device |
EP2292018B1 (en) | 2008-04-22 | 2012-06-20 | 3ality, Inc. | Position-permissive autostereoscopic display systems and methods |
JP2010068202A (en) | 2008-09-10 | 2010-03-25 | Michiyoshi Nagashima | Image display device |
BRPI0914482A2 (en) | 2008-10-28 | 2015-10-27 | Koninkl Philips Electronics Nv | three-dimensional display system, method of operation for a three-dimensional display system and computer program product |
WO2010065344A1 (en) | 2008-11-25 | 2010-06-10 | Refocus Imaging, Inc. | System of and method for video refocusing |
KR101324440B1 (en) | 2009-02-11 | 2013-10-31 | 엘지디스플레이 주식회사 | Method of controlling view of stereoscopic image and stereoscopic image display using the same |
DE102009009443B3 (en) | 2009-02-14 | 2010-09-23 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Monitor and method for displaying autostereoscopic images |
JP2010276965A (en) | 2009-05-29 | 2010-12-09 | Sony Corp | Stereoscopic display device and method |
US20110075257A1 (en) | 2009-09-14 | 2011-03-31 | The Arizona Board Of Regents On Behalf Of The University Of Arizona | 3-Dimensional electro-optical see-through displays |
KR101685343B1 (en) | 2010-06-01 | 2016-12-12 | 엘지전자 주식회사 | Image Display Device and Operating Method for the Same |
JP5494284B2 (en) | 2010-06-24 | 2014-05-14 | ソニー株式会社 | 3D display device and 3D display device control method |
JP5631106B2 (en) | 2010-08-18 | 2014-11-26 | キヤノン株式会社 | Image processing apparatus, control method thereof, program, and imaging apparatus |
US8798387B2 (en) | 2010-11-11 | 2014-08-05 | Panasonic Intellectual Property Corporation Of America | Image processing device, image processing method, and program for image processing |
HU1000640D0 (en) | 2010-11-29 | 2011-02-28 | Holografika Hologrameloeallito Fejlesztoe Es Forgalmazo Kft | Image coding and decoding method and apparatus for efficient encoding and decoding of 3d field content |
US8953242B2 (en) | 2011-03-31 | 2015-02-10 | Honeywell International Inc. | Varible focus stereoscopic display system and method |
JP5926500B2 (en) | 2011-06-07 | 2016-05-25 | ソニー株式会社 | Information processing apparatus, information processing method, and program |
US9414049B2 (en) | 2011-09-19 | 2016-08-09 | Écrans Polaires Inc./Polar Screens Inc. | Method and display for showing a stereoscopic image |
KR101269631B1 (en) | 2011-10-06 | 2013-05-30 | 한국과학기술연구원 | 3-dimensional image display apparatus and method of performing the same |
US9165401B1 (en) | 2011-10-24 | 2015-10-20 | Disney Enterprises, Inc. | Multi-perspective stereoscopy from light fields |
WO2013079607A1 (en) * | 2011-11-30 | 2013-06-06 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | View rendering for the provision of virtual eye contact using special geometric constraints in combination with eye-tracking |
JP6032856B2 (en) | 2011-12-23 | 2016-11-30 | トムソン ライセンシングThomson Licensing | Computer apparatus with power consumption management and method for managing power consumption of computer apparatus |
GB2499426A (en) | 2012-02-16 | 2013-08-21 | Dimenco B V | Autostereoscopic display device with viewer tracking system |
US8995785B2 (en) | 2012-02-28 | 2015-03-31 | Lytro, Inc. | Light-field processing and analysis, camera control, and user interfaces and interaction on light-field capture devices |
US9704220B1 (en) | 2012-02-29 | 2017-07-11 | Google Inc. | Systems, methods, and media for adjusting one or more images displayed to a viewer |
EP2648414B1 (en) | 2012-04-03 | 2016-03-23 | Samsung Electronics Co., Ltd | 3d display apparatus and method for processing image using the same |
KR101958447B1 (en) | 2012-05-16 | 2019-03-15 | 삼성디스플레이 주식회사 | 3 dimensional image display device and driving method thereof |
US9179126B2 (en) | 2012-06-01 | 2015-11-03 | Ostendo Technologies, Inc. | Spatio-temporal light field cameras |
US10129524B2 (en) | 2012-06-26 | 2018-11-13 | Google Llc | Depth-assigned content for depth-enhanced virtual reality images |
CN102752621A (en) | 2012-06-26 | 2012-10-24 | 京东方科技集团股份有限公司 | Depth-of-field retaining device, 3D (three-dimensional) display device and display method |
KR102008213B1 (en) | 2012-07-27 | 2019-08-08 | 삼성디스플레이 주식회사 | Method of displaying three dimensional image and three dimensional image display apparatus for performing the method |
NL2009616C2 (en) | 2012-10-11 | 2014-04-14 | Ultra D Co Peratief U A | Adjusting depth in a three-dimensional image signal. |
US9584797B2 (en) | 2012-10-31 | 2017-02-28 | Elwha Llc | Systems and methods to confirm that an autostereoscopic display is accurately aimed |
US9216133B2 (en) | 2013-01-16 | 2015-12-22 | Elwha Llc | Using a 3D display to train a weak eye |
US9456141B2 (en) | 2013-02-22 | 2016-09-27 | Lytro, Inc. | Light-field based autofocus |
KR102070800B1 (en) | 2013-03-11 | 2020-01-29 | 삼성전자주식회사 | Stereoscopic display apparatus, and display method thereof |
US9679360B2 (en) | 2013-05-10 | 2017-06-13 | Trustees Of Princeton University | High-resolution light-field imaging |
CA2922081C (en) | 2013-08-28 | 2016-12-06 | Ricoh Company, Ltd. | Image processing apparatus, image processing method, and imaging system |
KR20160058955A (en) | 2013-09-30 | 2016-05-25 | 코닌클리케 필립스 엔.브이. | Autostereoscopic display device |
WO2015104239A2 (en) * | 2014-01-07 | 2015-07-16 | Seereal Technologies S.A. | Display device for holographic reconstruction |
US10244223B2 (en) | 2014-01-10 | 2019-03-26 | Ostendo Technologies, Inc. | Methods for full parallax compressed light field 3D imaging systems |
CN104935905B (en) | 2014-03-20 | 2017-05-10 | 西蒙·丽兹卡拉·杰马耶勒 | Automated 3D Photo Booth |
CN103852819B (en) | 2014-03-28 | 2016-01-20 | 福州大学 | A kind of grating strengthening bore hole stereo display space continuity |
JP6648041B2 (en) | 2014-05-30 | 2020-02-14 | マジック リープ, インコーポレイテッドMagic Leap,Inc. | Method and system for displaying stereoscopic view using freeform optical system with addressable focus for virtual and augmented reality |
US9581821B2 (en) | 2014-06-24 | 2017-02-28 | Fakespace Labs, Inc. | Head mounted augmented reality display |
TWI556624B (en) | 2014-07-18 | 2016-11-01 | 友達光電股份有限公司 | Image displaying method and image dispaly device |
US10171792B2 (en) | 2014-08-15 | 2019-01-01 | The University Of Akron | Device and method for three-dimensional video communication |
KR101808129B1 (en) | 2014-10-14 | 2017-12-13 | 한국전자통신연구원 | Digital holographic table-top display apparatus and method thereof |
GB2533553B (en) | 2014-12-15 | 2020-09-09 | Sony Interactive Entertainment Inc | Image processing method and apparatus |
EP3041231A1 (en) | 2014-12-30 | 2016-07-06 | SeeFront GmbH | Autostereoscopic multi-view system |
US10152154B2 (en) | 2015-01-22 | 2018-12-11 | Superd Technology Co., Ltd. | 3D interaction method and display device |
US10853625B2 (en) | 2015-03-21 | 2020-12-01 | Mine One Gmbh | Facial signature methods, systems and software |
WO2016154123A2 (en) | 2015-03-21 | 2016-09-29 | Mine One Gmbh | Virtual 3d methods, systems and software |
US10419737B2 (en) | 2015-04-15 | 2019-09-17 | Google Llc | Data structures and delivery methods for expediting virtual reality playback |
US10341632B2 (en) | 2015-04-15 | 2019-07-02 | Google Llc. | Spatial random access enabled video system with a three-dimensional viewing volume |
US10546424B2 (en) | 2015-04-15 | 2020-01-28 | Google Llc | Layered content delivery for virtual and augmented reality experiences |
US10469873B2 (en) | 2015-04-15 | 2019-11-05 | Google Llc | Encoding and decoding virtual reality video |
US10567464B2 (en) | 2015-04-15 | 2020-02-18 | Google Llc | Video compression with adaptive view-dependent lighting removal |
US9983717B2 (en) | 2015-04-21 | 2018-05-29 | Dell Products L.P. | Disambiguation of false touch inputs at an information handling system projected user interface |
KR102415502B1 (en) | 2015-08-07 | 2022-07-01 | 삼성전자주식회사 | Method and apparatus of light filed rendering for plurality of user |
US10110884B2 (en) | 2015-09-15 | 2018-10-23 | Looking Glass Factory, Inc. | Enhanced 3D volumetric display |
US9912868B2 (en) | 2015-09-15 | 2018-03-06 | Canon Kabushiki Kaisha | Image-blur correction apparatus, tilt correction apparatus, method of controlling image-blur correction apparatus, and method of controlling tilt correction apparatus |
EP3360012A4 (en) | 2015-10-09 | 2019-05-29 | Visby Camera Corporation | Holographic light field imaging device and method of using the same |
US10511895B2 (en) | 2015-10-09 | 2019-12-17 | Warner Bros. Entertainment Inc. | Cinematic mastering for virtual reality and augmented reality |
US10416454B2 (en) | 2015-10-25 | 2019-09-17 | Facebook Technologies, Llc | Combination prism array for focusing light |
CN108476311B (en) | 2015-11-04 | 2021-04-27 | 奇跃公司 | Wearable display system and method for calibrating a wearable display |
US9813673B2 (en) * | 2016-01-20 | 2017-11-07 | Gerard Dirk Smits | Holographic video capture and telepresence system |
WO2017127897A1 (en) | 2016-01-27 | 2017-08-03 | Paul Lapstun | Shuttered waveguide light field display |
WO2017145154A1 (en) * | 2016-02-22 | 2017-08-31 | Real View Imaging Ltd. | Wide field of view hybrid holographic display |
CN109076148B (en) | 2016-04-12 | 2021-02-05 | 奎蒂安特有限公司 | Daily scene reconstruction engine |
US20190244432A1 (en) | 2016-07-05 | 2019-08-08 | Realfiction Aps | Exhibition system arranged for presenting a mixed reality and a method of using said system |
EP3488284B1 (en) | 2016-07-25 | 2023-06-28 | Magic Leap, Inc. | Light field processor system and method |
CN106125322A (en) | 2016-09-05 | 2016-11-16 | 万维云视(上海)数码科技有限公司 | Bore hole 3D display floater and bore hole 3D display device |
US9977248B1 (en) | 2016-12-21 | 2018-05-22 | PhantaField, Inc. | Augmented reality display system |
US20190057957A1 (en) | 2016-12-21 | 2019-02-21 | PhantaField, Inc. | Augmented reality display system |
US11119353B2 (en) | 2017-06-01 | 2021-09-14 | E-Vision Smart Optics, Inc. | Switchable micro-lens array for augmented reality and mixed reality |
WO2018227098A1 (en) * | 2017-06-09 | 2018-12-13 | Vid Scale, Inc. | External camera assisted virtual reality |
US10269159B2 (en) * | 2017-07-27 | 2019-04-23 | Rockwell Collins, Inc. | Neural network foreground separation for mixed reality |
EP3454098A1 (en) | 2017-09-12 | 2019-03-13 | Realfiction ApS | System with semi-transparent reflector for mixed/augmented reality |
US10948648B2 (en) | 2017-09-29 | 2021-03-16 | Reald Spark, Llc | Backlights having stacked waveguide and optical components with different coefficients of friction |
US11096765B2 (en) | 2018-06-22 | 2021-08-24 | Align Technology, Inc. | Light field intraoral 3D scanner with structured light illumination |
WO2020023669A1 (en) | 2018-07-25 | 2020-01-30 | Light Field Lab, Inc. | Light field display system based amusement park attraction |
US10867430B2 (en) | 2018-11-19 | 2020-12-15 | Intel Corporation | Method and system of 3D reconstruction with volume-based filtering for image processing |
US10904479B2 (en) | 2019-03-12 | 2021-01-26 | Light Field Lab, Inc. | Video communication including holographic content |
US11212514B2 (en) | 2019-03-25 | 2021-12-28 | Light Field Lab, Inc. | Light field display system for cinemas |
CA3150016A1 (en) | 2019-08-09 | 2021-02-18 | Light Field Lab, Inc. | Light field display system based digital signage system |
EP4022390A4 (en) | 2019-08-26 | 2023-08-23 | Light Field Lab, Inc. | Light field display system for sporting events |
US11474597B2 (en) | 2019-11-01 | 2022-10-18 | Google Llc | Light field displays incorporating eye trackers and methods for generating views for a light field display using eye tracking information |
GB2589575B (en) | 2019-12-02 | 2022-01-12 | Envisics Ltd | Pupil expander |
US11211433B2 (en) * | 2020-05-04 | 2021-12-28 | Intel Corporation | In-display sensors and viewing angle adjustment microassemblies |
-
2021
- 2021-06-23 WO PCT/US2021/038704 patent/WO2021262860A1/en active Application Filing
- 2021-06-23 US US17/356,129 patent/US11415935B2/en active Active
-
2022
- 2022-07-11 US US17/861,652 patent/US20220342365A1/en not_active Abandoned
Also Published As
Publication number | Publication date |
---|---|
WO2021262860A1 (en) | 2021-12-30 |
US20210397126A1 (en) | 2021-12-23 |
US11415935B2 (en) | 2022-08-16 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11960639B2 (en) | Virtual 3D methods, systems and software | |
US11995902B2 (en) | Facial signature methods, systems and software | |
US10602126B2 (en) | Digital camera device for 3D imaging | |
US11415935B2 (en) | System and method for holographic communication | |
EP3560195B1 (en) | Stereoscopic omnidirectional imaging | |
JP6027034B2 (en) | 3D image error improving method and apparatus | |
EP2299726B1 (en) | Video communication method, apparatus and system | |
US11849102B2 (en) | System and method for processing three dimensional images | |
JP5891424B2 (en) | 3D image creation apparatus and 3D image creation method | |
US20160021355A1 (en) | Preprocessor for Full Parallax Light Field Compression | |
KR100560464B1 (en) | Multi-view display system with viewpoint adaptation | |
US10631008B2 (en) | Multi-camera image coding | |
US20150195443A1 (en) | Systems and methods for real-time view-synthesis in a multi-camera setup | |
CN103167308A (en) | Stereoscopic image photographing system and play quality evaluation system and method thereof | |
JP4806088B1 (en) | Image conversion apparatus, image conversion apparatus control method, image conversion apparatus control program, and recording medium | |
US20230152883A1 (en) | Scene processing for holographic displays | |
US20230122149A1 (en) | Asymmetric communication system with viewer position indications | |
EP3564905A1 (en) | Conversion of a volumetric object in a 3d scene into a simpler representation model | |
US20240196065A1 (en) | Information processing apparatus and information processing method | |
US20230008125A1 (en) | Augmenting a view of a real-world environment with a view of a volumetric video object | |
Dehkordi | 3D Video Quality Assessment | |
JP2021196915A (en) | Stereoscopic image depth control device and program thereof | |
CN116823691A (en) | Light field image processing method and device | |
KR20150112461A (en) | Method of image extraction based on human factors and apparatus thereof | |
CN114882089A (en) | Image processing method, device, equipment and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: LOOKING GLASS FACTORY, INC., NEW YORK Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HORNSTEIN, ALEXIS;FRAYNE, SHAWN MICHAEL;SIGNING DATES FROM 20210628 TO 20210702;REEL/FRAME:060473/0104 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |