Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
  • H04N13/117—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/139—Format conversion, e.g. of frame-rate or size
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/30—Image reproducers
  • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
  • H04N13/344—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/30—Image reproducers
  • H04N13/366—Image reproducers using viewer tracking
  • H04N13/383—Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/124—Quantisation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/167—Position within a video image, e.g. region of interest [ROI]
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
  • H04N19/174—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a slice, e.g. a line of blocks or a group of blocks
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
  • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/46—Embedding additional information in the video signal during the compression process
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
  • H04N19/625—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding using discrete cosine transform [DCT]

Definitions

• a virtual reality, or VR, scenario typically involves presentation of digital or virtual image information without transparency to other actual real- world visual input;
• an augmented reality, or AR, scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the actual world around the viewer.
• FIG. 1 an augmented reality scene 100 is depicted.
• the user of an AR technology sees a real-world park-like setting featuring people, trees, buildings in the background, and a concrete platform 120.
• the present invention relates generally to methods and systems related to projection display systems including wearable displays. More particularly, embodiments of the present invention provide methods and systems that combine the concept of foveation (i.e., reduced video quality at sections where the human eye is not focused) with the concept of compression.
• the invention is applicable to a variety of applications in computer vision and image display systems and light field projection systems, including stereoscopic systems, systems that deliver beamlets of light to the retina of the user, or the like.
• Numerous benefits are achieved by way of the present invention over conventional techniques.
• FIG. 1 illustrates a user's view of augmented reality (AR) through an AR device.
• FIG. 1 illustrates a user's view of augmented reality (AR) through an AR device.
• FIG. 2A illustrates a cross-sectional, side view of an example of a set of stacked waveguides that each includes an incoupling optical element.
• FIG. 2B illustrates a perspective view of an example of the one or more stacked waveguides of FIG. 2A.
• FIG. 2C illustrates a top-down, plan view of an example of the one or more stacked waveguides of FIGS.2A and 2B.
• FIG. 3 is a simplified illustration of an eyepiece waveguide having a combined pupil expander according to an embodiment of the present invention.
• FIG. 4 illustrates an example of wearable display system according to an embodiment of the present invention.
• FIG. 3 is a simplified illustration of an eyepiece waveguide having a combined pupil expander according to an embodiment of the present invention.
• FIG. 5 shows a perspective view of a wearable device according to an embodiment of the present invention.
• FIG. 6 is a diagram illustrating run length encoding of a quantized DCT block according to an embodiment of the present invention.
• FIG. 7 is a diagram illustrating a JPEG header structure.
• FIG. 8 is a line drawing illustrating an image compressed using a single quality setting.
• FIG. 9 is a line drawing illustrating a foveated image with three foveated regions according to an embodiment of the present invention.
• FIG. 10 is a line drawing illustrating a foveated image with post-processing in the foveated regions according to another embodiment of the present invention. [0019] FIG.
• FIG. 11 is a foveated 3D generated image with three foveated regions according to yet another embodiment of the present invention.
• FIG. 12 is a line drawing illustrating an image that can be utilized in conjunction with multiple foveation maps according to an embodiment of the present invention.
• FIG. 13 is a simplified flowchart illustrating a method of compressing an image according to an embodiment of the present invention.
• FIG. 14 is a simplified schematic diagram illustrating a gaze-based image foveation system according to an embodiment of the present invention.
• FIG. 15 illustrates a compression-level obtained as a function of time, represented by successive frames versus frequency, for both a sparsity compression system implementation and a DSC-SPARSE system implementation, according to an embodiment of the present invention.
• FIG. 16 illustrates a histogram of frame count versus compression for a sparsity compression system implementation and a DSC-SPARSE system implementation according to an embodiment of the present invention.
• FIG. 17 is a simplified flowchart illustrating a method of compressing image frames using an alternating compression algorithm according to an embodiment of the present invention.
• FIG. 18 is a simplified image illustrating an image frame divided into a high quality region and a low quality region according to an embodiment of the present invention.
• FIG. 19 is a simplified flowchart illustrating a method of compressing an image using different compression ratios for a high quality region and a low quality region, according to an embodiment of the present invention.
• FIG. 28 FIG.
• FIG. 20 is a simplified image illustrating an image frame divided into high quality tiles and low quality tiles according to an embodiment of the present invention.
• FIG. 21 is a simplified flowchart illustrating a method of compressing an image using different compression ratios for high quality tiles and low quality tiles, according to an embodiment of the present invention.
• FIG. 22 is a simplified block diagram illustrating components of an AR system according to an embodiment of the present invention. DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS [0031] Reference will now be made to the drawings, in which like reference numerals refer to like parts throughout. Unless indicated otherwise, the drawings are schematic not necessarily drawn to scale. [0032] With reference now to FIG.
• FIG. 2A illustrates a cross-sectional, side view of an example of a set 200 of stacked waveguides that each includes an incoupling optical element.
• the waveguides may each be configured to output light of one or more different wavelengths, or one or more different ranges of wavelengths.
• the illustrated set 200 of stacked waveguides includes waveguides 202, 204, and 206.
• Each waveguide includes an associated incoupling optical element (which may also be referred to as a light input area on the waveguide), with, e.g., incoupling optical element 203 disposed on a major surface (e.g., an upper major surface) of waveguide 202, incoupling optical element 205 disposed on a major surface (e.g., an upper major surface) of waveguide 204, and incoupling optical element 207 disposed on a major surface (e.g., an upper major surface) of waveguide 206.
• incoupling optical element 203 disposed on a major surface (e.g., an upper major surface) of waveguide 202
• incoupling optical element 205 disposed on a major surface (e.g., an upper major surface) of waveguide 204
• incoupling optical element 207 disposed on a major surface (e.g., an upper major surface) of waveguide 206
• one or more of the incoupling optical elements 203, 205, 207 may be disposed on the bottom major surface of the respective waveguides 202, 204, 206 (particularly where the one or more incoupling optical elements are reflective, deflecting optical elements). As illustrated, the incoupling optical elements 203, 205, 207 may be disposed on the upper major surface of their respective waveguide 202, 204, 206 (or the top of the next lower waveguide), particularly where those incoupling optical elements are transmissive, deflecting optical elements. In some embodiments, the incoupling optical elements 203, 205, 207 may be disposed in the body of the respective waveguide 202, 204, 206.
• the incoupling optical elements 203, 205, 207 are wavelength-selective, such that they selectively redirect one or more wavelengths of light, while transmitting other wavelengths of light. While illustrated on one side or corner of their respective waveguides 202, 204, 206, it will be appreciated that the incoupling optical elements 203, 205, 207 may be disposed in other areas of their respective waveguides 202, 204, 206 in some embodiments. [0034] As illustrated, the incoupling optical elements 203, 205, 207 may be laterally offset from one another. In some embodiments, each incoupling optical element may be offset such that it receives light without that light passing through another incoupling optical element.
• each incoupling optical element 203, 205, 207 may be configured to receive light from a different projector and may be separated (e.g., laterally spaced apart) from other incoupling optical elements 203, 205, 207 such that it substantially does not receive light from the other ones of the incoupling optical elements 203, 205, 207.
• Each waveguide also includes associated light distributing elements, with, e.g., light distributing elements 210 disposed on a major surface (e.g., a top major surface) of waveguide 202, light distributing elements 212 disposed on a major surface (e.g., a top major surface) of waveguide 204, and light distributing elements 214 disposed on a major surface (e.g., a top major surface) of waveguide 206.
• the light distributing elements 210, 212, 214 may be disposed on a bottom major surface of associated waveguides 202, 204, 206, respectively.
• the light distributing elements 210, 212, 214 may be disposed on both top and bottom major surfaces of associated waveguides 202, 204, 206, respectively; or the light distributing elements 210, 212, 214 may be disposed on different ones of the top and bottom major surfaces in different associated waveguides 202, 204, 206, respectively.
• the waveguides 202, 204, 206 may be spaced apart and separated by, e.g., gas, liquid, and/or solid layers of material. For example, as illustrated, layer 208 may separate waveguides 202 and 204; and layer 209 may separate waveguides 204 and 206.
• the layers 208 and 209 are formed of low refractive index materials (that is, materials having a lower refractive index than the material forming the immediately adjacent one of waveguides 202, 204, 206).
• the refractive index of the material forming the layers 208, 209 is 0.05 or more, or 0.10 or less than the refractive index of the material forming the waveguides 202, 204, 206.
• the lower refractive index layers 208, 209 may function as cladding layers that facilitate total internal reflection (TIR) of light through the waveguides 202, 204, 206 (e.g., TIR between the top and bottom major surfaces of each waveguide).
• the layers 208, 209 are formed of air. While not illustrated, it will be appreciated that the top and bottom of the illustrated set 200 of waveguides may include immediately neighboring cladding layers. [0037] Preferably, for ease of manufacturing and other considerations, the material forming the waveguides 202, 204, 206 are similar or the same, and the material forming the layers 208, 209 are similar or the same. In some embodiments, the material forming the waveguides 202, 204, 206 may be different between one or more waveguides, and/or the material forming the layers 208, 209 may be different, while still holding to the various refractive index relationships noted above. [0038] With continued reference to FIG.
• light rays 218, 219, 220 are incident on the set 200 of waveguides. It will be appreciated that the light rays 218, 219, 220 may be injected into the waveguides 202, 204, 206 by one or more projectors (not shown). [0039] In some embodiments, the light rays 218, 219, 220 have different properties, e.g., different wavelengths or different ranges of wavelengths, which may correspond to different colors.
• the incoupling optical elements 203, 205, 207 each deflect the incident light such that the light propagates through a respective one of the waveguides 202, 204, 206 by TIR.
• the incoupling optical elements 203, 205, 207 each selectively deflect one or more particular wavelengths of light, while transmitting other wavelengths to an underlying waveguide and associated incoupling optical element.
• incoupling optical element 203 may be configured to deflect ray 218, which has a first wavelength or range of wavelengths, while transmitting rays 219 and 220, which have different second and third wavelengths or ranges of wavelengths, respectively.
• the transmitted ray 219 impinges on and is deflected by the incoupling optical element 205, which is configured to deflect light of a second wavelength or range of wavelengths.
• the ray 220 is deflected by the incoupling optical element 207, which is configured to selectively deflect light of a third wavelength or range of wavelengths.
• the deflected light rays 218, 219, 220 are deflected so that they propagate through a corresponding waveguide 202, 204, 206; that is, the incoupling optical elements 203, 205, 207 of each waveguide deflects light into that corresponding waveguide 202, 204, 206 to in-couple light into that corresponding waveguide.
• the light rays 218, 219, 220 are deflected at angles that cause the light to propagate through the respective waveguide 202, 204, 206 by TIR.
• the light rays 218, 219, 220 propagate through the respective waveguide 202, 204, 206 by TIR until impinging on the waveguide's corresponding light distributing elements 210, 212, 214, where they are outcoupled to provide out-coupled light rays 216.
• FIG. 2B a perspective view of an example of the stacked waveguides of FIG. 2A is illustrated.
• the in-coupled light rays 218, 219, 220 are deflected by the incoupling optical elements 203, 205, 207, respectively, and then propagate by TIR within the waveguides 202, 204, 206, respectively.
• the light rays 218, 219, 220 then impinge on the light distributing elements 210, 212, 214, respectively.
• the light distributing elements 210, 212, 214 deflect the light rays 218, 219, 220 so that they propagate towards the outcoupling optical elements 222, 224, 226, respectively.
• the light distributing elements 210, 212, 214 are orthogonal pupil expanders (OPEs).
• OPEs deflect or distribute light to the outcoupling optical elements 222, 224, 226 and, in some embodiments, may also increase the beam or spot size of this light as it propagates to the outcoupling optical elements.
• the light distributing elements 210, 212, 214 may be omitted and the incoupling optical elements 203, 205, 207 may be configured to deflect light directly to the outcoupling optical elements 222, 224, 226.
• the light distributing elements 210, 212, 214 may be replaced with outcoupling optical elements 222, 224, 226, respectively.
• the outcoupling optical elements 222, 224, 226 are exit pupils (EPs) or exit pupil expanders (EPEs) that direct light to the eye of the user.
• the OPEs may be configured to increase the dimensions of the eye box in at least one axis and the EPEs may be configured to increase the eye box in an axis crossing, e.g., orthogonal to, the axis of the OPEs.
• each OPE may be configured to redirect a portion of the light striking the OPE to an EPE of the same waveguide, while allowing the remaining portion of the light to continue to propagate down the waveguide.
• another portion of the remaining light is redirected to the EPE, and the remaining portion of that portion continues to propagate further down the waveguide, and so on.
• a single beam of in-coupled light may be "replicated" each time a portion of that light is redirected by an OPE or EPE, thereby forming a field of cloned beams of light.
• the OPE and/or EPE may be configured to modify a size of the beams of light.
• the functionality of the light distributing elements 210, 212, and 214 and the outcoupling optical elements 222, 224, 226 are combined in a combined pupil expander as discussed in relation to FIG. 2E.
• the set 200 of waveguides includes waveguides 202, 204, 206; incoupling optical elements 203, 205, 207; light distributing elements (e.g., OPEs) 210, 212, 214; and outcoupling optical elements (e.g., EPs) 222, 224, 226 for each component color.
• the waveguides 202, 204, 206 may be stacked with an air gap/cladding layer between each one.
• the incoupling optical elements 203, 205, 207 redirect or deflect incident light (with different incoupling optical elements receiving light of different wavelengths) into its waveguide. The light then propagates at an angle which will result in TIR within the respective waveguide 202, 204, 206.
• light ray 218 e.g., blue light
• the first incoupling optical element 203 is deflected by the first incoupling optical element 203, and then continues to bounce down the waveguide, interacting with the light distributing element (e.g., OPEs) 210 and then the outcoupling optical element (e.g., EPs) 222, in a manner described earlier.
• the light distributing element e.g., OPEs
• the outcoupling optical element e.g., EPs
• the light rays 219 and 220 (e.g., green and red light, respectively) will pass through the waveguide 202, with light ray 219 impinging on and being deflected by incoupling optical element 205.
• the light ray 219 then bounces down the waveguide 204 via TIR, proceeding on to its light distributing element (e.g., OPEs) 212 and then the outcoupling optical element (e.g., EPs) 224.
• light ray 220 (e.g., red light) passes through the waveguide 206 to impinge on the light incoupling optical elements 207 of the waveguide 206.
• FIG. 2C illustrates a top-down, plan view of an example of the stacked waveguides of FIGS. 2A and 2B.
• the waveguides 202, 204, 206, along with each waveguide's associated light distributing element 210, 212, 214 and associated outcoupling optical element 222, 224, 226, may be vertically aligned.
• the incoupling optical elements 203, 205, 207 are not vertically aligned; rather, the incoupling optical elements are preferably nonoverlapping (e.g., laterally spaced apart as seen in the top- down or plan view). As discussed further herein, this nonoverlapping spatial arrangement facilitates the injection of light from different resources into different waveguides on a one- to-one basis, thereby allowing a specific light source to be uniquely coupled to a specific waveguide.
• FIG. 3 is a simplified illustration of an eyepiece waveguide having a combined pupil expander according to an embodiment of the present invention.
• the eyepiece 310 utilizes a combined OPE/EPE region in a single-side configuration.
• the eyepiece 310 includes a substrate 320 in which in- coupling optical element 322 and a combined OPE/EPE region 324, also referred to as a combined pupil expander (CPE), are provided.
• CPE combined pupil expander
• Incident light ray 330 is incoupled via the incoupling optical element 320 and outcoupled as output light rays 332 via the combined OPE/EPE region 324.
• the combined OPE/EPE region 324 includes gratings corresponding to both an OPE and an EPE that spatially overlap in the x-direction and the y-direction.
• the gratings corresponding to both the OPE and the EPE are located on the same side of a substrate 320 such that either the OPE gratings are superimposed onto the EPE gratings or the EPE gratings are superimposed onto the OPE gratings (or both).
• the OPE gratings are located on the opposite side of the substrate 320 from the EPE gratings such that the gratings spatially overlap in the x-direction and the y-direction but are separated from each other in the z-direction (i.e., in different planes).
• the combined OPE/EPE region 324 can be implemented in either a single-sided configuration or in a two- sided configuration.
• FIG. 4 illustrates an example of wearable display system 430 into which the various waveguides and related systems disclosed herein may be integrated.
• the display system 430 includes a display 432, and various mechanical and electronic modules and systems to support the functioning of that display 432.
• the display 432 may be coupled to a frame 434, which is wearable by a display system user 440 (also referred to as a viewer) and which is configured to position the display 432 in front of the eyes of the user 440.
• the display 432 may be considered eyewear in some embodiments.
• a speaker 436 is coupled to the frame 434 and configured to be positioned adjacent the ear canal of the user 440 (in some embodiments, another speaker, not shown, may optionally be positioned adjacent the other ear canal of the user to provide stereo/shapeable sound control).
• the display system 430 may also include one or more microphones or other devices to detect sound.
• the microphone is configured to allow the user to provide inputs or commands to the system 430 (e.g., the selection of voice menu commands, natural language questions, etc.), and/or may allow audio communication with other persons (e.g., with other users of similar display systems).
• the microphone may further be configured as a peripheral sensor to collect audio data (e.g., sounds from the user and/or environment).
• the display system 430 may further include one or more outwardly directed environmental sensors configured to detect objects, stimuli, people, animals, locations, or other aspects of the world around the user.
• environmental sensors may include one or more cameras, which may be located, for example, facing outward so as to capture images similar to at least a portion of an ordinary field of view of the user 440.
• the display system may also include a peripheral sensor, which may be separate from the frame 434 and attached to the body of the user 440 (e.g., on the head, torso, an extremity, etc. of the user 440).
• the peripheral sensor may be configured to acquire data characterizing a physiological state of the user 440 in some embodiments.
• the sensor may be an electrode.
• the display 432 is operatively coupled by a communications link, such as by a wired lead or wireless connectivity, to a local data processing module which may be mounted in a variety of configurations, such as fixedly attached to the frame 434, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user 440 (e.g., in a backpack-style configuration, in a belt-coupling style configuration).
• the sensor may be operatively coupled by a communications link, e.g., a wired lead or wireless connectivity, to the local processor and data module.
• the local processing and data module may comprise a hardware processor, as well as digital memory, such as non- volatile memory (e.g., flash memory or hard disk drives), both of which may be utilized to assist in the processing, caching, and storage of data.
• the local processor and data module may include one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, and so on.
• the data may include data a) captured from sensors (which may be, e.g., operatively coupled to the frame 434 or otherwise attached to the user 440), such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, gyros, and/or other sensors disclosed herein; and/or b) acquired and/or processed using remote processing module 452 and/or remote data repository 454 (including data relating to virtual content), possibly for passage to the display 432 after such processing or retrieval.
• sensors which may be, e.g., operatively coupled to the frame 434 or otherwise attached to the user 440
• image capture devices such as cameras
• microphones such as inertial measurement units
• accelerometers compasses
• GPS units GPS units
• radio devices radio devices
• gyros gyros
• the local processing and data module may be operatively coupled by communication links 438 such as via wired or wireless communication links, to the remote processing and data module 450, which can include the remote processing module 452, the remote data repository 454, and a battery 460.
• the remote processing module 452 and the remote data repository 454 can be coupled by communication links 456 and 458 to remote processing and data module 450 such that these remote modules are operatively coupled to each other and available as resources to the remote processing and data module 450.
• the remote processing and data module 450 may include one or more of the image capture devices, microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, and/or gyros.
• the remote processing and data module 450 may comprise one or more processors configured to analyze and process data and/or image information, for instance including one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, and so on.
• the remote data repository 454 may comprise a digital data storage facility, which may be available through the internet or other networking configuration in a "cloud" resource configuration.
• the remote data repository 454 may include one or more remote servers, which provide information, e.g., information for generating augmented reality content, to the local processing and data module and/or the remote processing and data module 450.
• information e.g., information for generating augmented reality content
• all data is stored and all computations are performed in the local processing and data module, allowing fully autonomous use from a remote module.
• an outside system e.g., a system of one or more processors, one or more computers
• CPUs, GPUs, and so on may perform at least a portion of processing (e.g., generating image information, processing data) and provide information to, and receive information from, the illustrated modules, for instance, via wireless or wired connections.
• Wearable device 500 includes a frame 502 configured to support one or more projectors 504 at various positions along an interior-facing surface of frame 502, as illustrated.
• projectors 504 can be attached at positions near temples 506.
• another projector could be placed in position 508.
• Such projectors may, for instance, include or operate in conjunction with one or more liquid crystal on silicon (LCoS) modules, micro-LED displays, or fiber scanning devices.
• LoS liquid crystal on silicon
• micro-LED displays micro-LED displays
• fiber scanning devices a light from projectors 504 or projectors disposed in positions 508 could be guided into eyepieces 510 for display to eyes of a user.
• Projectors placed at positions 512 can be somewhat smaller on account of the close proximity this gives the projectors to the waveguide system. The closer proximity can reduce the amount of light lost as the waveguide system guides light from the projectors to eyepiece 510.
• the projectors at positions 512 can be utilized in conjunction with projectors 504 or projectors disposed in positions 508. While not depicted, in some embodiments, projectors could also be located at positions beneath eyepieces 510.
• Wearable device 500 is also depicted including sensors 514 and 516. Sensors 514 and 516 can take the form of forward-facing and lateral-facing optical sensors configured to characterize the real-world environment surrounding wearable device 500.
• Embodiments of the present invention utilize an eye tracking system to determine the eye gaze location of the user and utilize the eye gaze location for image compression processes.
• eye tracking cameras 505 are located on the frame 502 and can be utilized to track the eye gaze location of the user using the wearable device 500.
• other eye tracking systems are utilized to determine the eye gaze location and the eye tracking cameras 505 illustrated in FIG. 5 are merely exemplary.
• the image compression processes utilized to compress and decompress virtual content for storage in memory, internal communications, and display can be modified depending on the eye gaze location, for example, portions of an image or video stream corresponding to the eye gaze location can be compressed using a higher quality compression process compared to other portions of the image or video stream that are located more distant from the eye gaze location. Since these more distant portions of the image or video stream are in the user's peripheral vision, any impact on the user experience resulting from the reduction in compression quality can be less than the benefits achieved in terms of memory and processing efficiency and/or requirements.
• One of ordinary skill in the art would recognize many variations, modifications, and alternatives.
• image compression e.g., JPEG compression
• JPEG JPEG is a derivative of JPEG
• embodiments of the present invention are applicable to MPEG compression processes as appropriate.
• the quality i.e., the bandwidth
• embodiments of the present invention can reduce the quality (i.e., the bandwidth) at locations in an image where the user is not looking, i.e., locations in the image that are spatially separated from the eye gaze location, thereby decreasing the image quality in these regions and decreasing the overall need to send something at a superior quality setting that the human eye would not be able to discern, because the human eye is not currently focused on these non-gaze locations.
• embodiments of the present invention provide a video compression algorithm that takes human gaze into account and creates a foveated compression algorithm dependent on human gaze.
• the JPEG algorithm receives an image and segments it into macro-blocks (e.g., 16 pixels x 16 pixels). These macro-blocks are then subjected to a discrete cosine transform (DCT) process.
• DCT discrete cosine transform
• the DCT process generates a set of coefficients, which are filtered so that the high frequency values are eliminated (this is where the quality step resides). After this process occurs, the block is then run length encoded.
• Encoder Based Foveation Map [0056] Table 1 is a matrix illustrating an 8 x 8 pixel sub-image block according to an embodiment of the present invention.
• the 8 x 8 pixel sub-image block can also be referred to as a macro-block or a tile.
• the 8 x 8 pixels are represented by the pixel values illustrated in the matrix.
• Table 1 [0057]
• Table 2 is a matrix illustrating an example of an encoded 8 x 8 FDCT block according to an embodiment of the present invention.
• JPEG/MPEG compressions process a whole image at a fixed quality.
• the process of filtering results in the generation of the zero data illustrated in the quantized DCT block illustrated in Table 3. This filter occurs with a given quality setting.
• the magnitude of values generally decreases from the upper left portion of the matrix to the lower right portion of the matrix.
• Table 2 [0058]
• Table 3 is a matrix illustrating an example of a quantized DCT block according to an embodiment of the present invention. In Table 3, quantization results in a significant number of the values being reduced to zero.
• Table 3 [0059]
• FIG. 6 is a diagram illustrating run length encoding of a quantized DCT block according to an embodiment of the present invention. In order to encode the quantized DCT block, a run length encoding process starts at the upper left pixel and progresses to the lower right pixel. Referring to FIG. 6, pixel 610 is encoded, followed by the encoding of pixel 612. Next, encoding progresses to the next two rows of pixels, resulting in the encoding of pixel 614 and pixel 616.
• FIG. 7 is a diagram illustrating a JPEG header structure. As illustrated in FIG. 7, in the JPEG header structure, the default quality for the entire image is stored in the quantization table map area. Thus, a single quality setting is used to compress the entire image.
• the quantization table can apply to the unfoveated region(s), providing a 100% quality setting for regions corresponding to the location of the eye gaze, or the quantization table can apply to the regions more distance from the location of the eye gaze, providing a reduced quality setting for the foveated region.
• the Segments include Start of Image, Application0 (Default Header), Define Quantization Table (for luminance), Defined Quantization Table (for chrominance), Start of Frame, Define Huffman Table 1, Define Huffman Table 2, Define Huffman Table 3, Define Huffman Table 4, Start of Scan, Image Data (entropy-coded segment), and End of Image.
• Embodiments of the present invention maintain high quality on the blocks that the eye is focused on while reducing the quality setting on the blocks of the image that the eye is not focused on. These different quality settings are stored in a foveation map. Therefore, a foveation map can be passed to the compression engine. In turn, the compression engine can selectively alter predetermined video blocks corresponding to the eye gaze location in order to compress these predetermined video blocks with high quality, while other blocks can be compressed with low quality. [0064]
• the foveation map can be created based on eye gaze information, namely, by being able to actively tell where the human eye is currently focused or looking.
• the foveation map is supplied to the encoder and passed to the decoder.
• An added benefit provided by embodiments of that present invention is that, by using the concept of video blocks, the blocks with zero data (i.e., that are all black) will consume reduced memory space or power during the video display process.
• embodiments of the present invention utilize a video block compression algorithm that is modified to implement a variable quality per block.
• Decoder Based Foveation Map [0067] The decoder can use the current stream of DCT coefficients, which are included as part of the compression standard, that were passed to it. Therefore, some blocks would have more coefficients and some blocks would have fewer coefficients.
• the foveation map can be sent or passed along to the decoder so that the decoder will be able to use the locations of the reduced quality blocks/tile locations.
• the foveation map is used by the decoder to apply the desired quality setting to each tile/block. Additionally, this information can be used in order to apply a post processing image filtration in order to remove JPEG low quality artifacts.
• Map Implementation It should be noted that a particular implementation could have an inferred 100% quality and utilize the global table as the alternate table, or vice versa. Embodiments of the present invention can utilize a variety of mechanisms for implementing the quality map selection.
• FIG. 8 is a line drawing illustrating an image compressed using a single quality setting.
• FIG. 9 is a line drawing illustrating a foveated image with three foveated regions according to an embodiment of the present invention.
• the image in FIG. 9 is divided into multiple regions based on the eye gaze location. In this case, the user is gazing at the center of the image resulting in the eye gaze location being located at the center of the image.
• the eye gaze location can be determined using an eye tracking system as discussed in relation to FIGS. 5 and 22. Accordingly, the image can be divided into a central region corresponding to the eye gaze location and peripheral regions that are more distant from the eye gaze location.
• a foveation map is created based on the eye gaze location, with portions of the image close to the eye gaze location mapping to high quality settings and portions of the image more distant from the eye gaze location mapping to lower quality settings.
• the foveation map takes the form of two peripheral regions with a lower quality setting and a central region with a higher (e.g., 100%) quality setting.
• region 910 corresponding to the left quarter of the image (i.e., the left 1/4), has been compressed using a first quality setting.
• region 930 corresponding to the right quarter of the image (i.e., the right 1/4), has been compressed using the first quality setting.
• region 920 corresponding to the middle half of the image (i.e., the center 2/4), has been compressed using a second quality setting higher than the first quality setting.
• This division of the image into portions can be referred to as a tri-region division: left quarter (e.g., foveated at 70% quality setting), center half (e.g., un-foveated at 100% quality setting), and right quarter (e.g., foveated at 70% quality setting).
• the present invention is not limited to this implementation and the image can be divided in other manners.
• the quality setting for individual blocks or tiles e.g., 8 x 8 pixel blocks for JPEG compression
• the quality setting for individual blocks or tiles e.g., 8 x 8 pixel blocks for JPEG compression
• the blocks in region 910 are assigned a first quality setting (e.g., 70%)
• the blocks in region 920 are assigned a second quality setting (e.g., 100%)
• the blocks in region 930 are assigned the first quality setting (e.g., 70%)
• the foveation map can be more complex than the three region division illustrated in FIG. 9.
• a foveation map in which blocks in the peripheral regions are assigned quality settings that depend on the distance of the block from the eye gaze location while blocks in the central region have a uniform quality setting.
• the foveation map can be defined such that blocks in the peripheral regions are assigned a uniform quality setting while blocks in the central region are assigned quality settings that depend on the distance of the block from the eye gaze location.
• blocks in the central region are assigned quality settings that depend on the distance of the block from the eye gaze location.
• regions 920 i.e., the un- foveated section.
• the region that is unfoveated i.e., uncompressed or compressed using a lossless compression algorithm
• the foveation map could compress the right side using a higher quality setting and the left side of the image using a lower quality setting.
• region 910 and region 920 would be compressed using a first quality setting and region 930 would be compressed using a second quality setting higher than the first quality setting.
• region 930 could be compressed using a higher quality setting, for instance, a lossless compression
• region 920 could be compressed with an intermediate quality setting lower than the higher quality setting
• region 910 could be compressed using a lowest quality setting lower than the intermediate quality setting.
• the foveation of the image is a function of the eye gaze location, compressing or encoding the region including the eye gaze location with a higher quality setting than one or more regions more distant from the eye gaze location.
• a set of vertical regions is illustrated in FIG.
• FIG. 10 is a second foveated image with post-processing in the foveated regions according to another embodiment of the present invention. After post-processing of the image illustrated in FIG. 9, the blurring of the image content in the foveated regions, i.e., region 910 and region 930, reduces the artifacts present in these regions.
• FIG. 11 is a foveated 3D generated image with three foveated regions according to yet another embodiment of the present invention. In FIG.
• the regions are defined in a manner similar to that illustrated in FIGS. 9 and 10. However, the compression can be much higher since, for the 3D generated image, large portions of the image are black. Using the methods described herein, 87% compression was achieved while maintaining 100% quality in the center of the image corresponding to the eye gaze location.
• region 1120 was compressed using a 100% quality setting (un-foveated at 100% quality setting) while region 1110 and region 1130 were compressed at lower quality settings (foveated at 20% quality setting). Since, for many instances of virtual content, the image content is highest near the eye gaze location and peripheral regions are dark or black, embodiments of the present invention are particularly well suited for use with virtual reality and augmented reality implementations.
• all regions of the image can be compressed using the lower quality settings and the unfoveated region compressed with the higher quality setting.
• regions 910, 920, and 930 can each be compressed using the low quality setting of the foveated regions.
• the region 920 can also be compressed using the high quality setting.
• two decoders can be used to decode the compressed image.
• the decoded region 920 using the high quality settings can be overlaid on the decoded regions 910, 920, 930 (i.e., the entire image) using the low quality settings.
• FIG. 12 is a line drawing illustrating an image that can be utilized in conjunction with multiple foveation maps according to an embodiment of the present invention.
• an image is represented that includes a person 1206 located in section 1210, a tree 1202 located in sections 1220, 1222, 1230, 1232, and a house 1204 located in sections 1224, 1226, 1238, and 1240.
• different foveation maps can be created based on this image.
• a foveation map can be utilized in which the blocks in sections 1220, 1222, 1230, and 1232 are compressed using a 100% quality setting (un-foveated at 100% quality setting) while the blocks in the remaining sections (i.e., sections 1210, 1212, 1214, 1216, 1224, 1226, 1228, 1234, 1236, 1238, 1240, and 1242 are compressed using a lower quality settings (foveated at 70% quality setting).
• compression of the image can be implemented using a foveation map that maintains the quality in the region of the image corresponding to the eye gaze location and peripheral portions of the image can be compressed using a lower quality setting to save system resources including memory and processing.
• a foveation map can be utilized in which the blocks in sections 1224, 1226, 1238, and 1240 are compressed using a 100% quality setting (un-foveated at 100% quality setting) while the blocks in the remaining sections (i.e., sections 1210, 1212, 1214, 1216, 1220, 1222, 1228, 1230, 1232, 1234, and 1236, and 1242 are compressed using a lower quality settings (foveated at 70% quality setting).
• a foveation map can be utilized in which the blocks in section 1210 are compressed using a 100% quality setting (un-foveated at 100% quality setting) while the blocks in the remaining sections (i.e., sections 1212, 1214, 1216, 1220, 1222, 1224, 1226, 1228, 1230, 1232, 1234, and 1236, 1238, 1240, and 1242 are compressed using a lower quality settings (foveated at 70% quality setting).
• the quality settings used for the remaining sections are varied, for example, as a function of distance from the eye gaze location.
• blocks in sections 1212, 1214, and 1216 could be compressed using a quality setting of 90%
• blocks in sections 1220, 1222, 1224, 1226, and 1228 could be compressed using a quality setting of 80%
• blocks in sections 1230, 1232, 1234, and 1236, 1238, 1240, and 1242 could be compressed using a quality setting of 70%.
• the sections 1210-1242 may be compressed using techniques including DSC or VDC- X (e.g., using compression ratios).
• FIG. 13 is a simplified flowchart illustrating a method of compressing an image according to an embodiment of the present invention.
• the method 1300 includes receiving an image (1310), determining an eye gaze location of a user (1312), and generating a foveation map based on the eye gaze location (1314).
• the image may be an image included in a video stream. Determining the eye gaze location of the user can utilize an eye tracking system that provides the eye gaze location as a function of time.
• the foveation map defines the quality with which blocks are compressed and varies as a function of position in the image, with blocks in region(s) close to the eye gaze location being compressed using a higher quality setting and blocks in region(s) more distant from the eye gaze location being compressed using a lower quality setting.
• three regions are included in the foveation map, but the present invention is not limited to this particular implementation and two regions or more than three regions can be defined.
• the blocks in a given region can be compressed using a uniform quality setting or can be compressed with different quality settings depending on the particular implementation.
• the foveation map includes a first region of the image and a second region of the image.
• the method also includes compressing the first region of the image using a first quality setting and the second region of the image using a second quality setting (1316).
• the first quality setting is an uncompressed quality setting or lossless compression quality setting.
• the second quality setting is a lower quality setting, for example, a 70% quality setting that reduces the data corresponding to the compressed image in these regions.
• any loss in quality is offset by the savings in memory and processor usage.
• the data compression processes for the first region and the second region can be performed sequentially or in parallel, depending on the particular application.
• the compressed image or video which can be referred to as a foveated image or video, can be transmitted to a display system, along with the foveation map (1318), or can be stored in memory, along with the foveation map (1319).
• the method 1300 includes retrieving the foveated image and the foveation map from memory (1320) and decompressing the first region of the image using the first quality setting and the second region of the image using the second quality setting (1340).
• the method 1300 includes receiving the foveated image and the foveation map (1320) and decompressing the first region of the image using the first quality setting and the second region of the image using the second quality setting (1340).
• the decompression processes for the first region and the second region can be performed sequentially or in parallel, depending on the particular application.
• the two regions can be merged to form the final image suitable for display (1342).
• the final image is then displayed on the display device (1344).
• FIG. 14 is a simplified schematic diagram illustrating a gaze-based image foveation system according to an embodiment of the present invention.
• the gaze- based image foveation system 1400 includes a wearable 1410 (e.g., a wearable including an ASIC that performs the illustrated operations) that receives an image or a video suitable for display to a user.
• a wearable 1410 e.g., a wearable including an ASIC that performs the illustrated operations
• the image or video can be received using one or more communication interfaces 1420.
• WiFi, USB, DisplayPort (DP) or other communication protocols are utilized to receive the image or video content.
• the uncompressed content is MPEG video.
• the wearable 1410 also receives eye gaze information from an eye tracking system 1405.
• the eye tracking system 1405 can include one or more sensors suitable for measuring eye position and orientation and can provide data that can be utilized by eye gaze processor 1430 in calculating the user's eye gaze.
• the eye gaze processor 1430 is implemented using a CPU or neural processing unit (NPU) controller, although other processors can be utilized.
• NPU neural processing unit
• image compression processor 1422 in some embodiments, which implements a process to form a compressed image/video (e.g., a foveated image/video) based on the user's eye gaze as discussed more fully herein.
• a compressed image/video e.g., a foveated image/video
• Different foveation processes can be utilized as appropriate to the particular application, including tile-based foveation processes such as JPEG or DSC foveation processes as discussed more fully herein, sparsity-based compression processes, or the like.
• image compression processor 1422 is bypassed, for example, if the image was remoted compressed before being received by one or more communication interfaces 1420, and the image or video is passed to memory 1424 for storage.
• an image decompression process can be performed using decompression processor 1426 and the eye gaze information provided by eye gaze processor 1430.
• the decompression processor 1426 can decode the compressed image.
• the original or reconstructed image is then passed to warp / depth reprojection processor 1428.
• data provided by the eye gaze processor 1430 can be utilized once again to compress the warped image using variable quality encoder 1432 including processor component 1431 that represents image foveation based on eye gaze location.
• variable quality encoder 1432 including processor component 1431 is bypassed.
• a JPEG encoding process can be performed by variable quality encoder 1432 to form foveated images based on eye gaze in which the quality of the image varies across the image, providing high quality in the region of the image corresponding to the user's eye gaze and reduced quality in regions of the image more distant from the eye gaze location.
• foveated, as well as sparsity encoded images can be formed with reduced size while maintaining desired image quality.
• the encoded image is then provided to a mobile interface processor interface (MIPI) device 1434 for subsequent transmission to the display system.
• MIPI mobile interface processor interface
• the MIPI device 1434 of wearable 1410 can be connected to MIPI device 1442 of a display system 1440 that includes a variable quality decoder 1444 including a processor component 1443 that performs defoveation based on eye gaze location and a display device 1446, for example, an LCOS display or a micro-light emitting diode ( ⁇ LED) display.
• a variable quality decoder 1444 including a processor component 1443 that performs defoveation based on eye gaze location and a display device 1446, for example, an LCOS display or a micro-light emitting diode ( ⁇ LED) display.
• ⁇ LED micro-light emitting diode
• the JPEG/DSC tile-based encoded data or the N-way compression based encoded data can be received in a first communications channel and the quality map (Q-map), e.g., the foveation map, can be received in a second communications channel for use during the decoding process.
• the Q-map can be received using an embedded line format or other suitable format.
• the JPEG decoding process can be performed by variable quality decoder 1444 including a processor component 1443 to form final images based on the foveated images produced by variable quality encoder 1432 including a processor component 1431.
• embodiments of the present invention reduce system memory and transmission requirements, for example, the amount of data transmitted between the MIPI devices while maintaining desired image quality.
• the decoded image is then displayed using display device 1446.
• variable quality encoder 1432 is bypassed and the warped image is transmitted to the display system 1440 using MIPI device 1434 without variable quality image compression.
• the variable quality decoder 1444 is also bypassed.
• a tile-based (also referred to as a block-based) JPEG compression algorithm is utilized in the embodiments illustrated above, embodiments of the present invention are not limited to this particular compression standard and other compression standards can be utilized in conjunction with various embodiments of the present invention. As an example, FIGS.
• FIG. 15 illustrates a compression-level obtained as a function of time, represented by successive frames versus frequency, for both a sparsity compression system implementation and a DSC-SPARSE system implementation, according to an embodiment of the present invention.
• each frame was compressed using either the mask-based compression method or DSC in accordance with the alternating algorithm that implements either the mask-based compression method or the complete frame fixed compression, for example, DSC.
• each frame is analyzed and the number of lines having pixels characterized by a brightness level less than a threshold is determined.
• the frame is compressed using the mask-based compression method. In FIG. 15, this results in the first ⁇ 3800 frames being compressed using the mask-based compression method.
• a compression threshold e.g., 37%
• the DSC method is utilized. This results in these frames having a 37% compression value. Referring to FIG. 15, the frames represented by blue compression values less than 37% are compressed using DSC, effectively baselining the minimum compression at 37%.
• FIG. 16 illustrates a histogram of frame count versus compression for a sparsity compression system implementation and a DSC-SPARSE system implementation according to an embodiment of the present invention. As illustrated in FIG. 16, the number of frames with compression less than ⁇ 37% is reduced to zero since either the mask-based compression method was utilized for frames that could be compressed with a compression level greater than 37% or the frame-based compression method (e.g., DSC) was utilized for the remaining frames that could not be compressed with a compression level greater than 37% using the mask-based compression method.
• the frame-based compression method e.g., DSC
• the alternating method provided by embodiments of the present invention limits the lowest compression level to ⁇ 37% as illustrated in FIG. 16.
• the mask-based compression method provides high levels of compression while for frames with limited black pixel content, the frame-based compression method establishes a floor for the compression level, for example, 37% in this illustrated embodiment.
• the minimum compression level does not need to be 37%, which is merely exemplary and other minimum compression levels can be utilized depending on the particular application.
• One of ordinary skill in the art would recognize many variations, modifications, and alternatives.
• FIG. 17 is a simplified flowchart illustrating a method of compressing image frames using an alternating compression algorithm according to an embodiment of the present invention.
• the method 1700 includes receiving a frame of video data (1710).
• the method also includes determining a number of lines in the frame having pixel groups characterized by a brightness level less than a threshold (1712).
• a threshold 17.
• embodiments of the present invention alternate between compression methods for each frame depending on the level of compression that can be achieved by each compression method.
• FIG. 17 provides a particular method of compressing image frames using an alternating compression algorithm according to an embodiment of the present invention. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Moreover, the individual steps illustrated in FIG.
• 17 may include multiple sub-steps that may be performed in various sequences as appropriate to the individual step. Furthermore, additional steps may be added or removed depending on the particular applications.
• an embedded image-line control or alternate control mechanism that, per frame, would provide information to the endpoint display related to which system to use to decode the incoming MIPI frame.
• virtual MIPI channels could be utilized to indicate the compression ratio used by the endpoint display.
• Some embodiments of the present invention alter the compression quality based on eye tracking, thus giving the foveated regions a higher compression ratio at a loss of quality.
• Embodiments of the present invention reduce the amount of stream-based data sent over MIPI compression that occurs. Moreover, embodiments alter the compression quality based on eye tracking, thus giving the foveated regions a higher compression ratio at a loss of quality. Furthermore, embodiments allow for a higher compression ratio for steam-based compression techniques, and allow for quality to be preserved for the areas being observed by the user. As a result, embodiments allow for a much higher compression ratio while preserving quality.
• FIG. 18 is a simplified image illustrating an image frame divided into a high quality region and a low quality region according to an embodiment of the present invention.
• the image 1800 illustrated in FIG. 18 includes a high quality region 1810 and a low quality region 1820.
• the high quality region 1810 will be compressed and decompressed using a first quality setting or compression level and the low quality region 1820, or the entire image, will be compressed and decompressed using a second quality setting or compression level providing memory savings and other benefits.
• a single decoder can be utilized by not compressing the high quality region 1810 and compressing the low quality region using the single decoder. If the high quality region 1810 is small compared to the entire image, significant savings can be achieved. Additional description related to varying the size of the high quality region is provided in U.S. Provisional Patent Application No. 63/543,876, filed on October 12, 2023, the disclosure of which is hereby incorporated by reference in its entirety for all purposes. [0113] DSC [0114] Conventional DSC does not provide for variable quality compression. Rather, DSC takes a 24 bit color encoding and compresses it down to 15/12/10/8 bits. The higher the compression (24 ⁇ 8 bpp), the worse the impact to quality.
• embodiments are able to maintain, for example, a PSNR quality setting above 60dB as discussed above. From the use case analysis illustrated in FIG. 6, the inventors have determined that this only occurs at a 37% compression configuration (24 ⁇ 15 bpp). However, only the area in which the eye is currently focused on actually utilizes that compression setting.
• the outer foveated region e.g., the portion of the image more distant from the eye gaze location
• can afford to have a lower quality for example, 75% compression level (24 ⁇ 8 bpp).
• embodiments divide the main screen into a high quality region and low quality region (as shown in FIG. 18) or smaller sections (as shown in FIG. 20), each with a different compression ratio.
• the selected compression ratio will be a function of the current eye gaze location.
• the high quality region 1810 can be compressed with the lower compression level (e.g., 24 ⁇ 15 bpp), and the low quality region 1820 can be compressed with a higher compression level (e.g., 24 ⁇ 8 bpp).
• the low quality region 1820 can be compressed with an even higher compression level (e.g., 24 ⁇ 6 bpp). In embodiments in which the entire image is compressed using the higher compression level as described more fully herein, the high quality region 1810 can be overlaid on the entire image when the image is reconstructed.
• FIG. 19 is a simplified flowchart illustrating a method 1900 of compressing an image using different compression ratios for a high quality region and a low quality region, according to an embodiment of the present invention.
• the method 1900 includes determining an eye gaze location of a user (1910), generating a foveation map including a first region of an image and a second region of an image (1912), and compressing the first region using a first compression ratio and compressing the second region with a second compression ratio (1914).
• the image may be an image included in a video stream. Determining the eye gaze location of the user can utilize an eye tracking system that provides the eye gaze location as a function of time.
• the foveation map defines the compression ratio with which portions of the image are compressed and varies as a function of position in the image with respect to the eye gaze location, with region(s) close to the eye gaze location being compressed using a lower compression ratio and region(s) more distant from the eye gaze location being compressed using a higher compression ratio.
• two regions are included in the foveation map, but the present invention is not limited to this particular implementation and three regions or more than three regions can be defined.
• the foveation map includes a first region of the image and a second region of the image.
• the method 1900 may be referred to as an N-way compression (e.g., DSC, VDC- X, or JPEG), where N refers to the number of regions determined for the image. For example, based on the eye gaze location, a high quality region, a medium quality region surrounding the high quality region, and a low quality region can be determined for the image. The techniques of method 1900 can then be used as a 3-way compression, with different compression ratios for each region. [0118] Referring back to FIG. 18, in some examples the low quality region 1820 can encompass the entire image, including the portion of the image in the high quality region 1810 characterized by the eye gaze location.
• N refers to the number of regions determined for the image. For example, based on the eye gaze location, a high quality region, a medium quality region surrounding the high quality region, and a low quality region can be determined for the image.
• the techniques of method 1900 can then be used as a 3-way compression, with different compression ratios for each region.
• the low quality region 1820 can encompass
• the low quality region 1820 may be considered as the entire image.
• the low quality region 1820 may be the entire 4 megapixel image and may be compressed using a high compression level (e.g., 24 ⁇ 8 bpp).
• the high quality region 1810 may be determined based on the current eye gaze location and may be, for example, a 1 kilopixel by 1 kilopixel region (1 megapixel total).
• FIG. 20 is a simplified image illustrating an image frame divided into high quality sections and low quality sections according to an embodiment of the present invention.
• the sectioned image frame 2000 illustrated in FIG. 20 can be utilized to define a foveation map that defines the compression ratio with which different sections of the image are compressed in such a manner that the compression ratio or other compression quality metric varies as a function of position in the image with respect to the eye gaze location.
• sections close to the eye gaze location can be compressed using a lower compression ratio and sections that are more distant from the eye gaze location can be compressed using a higher compression ratio.
• the four sections 2010, 2012, 2014, and 2016 including the high quality region 2002 i.e., the region corresponding to the current eye gaze location
• a lower compression level e.g., 24 ⁇ 15 bpp
• peripheral sections or low quality sections e.g., 24 ⁇ 8 bpp
• all sections 2010-2046 of the image may be compressed at the high compression ratio (e.g., 24 ⁇ 8 bpp).
• the four sections 2010, 2012, 2014, and 2016 including the high quality region can also be compressed with a lower compression ratio (e.g., 24 ⁇ 15 bpp).
• the foveation map may define sections that are coincident with the high quality region.
• sections 2010-2016 may include only the high quality region characterized by the eye gaze location, without including portions of the image in the low quality regions.
• DSC decoders it may be desirable to use multiple DSC decoders to decode the compressed image in the section-based DSC technique.
• four DSC decoders can be used to decode the compressed image, with one decoder used to decode the high quality sections 2010-2016, another decoder used to decode the sections 2020-2026, a third decoder used to decode the sections 2030-2036, and a fourth decoder used to decode the sections 2040-2046, with each decoder using a compression ratio for each group of sections based on proximity to the eye gaze location.
• a single decoder may be implemented with acceptable latency when decoding the compressed image.
• the image may be an image included in a video stream. Determining the eye gaze location of the user can utilize an eye tracking system that provides the eye gaze location as a function of time.
• the foveation map defines the compression ratio with which different sections (e.g., sections 2010-2016, sections 2020-2026, sections 2030-2036, and sections 2040-2046) of the image are compressed and varies as a function of position in the image with respect to the eye gaze location, with sections close to the eye gaze location being compressed using a lower compression ratio and sections more distant from the eye gaze location being compressed using a higher compression ratio.
• sections e.g., sections 2010-2016, sections 2020-2026, sections 2030-2036, and sections 2040-2046
• 16 sections are included in the foveation map, but the present invention is not limited to this particular implementation and more or fewer than 16 sections can be defined.
• the methods described herein may be referred to as section-based compression (e.g., DSC, VDC-X, or JPEG) methods.
• sections 2010-2014 could be compressed using a 37% compression level (i.e., 24 ⁇ 15 bpp) while sections 2020, 2022, 2024, and 2026, which are more distant from the high quality region, could be compressed using a 50% compression level (i.e., 24 ⁇ 15 bpp), sections 2030, 2032, 2034, and 2036, which are more distant from the high quality region than sections 2020-2026, could be compressed using a 58% compression level (i.e., 24 ⁇ 12 bpp), and sections 2040, 2042, 2044, and 2046, which are most distant from the high quality region than sections 2010-2016, could be compressed using a 67% compression level (i.e., 24 ⁇ 8 bpp).
• the compression level may be 0%, i.e., uncompressed, including sections corresponding to the eye gaze location and high quality region.
• the compressed image could have uncompressed sections as well as compressed sections.
• One of ordinary skill in the art would recognize many variations, modifications, and alternatives.
• sixteen uniform area sections are illustrated in FIG. 20, this is not required and other numbers of sections, including sections with differing sizes can be utilized, with smaller sections adjacent the high quality region and larger sections, for example, sections compressed at higher levels, at greater distances from the high quality region.
• the number of compression levels, the levels of compression, the number of the sections, and the sizes of the sections can be varied as appropriate to the particular application.
• FIG. 21 is a simplified flowchart illustrating a method 2100 of compressing an image using different compression ratios for high quality sections and low quality sections, according to an embodiment of the present invention.
• the method 2100 includes determining an eye gaze location of a user (2110), generating a foveation map including first sections of an image and second sections of an image (2112), and compressing the first region using a first compression ratio and compressing the second region with a second compression ratio (2114).
• FIGS. 19 and 21 provide particular methods of compressing an image according to an embodiment of the present invention. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Moreover, the individual steps illustrated in FIGS. 19 and 21 may include multiple sub-steps that may be performed in various sequences as appropriate to the individual step.
• VDC-X [0130]
• VDC-X compression standard e.g., VDC-M
• VDC-M uses a tile-based approach instead of a nearest neighbor approach.
• This compression standard encodes different tiles at different quality settings, however the goal of this conventional compression is to maintain an overall constant frame size (i.e., bit rate). So once a compression ratio is selected, it varies each tile in order to maintain the constant bit rate.
• video images are compressed, not solely based on bit rate, but based on the user's eye gaze location.
• the four sections 2010, 2012, 2014, and 2016 including the high quality region will be compressed with a higher quality setting than the remaining sections, which can be referred to as peripheral sections, which will be compressed with a lower quality setting that that used for the sections 2010-2016.
• Some embodiments of the present invention do not maintain a constant bit rate, so that each frame size varies over time, and that the transport interface, for example, MIPI, is put into a low power mode when not in use.
• embodiments In a manner similar to the DSC-based approach discussed above, for a VDC-X tile- based approach, embodiments encode the quality of each tile based on the current location of the user's eye-gaze.
• embodiments of the present invention are able to vary the frame size or bit rate per frame, and to use the current eye-gaze information in order to select which tile (VDC-X) or section (DSC) has a higher quality vs the foveated regions that have a lower quality setting.
• the N-way compression or the section-based compression described above can implement JPEG as the compression standard rather than DSC or VDC- X.
• FIG. 22 is a simplified block diagram illustrating components of an AR system according to an embodiment of the present invention.
• AR system 2200 as illustrated in FIG. 22 may be incorporated into the AR devices as described herein.
• FIG. 22 provides a schematic illustration of one embodiment of AR system 2200 that can perform some or all of the steps of the methods provided by various embodiments. It should be noted that FIG. 22 is meant only to provide a generalized illustration of various components, any or all of which may be utilized as appropriate. FIG. 22, therefore, broadly illustrates how individual system elements may be implemented in a relatively separated or relatively more integrated manner.
• AR system 2200 is shown comprising hardware elements that can be electrically coupled via a bus 2205, or may otherwise be in communication, as appropriate.
• the hardware elements may include one or more processors 2210, including without limitation one or more general-purpose processors and/or one or more special-purpose processors such as digital signal processing chips, graphics acceleration processors, and/or the like; one or more input devices 2215, which can include, without limitation, a mouse, a keyboard, a camera, and/or the like; and one or more output devices 2220, which can include, without limitation, a display device, a printer, and/or the like.
• AR system 2200 includes an eye tracking system 2255 that can provide the user's eye gaze location to the AR system.
• AR system 2200 may further include and/or be in communication with one or more non-transitory storage devices 2225, which can comprise, without limitation, local and/or network accessible storage, and/or can include, without limitation, a disk drive, a drive array, an optical storage device, a solid-state storage device, such as a random access memory (RAM), and/or a read-only memory (ROM), which can be programmable, flash-updateable, and/or the like.
• RAM random access memory
• ROM read-only memory
• Such storage devices may be configured to implement any appropriate data stores, including without limitation, various file systems, database structures, and/or the like.
• AR system 2200 might also include a communications subsystem 2219, which can include, without limitation, a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device, and/or a chipset such as a Bluetooth ⁇ device, an 802.11 device, a WiFi device, a WiMax device, cellular communication facilities, etc., and/or the like.
• Communications subsystem 2219 may include one or more input and/or output communication interfaces to permit data to be exchanged with a network such as the network described below to name one example, other computer systems, television, and/or any other devices described herein.
• a portable electronic device or similar device may communicate image and/or other information via communications subsystem 2219.
• a portable electronic device e.g., the first electronic device
• AR system 2200 may be incorporated into AR system 2200, e.g., an electronic device as an input device 2215.
• AR system 2200 will further comprise a working memory 2260, which can include a RAM or ROM device, as described above.
• AR system 2200 also can include software elements, shown as being currently located within working memory 2260, including an operating system 2262, device drivers, executable libraries, and/or other code, such as one or more application programs 2264, which may comprise computer programs provided by various embodiments, and/or may be designed to implement methods, and/or configure systems, provided by other embodiments, as described herein.
• an operating system 2262 device drivers, executable libraries, and/or other code
• application programs 2264 which may comprise computer programs provided by various embodiments, and/or may be designed to implement methods, and/or configure systems, provided by other embodiments, as described herein.
• code and/or instructions can be used to configure and/or adapt a general purpose computer or other device to perform one or more operations in accordance with the described methods.
• a set of these instructions and/or code may be stored on a non-transitory computer- readable storage medium, such as storage device(s) 2225 described above.
• the storage medium might be incorporated within a computer system, such as AR system 2200.
• the storage medium might be separate from a computer system e.g., a removable medium, such as a compact disc, and/or provided in an installation package, such that the storage medium can be used to program, configure, and/or adapt a general purpose computer with the instructions/code stored thereon.
• These instructions might take the form of executable code, which is executable by AR system 2200 and/or might take the form of source and/or installable code, which, upon compilation and/or installation on AR system 2200, e.g., using any of a variety of generally available compilers, installation programs, compression/decompression utilities, etc., then takes the form of executable code.
• executable code which is executable by AR system 2200 and/or might take the form of source and/or installable code, which, upon compilation and/or installation on AR system 2200, e.g., using any of a variety of generally available compilers, installation programs, compression/decompression utilities, etc., then takes the form of executable code.
• some embodiments may employ a computer system such as AR system 2200 to perform methods in accordance with various embodiments of the technology. According to a set of embodiments, some or all of the procedures of such methods are performed by AR system 2200 in response to processor 2210 executing one or more sequences of one or more instructions, which might be incorporated into operating system 2262 and/or other code, such as an application program 2264, contained in working memory 2260. Such instructions may be read into working memory 2260 from another computer-readable medium, such as one or more of storage device(s) 2225. Merely by way of example, execution of the sequences of instructions contained in working memory 2260 might cause processor(s) 2210 to perform one or more procedures of the methods described herein.
• machine-readable medium and computer-readable medium refer to any medium that participates in providing data that causes a machine to operate in a specific fashion.
• various computer-readable media might be involved in providing instructions/code to processor(s) 2210 for execution and/or might be used to store and/or carry such instructions/code.
• a computer-readable medium is a physical and/or tangible storage medium.
• Such a medium may take the form of a non-volatile media or volatile media.
• Non- volatile media include, for example, optical and/or magnetic disks, such as storage device(s) 2225.
• Volatile media include, without limitation, dynamic memory, such as working memory 2260.
• Common forms of physical and/or tangible computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, EPROM, a FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read instructions and/or code.
• Various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to processor(s) 2210 for execution.
• the instructions may initially be carried on a magnetic disk and/or optical disc of a remote computer.
• a remote computer might load the instructions into its dynamic memory and send the instructions as signals over a transmission medium to be received and/or executed by AR system 2200.
• Communications subsystem 2219 and/or components thereof generally will receive signals, and bus 2205 then might carry the signals and/or the data, instructions, etc. carried by the signals to working memory 2260, from which processor(s) 2210 retrieves and executes the instructions.
• the instructions received by working memory 2260 may optionally be stored on a non-transitory storage device 2225 either before or after execution by processor(s) 2210.
• Various examples of the present disclosure are provided below.
• Example 1 is a method of compressing an image, the method comprising: determining an eye gaze location of a user; generating a foveation map based on the eye gaze location, wherein the foveation map includes a first region of the image and a second region of the image; and compressing the first region of the image using a first quality setting and the second region of the image using a second quality setting.
• Example 2 is the method of example 1 wherein determining the eye gaze location comprises use of an eye tracking camera of an augmented reality device.
• Example 3 is the method of example(s) 1-2 wherein the foveation map includes a central region and a peripheral region.
• Example 4 is the method of example(s) 1-3 wherein the image comprises virtual content generated by an augmented reality device.
• Example 5 is the method of example(s) 1-4 wherein the image is included in a virtual content video stream.
• Example 6 is the method of example(s) 1-5 wherein compressing the first region of the image using the first quality setting comprises compressing all blocks in the first region using the first quality setting.
• Example 7 is the method of example(s) 1-6 wherein the first quality setting is greater than the second quality setting.
• Example 8 is the method of example(s) 1-7 wherein the first quality setting is 100%.
• Example 9 is the method of example(s) 1-8 further comprising post-processing image content in at least one of the first region or the second region.
• Example 10 is the method of example(s) 1-9 wherein the compressing produces a compressed image, the method further comprising decoding the compressed image using the foveation map.
• Example 11 is the method of example(s) 1-10 wherein: the first region of the image includes a plurality of first blocks; the second region of the image includes a plurality of second blocks; compressing the first region of the image comprises compressing each of the plurality of first blocks using the first quality setting; and compressing the second region of the image comprises compressing each of the plurality of second blocks using the second quality setting.
• Example 12 is the method of claim example(s) 1-11 further comprising: decompressing the first region of the image using the first quality setting; decompressing the second region of the image using the second quality setting; and displaying the image to the user.
• Example 13 is the method of example(s) 1-12 wherein the second region of the image includes the first region of the image.
• Example 14 is the method of example(s) 1-13 wherein the compressing produces a compressed image, the method further comprising: decoding the compressed image using the foveation map to produce a decoded first region and a decoded second region; and reconstructing the image by overlaying the decoded first region over the decoded second region.
• Example 15 is an augmented reality (AR) system comprising: a wearable device including: a frame; a projector coupled to the frame; a display optically coupled to the projector; and an eye tracking system; a memory; and a processor configured to: receive an eye gaze location from the eye tracking system; generate an image; generate a foveation map based on the eye gaze location, wherein the foveation map includes a first region of the image and a second region of the image; and compress the first region of the image using a first quality setting and the second region of the image using a second quality setting.
• AR augmented reality
• Example 16 is the AR system of example 15 wherein the projector comprises one projector of a set of projectors, the display comprises one display of a set of displays, and the eye tracking system includes a set of eye tracking devices.
• Example 17 is the AR system of example(s) 15-16 wherein determining the eye gaze location comprises use of an eye tracking camera of an augmented reality device.
• Example 18 is the AR system of example(s) 15-17 wherein the foveation map includes a central region and a peripheral region.
• Example 19 is the AR system of example(s) 15-18 wherein the image comprises virtual content generated by an augmented reality device.
• Example 20 is the AR system of example(s) 15-19 wherein the image is included in a virtual content video stream.
• Example 21 is the AR system of example(s) 15-20 wherein compressing the first region of the image using the first quality setting comprises compressing all blocks in the first region using the first quality setting.
• Example 22 is the AR system of example(s) 15-21 wherein the first quality setting is greater than the second quality setting.
• Example 23 is the AR system of example(s) 15-22 wherein the first quality setting is 100%.
• Example 24 is the AR system of example(s) 15-23 wherein the processor is further configured to post-process image content in at least one of the first region or the second region.
• Example 25 is the AR system of example(s) 15-24 wherein the compressing produces a compressed image, wherein the processor is further configured to decode the compressed image using the foveation map.
• Example 26 is the AR system of example(s) 15-25 wherein: the first region of the image includes a plurality of first blocks; the second region of the image includes a plurality of second blocks; compressing the first region of the image comprises compressing each of the plurality of first blocks using the first quality setting; and compressing the second region of the image comprises compressing each of the plurality of second blocks using the second quality setting.
• Example 27 is the AR system of example(s) 15-26 wherein the processor is further configured to: decompress the first region of the image using the first quality setting; decompress the second region of the image using the second quality setting; and display the image to the user.
• Example 28 is the AR system of example(s) 15-27 wherein the second region of the image includes the first region of the image.
• Example 29 is the AR system of example(s) 15-28 wherein compressing produces a compressed image and the processor is further configured to: decode the compressed image using the foveation map to produce a decoded first region and a decoded second region; and reconstruct the image by overlaying the decoded first region over the decoded second region.
• Example 30 is a non-transitory computer-readable medium comprising program code that is executable by a processor of a device that is wearable by a user, the program code being executable by the processor to: determine an eye gaze location of a user; generate a foveation map based on the eye gaze location, wherein the foveation map includes a first region of the image and a second region of the image; and compress the first region of the image using a first quality setting and the second region of the image using a second quality setting.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Physics & Mathematics (AREA)
• Discrete Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Processing Or Creating Images (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)
• Controls And Circuits For Display Device (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=92842415

Family Applications (1)

Country Status (5)

Family Cites Families (4)

• 2024
  • 2024-03-19 EP EP24775555.6A patent/EP4684530A1/de active Pending
  • 2024-03-19 WO PCT/US2024/020491 patent/WO2024196910A1/en not_active Ceased
  • 2024-03-19 CN CN202480020041.9A patent/CN120917751A/zh active Pending
  • 2024-03-19 JP JP2025555103A patent/JP2026510991A/ja active Pending
• 2025
  • 2025-09-18 US US19/332,513 patent/US20260019545A1/en active Pending

Also Published As

Similar Documents

Legal Events