Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/70—Denoising; Smoothing
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/0075—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for altering, e.g. increasing, the depth of field or depth of focus
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/01—Head-up displays
  • G02B27/017—Head mounted
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
  • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
  • G06V40/18—Eye characteristics, e.g. of the iris
  • G06V40/19—Sensors therefor
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/122—Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/30—Image reproducers
  • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
  • H04N13/344—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/30—Image reproducers
  • H04N13/366—Image reproducers using viewer tracking
  • H04N13/383—Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10004—Still image; Photographic image
  • G06T2207/10012—Stereo images

Definitions

• This disclosure generally relates to the field of computing systems. More particularly, the disclosure relates to virtual reality systems.
• VR virtual reality
• the simulation is artificially generated by providing the user with various sensory experiences, e.g., sight, sound, touch, smell, etc.
• Some current VR systems are implemented via a stereoscopic display device.
• the stereoscopic display device provides an illusion of depth in an image via stereopsis, i.e., presenting a first image to a first eye and a second image to a second eye of a user to artificially generate a 3D image from 2D images.
• VR glasses are an example of a VR system that utilizes stereopsis.
• VR glasses typically include head mounted display screens that cover the direct and peripheral vision of both eyes of a user.
• the VR glasses display two images of the same 3D scene, which are then synthesized by a computer graphics system with a corresponding amount of parallax that renders the 3D scene in a realistic manner.
• Parallax is the difference in the perceived position of an object viewed from two different lines of sight.
• a realistic rendering of a 3D scene typically has a larger parallax for nearby objects and a smaller parallax for distant objects.
• Some VR systems may also utilize a stereographic image capture rig to capture the images for display by the VR glasses.
• the stereographic image capture rig captures an image as a three hundred sixty degree panorama so that a viewer utilizing the VR system may perform head rotation when utilizing the VR system to look around at different objects in the virtual environment provided by the VR system.
• a virtual reality apparatus includes a virtual reality display screen. Further, the virtual reality apparatus includes an eye tracking system that tracks a gaze direction of one or more eyes of a user to determine an object of focus in a virtual reality display. In addition, the virtual reality apparatus includes a processor. The virtual reality apparatus also includes a memory having a set of instructions that when executed by the processor causes the virtual reality apparatus to estimate one or more pupil dimensions of the one or more eyes based upon a luminance of virtual reality imagery displayed by the virtual reality display. The virtual reality apparatus is further caused to determine a focal plane based upon the estimated one or more pupil dimensions. Further, the virtual reality apparatus is caused to generate a synthetic depth of field based upon the focal plane. In addition, the virtual reality apparatus is caused to apply a blurring effect to one or more pixels of the virtual reality display based upon the one or more pixels being outside of the synthetic depth of field.
• a process tracks, with an eye tracking system, a gaze direction of one or more eyes of a user to determine an object of focus in a virtual reality display of a virtual reality apparatus.
• the process estimates, with a processor one or more pupil dimensions of the one or more eyes based upon a luminance of virtual reality imagery displayed by the virtual reality display.
• the process also determines, with the processor, a focal plane based upon the estimated one or more pupil dimensions.
• the process generates, with the processor, a synthetic depth of field based upon the focal plane.
• the process applies, with the processor, a blurring effect to one or more pixels of the virtual reality display based upon the one or more pixels being outside of the synthetic depth of field.
• FIG 1. Illustrates a VR configuration in which a user utilizes a head mounted VR device to view a VR environment.
• FIG. 2 illustrates a top view of the internal components of the VR head mounted device as viewed by the user illustrated in FIG. 1 .
• FIG. 3 illustrates the top view of the internal components of the VR head mounted device with a different focal plane and corresponding depth of field ("DoF").
• FIG. 4 illustrates the internal components of the adaptive focal plane system illustrated in FIGS. 2 and 3.
• FIG. 5 illustrates a process that is utilized by the adaptive focal plane system to adapt a focal plane based upon luminance.
• a configuration for rendering VR with an adaptive focal plane is provided.
• prior VR systems were able to generate parallax in the context of synthetic 3D scenes where the 3D world geometry was lighted and rendered utilizing a computer graphics pipeline, such VR systems typically displayed all objects in focus.
• all virtual objects remained in focus in the foreground and background of an object at which the user was focused.
• Such an environment has an unrealistic appearance as the human visual system relies on the focal plane in addition to parallax.
• a realistic environment has an object in the plane of focus at a particular depth that appears clear to the user whereas objects in planes that are in the foreground or background have a certain amount of blur.
• the configuration for rendering VR with an adaptive focal plane estimates the diameter of a user's pupils based on the luminance of a virtual scene.
• the computational efficiency, e.g., processing speed, of a VR system is improved as the pupil diameter estimation is more efficient than image processing of pupil dilation.
• the configuration for rendering VR with an adaptive focal plane reduces the cost of prior VR systems as equipment for complex image analysis is unnecessary for the configuration for rendering VR with an adaptive focal plane in contrast with prior VR systems.
• the configuration for rendering VR with an adaptive focal plane is also more accurate than prior VR systems.
• a user's pupils in a real outdoor scene would be constricted such that all objects in the scene are in focus.
• the user may reduce the display brightness, which may lead to pupil dilation that is not commensurate with that of the real world scene.
• Prior VR systems applied blur to foreground and background objects based on the measured pupil dilation even though the user was looking at a scene that was brightly lit in the virtual environment.
• the user may have expected to view all of the objects in the virtual scene in focus, but may view blur with respect to certain foreground and background objects as a result of prior VR systems utilizing measured pupil dilation for blur application.
• the configuration for rendering VR with an adaptive focal plane improves upon prior VR systems by providing more accuracy for blur application since the luminance of the scene is utilized as a criteria for blur application.
• the configuration for rendering VR with an adaptive focal plane would not apply as much blur to a sunlit VR scene for which the user has reduced the display brightness since the luminance of the sunlit VR scene has not been changed by the user.
• FIG 1 Illustrates a VR configuration 100 in which a user 101 utilizes a head mounted VR device 102 to view a VR environment.
• the user 101 views a VR display screen 203, as illustrated in FIG. 2, positioned within the head mounted VR device 102.
• the VR environment has captured stereographic images of real world imagery, which are rendered in the VR environment as synthetic imagery.
• the user 101 may navigate the VR environment via various modalities, e.g., head turning, providing voice inputs to a microphone positioned on head mounted VR device 102, providing touch inputs to actuators positioned on the head mounted VR device 102, providing touch inputs to actuators located in proximity to the VR head mounted device 102, etc.
• a variety of head mounting devices may be utilized for effectuating the head mounting of the VF device 102 to the head of the user 101 .
• the VR configuration 100 is illustrated as utilizing a head mounted VR device 102, various other embodiments of VR systems include a VR display screen that is positioned at a distance from the user and changes the VR environment based upon tracked head movement of the user with respect to the VR display screen. Therefore, the VR configuration 100 is not limited to a particular VR system.
• FIG. 2 illustrates a top view of the internal components of the VR head mounted device 102 as viewed by the user 101 illustrated in FIG. 1 .
• the user 101 views the display screen 203 to perceive various objects 207-21 1 in the VR environment.
• the left eye 201 and the right eye 202 of the user 101 gaze at an object of focus 207.
• An eye tracking system 204 e.g., one or more image capture devices, captures the gaze of the eyes 201 and 202 to determine the object of focus 207.
• the eye tracking system 204 provides the captured eye tracking data to an adaptive focal plane system 205.
• the eye tracking system 204 may have a camera coordinate system, e.g., x and y coordinates, for each of the eyes 201 and 202.
• a world geometry may be represented according to the camera coordinate system for each of the eyes 201 and 202 such that each of the eyes 201 and 202 is considered to be at the origin of the world geometry.
• the eye tracking system 204 estimates the viewing direction of each of the eyes 201 and 202 in terms of (x,y) pixel coordinates on the VR display screen 203.
• the adaptive focal plane system 205 utilizes the viewing direction of the captured eye tracking data to estimate a focal plane 206 for the eyes 201 and 202.
• the adaptive focal plane system 205 estimates the focal plane 206 as a single focal plane 206 on the basis that the eyes 201 and 202 are gazing at the same point of the object of focus 207.
• the adaptive focal plane system 205 estimates different focal planes for each of the eyes 201 and 202 based on the eyes 201 and 202 gazing at different points of an object or different objects in the VR environment.
• the adaptive focal plane system 205 estimates a pupil dilation of the eyes
• the adaptive focal plane system 205 may estimate a large pupil dilation of the eyes 201 and 202 based upon the luminance of the VR display screen 203.
• the adaptive focal plane system 205 may estimate a large pupil dilation of the eyes 201 and
• the adaptive focal plane system 205 may generate a small pupil dilation of the eyes 201 and 202 if the VR display screen 203 emits imagery having a large luminance.
• Such pupil dilation correlates to pupil constriction of a real world scene having optimal lighting or nearly optimal lighting such that pupil dilation is unnecessary to perceive objects in that real world scene.
• the adaptive focal plane system 205 generates a synthetic DoF based on the estimated focal plane 206.
• the adaptive focal plane system 205 allows, or modifies, the pixels of objects within the synthetic DoF to be clear whereas the adaptive focal plane system 205 applies blur to objects outside of the synthetic DoF if blur has not already been applied.
• the pixels of the object of focus 207 are unmodified, or are modified by the adaptive focal plane system 205 to remove any blur, as the object of focus 207 is intended to be clearly viewed and perceived by the user 101 .
• foreground objects and/or background objects within the synthetic DoF are also intended to be clearly viewed and perceived by the user 101 . Therefore, the adaptive focal plane system 205 allows, or modifies, pixels of such objects to be viewed without any blur.
• the example of the DoF illustrated by FIG. 2 is a low DoF that results from an estimation of significant pupil dilation based on low luminance emitted by the VR display screen 203.
• the DoF is illustrated as only extending forward and backward from the object of focus 207 a minimal distance that is less than the entirety of the VR display screen 203.
• the adaptive focal plane system 205 does not apply blur to the foreground object 209 and the background object 208 that are present within the DoF, but applies blur to the foreground object 210 and the background object 21 1 that are outside of the DoF.
• the luminance is predetermined since the imagery of the VR environment displayed by the VR display screen 203 is synthetic imagery.
• the luminance of a particular scene does not change even if the user adjusts various inputs such as brightness.
• luminance scene values may be encoded into video frames that are provided to the adaptive focal plane system 205, e.g., via streaming, downloading, etc., so that the adaptive focal plane system 205 determines a predetermined luminance associated with a particular scene.
• the adaptive focal plane system 205 may determine the particular scene by analyzing a code displayed by the VR display screen 203 that is associated with a particular VR scene, analyzing metadata received from the VR display screen 203, performing image analysis to determine a scene that is predetermined to be associated with particular images, etc.
• the blur data is determined by the adaptive focal plane system 205, which may send such data to the VR display screen 203 for rendering.
• the adaptive focal plane system 205 may alter the display data to blur certain objects or portions of objects that are outside of the synthetic DoF.
• the VR display screen 203 may then render that altered data.
• the adaptive focal plane system 205 is illustrated as a single system, in various embodiments, multiple systems may be utilized to perform the functionality of the adaptive focal plane system 205.
• a first system may estimate the pupil dimensions based upon luminance
• a second system may determine the focal plane or adapt an existing focal plane based upon the pupil dimension estimates
• a third system may generate a synthetic DoF based upon the focal plane
• a fourth system may perform blur application based upon the synthetic DoF.
• Various other numbers of systems may be utilized to perform the different functionality.
• the adaptive focal plane system 205 adapts the focal plane to various pupil size estimations.
• FIG. 3 illustrates the top view of the internal components of the VR head mounted device 102 with a different focal plane 301 and corresponding DoF.
• the adaptive focal plane system 205 of FIG. 3 estimates a small pupil dilation of the eyes 201 and 202, i.e., pupil constriction that results from optimal or nearly optimal luminance emitted from the VR display screen 203.
• the adaptive focal plane system 205 calculates a high DoF such that all of the objects 207-21 1 are rendered clearly by the VR display device 203.
• FIG. 4 illustrates the internal components of the adaptive focal plane system 205 illustrated in FIGS. 2 and 3.
• the adaptive focal plane system 205 comprises a processor 401 , various input/output devices 402, e.g., audio/video outputs and audio/video inputs, storage devices, including but not limited to, a tape drive, a floppy drive, a hard disk drive or a compact disk drive, a receiver, a transmitter, a speaker, a display, an image capturing sensor, e.g., those used in a digital still camera or digital video camera, a clock, an output port, a user input device such as a keyboard, a keypad, a mouse, and the like, or a microphone for capturing speech commands, a memory 403, e.g., random access memory (“RAM”) and/or read only memory (“ROM”), a data storage device 404, and synthetic DoF generation code 405.
• RAM random access memory
• ROM read only memory
• the processor 401 may be a specialized processor that is specifically configured execute the synthetic DoF generation code 405 to generate a synthetic DoF that is utilized to determine which portions of a VR scene are blurred or are not blurred.
• the processor 401 utilizes a more computationally efficient estimation of pupil dilation that is based upon luminance data. Therefore, the processor 401 improves the functioning of a computer by improving the processing speed of blur application to render realistic VR scenery.
• the processor 401 improves the accuracy of a computer by utilizing luminance values to generate a more accurate synthetic DoF than via determining pupil dilation via image processing.
• the processor 401 determines what pupil dilation should be based upon a particular luminance value of predetermined synthetic imagery rather than a particular brightness level that may have been adjusted by the user 101 and may have an effect on pupil dilation.
• the use of luminance of synthetic imagery is a more accurate determining factor for generating a synthetic DoF than pupil dilation that may be affected by other constraints such as brightness adjustments.
• FIG. 5 illustrates a process 500 that is utilized by the adaptive focal plane system 205 to adapt a focal plane based upon luminance.
• the process 500 tracks, with the eye tracking system 204, a gaze direction of one or more eyes 201 and 202 of a user 101 to determine an object of focus 207 in a virtual reality display 203 of a virtual reality apparatus 102.
• the process 500 estimates, with a processor 401 , one or more pupil dimensions of the one or more eyes 201 and 202 based upon a luminance of virtual reality imagery displayed by the virtual reality display 203.
• the process 500 also determines, with the processor 401 , a focal plane 206 based upon the estimated one or more pupil dimensions. Further, at a process block 504, the process 500 generates, with the processor 401 , a synthetic DoF based upon the focal plane 206. In addition, at a process block 505, the process 500 applies, with the processor 401 , a blurring effect to one or more pixels of the virtual reality display 203 based upon the one or more pixels being outside of the synthetic depth of field. [0033]
• the processes described herein may be implemented by the processor 401 illustrated in Figure 4. Such a processor will execute instructions, either at the assembly, compiled or machine-level, to perform the processes.
• a computer readable medium such as a computer readable storage device.
• the instructions may also be created using source code or any other known computer-aided design tool.
• a computer readable medium may be any medium capable of carrying those instructions and include a CD-ROM, DVD, magnetic or other optical disc, tape, silicon memory, e.g., removable, non-removable, volatile or non-volatile, packetized or non-packetized data through wireline or wireless transmissions locally or remotely through a network.
• a computer is herein intended to include any device that has a general, multi-purpose or single purpose processor as described above.
• AR systems typically utilize a display system, e.g., a head mounted system such as a pair of glasses, that overlays virtual imagery over real world imagery.
• a display system e.g., a head mounted system such as a pair of glasses
• the configurations provided for herein may be utilized to adjust the focal plane of the virtual imagery based upon a pupil dilation estimate generated from luminance values of that virtual imagery so that the virtual imagery appears more realistic.
• such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C).
• This may be extended for as many items as listed.

Landscapes

• Physics & Mathematics (AREA)
• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• General Physics & Mathematics (AREA)
• Optics & Photonics (AREA)
• Theoretical Computer Science (AREA)
• Health & Medical Sciences (AREA)
• Ophthalmology & Optometry (AREA)
• Human Computer Interaction (AREA)
• General Health & Medical Sciences (AREA)
• Controls And Circuits For Display Device (AREA)
• User Interface Of Digital Computer (AREA)
• Processing Or Creating Images (AREA)

Priority Applications (5)

Applications Claiming Priority (2)

Publications (1)

Family

ID=57681595

Family Applications (1)

Country Status (6)

Cited By (4)

Families Citing this family (6)

Citations (3)

Family Cites Families (17)

• 2016
  • 2016-12-22 KR KR1020187018759A patent/KR20180099703A/ko not_active Withdrawn
  • 2016-12-22 US US16/066,310 patent/US10713760B2/en active Active
  • 2016-12-22 WO PCT/EP2016/082428 patent/WO2017114755A1/en not_active Ceased
  • 2016-12-22 JP JP2018534138A patent/JP2019510991A/ja active Pending
  • 2016-12-22 EP EP16819930.5A patent/EP3398004B1/en active Active
  • 2016-12-22 CN CN201680077396.7A patent/CN108885339B/zh active Active

Patent Citations (3)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events