DE112014002469B4 - System and method for generating images for a near-eye light field display - Google Patents

Abstract

A method (100) comprising: identifying (105) a ray defined by a pixel of a micro-display (250) and a micro-lens (253) of a near-eye light field display device (200); Generating imaging coordinates corresponding to the pixel by intersecting (100) the beam with a two-dimensional virtual display plane (210), wherein generating the imaging coordinates involves determining a displacement of an optical center of the microlens (253) relative to a reference point on the microdisplay ( 250); and calculating (115) a color for the pixel based on the mapping coordinates.

Description

FIELD OF THE INVENTION

The present invention relates to the generation of images and more particularly relates to the generation of images for a near-eye light field display.

BACKGROUND

Near-Eye Displays (NEDs) include head-mounted displays (HMDs) that can project images directly into a viewer's eyes. Such displays may overcome the screen size limitations dictated by form factors of other mobile displays by compositing virtual large format display surfaces, or such displays may be deployed for virtual or augmented reality applications. One type of NED is a near-eye light field display that can be implemented as a thin, lightweight head-mounted display that has a pair of microlenses positioned between a viewer's eye and a corresponding pair of microdisplays. Such a near-eye light field display is capable of synthesizing light fields corresponding to virtual objects within a viewer's natural range of accommodation.

The images viewed using the near-eye light field display comprise an array of elemental images, each elemental image corresponding to one of the microlenses in the microlens array. The images displayed by the microdisplays therefore differ significantly from images displayed by conventional display devices. Consequently, content (e.g., images, movies, and video games) designed for conventional two-dimensional displays and stereoscopic displays (e.g., 3DTV) cannot be directly displayed using the near-eye light field display. There is therefore a need to address this issue and/or other issues associated with the prior art.

the US6097394A describes a system for generating views from freely selectable camera positions. U.S. 2013/0107024 A1 describes a light field display with an array of microlenses.

OVERVIEW

There is provided a system according to claim 15, a method according to claim 1 and a storage medium according to claim 18 for generating images for a near-eye light field display. A ray defined by a pixel of a micro-display and an optical device of a near-eye light field display is identified (determined) and the ray is intersected with a two-dimensional virtual display plane to generate map coordinates corresponding to the match pixels. A color is calculated for the pixel based on the mapping coordinates. The optical device of the near-eye light field display may be, for example, a microlens of a microlens array placed between a viewer and an emissive microdisplay, or a spot of a light spot array placed behind a transmissive microdisplay relative to the viewer is.

character list

• 1 12 shows a flow diagram of a method for generating images for a near-eye light field display according to an embodiment.
• 2A 12 shows an eye of a viewer and a near-eye light field display according to an embodiment.
• 2 B 12 shows a near-eye goggle light field display according to one embodiment.
• 2C 12 shows another near-eye goggle light field display according to an embodiment.
• 3A FIG. 12 shows an existing image to be processed for display by a near-eye light field display according to one embodiment.
• 3B 12 shows a corresponding micro-display image with an elementary image corresponding to each micro-lens, according to an embodiment.
• 3C shows the picture of the micro display 3B , when viewed through a microlens array, according to one embodiment.
• 3D FIG. 12 shows an elementary image generated without image defect removal and an elementary image generated with image defect removal, according to one embodiment.
• 3E FIG. 12 shows another existing image to be processed for display by a near-eye light field display according to one embodiment.
• 3F FIG. 12 shows an image of a micro-display with an elementary image corresponding to each light spot in a light spot array, according to an embodiment.
• 4 Figure 12 shows a parallel processing unit according to one embodiment;
• 5A Figure 12 shows a general processing cluster of the parallel processing unit 4 according to one embodiment.
• 5B shows a partition unit of the parallel processing unit 4 according to one embodiment.
• 6A shows the data stream multiprocessor 5A according to one embodiment.
• 6B FIG. 12 shows a flow chart of a method for generating images for the near-eye light field displays of FIG 2A , 2D and 2C according to one embodiment.
• 7 Figure 1 shows an illustrative computer system in which the various architectures and/or functions of the various previous embodiments may be implemented.

DETAILED DESCRIPTION

Reference will now be made in detail to the various embodiments of the present disclosure, examples of which are shown in the accompanying drawings. Although description will be made in connection with these embodiments, it should be understood that the disclosure is not intended to be limited to these embodiments. On the contrary, the disclosure is intended to cover alternatives, modifications, and equivalents, which may be included within the spirit and scope of the disclosure as defined by the appended claims. Furthermore, in the following detailed description of the present disclosure, numerous specific details are set forth in order to provide a more thorough understanding of the present disclosure. However, it should be noted that the present disclosure may be practiced without these specific details. In other instances, well known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects of the present disclosure.

Some portions of the detailed descriptions that follow are presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within computer memory. These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. In the present application, a procedure, logic block, process or the like is understood to be a self-consistent sequence of steps or instructions leading to a desired result. These steps are those using physical manipulations of physical quantities. Usually, but not required, these quantities take the form of electrical or magnetic signals capable of being stored, transmitted, combined, compared, or otherwise manipulated in a computer system. It has proven convenient at times, particularly for reasons of common usage, to refer to these signals as transactions, bits, values, elements, symbols, characters, samples, pixels, or the like.

However, it should be borne in mind that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient names given to those quantities. Unless otherwise stated in the following explanations, it should be noted that throughout this disclosure, explanations using terms such as "indicate", "generate", "produce", "calculate", "determine", "radiate" , "emit,""attenuate,""modulate,""transmit,""receive," or the like, actions and processes (e.g., flowcharts 100 from FIG 1 ) of a computer system or similar electronic computing device or processor (e.g., computing system 710 7 ) describe. The computer system or similar electronic computing device processes or transforms data as physical (electronic) quantities within the computer system's memories, registers or other information storage, transmission or display devices.

Embodiments described herein may be discussed in the general context of computer-executable instructions embodied on some form of computer-readable storage medium, such as program modules, being executed by one or more computers or other devices. For example, without limitation, computer-readable storage media includes non-transitory computer-readable storage media and communications media; non-transitory computer-readable media includes any computer-readable media other than a transient, propagated signal. In general, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The function of the program modules can be combined or distributed as desired in various embodiments.

Computer storage media include volatile and non-volatile, removable and stationary media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read only memory (ROM), electrically erasable programmable ROM (EEPROM), flash memory or other storage technology, compact disk ROM (CD ROM), digital versatile disks (DVD), or other optical storage, magnetic cartridge, magnetic tape storage, or magnetic disk storage, or other magnetic storage device, or other medium that can be used to store and be accessed to store the desired information to retrieve information.

Communication media may embody computer-executable instructions, data structures, and program modules, and includes any information delivery media. For example, without limitation, communication media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, radio frequency (RF), infrared, or other wireless media. Combinations of any of the above components may also be included within the scope of computer-readable media.

A near-eye light field display allows a user wearing an eyeglass-like device to view a synthetic image that appears positioned within the user's range of accommodation.

As described herein, near-eye light field displays enable thin, lightweight head-mounted displays (HMD) capable of presenting near-correct convergence, accommodation, differences in binocularity, and retinal defocus depth markers. The near-eye light field display device is placed close to the user's eyes, that is, closer than a near-focal plane defining the user's range of accommodation. Typically, objects positioned closer than a near focal plane appear blurry. However, images displayed using the near-eye light field display device may appear to be in focus to the viewer. Clear images can be displayed by the near-eye light field display device when light fields corresponding to virtual objects within the user's natural range of accommodation are composed. Merely displaying existing two-dimensional monoscopic, stereoscopic, and multiscopic content intended for display by conventional display devices will not produce sharp images when displayed by the near-eye light field display device.

Therefore, existing two-dimensional monoscopic, stereoscopic, and multiscopic content suitable for display by conventional display devices should be processed to generate an image or images for display by the near-eye light field display device. An existing image to be displayed can be processed to produce a processed image containing an array of elementary images. Each elemental image is a view of the desired existing image, corresponding to a single microlens within a microlens array through which a user views the processed image. When the processed image is viewed through the microlens array, the user sees the desired existent image mapped onto a virtual display plane that is within the user's natural range of accommodation. If a near-eye light field display based on points of light is used, each elementary image is one View of the desired existing image corresponding to a single point of light within a point of light array. The user views the array of light spots through a transmissive display layer, which encodes the processed image (ie, light rays generated by the array of light spots are projected through the display layer). When the processed image is viewed through the transmissive display layer placed in front of the light spot array, the user sees the desired existent image mapped onto a virtual display plane that is within the user's natural range of accommodation.

1 10 shows a flow diagram of a method 100 for generating images for a near-eye light field display according to an embodiment. In one embodiment, the in 1 The steps shown are performed by a processor configured to execute a graphics shading program. In step 105, a ray defined by a pixel of a micro-display and an optical device of the near-eye light field display device is determined. In the context of the following description, a microdisplay is a backlit transmissive display, such as an LCD (Liquid Crystal Display) panel, or an emissive display, such as an OLED (Organic Light Emitting Diode) panel or chip. The near-eye light field display includes the micro-display.

In one embodiment, the micro-display is an emissive display configured to be viewed through a two-dimensional array of micro-lenses, each micro-lens being an optical device that, in conjunction with a pixel of the micro-display, defines the beam of step 105 can. In the context of the following description, an array of microlenses is a two-dimensional array containing at least two microlenses. In one embodiment, the near-eye light field display is implemented in the form of an eyeglass form factor that includes two micro-displays and two micro-lens arrays to form a dual-eye near-eye light field display, with a first set of a micro-display and a micro-lens array in a left eye of the user, and a second set of micro-display and micro-lens array is associated with a right eye of the user.

In another embodiment, the microdisplay is a transmissive (i.e., of variable opacity and color and capable of becoming substantially transparent) display positioned between a viewer and a two-dimensional array of points of light, each point of light being an optical device . The array of dots of light is viewed through the micro-display. The ray from step 105 is defined by a pixel of the micro-display and a point of light in the point of light array.

In one embodiment, the near-eye light field display is implemented as a tensor display, such as is described by Wetzstein et al., "Tensor Displays: Compressive Light Field Synthesis Using Multilayer Directional Backlit Displays," ACM Transactions via Graphics SIGGRAPH 2012 Conference Notes, Vol. 31, Issue 4, July 2012. A display assembly may include a microdisplay and an optical device, where the optical device may include one or more microlenses, one or more points of light, one or more pinholes, or one or more transmissive layers. included in the tensor display array.

In step 110, the ray is intersected with a two-dimensional virtual display plane to generate map coordinates corresponding to the pixel. In the context of the present description, the virtual display plane is a virtual plane that appears to the viewer to be positioned at or behind the near focus plane. In one embodiment, the virtual display plane is in front of a distant plane beyond which objects appear out of focus to the viewer.

In step 115, a color for the pixel is calculated based on the mapping coordinates. In one embodiment, the mapping coordinates correspond to a texture map that stores an existing image. In one embodiment, the existing image, stored as a texture map, may be pre-processed to generate pixel colors for a defect corrected image. In the context of the following description, colors can be calculated for each pixel of the micro-display to process an existing image for display on the near-eye light field display device. If the existing image is a single image of an existing stereo image pair, the existing image for the other eye (left eye or right eye) can be processed to produce a corresponding left eye image or right eye image for display using the to generate near-eye light field display device. The stereo image pair that is created can be viewed simultaneously by the user of the near-eye light field display device to create a virtual image on the virtual display plane that appears three-dimensional. Thus, the three-dimensional qualities of the existing stereoscopic pair can also be present in the virtual image.

More illustrative information is now provided regarding various optional architectures and features with which the foregoing technique may or may not be implemented according to the user's desires. However, it is important to note that the following information is provided for illustrative purposes only and should not be taken as limiting in any way. Each of the following features may optionally be employed with or without the exclusion of other described features.

2A 12 shows an eye 205 of a viewer and a near-eye light field display device 200 according to an embodiment. In one embodiment, an eye box 206 defines characteristics of the eye 205, such as expected diameters, orientations (ie, rotations), and positions of the pupil. A width of the eye box 206 is in 2A as w _e shown. Objects between the eye 205 and a near focal plane 220 are too close to the eye 205 to be clearly perceived because the eye 205 cannot focus on objects that are closer to the eye 205 than the near focal plane 220. The eye 205 is in able to focus on objects that are at or beyond (ie, further away from) the near focus plane 220 . Objects that are very far away may also not appear clearly, and an area called an accommodation range within which objects appear clearly is bounded by the near focal plane 220 and a far plane (not shown). which is a distance from the near focus plane 220 equal to the accommodation range. Because conventional eyeglasses are positioned closer to the eye 205 than the near focal plane 220, an image displayed on the surface of conventional eyeglasses is not clearly visible to the viewer. Similarly, an existing image displayed by a micro-display 250 will also not be clear to the viewer.

The micro-display 250 is spaced from a micro-lens array 255 by a distance _dl . The microlens array 255 is arranged at a distance d _e which corresponds to an eye relief distance. A micro-display image displayed by the micro-display 250 appears to the viewer as being mapped onto a virtual display plane 210 located at a virtual display distance d _v from the eye 205 . The micro-display image can be created from an existing image. The distance _dv may be different for each eye 205, thereby making it possible to correct for optical aberration (ie, eyeglass prescription) of the viewer. The width of the virtual display plane 210 is w _v and the width of the micro-display 250 is w _s .

Typically, a three-dimensional scene is imaged to create a two-dimensional (2D) image for display on the micro-display 250 . A micro-display image displayed by micro-display 250 includes an elemental image corresponding to a view for each micro-lens in micro-lens array 255 . For example, an elementary image in the elementary image area 225 corresponding to the micro lens 253 is displayed. Elementary image area 225 is a two-dimensional area with a width of Δw _s and is defined as the intersection of micro-display 250 with a projection from the pupil of eye 205 through microlens 253. Each elementary image area and corresponding microlenses (e.g., elemental image area 225 and microlens 252 ) serve as an independent lens-limited magnification unit, synthesizing an off-axis perspective projection of a virtual image located at a distance d _v from the eye 205 (ie, in the virtual display plane 210). Parts of the virtual image are repeated across the different elementary image areas, as in 3B is shown.

Generation of a 3D scene to generate a micro-display image for display by micro-display 250 requires that each elementary image contained in the micro-display image be generated. In contrast, only a single image is generated for display on a conventional display device. In some cases, the 3D scene is not available for image generation, such as for existing content with conventional 2D images. However, existing content intended for display by a conventional display device can be rendered for display on the micro-display 250. In particular, existing stereoscopic image pairs are widely available for 3D television, and the existing stereoscopic image pairs can be processed for display on the micro-display 250.

2 B 12 shows a binocular near-eye light field display 260 according to one embodiment. The binocular near-eye light field display 260 includes a left micro-display 251 paired with a left micro-lens array 256 and a right micro-display 252 paired with a right micro-lens array 257 . The left micro-display 251 and the right micro-display 252 are separated by a distance b _s measured from the right edge of the left micro-display 251 to the left edge of the right micro-display 252 . The interpupillary distance between the centers of the pupils of the viewer's left and right eyes is b _e . The near-eye light field display 260 simulates the appearance of a conventional stereoscopic display placed at a distance dv _from the viewer (ie, at the virtual display plane 210). In one embodiment, b _e is a fixed mean distance, such as an inter-eye distance, rather than a dynamically sampled inter-pupillary distance.

A viewer looking through the near-eye light field display device 260 perceives a rectangular virtual display surface at the virtual image plane 210 on which is mapped a texture corresponding to the corresponding left and right eye views of an existing stereoscopic image pair. Therefore, a pair of texture maps is stored for an existing stereoscopic image pair. The truncated cones of the left and right eyes are indicated by dashed lines in 2 B specified. For a monoscopic existent image, only a single existent image is stored as a texture map that is used to determine the pixel color for both the left 251 and right 252 micro-displays. The existing image is mapped as a texture onto the virtual display plane 210 to create a virtual image. Each pixel of the virtual image in the virtual display plane 210 corresponds to a color defined by the existing image. The virtual image is then used to calculate colors for each pixel of the left micro-display 251 and the right micro-display 252. Simply put, a color of a pixel on a micro-display is calculated by outputting a ray from the pixel on the micro-display so that it intersects the virtual display plane 210 and then under a color of the pixel on a micro-display applying one or more samples of the virtual image at the point of intersection.

The pseudo-code shown in TABLE 1 may be included in a vertex shader program to map the existing image into the virtual display plane 210. TABLE 1 Void (main(){ //Evaluate the interpolated texture coordinates. gl_TexCoord[0] = gl_MultiTExCoord0; // Simulate the vertex transform used by a pipeline // specified function is executed gl_Position = gl_ModelViewProjectionMatrix*gl_Vertex; }

The generation of images for the near-eye light field display device 260 requires mapping the color of the pixel of the left micro-display 251 and the right micro-display 252 such that the emitted light field approximates a virtual image on the virtual display plane 210 . The virtual image may include objects, such as virtual object 230, appearing at different depths relative to virtual display plane 210. FIG. For a pixel located at a distance x _s from the center 215 of the left micro-display 251, ray tracing can be used to compute a color. The virtual object 230 appears to be displaced in front of the virtual display plane 210 as intended by the existing stereoscopic image pair. The depth of the virtual object 230 is not readily available after the 3D scene is created, but the intended displacements are substantially preserved when creating a stereoscopic image pair for the micro-display for the existing stereoscopic image pair.

The pixel is visible to the viewer's left eye through the left microlens array 256 which has an optical center that is laterally shifted a distance x _l from the center 215 of the left microdisplay 251 . Each optical ray is defined by a point of a pixel on a micro-display and a point of the optical center of the cut micro-lens. As in 2 B As shown, the 3D coordinates of the center of the microlens and the pixel define an optical ray 235. Note that the distances are described by the coordinate x and corresponding distances can also be defined as the y coordinate to define distances y _l and y _s relative to a reference point on the left micro-display 251.

The pseudo-code shown in TABLE 2 can be included in a fragment shader program to convert the in 2A dimensions shown and other system parameters as uniform variables. TABLE 2 uniform sampler2D viewTexture; // Texture Mapping Of The Existing Image // Micro-display variables uniform vec2 microdisplayDim; // physical dimensions mm [ w _s ] uniform int microdisplayOrientation; // 0: upright, 1: inverted //Variables of the microlens array uniform float microlensPitch; // distance (mm) [ w _l ] uniform float microlensDist; // distance to micro display (mm) [ d _l ] uniform float microlens rotation; // angle of rotation (radians) uniform vec2 microlensOffset; // offsets to align (mm) uniform bool microlensExcludeBorder; // exclude the partial microlenses // along the border of the micro-display // elementary image variables uniform float elemenalImagePitch; // distance (mm) [Δ w _s ] uniform vec2 elementalImageOffset; // Move to origin of micro display //(mm) uniform vec4 elementallmagerange, // index range(min column, max column, // min line, max line) // Virtual display layer variables uniform vec2 screenDim; // physical dimensions (mm) [ w _v ] uniform float screenDist; // Distance to microlens array (mm) [ d _v ] uniform vec2 screenPosition; // Shift to micro-display origin(mm) [ o _v ]

The displacement x _s can be evaluated for a texture element with the texture map coordinates (s, t) based on the orientation and dimensions of the micro-display. The pseudo-code shown in TABLE 3 can be included in a fragment shader program to calculate the displacement x _s for a pixel relative to the reference point on a microdisplay (microdisplayCoord). TABLE 3 vec2 microdisplayCoord; if (microdisplayOrientation == 0) microdisplayCoord = microdisplayDim*(gl_TexCoord[0].st - 0.5); else microdisplayCoord = microdisplayDim*(0.5 - gl_TexCoord[0].st);

The position of the optical center of the corresponding microlens, x _l , can be evaluated based on the orientation of the microlens array and the displacement x _s . The pseudo-code shown in TABLE 4 can be included in a fragment shader program to calculate the optical center of the corresponding microlens, x _{1 ,} (microlensCoord). TABLE 4 mat2 R = mat2(cos(microlensRotation), -sin(microlensRotation), sin(microlensRotation), cos(microlensRotation) );; vec2 microlensIndex = floor(R* (microdisplayCoord -elementalImageOffset)/elementalImagePitch); vec2 microlensCoord = transpose(R)*microlensPitch*microlensIndex + microlensOffset;

Then the horizontal coordinate or shift of the centroid of the optical ray 235 and the virtual display plane 210 (ie, the virtual image) x _v can be calculated by: $$x_v=x_l+\left(\frac{{\mathrm{i.e}}_v-{\mathrm{i.e}}_e}{{\mathrm{i.e}}_l}\right)\left(x_s-x_s\right)-O_v.$$

A vertical coordinate can be calculated for the intersection of the optical ray 235 with the virtual display plane 210 in a similar manner. In other embodiments, another reference position on the micro-display 251 can be used instead of the center point 251 to determine the distances x _s , x _{l , ov and x v .}

The pseudo-code shown in TABLE 5 can be included in a fragment shader program for x _v (screenCoord). TABLE 5 vec2 screenCoord = microlensCoord + (screenDist/microlensDist)*(microdisplayCoord- microlensCoord) - screenPosition;

The color of the pixel gl_FragColor can be calculated by converting the screen coordinates (x _v ,y _v ) to texture map coordinates (s,t). Because artifacts can result when part of the virtual image is visible through peripheral microlenses, pixels lying outside of the virtual display plane 210 are set to black when the boolean variable microlensExcludeBorder is TRUE. TABLE 6 vec2 texCoord = screenCoord/screenDim; if( ((texCoord.x>0.0) &&(texCoord.x<1.0))&&((texCoord.y>0.0)&&(texCoord.y<1.0)) ) { if ( ! microlensExcludeBorder || ( ((microlenslndex.x > elementalImageRange.x) && (microlenslndex.x < elementalImageRange.y)) && (microlensindex.y > elementallmageRange.z) && (microlenslndex.y < elementalImageRange.w)) ) ) gl_FragColor = vec4(texture2D(viewTexture, texCoord).rgb, 1); else gl_FragColor = vec4(0.0, 0.0, 0.0, 1.0); } else gl_FragColor = vec4(0.0, 0.0, 0.0, 1.0);

Ray tracing provides a direct estimate of the color of the left micro-display 251 pixel. Generating a set of rays defined by each pixel contained in the elementary image region corresponding to a microlens provides estimates of the color of each pixel in an elementary picture. Generating groups of rays for each elementary image area corresponding to a microlens provides estimates of the color of each pixel in the micro-display image for left micro-display 251 and in the micro-display image for right micro-display 252.

2C 12 shows a binocular near-eye light field display 280 according to one embodiment. Unlike the near-eye light field display device 260 shown in 2 B as shown, in which a lens (e.g., the lens in left microlens array 256 and in right microlens array 257) is used as an optical device positioned between a viewer and an emissive micro-display (e.g., left micro- Display 251 and the right micro-display 252), the near-eye light field display 280 uses a spot of light array located behind a transmissive micro-display relative to the viewer. The binocular near-eye light field display 260 includes a left micro-display 271 paired with a left light spot array 276 and a right micro-display 272 paired with a right light spot array 277 .

The left micro-display 271 and the right micro-display 272 are separated by a distance b _s measured from the right edge of the left micro-display 271 to the left edge of the right micro-display 272. The interpupillary distance between the centers of the pupils of the viewer's left and right eyes is b _e . The near-eye light field display 280 simulates the appearance of a conventional stereoscopic display placed at a distance dv _from the viewer (ie, at the virtual display plane 210). In one embodiment, b _e is a fixed mean distance, such as an interocular distance, rather than a dynamically sampled interpupillary distance.

A viewer looking through the near-eye light field display device 280 will perceive a rectangular virtual display surface at the virtual image plane 210 that has a corresponding left and right eye view of an existing stereoscopic image pair texture mapped. Therefore, a pair of texture maps is stored for an existing stereoscopic image pair. In other embodiments, the virtual display surface may have a different shape that is not necessarily rectangular. The left and right eye truncated cones are indicated by the dashed lines in 2C displayed. For a monoscopic existent image, only a single existent image is stored as a texture map that is used to determine the pixel color for both the left micro-display 271 and the right micro-display 272 . The existing image is mapped to a texture on the virtual display plane 210 to create a virtual image. Each pixel of the virtual image on the virtual display plane 210 corresponds to a color specified by the existing image. The virtual image is then used to calculate colors for each pixel of the left micro-display 271 and the right micro-display 272. Simply put, a color of a pixel on a micro-display is calculated by extending a ray from the pixel on the micro-display to intersect the virtual display plane 210, and then a color of the pixel on a micro-display is calculated. Display computed using one or more samples of the virtual image at the intersection.

As previously in connection with 2 B 1, the pseudo-code shown in TABLE 1 may be included in a vertex shader program to map the existing image onto virtual display plane 210. The generation of images for the near-eye light field display device 280 requires the assignment of the color of the pixel of the left micro-display 271 and the right micro-display 272 such that the emitted light field approximates a virtual image in the virtual display plane 210. The virtual image may include objects, such as virtual object 230, that appear at different depths relative to virtual display plane 210. FIG. For a pixel located at a distance x _s from the center 275 of the left micro-display 271, ray tracing can be applied to compute a color. The virtual object 230 appears displaced in front of the virtual display plane 210 as intended by the existing stereoscopic image pair. The depth of the virtual object 230 is not readily available after the 3D scene has been generated to produce the existing stereoscopic image pair, but the intended displacements are essentially preserved when a stereoscopic image pair is used for a micro-display for the existing stereoscopic image pair is generated.

The pixel is visible to the viewer's left eye in the left micro-display 271, which has an optical center that is laterally offset from the center 275 of the left micro-display 271 by a distance x ₁ . Each optical ray is defined by a point of a pixel on a microdisplay and by a point at the optical center of the intersected light spot in the corresponding light spot array. As in 2C is shown, the 3D coordinates of the center of the intersected spot of light and the pixel define an optical ray 285. Note that the distances are described using the x coordinate, but equivalent distances can also be defined using the y coordinate, to generate distances y _l and y _s relative to a reference point on the left micro-display 271.

The displacement x _l can be evaluated for a texture element or texel with the texture map coordinates (s, t) based on the orientation and the dimensions of the micro-display. The position of the optical center of the corresponding light spot x _s can be evaluated based on the orientation of the light spot array and the displacement s _l . Then the horizontal coordinate or displacement of the point of intersection of the optical ray 285 with the virtual display plane 210 (ie, virtual image) x _v can be calculated by: $$x_v=x_l+\left(\frac{{\mathrm{i.e}}_v-{\mathrm{i.e}}_e}{{\mathrm{i.e}}_l}\right)\left(x_s-x_s\right)-O_v.$$

A vertical coordinate can be calculated for the intersection of the optical ray 285 and the virtual display plane 210 in a similar manner. In other embodiments, a reference position on the micro-display 271 other than the center 275 can be used to determine the distances x _s , x _{l , ov base x v .}

A fragment shader program can be set up to calculate the displacement for a pixel relative to a reference point on a microdisplay x _l (microdisplayCoord), the optical center of the corresponding point of light x _s (pinlightCoord) and the horizontal coordinate x _v (screenCoord).

The color of the pixel gI_FragColor can be calculated by converting the screen coordinates (x _v ,y _v ) to texture map coordinates (s,t). Ray tracing provides a direct estimate of the color of the pixel for the left micro-display 271. Generating a set of rays defined by each pixel that lies within the elementary image region that corresponds to a point of light provides estimates of the color of the pixel in an elementary image. Generating groups of rays for each elementary image area corresponding to a point of light provides estimates of the color of each pixel in the micro-display image for left micro-display 271 and in the micro-display image for right micro-display 272.

3A FIG. 300 shows an existing image 300 to be processed for display on a near-eye light field display according to one embodiment. The existing image 300 is stored as a texture map (e.g., viewTexture) that is used to compute a color for each pixel of a micro-display image.

3B FIG. 310 shows an image 310 of a micro-display including an elemental image corresponding to each micro-lens in the micro-lens array 255 according to one embodiment. Micro-display image 310 may be displayed by micro-display 250, micro-display 251, and/or micro-display 252.

3C FIG. 3 shows a virtual image 320 representing the image 310 of the micro-display 3B is viewed through microlens array 255 according to one embodiment. When the micro-display 250 is viewed through the micro-lens array 255, the virtual image 320 corresponding to the existing image 300 is created in the virtual display plane 210. FIG.

3D FIG. 3 shows an elemental image 330 generated without de-corrosion and an elementary image generated with de-corrosion 340 according to one embodiment. The existing image 300 may be pre-processed prior to generating the image 310 for the micro-display to generate artifact corrected elemental images. When the artifact-removed elementary images are included as the micro-display image 310, a artifact-removed virtual image is generated in the virtual display plane 210. FIG.

wobei die Länge in normalisierten Textur-Koordinateneinheiten ausgedrückt ist, M einem Vergrößerungsfaktor entspricht und p die normalisierte Textur-Koordinate ist, die die Position eines Pixels repräsentiert, das nahe bei dem aktuellen Pixel p0 liegt, das gerade verarbeitet wird, wie dies nachfolgend detaillierter beschrieben ist.In one embodiment, stereoscopic image pairs are assumed to be oversampled relative to the near-eye light field display 260 resolution. A length of a low-pass filter p - _v can be determined by the size of the pixels of the micro-display as they are enlarged by projection into the virtual display plane 210 . The parameter p̂ _v can be calculated by: $${\widehat{p}}_v=\frac{M_p}{w_v}=\left(\frac{{\mathrm{i.e}}_v-{\mathrm{i.e}}_e}{{\mathrm{i.e}}_l}\right)\left(\frac{p}{w_v}\right),$$

where the length is expressed in normalized texture coordinate units, M corresponds to a magnification factor, and p is the normalized texture coordinate representing the position of a pixel that is close to the current pixel p0 being processed, as described in more detail below is.

The pseudo-code shown in TABLE 7 can be included in a fragment shader program to perform pre-processing for image defect removal. The variable antialiasLength is the parameter. As shown in the pseudo-code in TABLE 7, p0 is eventually assigned a new color during execution of a fragment shader program. The positions of neighboring pixels are stored in the temporary variable p, and then the colors of the neighboring pixels are averaged to find a color associated with p0. Although the pseudo-code in TABLE 7 implements a separable filter, which is typically chosen for reasons of efficiency, those skilled in the art will recognize that other filters are also possible, including non-separable filters. TABLE 7 uniform sampler2D viewTexture; // Variables for image debugging uniform int ; antialiasDirection // filter direction (0: horizontal, 1: vertical) uniform float antialiasLength; // Filter length in normalized texture coordinates uniform int antialiasNumSamples; // number of samples along the filter length void main() { // Evaluate texture coordinate vec2 p0 = gl_TexCoord[0].st; // Apply separable image error correction filter if needed if(antialiasNumSamples > 0) { // Evaluate temporary image error fixing variables float antialiasSampleOffset = antialiasLength/float(antialiasNumSamples - 1); float antialiasHalfLength = antialiasLength/2.0; float antialiasWeight = 1.0/float(antiallasNumSamples, vec3 color = vec3(0.0, 0.0, 0.0); vec2 p; // Apply ID filter along horizontal or vertical axis if (antialiasDirection == 0) { for (int i=0; i<antialiasNumSamples; i++){ p=p0+vec2(antialiasSampleOffset*float(i)-antialiasHalfLength, 0.0); color += antialiasWeight*texture2D(viewTexture, p).rgb; } } else { for (int i=0; i<antialiasNumSamples; i++){ p=p0+vec2(0.0, antialiasSampleOffset*float(i)-antialiasHalfLength); color += antialiasWeight*texture2D(viewTexture, p).rgb; } } gl_FragColor = vec4(color, 1.0); } else{ // Image error correction is not required, so pass input color gl FragColor = vec4(texture2D(viewTexture, p0).rgb, 1.0); } }

3E 350 shows an existing image 350 to be processed for display on a near-eye light field display according to one embodiment. The pattern on the tea kettle consists of diagonal stripes alternating in red, green and blue (the image is shown as a grayscale image).

3F FIG. 3 shows a micro-display image 310 with the red channel of an elementary image corresponding to each light spot in the light spot array 276 or 277 when the light spots are shaped as hexagons and arranged in a hexagonal array according to one embodiment. Micro display image 360 may be displayed by micro display 271 and/or micro display 272 . In other embodiments, the light spots may have a different shape, such as a square, a rectangle, or a circle.

4 FIG. 4 shows a parallel processing unit (PPU) 400 according to one embodiment. In one embodiment, PPU 400 is a multi-threaded processor implemented in one or more integrated circuit devices. The PPU 400 is a processing time-hiding architecture designed to process a large number of threads in parallel. A thread (ie, thread of execution) is an instantiation of a group of instructions suitable for execution by PPU 400 . In one embodiment, PPU 400 is a graphics processing unit (GPU) configured to implement a graphics processing pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device, such as a liquid crystal display (LCD -) facility to generate. In other embodiments, PPU 400 may be used to perform general purpose computations. Although a single exemplary parallel processor is provided herein for illustrative purposes, it should be understood that such a processor is provided for illustrative purposes only and any processor may be used to supplement and/or replace it.

As in 4 As shown, the PPU 400 includes an input/output (I/O) unit 405, a host interface unit 410, a front end unit 415, a scheduler unit 420, a work distribution unit 425, a hub 430, a cross-connect unit (Xbar) 670, one or more general processing clusters (GPCs) 450; and one or more partition units 480. The PPU 400 can be connected to a host processor or other peripheral devices via a system bus 402 . The PPU 400 may also be connected to a local memory comprising a number of storage devices 404 . In one embodiment, the local memory may include a number of dynamic random access memory (DRAM) devices.

I/O unit 405 is configured to send and receive communication events (i.e., commands, data, etc.) from a host processor (not shown) over system bus 402. The I/O unit 405 may communicate with the host processor directly over the system bus 402 or through one or more intermediaries such as a memory bridge. In one embodiment, I/O unit 405 implements a Peripheral Component Interconnect Express (PCIe) interface for communication over a PCIe bus. In alternate embodiments, I/O unit 405 may implement other types of well-known interfaces for communicating with external devices.

The I/O unit 405 is connected to a host interface unit 410 which decodes packets received over the system bus 402. In one embodiment, the packets represent instructions arranged to cause PPU 400 to perform various operations. The host interface unit 410 sends the decoded commands to various other units of the PPU 400 as indicated by the commands. For example, some commands may be sent to front end unit 415 . Other commands may be sent to hub 430 or other units of PPU 400, such as one or more copy units, a video encoder, a video decoder, a power management unit, etc. (not explicitly shown). In other words, the host interface unit 410 is configured to distribute communication events between the various logical units of the PPU 400.

In one embodiment, a program executed by the host processor encodes an instruction stream in a buffer that provides work orders to PPU 400 for processing. A work order or workload may include a set of commands and data to be processed by those commands. The buffer is an area in memory that is accessible (i.e., read/write) by both the host processor and the PPU 400. For example, host interface unit 410 may be configured to access the buffer in system memory connected to system bus 402 via memory requests communicated by I/O unit 405 . In one embodiment, the host processor writes the instruction stream to the buffer and then sends a pointer to the beginning of the instruction stream to the PPU 400. The host interface unit 410 provides the front end unit 415 with pointers to one or more instruction streams. The front end unit 415 manages the one or more streams by reading commands from the streams and forwarding commands to the various PPU 400 units.

The front end unit 415 is connected to a scheduling unit 420 that configures the various GPCs 450 to process tasks defined by the one or more streams. The scheduler 420 is configured to monitor status information related to the various tasks managed by the scheduler 420 . The state may indicate which GPC 450 a task is assigned to, whether the task is active or inactive, a priority level assigned to the task, etc. The scheduler 420 manages the execution of multiple tasks in the one or more GPCs 450.

The scheduler 420 is connected to a work distribution unit 425 configured to issue tasks for execution in the GPCs 450 . The work distribution unit 425 may monitor a series of scheduled tasks received from the scheduler 420 . In one embodiment, the work distribution unit 425 maintains a pending task collection and an active task collection for each of the GPCs 450. The pending task collection may have a number of sections (e.g., 16 sections) that contain tasks that are scheduled for processing by a particular GPC 450 are assigned. The active task collection includes a number of bins (e.g., 4 bins) for tasks that are currently being actively processed by the GPCs 450 . When a GPC 450 completes execution of a task, then the task is removed from the collection of active tasks for the GPC 450 and one of the other tasks from the collection of pending tasks is selected and scheduled for execution in the GPC 450. If an active task in the GPC 450 has been dormant, such as waiting for a data dependency to be resolved, then the active task may be taken out of the GPC 450 and returned to the pending task pool while another task is in the pending task pool selected and scheduled for execution in the GPC 450.

The work distribution unit 425 communicates with the one or more GPCs 450 via an Xbar 470. The Xbar 470 is an interconnection network that connects many of the PPU 400 units to other PPU 400 units. For example, the Xbar 470 may be configured to connect the work distribution unit 425 to a specific GPC 450 . Although not explicitly shown, one or more other units of PPU 400 are connected to host unit 410 . The other units may also be connected to the Xbar 470 via a hub 430 .

The tasks are managed by the scheduling unit 420 and dispatched to a GPC 450 by the work distribution unit 425 . The GPC 450 is configured to process the task and generate results. The results may be consumed by other tasks within the GPC 450, forwarded to another GPC 450 via the Xbar 470, or stored in memory 404. The results may be written to memory 404 via partition units 480 that implement a memory interface for reading and writing data to/from memory 404 . In In one embodiment, PPU 400 has a number U of partition units 480 equal to the number of separate and individual storage devices 404 connected to PPU 400 . A partition unit 480 is discussed in more detail below in connection with FIG 5B described.

In one embodiment, a host processor executes a driver kernel that implements an application programming interface (API) that allows one or more applications running on the host processor to schedule operations for execution on PPU 400 . An application may issue commands (i.e., API calls) that cause the driver kernel to create one or more tasks for execution on the PPU 400. The driver kernel issues tasks to one or more streams being processed by the PPU 400. Each task may have one or more groups of related threads, which group is referred to herein as a thread group or warp. A thread block can refer to multiple groups of threads with instructions to perform the task. Threads in the same group of threads can exchange data via shared memory. In one embodiment, a set of strands includes 32 associated strands.

5A shows a GPC 450 of the PPU 400 4 according to one embodiment. As in 5A As shown, each GPC 450 includes a number of hardware units for processing tasks. In one embodiment, each GPC 450 includes a pipeline manager 510, a pre-raster operation unit (PROP) 515, a rasterizer unit 525, a work distribution cross-connect unit (WDX) 580, a memory management unit (MMU) 590, and one or more texture processing clusters (TPCs) 520. Note that the GPC is 450 from 5A other hardware units instead of or in addition to those in 5A shown units may have.

In one embodiment, the operation of GPC 450 is controlled by pipeline manager 510 . The pipeline manager 510 manages the configuration of the one or more TPCs 520 to process tasks associated with the GPC 450 . In one embodiment, the pipeline manager 510 may configure at least one of the one or more TPCs 520 to implement at least a portion of a graphics processing pipeline. For example, a TPC 520 can be configured to run a vertex shader program in the stream programmable multiprocessor (SM) 540 . Pipeline manager 510 may be further configured to distribute packets received from work distribution unit 425 to appropriate logical units within GPC 450 . For example, some packets may be forwarded to dedicated function hardware units in PROP 515 and/or rasterizer unit 525, while other packets may be forwarded to TPCs 520 for processing by primitives unit 535 or SM 540.

TPC unit 515 is configured to forward data generated by rasterizer 525 and TPCs 520 to a raster operations (ROP) unit in partition unit 480, as described in more detail below. The PROP unit 515 may also be configured to perform color mixing optimizations, organize pixel data, perform address translations, and the like.

The rasterizer 525 includes a number of dedicated function hardware units configured to perform various raster operations. In one embodiment, the rasterizer 525 includes a setup unit, a coarse rasterizer, a culling unit, a clipping unit, a fine rasterizer, and a tile merging unit. The setup unit receives transformed vertices and generates plane equations associated with the geometric primitive defined by the vertices. The plane equations are sent to the coarse rasterizer to generate coverage information (e.g., an x,y coverage mask for a tile) for the primitive. The output of the coarse rasterizer can be sent to the culling unit, in which fragments belonging to the baseform that fail a z-test are culled, and can then be transmitted to a clipping unit, in which fragments falling outside of a viewing truncated cone are cut off. Those fragments that survive clipping and culling can be passed to a fine rasterizer to generate attributes for the pixel fragments based on the plane equations generated by the setup engine. The output of the rasterizer 580 includes fragments to be processed by a fragment shader implemented in a TPC 520, for example.

Each TPC 520 included in the GPC 450 includes an M-channel controller (MPC) 530, a primitive unit 535, an SM 540, and one or more texture units 545. The MPC 530 controls the operation of the TPC 520 wherein packets received from the pipeline manager 510 are forwarded to the appropriate entities in the TPC 520. For example, packets associated with a vertex may be forwarded to primitives unit 535 configured to fetch from memory 404 vertex attributes associated with the vertex. On the other hand, packets belonging to a shadowing program can be transmitted to the SM 540.

In one embodiment, texture units 545 are configured to load texture maps (e.g., a 2D array of texture elements) from memory 404 and sample the texture maps to generate sampled texture values for use in shading programs executed by SM 540 will. The texture units 545 implement texture operations, such as filtering operations, using mipmaps (i.e., texture maps with varying degrees of detail). In one embodiment, each TPC 520 contains four (4) texture units 545.

SM 540 includes a programmable data stream processor configured to process tasks represented by a number of threads. Each SM 540 is multi-threaded and configured to execute multiple threads (e.g., 32 threads) from a particular set of threads simultaneously. In one embodiment, SM 540 implements a SIMD (single instruction, multiple data) architecture in which each thread in a group of threads (ie, a thread group) is configured to assign a different set of data based on the same group of instructions to process. All threads in the group of threads execute the same instructions. In another embodiment, SM 540 implements a SIMT (single instruction, multiple threads) architecture in which each thread in a group of threads is configured to process a different data set based on the same group of instructions, but with individual threads in the group of threads are allowed to diverge during execution. In other words, when an instruction for the set of threads is issued for execution, some threads in the set of threads may be active, causing the instruction to execute, while other threads in the set of threads may be inactive, causing instead of execution of the instruction a no operation (NOP) is executed. The SM 540 is below in connection with 6A described in more detail.

The MMU 590 provides an interface between the GPC 450 and the partition unit 480 . The MMU 590 may provide virtual address to physical address translation, memory protection, and distribution of memory requests. In one embodiment, MMU 500 provides one or more translation lookaside buffers (TLBs) to enhance virtual address-to-physical address translation in memory 404 .

5B 12 shows a partition unit 480 of the PPU 400. FIG 4 according to one embodiment. As in 5B As shown, the partition unit 480 includes a raster operation (ROP) unit 550, a level two (L2) cache 560, a memory interface 570, and an L2 cross-connect unit (Xbar) 565. The memory interface 570 is connected to the memory 404 tied together. Memory interfaces 570 may implement 16, 32, 64, 128 bit data buses or the like for high speed data transfer. In one embodiment, the PPU 400 U includes memory interfaces 570, ie one memory interface 570 per partition unit 480, with each partition unit 480 being connected to a corresponding memory device 404. For example, the PPU 400 may be coupled to up to U memory devices 404, such as version 5 double data rate graphics synchronous dynamic random access memory (GDDR5 SDRAM). In one embodiment, memory interface 570 implements a DRAM interface and U=6.

In one embodiment, PPU 400 implements a multi-level memory hierarchy. Memory 404 is located off-chip in SDRAM coupled to PPU 400 . Data from memory 404 may be fetched and stored in L2 cache 560 located on chip and shared among the various GPCs 450 . As shown, each partition unit 480 includes a portion of L2 cache 560 associated with a corresponding storage device 404 . Lower level caches can then be implemented in various units within the GPCs 450. For example, each of the SMs 540 may implement a level one (L1) cache. The L1 cache memory is private memory dedicated to a specific SM 540. Data from the L2 cache 560 may be fetched and stored in any of the L1 caches for processing in the SMs 540 functional units. The L2 cache memory 560 is connected to the memory interface 570 and the Xbar 470.

The ROP unit 550 includes a ROP manager 355, a color ROP (CROP) unit 552, and a Z-ROP (ZROP) unit 554. The CROP unit 552 performs raster operations that affect pixel color, such as color compression , pixel blending and the like. The ZROP unit 554 implements a depth check in conjunction with the rasterizer unit 525. The ZROP unit 554 receives a depth for a sample position associated with a pixel fragment from the screening unit of the rasterizer unit 525. The ZROP unit 554 checks the depth versus a corresponding depth in a depth buffer for a sample position associated with the fragment. If the fragment passes the depth test for the sample position, then the ZROP unit 554 updates the depth buffer and sends a depth test result to the rasterizer 525. The ROP manager 355 controls the operation of the ROP unit 550. It should be noted that the number of partition units 480 can differ from the number of GPCs 450 and therefore each ROP unit 550 can be connected to each of the GPCs 450. Therefore, the ROP manager 355 monitors packets received from the different GPCs 450 and determines to which GPC 450 a result generated by the ROP unit 550 is to be forwarded. The CROP unit 552 and the ZROP unit 554 are connected to the L2 cache memory 560 via an L2 xbar 565 .

6A Figure 12 shows the data stream multiprocessor 540 5A according to one embodiment. As in 6A As shown, the SM 540 includes an instruction cache 605, one or more scheduler units 610, a register file 620, one or more processing cores 650, one or more special purpose functional units (SFUs) 652, one or more load/store units (LSUs ) 654, an interconnection network 680, and a shared memory/L1 cache 670.

As previously described, the work distribution unit 425 issues tasks for execution in the GPCs 450 of the PPU 400 . The tasks are assigned to a specific TPC 520 within a GPC 450, and if the task is associated with a shader program, the task can be assigned to an SM 540. The scheduler 610 receives the tasks from the work distribution unit 425 and manages instruction scheduling for one or more groups of threads (ie, thread groups) associated with the SM 540 . The scheduler 610 schedules threads for execution in groups of parallel threads, each group being referred to as a thread group. In one embodiment, each strand group contains 32 strands. The scheduler 610 can manage multiple different thread groups, thereby scheduling the thread groups for execution and then issuing instructions from the multiple different thread groups to the various functional units (i.e., cores 650, SFUs 652, and LSUs 654) during each clock cycle.

In one embodiment, each scheduler 610 includes one or more instruction issue units 615. Each issue unit 615 is configured to dispatch instructions to one or more of the functional units. in the in 6A In the embodiment shown, the scheduler 610 includes two issue units 615 that allow two different instructions from the same thread group to be issued during each clock cycle. In alternative embodiments, each scheduler 610 may have a single scheduler 615 or additional schedulers 615 .

Each SM 540 has a register file 620 that provides a set of registers for the SM 540's functional units. In one embodiment, register file 620 is partitioned among all functional units such that each functional unit is associated with a specific portion of register file 620 . In another embodiment, register file 620 is partitioned among the different thread groups being executed by SM 540 . Register file 620 provides temporary storage for operands associated with functional unit data paths.

Each SM 540 includes L processing cores 450. In one embodiment, the SM 540 includes a large number (e.g., 192, etc.) of individual processing cores 450. Each core 450 may include a fully pipelined, single-precision processing unit having a floating-point Arithmetic logic unit and an integer arithmetic logic unit. The core 450 may further include a double precision processing unit that includes a floating point arithmetic logic unit. In one embodiment, the floating point arithmetic logic units implement the IEEE 754-4008 standard for floating point arithmetic. Each SM 540 also includes M SFUs 652 that perform specialized functions (e.g., pixel blending operations and the like) and N LSUs 654 that perform load and store operations between shared memory/L1 cache 670 and register file 620. In one embodiment, the SM 540 includes 192 cores 650, 32 SFUs 652, and 32 LSUs 654.

Each SM 540 has an interconnection network 680 that connects each of the functional units to register file 620 and shared memory/L1 cache 670 . In one embodiment, interconnect network 680 is a cross-connect unit that may be configured to connect any functional unit to any register in register file 620 or locations in shared memory/L1 cache 670 .

Shared memory/L1 cache 670 is an array of internal memory that, in one embodiment, may be shared memory or an L1 cache, or a combination of both as required by the application . For example, shared memory/L1 cache 670 may have 64 KB of storage capacity. The shared memory/L1 cache 670 may be 64 KB of shared memory or an L1 cache, or a combination of the two, such as 16 KB of L1 cache and 48 KB of shared memory be configured.

The PPU 400 described above can be configured to perform highly parallel computations much faster than conventional CPUs. Parallel computation has advantages in graphics processing, data compression, biometric processes, data stream processing algorithms, and the like.

In one embodiment, PPU 400 includes a graphics processing unit (GPU). The PPU 400 is configured to receive commands specifying shadowing programs for processing graphics data. Graphics data can be defined as a set of primitive shapes, such as points, lines, triangles, squares, triangle strips, and the like. Typically, a primitive includes data specifying a number of vertices for the primitive (e.g., in a model space coordinate system) and attributes associated with each vertex of the primitive. The PPU 400 may be configured to process the graphics primitives to generate an image block buffer (ie, pixel data for each pixel of the display).

An application writes model data for a scene (i.e., a collection of vertices and attributes) to a memory, such as system memory or memory 404. The model data defines each of the objects that may be visible on a display. The application then makes an API call into the driver kernel requesting that the model data be rendered and displayed as an image. The driver kernel reads the model data and writes instructions to the one or more streams to perform operations to process the model data. The instructions may address different shaders to be implemented in the SMs 540 of the PPU 400, including a vertex shader and/or a skin shader, a region shader, a geometry shader, and a pixel shader. For example, one or more of the SMs 540 may be configured to run a vertex shader program that processes a number of vertices defined by the model data. In one embodiment, the different SMs 540 may be configured to run different shading programs concurrently. For example, a first subset of SMs 540 may be configured to execute a vertex shader program, while a second subset of SMs 540 may be configured to execute a pixel shader program. The first subset of SMs 540 processes vertex data to generate processed vertex data and writes the processed vertex data to L2 cache 560 and/or memory 404. After rasterizing the processed vertex data (i.e., transformed from the three-dimensional data into two-dimensional data in screen space) to produce fragment data, the second subset of SMs 540 performs a pixel shader to produce processed fragment data, which is then mixed with other processed fragment data and written to the image block buffer in memory 404. The vertex shader program and the pixel shader program can run concurrently, thereby processing different data from the same scene in a pipeline manner until all model data for the scene is imaged into the image block buffer. The contents of the image block buffer can then be transferred to a display controller for display on a display device.

The PPU 400 may be incorporated into a desktop computer, a portable computer, a tablet computer, a smart phone (e.g., a wireless handset), a personal digital assistant (PDA), a digital camera, a handheld electronic device, and the like. In one embodiment, PPU 400 is formed as a single semiconductor substrate. In another embodiment, the PPU 400 is in a system-on-a-chip (SoC) along with one or more others other logic units, such as a CPU with a reduced instruction set computer (RISC), a memory management unit (MMU), a digital-to-analog converter (DAC), and the like.

In one embodiment, PPU 400 may be included in a graphics card that includes one or more memory devices 404, such as GDDR5 SDRAM. The graphics card may be configured to interface with a PCIe portion on a motherboard of a desktop computer having, for example, a north bridge chipset and a south bridge chipset. In yet another embodiment, the PPU 400 may be an integrated graphics processing unit (iGPU) included in the chipset (i.e., north bridge) of the motherboard.

This invention describes a method of generating a near-eye light field - an array of rays emitted from a light field display in close proximity to the eye. The method allows more traditional image sources to be used to determine the color of each ray, rather than necessarily using ray tracing. Light sources can be monoscopic (i.e., normal 2D images), or stereoscopic (one image for each eye, imaging with correct binocular perspective), or multiscopic, where multiple views are generated for a set of different viewing positions and camera orientations. Multiscopic data is the data format used for multiview 3D television, but also includes other forms such as multiple panoramas provided by applications such as Google Earth®. A set of source images in many different data formats can be used to estimate the color of a ray knowing how the source images were generated and knowing the configuration of the particular near-eye light field display.

6B FIG. 6 shows a flow diagram of a method 625 for generating images for a near-eye light field display device 200 or 260 of FIG 2A , 2D and 2C according to one embodiment. Although method 625 is described in the context of a program executed by a processor, method 625 may also be executed by application specific circuitry or by a combination of application specific circuitry and a program.

In step 630, existing images are received. The existing images can be individual two-dimensional images, stereoscopic image pairs or multiscopic image groups. When eye tracking is employed, source images in a best match microscopic image group can be selected for each eye. In one embodiment, two or more original images of a microscopic image group are interpolated to generate a matching original image. In step 635, the existing images are pre-processed if image defect correction is enabled. In one embodiment, the existing images are filtered using the low pass filter length. If image error correction is disabled, step 635 can be omitted. In step 640, the pre-processed existing images are stored as texture maps.

At step 645, a texture element of a texture map corresponding to one of the existing images is mapped to a pixel on the micro-display 250, the left micro-display 251, or the right micro-display 252. If the texture map corresponds to an individual two-dimensional image, the texture element is mapped to, or associated with, a pixel on the left micro-display 251 and a pixel on the right micro-display 252 . If the texture map corresponds to the left eye image of a stereoscopic image pair, then the texture element is mapped to a pixel on the left micro-display 251 . If the texture map corresponds to the right eye image of a stereoscopic image pair, then the texture element is mapped to the right micro-display 252 . Steps 645-672 (shown as steps 685) may be performed in parallel to process multiple pixels of the left micro-display 251 and the right micro-display 252. For example, each thread may receive coordinates of a texture element and map the texture element coordinates to left micro-display 251 or right micro-display 252 coordinates.

In step 655, an optical device (for example, microlens in microlens array 255, left microlens array 256, right microlens array 257, light spot in left light spot array 276, or light spot in right light spot array 277 ) corresponding to the pixel on the micro-display (eg, micro-display 250, left micro-display 251, right micro-display 252, left micro-display 271, or right micro-display 272). At step 660, a ray defined by the pixel and the optical device is determined. For example, if the optical device is a microlens array, the ray is defined by the (x,y,z) position of the pixel and the (x,y,z) position of the optical center of the microlens. If the optical device is a spot array, then the ray is defined by the (x,y,z) position of the pixel and the (x,y,z) position of the center of the light spot. In step 665, the ray is made to intersect with the two-dimensional virtual display plane 210 to generate texture map coordinates corresponding to the pixel. In step 672, a color for the pixel is calculated based on the texture map coordinates. Multiple texture elements (e.g., color samples) can be extracted from the texture map and combined (ie, filtered) to calculate the colors for the pixel. When steps 645-672 are executed in parallel, each thread computes a color for the left micro-display 251 or the right micro-display 252.

At step 675, the computed colors for the pixels are stored to create at least one micro-display image. The colors may be stored in a data structure (e.g., image block buffer) for a micro-display image in memory. The pixel colors for the left micro-display 251 define a left micro-display image and the pixel colors for the right micro-display 252 define a right micro-display image. In step 682, the shader program determines whether there is another existing image to process, and if not, the shader program ends. Otherwise, steps 685 are repeated to generate one or more additional micro-display images.

7 7 is a block diagram of an example of a computing system 710 capable of implementing embodiments of the present disclosure. Computing system 710 broadly represents a single processor or multiple processor computing device or system capable of executing computer-readable instructions. Examples of computing system 710 include, without limitation, workstations, portable computers, client-side terminals, service providers, distributed computing systems, embedded devices, computing devices for automotive applications, handheld devices (e.g., cellular phone, tablet computer, digital camera, etc.), wearable devices ( e.g. head mounted or waist worn devices), or any other computing system or device. In the most basic configuration, the computing system 710 may include at least a processor 714 and system memory 716 .

Processor 714 generally represents any type or form of processing unit capable of processing data or translating and executing instructions. In certain embodiments, processor 714 may receive instructions from a software application or module. These instructions may cause processor 714 to perform the functions of one or more of the illustrative embodiments described and/or illustrated herein.

System memory 716 generally represents any type or form of volatile or non-volatile storage device or medium capable of storing data and/or other computer-readable instructions. Examples of system memory 716 include, without limitation, RAM, ROM, flash memory, or other suitable memory devices. Although not required, in certain embodiments, computing system 710 may include both a volatile storage device (e.g., system memory 716) and a non-volatile storage device (e.g., primary storage device 732).

Computing system 710 may further include one or more components or elements in addition to processor 714 and system memory 716 . For example, in the embodiment of FIG 7 The computing system 710 includes a memory controller 718, an input/output (I/O) controller 720, and a communications interface 722, each of which may be interconnected via a communications infrastructure 712. Communication infrastructure 712 generally represents any type or form of infrastructure capable of facilitating communication between one or more components of a computing device. Examples of communications infrastructure 712 include, without limitation, a communications bus (such as an Industry Standard Architecture (ISA), Peripheral Component Interconnect (PCI), PCI Express (PCle), or similar bus) and a network.

Memory controller 718 generally represents any type or form of device capable of manipulating memory, data, or controlling communication between one or more components of computing system 710 . For example, memory controller 718 may control communications between processor 714, system memory 716, and I/O controller 720 via communications infrastructure 712.

I/O controller 720 generally represents any type or form of module capable of coordinating and/or controlling the input and output functions of a computing device. For example, I/O controller 720 may control the exchange of data between one or more elements of computing system 710, such as processor 714, system memory 716, communications interface 722, a display adapter 726, input interface 730, and memory interface 734, or enable.

Communications interface 722 broadly represents any type or form of communication device or adapter capable of facilitating communication between illustrative computing system 710 and one or more other devices. For example, communication interface 722 may enable communication between computing system 710 and a private or public network containing other computing systems. Examples of communication interface 722 include, without limitation, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, or other suitable interface. In one embodiment, communication interface 722 provides a direct connection to a remote service provider via a direct connection to the network, such as the Internet. Communications interface 722 may also provide such a connection indirectly via another suitable connection.

The communication interface 722 may further represent a host adapter configured to enable communication between the computing system 710 and one or more other network or storage devices via an external bus or communication channel. Examples of host adapters include, without limitation, small computer systems interface (SCSI) host adapters, universal serial bus (USB) host adapters, IEEE (Institute of Electrical and Electronic Engineers) 1394- Host adapters, Serial Advanced Technology Adder (SATA) and external SATA (eSATA) adapters, Advanced Technology Adder (ATA) and Parallel ATA (PATA) host adapters, Fiber Channel interface adapters, Ethernet adapters, or the like. Communications interface 722 may also allow computing system 710 to participate in distributed or remote computing configurations. For example, communications interface 722 may receive commands from a remote device or send commands to a remote device for execution.

As in 7 As shown, the computing system 710 may further include at least one display device 724 coupled to the communications infrastructure 712 via a display adapter 726 . Display devices 724 generally represent any type or form of device capable of visually displaying information passed by display adapter 726 . Similarly, display adapter 726 generally represents any type or form of device configured to pass graphics, text, or other data for display on display device 724 . In one embodiment, the display device 724 may be the near-eye light field display device 200 or the binocular near-eye light field display device described in FIG 2 B or 2C and one or more components shown in the computing system 710 may be integrated into an eyeglass form factor of the near-eye light field display device 200 or the binocular near-eye light field display device shown in FIG 2 B or 2C are shown. In one embodiment, one or more of the components shown in computing system 710 are outside of the eyeglass form factor and may be carried by a user elsewhere or may be entirely remote (ie, tethered via a wireless or wired communication system). A power supply (ie, battery, power source, or power interface) may be integrated within the goggle form factor or may be external to the goggle form factor.

As in 7 As shown, the computing system 710 may further include at least one input device 728 that is connected to the communication infrastructure 712 via an input interface 730 . Input device 728 generally represents any type or form of input device capable of providing input, whether computer generated or human generated, to computing system 710 . Examples of the input device 728 include, without limitation, a keyboard, a pointing device, a speech recognizer, an eye tracking system, an ambient light detection system, a depth sensor, a light field camera, an ambient motion tracking sensor, an internal motion detection sensor, a gyroscope sensor , an accelerometer, an electronic compass sensor, a charge coupled device (CCD) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, or other input device. In one embodiment, the input device 728 may be configured to display pupillary properties, such as the provide diameter, orientation and position for the 714 processor. In one embodiment, input device 728 may also originate communication events from other near-eye light field display devices 200 (ie, near-eye light field goggles) in the vicinity, for example, when multiple near-eye light field display device users are nearby.

As in 7 As shown, the computing system 710 may further include a primary storage device 732 and a backup storage device 733 coupled to the communications infrastructure 712 via a storage interface 734 . Storage devices 732 and 733 generally represent any type or form of storage device or medium capable of storing data and/or other computer-readable instructions. For example, the storage devices 732 and 733 may be a magnetic disk drive (e.g., a so-called hard disk drive), a floppy disk drive, a magnetic tape drive, an optical disk drive, a flash drive, or the like. Storage interface 734 generally represents any type or form of interface or device to exchange data between storage devices 732 and 733 and other components of computing system 710 .

In one example, databases 740 may be stored on primary storage device 732 . Databases 740 may represent portions of a single database or computing device, or they may represent multiple databases or computing devices. For example, databases 740 may represent (or be stored within) only a portion of computing system 710 . Alternatively, databases 740 may represent (or be stored on) one or more physical separate devices that are accessible by a computing device, such as computing system 710 .

It's still open 7 referred; storage devices 732 and 733 may be configured to read from and/or write to a removable storage device configured to store computer software, data, or other computer-readable information. Examples of suitable removable storage devices include, without limitation, a floppy disk, magnetic tape, optical disk, flash memory device, or the like. Storage devices 732 and 733 may also include other similar structures or devices to enable computer software, data, or other computer-readable instructions to be loaded into computing system 710. For example, storage devices 732 and 733 may be configured to read and write software, data, or other computer-readable information. Storage devices 732 and 733 may also be part of a computing system 710 or may be separate devices accessed through other interface systems.

Many other devices or subsystems may be connected to computing system 710. On the other hand, not all components and facilities that are used in 7 shown may be present to implement the embodiments described herein. The facilities and subsystems identified above may also be interconnected in ways other than those specified in 7 is shown. Computing system 710 may also employ a variety of software, firmware, and/or hardware configurations. For example, the illustrative embodiments disclosed herein may be encoded as a computer program (also referred to as computer software, software applications, computer-readable instructions, or computer control logic) on a computer-readable medium.

The computer-readable medium containing the computer program can be loaded into the computing system 710 . All or a portion of the computer program stored on the computer-readable media may then be stored in system memory 716 and/or various portions of storage devices 732 and 733 . When executed by processor 714, a computer program loaded into computing system 710 may cause processor 714 to perform and/or be a means of performing the functions of the illustrative embodiments described and/or illustrated herein. Additionally or alternatively, the illustrative embodiments described and/or shown herein may be implemented in firmware and/or hardware.

For example, a computer program for determining an image and/or mask to be encoded in a near-eye display layer 515 based on a desired image 525 may be stored on the computer-readable media and then in system memory 716 and/or in various parts of the memory devices 732 and 733 are stored. When executed by the processor 714, the computer program may cause the processor 714 to perform and/or be a means of performing the functions required to acquire an image and/or the mask encoding different attenuation patterns as discussed above.

Although the foregoing disclosure provides various embodiments using specific block diagrams, flowcharts, and examples, each block diagram component, flowchart step, operation, and/or component described and/or illustrated herein may individually and/or collectively using a broad Any number of hardware, software, or firmware configurations (or any combination thereof) can be implemented. Furthermore, the disclosure of components contained within other components should be considered as examples, as many other architectures can be implemented to achieve the same function.

Although various embodiments have been described above, it should be understood that these are presented by way of example and not limitation. Thus, the breadth and scope of a preferred embodiment are not limited by any of the illustrative embodiments described above, but are defined only in accordance with the following claims and their equivalents.

The process parameters and sequence of steps described and/or illustrated herein are given by way of example only. For example, although the steps illustrated and/or described herein are shown or discussed in a particular order herein, these steps do not necessarily have to be performed in the order illustrated or discussed. The various illustrative methods described and/or illustrated herein may also omit one or more of the steps described or illustrated herein, or may include additional steps in addition to the steps disclosed.

Although various embodiments are described and/or illustrated herein in the context of fully functional computing systems, one or more of these illustrative embodiments may be distributed as a program product in a variety of forms, regardless of the particular type of computer-readable media used to actually carry out the distribution will. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include script, batch, or other executable files that may be stored on a computer readable storage medium or computing system. These software modules can configure a computing system to perform one or more of the example embodiments disclosed herein. One or more of the software modules disclosed herein can be implemented in a cloud computing environment. Cloud computing environments can provide various services and applications over the Internet. These cloud-based services (e.g., software as a service, platform as a service, infrastructure as a service, etc.) can be accessed via web browser or other remote interface. Various functions described herein may be provided by a remote desktop environment or other cloud-based computing environment.

Claims (18)

A method (100) comprising: Identifying (105) a ray passing through a pixel of a micro-display (250) and a microlens (253) of a near-eye light field display device (200); Generating imaging coordinates corresponding to the pixel by intersecting (100) the beam with a two-dimensional virtual display plane (210), wherein generating the imaging coordinates involves determining a displacement of an optical center of the microlens (253) relative to a reference point on the microdisplay ( 250); and calculating (115) a color for the pixel based on the mapping coordinates. The procedure after claim 1 , wherein an existing two-dimensional image (300, 350) is stored as a texture map indexed using the map coordinates. The procedure after claim 1 Further comprising: pre-processing an existing two-dimensional image (300, 350) to produce a pre-processed existing two-dimensional image in which artifacts are corrected; and storing the previous verar processed the existing two-dimensional image as a texture map indexed using the map coordinates. The procedure after claim 1 , further comprising: determining additional rays, generating additional mapping coordinates, and computing additional colors for additional pixels of the micro-display (250) to generate a micro-display image (310, 360). The procedure after claim 1 further comprising storing the color for the pixel in a data structure for a micro-display image (310, 360) in a memory (404) The procedure after claim 5 wherein the micro-display image (310, 360) corresponds to a left eye image or a right eye image of a stereoscopic image pair. The procedure after claim 5 , wherein the micro-display image (310, 360) corresponds to an image of a multiscopic image group. The procedure after claim 1 , further comprising: sampling a plurality of texture elements of a texture map based on the map coordinates; and combining the multiple texture elements to calculate the color for the pixel. The procedure after claim 1 wherein a depth of the two-dimensional virtual display plane (210) relative to the viewer's eye is based on an eyeglass prescription. The procedure after claim 1 wherein multiple processing threads are configured to execute a fragment shader program to compute colors for pixels of the micro-display (250) in parallel. The procedure after claim 1 , where the color is black if the mapping coordinates are outside the virtual display plane (210). The procedure after claim 1 wherein the virtual image includes one that appears to a user of the near-eye light field display device (200) at a depth that is offset from the virtual display plane (210). The procedure after claim 1 , wherein generating the mapping coordinates comprises determining a displacement of the pixel relative to the reference point on the micro-display (250) The procedure after claim 1 , wherein generating the mapping coordinates comprises determining a displacement of a point within the two-dimensional virtual display plane (210) intersected by the ray. Having a system: a microdisplay (250) positioned between a near focal plane (220) and a pupil; a microlens (253) positioned between the microdisplay (250) and the pupil; and a processor (714) configured to: identify a ray defined by a pixel of the micro-display (250) and the optical device; generate mapping coordinates corresponding to the pixel by intersecting the beam with a two-dimensional virtual display plane (210); wherein generating the mapping coordinates comprises determining a displacement of an optical center of the microlens (253) relative to a reference point on the microdisplay (250); and calculate a color for the pixel based on the mapping coordinates. The system after claim 15 , further comprising a memory (404) configured to store an existing two-dimensional image (300, 350) as a texture map indexed using the texture coordinates. The system after claim 15 wherein the processor (714) is further configured to process multiple threads of processing, each executing a fragment shader program to compute colors for pixels of the micro-display (250) in parallel. A computer-readable storage medium that stores instructions that, when executed by a processor (714), cause the processor to perform steps comprising: Identifying (105) a ray passing through a pixel of a micro-display (250) and a microlens (253) of a near-eye light field display device (200); Generating mapping coordinates corresponding to the pixel by intersecting (100) the beam with a two-dimensional virtual display plane (210), wherein generating the mapping coordinates includes determining a displacement of an optical center of the microlens (253) relative to a reference point on the microdisplay (250); and calculating (115) a color for the pixel based on the mapping coordinates.

Images